[Perf] Linux/arm64: 1275 Improvements on 5/19/2023 1:23:34 PM #18145
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | f73356c1f4b9b292dc4fb02f1dd47e4a61e423f5 |
| Compare | 1ec5ff876baf614275dcdc1d6ba435e94d78e4b1 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Collections.ContainsKeyTrue<Int32, Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ImmutableSortedDictionary - Duration of single invocation | 28.44 μs | 19.85 μs | 0.70 | 0.40 | False | |||||
| SortedDictionary - Duration of single invocation | 41.11 μs | 28.95 μs | 0.70 | 0.38 | False | |||||
| IDictionary - Duration of single invocation | 6.99 μs | 4.41 μs | 0.63 | 0.03 | True | |||||
| SortedList - Duration of single invocation | 28.19 μs | 22.50 μs | 0.80 | 0.01 | True | |||||
| ImmutableDictionary - Duration of single invocation | 21.41 μs | 12.70 μs | 0.59 | 0.14 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue<Int32, Int32>*'Payloads
Histogram
System.Collections.ContainsKeyTrue<Int32, Int32>.ImmutableSortedDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.845173131443303 < 27.04157568262564.
IsChangePoint: Marked as a change because one of 4/4/2023 3:37:20 PM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 64.24831238713575 (T) = (0 -19711.357975113613) / Math.Sqrt((10442347.66756669 / (299)) + (21903.516015626705 / (10))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (10) - 2, .975) and 0.38572640402067115 = (32088.890201585236 - 19711.357975113613) / 32088.890201585236 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.ContainsKeyTrue<Int32, Int32>.SortedDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 28.945020317313617 < 39.14906951306867.
IsChangePoint: Marked as a change because one of 3/7/2023 1:52:33 AM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 21.472277173351426 (T) = (0 -23395.24281987641) / Math.Sqrt((11537708.18385077 / (299)) + (7268419.791378925 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.4342406839385659 = (41351.93563005508 - 23395.24281987641) / 41351.93563005508 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 4.405113722940681 < 6.5899548041723595.
IsChangePoint: Marked as a change because one of 4/27/2023 1:05:59 AM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 103.12809209773579 (T) = (0 -4315.327517342131) / Math.Sqrt((54540.45904620398 / (299)) + (5743.582384300903 / (10))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (10) - 2, .975) and 0.39665352761270845 = (7152.320788861922 - 4315.327517342131) / 7152.320788861922 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Collections.ContainsKeyTrue<Int32, Int32>.SortedList(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 22.501847370607784 < 26.6740789767256.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 123.57486190259873 (T) = (0 -22396.154704592882) / Math.Sqrt((583104.0538311899 / (299)) + (16187.898195294174 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.24400698961639 = (29624.817157011155 - 22396.154704592882) / 29624.817157011155 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Collections.ContainsKeyTrue<Int32, Int32>.ImmutableDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 12.704356043624783 < 20.3472417208219.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 224.18218498029682 (T) = (0 -12688.148237350291) / Math.Sqrt((252321.94261828726 / (299)) + (7979.354076469249 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.4117397757369856 = (21568.937885008778 - 12688.148237350291) / 21568.937885008778 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | f73356c1f4b9b292dc4fb02f1dd47e4a61e423f5 |
| Compare | 1ec5ff876baf614275dcdc1d6ba435e94d78e4b1 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Diagnostics.Perf_Activity
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| EnumerateActivityEventsLarge - Duration of single invocation | 26.53 μs | 5.82 μs | 0.22 | 0.65 | False | |||||
| EnumerateActivityEventsSmall - Duration of single invocation | 132.74 ns | 43.70 ns | 0.33 | 0.60 | False | |||||
| EnumerateActivityLinksLarge - Duration of single invocation | 21.60 μs | 5.86 μs | 0.27 | 0.67 | False | |||||
| EnumerateActivityLinksSmall - Duration of single invocation | 161.04 ns | 38.55 ns | 0.24 | 0.59 | False | |||||
| EnumerateActivityLinkTagsLarge - Duration of single invocation | 15.87 μs | 9.60 μs | 0.61 | 0.64 | False | |||||
| EnumerateActivityTagObjectsLarge - Duration of single invocation | 17.01 μs | 6.88 μs | 0.40 | 0.70 | False | |||||
| ActivityAllocations - Duration of single invocation | 347.94 ns | 303.58 ns | 0.87 | 0.10 | False | |||||
| EnumerateActivityLinkTagsSmall - Duration of single invocation | 100.70 ns | 71.27 ns | 0.71 | 0.58 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Diagnostics.Perf_Activity*'Payloads
Histogram
System.Diagnostics.Perf_Activity.EnumerateActivityEventsLarge
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 5.819415657422187 < 23.983910996141105.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 29.03471030364613 (T) = (0 -6565.5021771157935) / Math.Sqrt((45848477.872694485 / (299)) + (2047969.8275757022 / (10))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (10) - 2, .975) and 0.7257652863757126 = (23941.17830797598 - 6565.5021771157935) / 23941.17830797598 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Diagnostics.Perf_Activity.EnumerateActivityEventsSmall
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 43.70324451508215 < 126.2717161519518.
IsChangePoint: Marked as a change because one of 4/9/2023 11:24:15 AM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 25.1155136055156 (T) = (0 -48.90349027729031) / Math.Sqrt((609.1156307044935 / (299)) + (111.29177971952006 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.6416404390744679 = (136.46486827639728 - 48.90349027729031) / 136.46486827639728 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Diagnostics.Perf_Activity.EnumerateActivityLinksLarge
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 5.862867127081095 < 25.611514765715118.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 35.894039974231795 (T) = (0 -6959.012769766078) / Math.Sqrt((76280793.12814906 / (299)) + (1593556.068810393 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.7653743564194744 = (29660.06896589581 - 6959.012769766078) / 29660.06896589581 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Diagnostics.Perf_Activity.EnumerateActivityLinksSmall
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 38.55456469915433 < 120.8827772875011.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 28.42475501153626 (T) = (0 -47.72479931872151) / Math.Sqrt((950.0608588109176 / (299)) + (120.99933265715607 / (10))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (10) - 2, .975) and 0.6995164660852041 = (158.82667078940239 - 47.72479931872151) / 158.82667078940239 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Diagnostics.Perf_Activity.EnumerateActivityLinkTagsLarge
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 9.60377301331049 < 13.47592770871697.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 19.170674708110017 (T) = (0 -7615.190709000814) / Math.Sqrt((12415503.611338325 / (299)) + (1636786.4066557628 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.5234124322031447 = (15978.576076174062 - 7615.190709000814) / 15978.576076174062 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Diagnostics.Perf_Activity.EnumerateActivityTagObjectsLarge
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 6.8776930208398905 < 15.739235483373776.
IsChangePoint: Marked as a change because one of 3/31/2023 7:02:21 PM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 16.506202503544568 (T) = (0 -8368.563086120383) / Math.Sqrt((14465326.731112704 / (299)) + (2041783.1824394346 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.48825908233136556 = (16353.12478870264 - 8368.563086120383) / 16353.12478870264 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Diagnostics.Perf_Activity.ActivityAllocations(idFormat: W3C)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 303.57941268721083 < 332.5703102667667.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 12.825031527846349 (T) = (0 -308.75705534426504) / Math.Sqrt((58.73137173790133 / (299)) + (133.38156555478676 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.127253739336249 = (353.7764287978107 - 308.75705534426504) / 353.7764287978107 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Diagnostics.Perf_Activity.EnumerateActivityLinkTagsSmall
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 71.26704899623834 < 95.74071278471314.
IsChangePoint: Marked as a change because one of 4/26/2023 2:24:59 PM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 12.051031930884077 (T) = (0 -65.28751381383026) / Math.Sqrt((262.9984265823687 / (299)) + (86.39462406582483 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.3529587591674877 = (100.90162681103358 - 65.28751381383026) / 100.90162681103358 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | f73356c1f4b9b292dc4fb02f1dd47e4a61e423f5 |
| Compare | 1ec5ff876baf614275dcdc1d6ba435e94d78e4b1 |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Improvements in System.Collections.TryGetValueTrue<Int32, Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SortedList - Duration of single invocation | 28.75 μs | 22.67 μs | 0.79 | 0.00 | True | |||||
| ImmutableSortedDictionary - Duration of single invocation | 30.35 μs | 19.72 μs | 0.65 | 0.41 | False | |||||
| Dictionary - Duration of single invocation | 5.58 μs | 4.34 μs | 0.78 | 0.07 | False | |||||
| SortedDictionary - Duration of single invocation | 41.48 μs | 29.84 μs | 0.72 | 0.36 | False | |||||
| FrozenDictionaryOptimized - Duration of single invocation | 3.74 μs | 3.04 μs | 0.81 | 0.01 | True | |||||
| ImmutableDictionary - Duration of single invocation | 21.42 μs | 12.63 μs | 0.59 | 0.20 | True | |||||
| FrozenDictionary - Duration of single invocation | 7.53 μs | 4.54 μs | 0.60 | 0.04 | True | |||||
| ConcurrentDictionary - Duration of single invocation | 6.16 μs | 3.63 μs | 0.59 | 0.11 | False | |||||
| IDictionary - Duration of single invocation | 7.26 μs | 4.25 μs | 0.59 | 0.02 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue<Int32, Int32>*'Payloads
Histogram
System.Collections.TryGetValueTrue<Int32, Int32>.SortedList(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 22.673975243309 < 27.311584862672902.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 155.40022530754317 (T) = (0 -22447.00020990847) / Math.Sqrt((359580.6837449 / (299)) + (11694.491626854378 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.24785624009936258 = (29844.029036249467 - 22447.00020990847) / 29844.029036249467 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.ImmutableSortedDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 19.715417243374482 < 28.83396735687233.
IsChangePoint: Marked as a change because one of 4/10/2023 1:33:42 PM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 39.398373854590396 (T) = (0 -19192.590949155016) / Math.Sqrt((9850610.94136418 / (299)) + (893337.7200562682 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.40953524739966585 = (32504.21107209737 - 19192.590949155016) / 32504.21107209737 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.Dictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 4.3402575175561795 < 5.41161255514664.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 81.59152023611438 (T) = (0 -4272.485588643293) / Math.Sqrt((100135.2294227967 / (299)) + (3172.0921216040815 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.32284369452616507 = (6309.452565244378 - 4272.485588643293) / 6309.452565244378 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.SortedDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 29.835016276041667 < 39.411968135196204.
IsChangePoint: Marked as a change because one of 4/11/2023 12:48:48 AM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 20.73437345364606 (T) = (0 -24055.229794980158) / Math.Sqrt((12090017.09769509 / (299)) + (7413145.885677353 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.42146906507390713 = (41579.850519234904 - 24055.229794980158) / 41579.850519234904 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.FrozenDictionaryOptimized(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.0442017905932865 < 3.544802470944282.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 211.74230489708003 (T) = (0 -3045.275018333855) / Math.Sqrt((1779.108471669248 / (175)) + (18.585192061491824 / (11))) is greater than 1.972940542389176 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (175) + (11) - 2, .975) and 0.1931666568445461 = (3774.354448991973 - 3045.275018333855) / 3774.354448991973 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.ImmutableDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 12.628815715357337 < 20.192556119592705.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 93.64446561712187 (T) = (0 -12862.167124931952) / Math.Sqrt((292405.35448892775 / (299)) + (90333.02544955087 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.41105578832300377 = (21839.364187496503 - 12862.167124931952) / 21839.364187496503 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.FrozenDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 4.544386865219959 < 7.264293925058213.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 13.582330366494054 (T) = (0 -4794.603885535182) / Math.Sqrt((49887.3301507353 / (176)) + (427107.39088730735 / (10))) is greater than 1.972940542389176 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (10) - 2, .975) and 0.37003503156310125 = (7610.905567387021 - 4794.603885535182) / 7610.905567387021 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.ConcurrentDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.6310388103448274 < 5.841206162844954.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 113.22548098162642 (T) = (0 -3618.3365354168345) / Math.Sqrt((153261.57794635984 / (299)) + (512.4982368865137 / (10))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (10) - 2, .975) and 0.42628815134057496 = (6306.888281759721 - 3618.3365354168345) / 6306.888281759721 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.TryGetValueTrue<Int32, Int32>.IDictionary(Size: 512)
Description of detection logic
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 4.2513812310762455 < 6.910571291406044.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 5/23/2023 1:41:12 PM falls between 5/14/2023 7:30:23 PM and 5/23/2023 1:41:12 PM.
IsImprovementStdDev: Marked as improvement because 146.87407548721916 (T) = (0 -4326.888270540316) / Math.Sqrt((45302.615357313014 / (299)) + (3383.1250046408777 / (11))) is greater than 1.9676960046166507 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (11) - 2, .975) and 0.421059308187883 = (7473.802293974037 - 4326.888270540316) / 7473.802293974037 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository