-
Notifications
You must be signed in to change notification settings - Fork 4
Open
Description
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 0202b242169f8d5955e0411dc5e4f42a5d016c1b |
Compare | 24107379cd4fbfe236f0a1cf92dbd6c4ad944fec |
Diff | Diff |
Improvements in System.Tests.Perf_Enum
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
ToString_NonFlags_Small - Duration of single invocation | 953.09 ns | 720.20 ns | 0.76 | 0.11 | False | |||||
ToString_Format_Flags_Large - Duration of single invocation | 817.08 ns | 582.00 ns | 0.71 | 0.14 | False | |||||
StringFormat - Duration of single invocation | 8.22 μs | 6.93 μs | 0.84 | 0.11 | False | |||||
InterpolateIntoStringBuilder_NonFlags - Duration of single invocation | 13.15 μs | 11.27 μs | 0.86 | 0.07 | False | |||||
InterpolateIntoString - Duration of single invocation | 13.77 μs | 12.01 μs | 0.87 | 0.05 | False | |||||
ToString_Flags - Duration of single invocation | 1.09 μs | 827.19 ns | 0.76 | 0.17 | False | |||||
InterpolateIntoStringBuilder_Flags - Duration of single invocation | 12.67 μs | 11.35 μs | 0.90 | 0.06 | False | |||||
ToString_NonFlags_Large - Duration of single invocation | 859.71 ns | 619.81 ns | 0.72 | 0.16 | False | |||||
ToString_Flags - Duration of single invocation | 1.12 μs | 847.41 ns | 0.75 | 0.11 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'
Payloads
Histogram
System.Tests.Perf_Enum.ToString_NonFlags_Small(value: -1)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 720.203303281232 < 936.6130330016003.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/15/2023 7:15:22 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 35.249899840322435 (T) = (0 -734.5593417431369) / Math.Sqrt((7771.825305781427 / (194)) + (473.89661609312645 / (32))) is greater than 1.9706109611022053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (194) + (32) - 2, .975) and 0.26224692196468125 = (995.670995638965 - 734.5593417431369) / 995.670995638965 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 582.0009367748637 < 754.8838858255058.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 23.798933708350674 (T) = (0 -596.3424029089945) / Math.Sqrt((12773.723670705198 / (194)) + (252.0049371920753 / (32))) is greater than 1.9706109611022053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (194) + (32) - 2, .975) and 0.25520472290620283 = (800.6796246559617 - 596.3424029089945) / 800.6796246559617 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.StringFormat(value: 32)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.929294960346256 < 7.901238860994894.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 26.096534015279584 (T) = (0 -6948.659460474508) / Math.Sqrt((1311801.5030933253 / (194)) + (55391.23441282889 / (32))) is greater than 1.9706109611022053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (194) + (32) - 2, .975) and 0.2571160714029868 = (9353.627387790611 - 6948.659460474508) / 9353.627387790611 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.268638191893219 < 12.179635376252927.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 7.777918305092729 (T) = (0 -11470.650397209016) / Math.Sqrt((1898370.6616823785 / (193)) + (37997.92541861701 / (32))) is greater than 1.9706589608356722 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (193) + (32) - 2, .975) and 0.06646124532748722 = (12287.278208640564 - 11470.650397209016) / 12287.278208640564 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.InterpolateIntoString(value: 32)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.010418258212374 < 12.842441585561138.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 1/15/2023 10:25:39 PM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 8.831736644254791 (T) = (0 -12181.7619402098) / Math.Sqrt((1738702.7511535508 / (195)) + (46435.448521687074 / (32))) is greater than 1.9705633902925863 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (195) + (32) - 2, .975) and 0.06874517664835579 = (13081.018894878718 - 12181.7619402098) / 13081.018894878718 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_Flags(value: 32)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 827.1881842248863 < 1.0485795461674177.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 34.53372202962532 (T) = (0 -815.52673279313) / Math.Sqrt((7377.452199015507 / (195)) + (590.6052572897656 / (32))) is greater than 1.9705633902925863 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (195) + (32) - 2, .975) and 0.24110262676917632 = (1074.6205765889272 - 815.52673279313) / 1074.6205765889272 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.InterpolateIntoStringBuilder_Flags(value: 32)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 11.345271415678186 < 11.822569956623054.
IsChangePoint: Marked as a change because one of 1/12/2023 3:42:01 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 8.342950605926932 (T) = (0 -11237.086838265774) / Math.Sqrt((1800350.502344818 / (192)) + (39521.06018303514 / (32))) is greater than 1.9707073953201992 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (192) + (32) - 2, .975) and 0.07104846821319175 = (12096.526518075276 - 11237.086838265774) / 12096.526518075276 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_NonFlags_Large(value: 42)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 619.8104680394317 < 822.6725513279544.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 1/12/2023 3:42:01 AM, 1/15/2023 4:08:29 PM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 23.839697624176495 (T) = (0 -641.3830462586478) / Math.Sqrt((20089.012642210015 / (194)) + (457.10674993000526 / (32))) is greater than 1.9706109611022053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (194) + (32) - 2, .975) and 0.2874856831883279 = (900.1686438087036 - 641.3830462586478) / 900.1686438087036 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Enum.ToString_Flags(value: 36)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 847.4135177508504 < 1.0946999415456142.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 31.071613780238597 (T) = (0 -842.616156693363) / Math.Sqrt((7068.7545966472435 / (194)) + (1170.6915718188015 / (32))) is greater than 1.9706109611022053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (194) + (32) - 2, .975) and 0.2396055306900975 = (1108.1303069682779 - 842.616156693363) / 1108.1303069682779 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 0202b242169f8d5955e0411dc5e4f42a5d016c1b |
Compare | 24107379cd4fbfe236f0a1cf92dbd6c4ad944fec |
Diff | Diff |
Improvements in System.Tests.Perf_Version
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
TryFormat4 - Duration of single invocation | 1.88 μs | 992.86 ns | 0.53 | 0.11 | False | |||||
ToString3 - Duration of single invocation | 1.70 μs | 1.05 μs | 0.62 | 0.06 | False | |||||
ToStringL - Duration of single invocation | 2.45 μs | 1.76 μs | 0.72 | 0.09 | False | |||||
ToString4 - Duration of single invocation | 2.13 μs | 1.28 μs | 0.60 | 0.08 | False | |||||
TryFormat2 - Duration of single invocation | 917.72 ns | 518.13 ns | 0.56 | 0.09 | False | |||||
TryFormat3 - Duration of single invocation | 1.40 μs | 729.79 ns | 0.52 | 0.09 | False | |||||
TryFormatL - Duration of single invocation | 2.18 μs | 1.49 μs | 0.68 | 0.09 | False | |||||
ToString2 - Duration of single invocation | 1.24 μs | 806.70 ns | 0.65 | 0.09 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'
Payloads
Histogram
System.Tests.Perf_Version.TryFormat4
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 992.860077158324 < 1.741770321024235.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 33.914861440937564 (T) = (0 -991.9501992425254) / Math.Sqrt((121261.67898038065 / (299)) + (1160.6088195635218 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.4181527934196376 = (1704.8293573022788 - 991.9501992425254) / 1704.8293573022788 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToString3
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.0522938944031506 < 1.6112435431371757.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 42.02507418705528 (T) = (0 -1038.9232369445745) / Math.Sqrt((51295.002386310916 / (299)) + (1185.5122453880688 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.36877899324358693 = (1645.894584977735 - 1038.9232369445745) / 1645.894584977735 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToStringL
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.759734170347981 < 2.3714869509610077.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/15/2023 4:08:29 PM, 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 58.630484397544656 (T) = (0 -1752.3469947296305) / Math.Sqrt((87500.96496173713 / (299)) + (2645.621073079174 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.39327520460843274 = (2888.207319100422 - 1752.3469947296305) / 2888.207319100422 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToString4
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.2773342048199834 < 2.0173746366166463.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 38.84384159377969 (T) = (0 -1304.311543290916) / Math.Sqrt((96260.83418067398 / (299)) + (1897.6849864039011 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.3676853351333497 = (2062.757066635461 - 1304.311543290916) / 2062.757066635461 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.TryFormat2
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 518.1311255992244 < 880.1244498680181.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 34.2167451024951 (T) = (0 -525.0333459397746) / Math.Sqrt((30015.323269150696 / (299)) + (204.56373606478658 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.4024254445978987 = (878.6072653084859 - 525.0333459397746) / 878.6072653084859 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.TryFormat3
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 729.7945119526396 < 1.3210621384923098.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 34.62216904906278 (T) = (0 -751.6824621483197) / Math.Sqrt((67212.88168789464 / (299)) + (610.0615703044 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.41835511773954637 = (1292.3391661713706 - 751.6824621483197) / 1292.3391661713706 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.TryFormatL
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.486235989131342 < 2.037452798548641.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 1/12/2023 3:42:01 AM, 1/15/2023 4:08:29 PM, 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 63.55858470161165 (T) = (0 -1410.362470835293) / Math.Sqrt((67980.23808052542 / (299)) + (1296.642703539459 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.42448888026706155 = (2450.625926202366 - 1410.362470835293) / 2450.625926202366 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Version.ToString2
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 806.6976407488386 < 1.183086656598896.
IsChangePoint: Marked as a change because one of 1/15/2023 4:08:29 PM, 2/1/2023 12:50:52 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 42.03770357063851 (T) = (0 -806.8239625525238) / Math.Sqrt((19876.602651253037 / (299)) + (1244.9980230023202 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.3484777079102295 = (1238.367393331424 - 806.8239625525238) / 1238.367393331424 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 0202b242169f8d5955e0411dc5e4f42a5d016c1b |
Compare | 24107379cd4fbfe236f0a1cf92dbd6c4ad944fec |
Diff | Diff |
Improvements in System.Tests.Perf_Decimal
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Round - Duration of single invocation | 374.85 ns | 168.29 ns | 0.45 | 0.06 | False | |||||
Floor - Duration of single invocation | 312.27 ns | 108.96 ns | 0.35 | 0.05 | False | |||||
Divide - Duration of single invocation | 1.71 μs | 831.15 ns | 0.49 | 0.06 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Decimal*'
Payloads
Histogram
System.Tests.Perf_Decimal.Round
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 168.28632164937076 < 355.2228549720679.
IsChangePoint: Marked as a change because one of 1/13/2023 11:01:49 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 25.268794834230206 (T) = (0 -169.24566418931315) / Math.Sqrt((9620.159988871917 / (299)) + (3.38828547156693 / (31))) is greater than 1.9672228266464409 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (31) - 2, .975) and 0.45896771157944555 = (312.81989598697544 - 169.24566418931315) / 312.81989598697544 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Decimal.Floor
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 108.95687509549275 < 298.1062596391903.
IsChangePoint: Marked as a change because one of 1/13/2023 11:01:49 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 28.76282852439328 (T) = (0 -110.12381189581542) / Math.Sqrt((8924.69071801176 / (299)) + (4.331499159417129 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.5885096095285645 = (267.62183138626636 - 110.12381189581542) / 267.62183138626636 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Decimal.Divide
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 831.1528819482463 < 1.6344916751246217.
IsChangePoint: Marked as a change because one of 1/13/2023 11:01:49 AM, 2/15/2023 6:58:43 PM, 2/21/2023 8:58:51 AM falls between 2/12/2023 2:35:10 PM and 2/21/2023 8:58:51 AM.
IsImprovementStdDev: Marked as improvement because 31.59828232508761 (T) = (0 -846.4472227867492) / Math.Sqrt((143824.18006140908 / (299)) + (219.2482947279804 / (32))) is greater than 1.9672006833568965 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (32) - 2, .975) and 0.4519185862504692 = (1544.382278895466 - 846.4472227867492) / 1544.382278895466 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository