-
Notifications
You must be signed in to change notification settings - Fork 4
Description
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 6123cb00bbec8ae3500c970890f5b0efcf7950e8 |
Compare | ffbcb8336ac67496b0565112f663f74fda6cac34 |
Diff | Diff |
Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
EncodeUtf8 - Duration of single invocation | 79.49 ns | 86.50 ns | 1.09 | 0.01 | False | |||||
EncodeUtf8 - Duration of single invocation | 66.06 ns | 74.62 ns | 1.13 | 0.04 | False | |||||
EncodeUtf8 - Duration of single invocation | 57.32 ns | 66.33 ns | 1.16 | 0.04 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'
Payloads
Histogram
System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,&lorem ipsum=dolor sit amet,512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.49511848181112 > 83.5538407638473.
IsChangePoint: Marked as a change because one of 2/24/2023 8:06:06 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -119.57934323858325 (T) = (0 -86.18799593706828) / Math.Sqrt((0.01989936205140299 / (24)) + (0.08040259258575326 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.08316597032593837 = (79.57044284832288 - 86.18799593706828) / 79.57044284832288 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: JavaScript,&Hello+<World>!,512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 74.6237372864477 > 69.41757445712778.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -116.76856758117555 (T) = (0 -74.6062875482607) / Math.Sqrt((0.010243949585289716 / (24)) + (0.17093653979911036 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.12687771958258215 = (66.20619633503478 - 74.6062875482607) / 66.20619633503478 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,hello "there",512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 66.32828236958538 > 60.18437576911939.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -144.8997615076406 (T) = (0 -66.77881697920037) / Math.Sqrt((0.011856860513257208 / (24)) + (0.1346398819420697 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.16440283453627252 = (57.350270025575185 - 66.77881697920037) / 57.350270025575185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 6123cb00bbec8ae3500c970890f5b0efcf7950e8 |
Compare | ffbcb8336ac67496b0565112f663f74fda6cac34 |
Diff | Diff |
Regressions in System.Memory.Span<Char>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
IndexOfAnyTwoValues - Duration of single invocation | 12.10 ns | 14.07 ns | 1.16 | 0.01 | False | |||||
IndexOfAnyThreeValues - Duration of single invocation | 5.50 ns | 6.58 ns | 1.20 | 0.01 | False | |||||
Clear - Duration of single invocation | 3.67 ns | 4.93 ns | 1.35 | 0.26 | False | |||||
IndexOfAnyTwoValues - Duration of single invocation | 3.55 ns | 4.56 ns | 1.28 | 0.01 | False | |||||
IndexOfAnyThreeValues - Duration of single invocation | 13.86 ns | 15.91 ns | 1.15 | 0.02 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span<Char>*'
Payloads
Histogram
System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.074189979373168 > 12.731450378667962.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/17/2023 9:53:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -231.83331520873986 (T) = (0 -14.065880044275872) / Math.Sqrt((0.0015631750257780485 / (24)) + (0.000158012843443573 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.15948724186938523 = (12.131121013111049 - 14.065880044275872) / 12.131121013111049 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 33)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.5822732227539635 > 5.775606499312917.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -742.767446226959 (T) = (0 -6.57915016805052) / Math.Sqrt((6.636319589554143E-06 / (24)) + (6.409153022641278E-05 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.19603376949097623 = (5.500806361722179 - 6.57915016805052) / 5.500806361722179 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.Span<Char>.Clear(Size: 33)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.9320827964007865 > 4.065434581487989.
IsChangePoint: Marked as a change because one of 1/7/2023 12:45:06 AM, 1/10/2023 1:28:02 AM, 2/17/2023 9:53:52 PM, 3/5/2023 11:54:18 PM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.113556971926728 (T) = (0 -4.738262217306085) / Math.Sqrt((0.04425202676076872 / (22)) + (0.17709462286183417 / (37))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (37) - 2, .025) and -0.21356612783079607 = (3.9044120535693847 - 4.738262217306085) / 3.9044120535693847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 4)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.561214224648973 > 3.7283327037432197.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/17/2023 9:53:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -1307.2256261227444 (T) = (0 -4.559385808248598) / Math.Sqrt((2.648202526564375E-06 / (24)) + (1.695824428428527E-05 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.2839155343246184 = (3.5511571332821235 - 4.559385808248598) / 3.5511571332821235 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.90542924475584 > 14.547626528731172.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -168.06203308429758 (T) = (0 -16.249888418820845) / Math.Sqrt((2.316577996253899E-05 / (24)) + (0.007067520835814389 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.17276943213676807 = (13.855995879099437 - 16.249888418820845) / 13.855995879099437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 6123cb00bbec8ae3500c970890f5b0efcf7950e8 |
Compare | ffbcb8336ac67496b0565112f663f74fda6cac34 |
Diff | Diff |
Regressions in System.Tests.Perf_Double
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
TryParse - Duration of single invocation | 125.20 ns | 136.58 ns | 1.09 | 0.11 | False | |||||
ToString - Duration of single invocation | 185.24 ns | 199.66 ns | 1.08 | 0.04 | False | |||||
ToStringWithFormat - Duration of single invocation | 167.71 ns | 180.04 ns | 1.07 | 0.04 | False | |||||
ToString - Duration of single invocation | 182.68 ns | 197.84 ns | 1.08 | 0.04 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'
Payloads
Histogram
System.Tests.Perf_Double.TryParse(value: "1.7976931348623157e+308")
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 136.58422990731475 > 132.1264845186577.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 2/8/2023 3:55:57 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -12.347099224407398 (T) = (0 -144.32329930155305) / Math.Sqrt((33.28839344479712 / (24)) + (11.390719690484914 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.12606772630697854 = (128.16573633174966 - 144.32329930155305) / 128.16573633174966 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.ToString(value: -1.7976931348623157E+308)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 199.65683680795124 > 193.63268538069798.
IsChangePoint: Marked as a change because one of 1/19/2023 8:10:32 PM, 1/26/2023 7:55:58 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:36:59 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -10.942253093135342 (T) = (0 -200.52522225292913) / Math.Sqrt((24.846025989354953 / (25)) + (3.519626324509351 / (35))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (35) - 2, .025) and -0.06054159426487934 = (189.0781307751766 - 200.52522225292913) / 189.0781307751766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.ToStringWithFormat(value: 1.7976931348623157E+308, format: "G17")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.03961044996865 > 175.81341869506457.
IsChangePoint: Marked as a change because one of 1/13/2023 11:20:20 PM, 2/14/2023 11:21:41 AM, 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -16.742243180659532 (T) = (0 -181.1576209930889) / Math.Sqrt((4.760415377768425 / (24)) + (10.9727263039687 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07015382872748935 = (169.28185101061766 - 181.1576209930889) / 169.28185101061766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Double.ToString(value: 1.7976931348623157E+308)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 197.8402573512462 > 192.09549389273545.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 2/17/2023 5:12:50 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:36:59 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -12.561049854828598 (T) = (0 -199.00047558944723) / Math.Sqrt((16.35165569652949 / (25)) + (8.578536276383302 / (35))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (35) - 2, .025) and -0.06366440657833547 = (187.08953158412515 - 199.00047558944723) / 187.08953158412515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 6123cb00bbec8ae3500c970890f5b0efcf7950e8 |
Compare | ffbcb8336ac67496b0565112f663f74fda6cac34 |
Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
ReplaceWords - Duration of single invocation | 56.35 μs | 60.10 μs | 1.07 | 0.18 | False | |||||
MatchesWord - Duration of single invocation | 1.04 μs | 1.18 μs | 1.13 | 0.05 | False | |||||
MatchesBoundary - Duration of single invocation | 49.99 μs | 62.35 μs | 1.25 | 0.05 | False | |||||
MatchesBoundary - Duration of single invocation | 128.42 μs | 137.79 μs | 1.07 | 0.18 | False | |||||
MatchesBoundary - Duration of single invocation | 50.42 μs | 63.28 μs | 1.26 | 0.02 | False | |||||
MatchesWord - Duration of single invocation | 897.42 ns | 952.33 ns | 1.06 | 0.04 | False | |||||
Uri_IsNotMatch - Duration of single invocation | 90.57 ns | 98.96 ns | 1.09 | 0.09 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: None)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 60.099919250645996 > 59.12534628787353.
IsChangePoint: Marked as a change because one of 1/3/2023 9:28:36 AM, 1/20/2023 12:33:18 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -5.150253894798643 (T) = (0 -61492.338521293845) / Math.Sqrt((4929824.096610276 / (24)) + (19585413.171780925 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.07817498476091901 = (57033.72772549488 - 61492.338521293845) / 57033.72772549488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1767921299447848 > 1.1082952884529715.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -23.470873459607 (T) = (0 -1213.1073300192284) / Math.Sqrt((260.39742048214026 / (24)) + (1020.8398051689537 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.13784552285538618 = (1066.1441343768486 - 1213.1073300192284) / 1066.1441343768486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: IgnoreCase, Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 62.35373861696072 > 52.47879842252397.
IsChangePoint: Marked as a change because one of 1/20/2023 8:02:54 AM, 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/3/2023 4:59:02 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -4.242580088948566 (T) = (0 -57757.51326160844) / Math.Sqrt((25627734.76320062 / (24)) + (3468110.623573912 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.08607724735036967 = (53179.931172037344 - 57757.51326160844) / 53179.931172037344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: None)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.79348621511232 > 134.64698924036156.
IsChangePoint: Marked as a change because one of 1/3/2023 6:29:42 AM, 2/1/2023 6:08:01 AM, 2/17/2023 9:53:52 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -15.42815884980818 (T) = (0 -137755.8522680183) / Math.Sqrt((10178935.297662929 / (24)) + (527925.319306239 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.08013234255888073 = (127536.08686660438 - 137755.8522680183) / 127536.08686660438 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesBoundary(Options: Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.28360956477732 > 52.568392088043744.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -57.71819385049054 (T) = (0 -61285.16258625139) / Math.Sqrt((450311.7474320481 / (24)) + (657056.3589406938 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.22129041596132856 = (50180.66283440969 - 61285.16258625139) / 50180.66283440969 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 952.3261018717368 > 943.4432166812592.
IsChangePoint: Marked as a change because one of 1/18/2023 3:20:52 PM, 2/9/2023 4:27:10 AM, 2/28/2023 5:58:27 PM, 3/6/2023 11:36:59 AM, 3/13/2023 2:14:21 PM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -17.72720879187587 (T) = (0 -958.5371067781595) / Math.Sqrt((393.53269149773416 / (57)) + (42.8579794795631 / (3))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (57) + (3) - 2, .025) and -0.09305499611614286 = (876.9340153826165 - 958.5371067781595) / 876.9340153826165 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 98.95733409121391 > 95.37176299970194.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 6:24:13 PM, 1/27/2023 7:12:54 PM, 3/10/2023 12:53:36 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -11.287470726246964 (T) = (0 -100.52911394437659) / Math.Sqrt((9.3495984907112 / (43)) + (1.7230851213099918 / (16))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (16) - 2, .025) and -0.06840134140125713 = (94.09302482953463 - 100.52911394437659) / 94.09302482953463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 6123cb00bbec8ae3500c970890f5b0efcf7950e8 |
Compare | ffbcb8336ac67496b0565112f663f74fda6cac34 |
Diff | Diff |
Regressions in Span.IndexerBench
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
CoveredIndex2 - Duration of single invocation | 757.48 ns | 868.67 ns | 1.15 | 0.03 | False | |||||
WriteViaIndexer2 - Duration of single invocation | 638.48 ns | 683.82 ns | 1.07 | 0.02 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Span.IndexerBench*'
Payloads
Histogram
Span.IndexerBench.CoveredIndex2(length: 1024)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 868.6694328219432 > 793.7614795127765.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -528.7165591326337 (T) = (0 -868.644605622546) / Math.Sqrt((1.0404024993999381 / (24)) + (0.07327696756848821 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.14909919548168005 = (755.9352656742807 - 868.644605622546) / 755.9352656742807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Span.IndexerBench.WriteViaIndexer2(length: 1024)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 683.8237477690978 > 670.8679993054843.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 1/18/2023 3:20:52 PM, 1/27/2023 7:12:54 PM, 2/9/2023 12:40:40 PM, 2/24/2023 9:01:57 AM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -111.26261905677086 (T) = (0 -681.3638234450007) / Math.Sqrt((1.7666489030307402 / (24)) + (2.499640908718883 / (36))) is less than -2.001717484144427 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (36) - 2, .025) and -0.06582521674155604 = (639.2828887348604 - 681.3638234450007) / 639.2828887348604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | Windows 10.0.19042 |
Baseline | 6123cb00bbec8ae3500c970890f5b0efcf7950e8 |
Compare | ffbcb8336ac67496b0565112f663f74fda6cac34 |
Diff | Diff |
Regressions in System.Collections.IterateFor<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
ImmutableList - Duration of single invocation | 5.57 μs | 6.27 μs | 1.12 | 0.02 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateFor<Int32>*'
Payloads
Histogram
System.Collections.IterateFor<Int32>.ImmutableList(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.268370710303749 > 5.868966593351013.
IsChangePoint: Marked as a change because one of 2/23/2023 8:54:44 PM, 3/6/2023 11:14:52 AM, 3/14/2023 5:57:45 AM falls between 3/5/2023 5:50:49 PM and 3/14/2023 5:57:45 AM.
IsRegressionStdDev: Marked as regression because -48.430665834731016 (T) = (0 -6405.035245773167) / Math.Sqrt((2769.2766252571782 / (24)) + (5453.576731893705 / (35))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (35) - 2, .025) and -0.14223318578705976 = (5607.467306563813 - 6405.035245773167) / 5607.467306563813 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository