-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Closed as not planned
Labels
area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark
Milestone
Description
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 9e05d3347be231ae1bfcc7037ddecfb5c4ca51a1 |
Compare | 120b7215c9d029fccee94f46864494d79074a0eb |
Diff | Diff |
Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Count - Duration of single invocation | 280.39 ms | 303.10 ms | 1.08 | 0.01 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig*'
Payloads
Histogram
System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 303.0988273333333 > 291.22724740673084.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 1/27/2023 7:12:54 PM, 2/1/2023 8:39:12 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -6.8991552954396305 (T) = (0 -303268185.5826008) / Math.Sqrt((192513142277717.06 / (27)) + (51382853519.585594 / (32))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (32) - 2, .025) and -0.06468243531225498 = (284843795.2239316 - 303268185.5826008) / 284843795.2239316 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 9e05d3347be231ae1bfcc7037ddecfb5c4ca51a1 |
Compare | 120b7215c9d029fccee94f46864494d79074a0eb |
Diff | Diff |
Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
Get_EnumerateHeaders_Unvalidated - Duration of single invocation | 8.95 ms | 17.44 ms | 1.95 | 0.90 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'
Payloads
Histogram
System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: True, chunkedResponse: True, responseLength: 100000)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.440992809375 > 11.242045705142887.
IsChangePoint: Marked as a change because one of 2/1/2023 8:39:12 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -3.548097165978861 (T) = (0 -15794333.253045475) / Math.Sqrt((17374797894054.414 / (27)) + (37834194257680.664 / (32))) is less than -2.0024654592901125 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (32) - 2, .025) and -0.4358447430604137 = (11000028.609904464 - 15794333.253045475) / 11000028.609904464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Architecture | x64 |
---|---|
OS | ubuntu 18.04 |
Baseline | 9e05d3347be231ae1bfcc7037ddecfb5c4ca51a1 |
Compare | 120b7215c9d029fccee94f46864494d79074a0eb |
Diff | Diff |
Regressions in SciMark2.kernel
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
---|---|---|---|---|---|---|---|---|---|---|
benchSparseMult - Duration of single invocation | 609.18 ms | 696.21 ms | 1.14 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'SciMark2.kernel*'
Payloads
Histogram
SciMark2.kernel.benchSparseMult
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 696.2089355333333 > 639.6030126750002.
IsChangePoint: Marked as a change because one of 2/1/2023 8:39:12 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -657.0518642017826 (T) = (0 -696370455.7566222) / Math.Sqrt((209470639230.72266 / (26)) + (303351012897.6243 / (32))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (26) + (32) - 2, .025) and -0.14278876574405194 = (609360606.8162792 - 696370455.7566222) / 609360606.8162792 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Metadata
Metadata
Assignees
Labels
area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMICLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMItenet-performancePerformance related issuePerformance related issuetenet-performance-benchmarksIssue from performance benchmarkIssue from performance benchmark