[Perf] Linux/arm64: 4 Regressions on 9/22/2023 12:22:36 AM #92774
Description
Run Information
| Name | Value |
|---|---|
| Architecture | arm64 |
| OS | ubuntu 20.04 |
| Queue | AmpereUbuntu |
| Baseline | 736dabeca728ccf8b911d96d1b4c575b4d0db7d2 |
| Compare | a54ccf39b17586e7c2a38b882e80a3cc62d83d7d |
| Diff | Diff |
| Configs | CompilationMode:tiered, RunKind:micro |
Regressions in System.Collections.Tests.Perf_BitArray
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 27.12 ns | 30.38 ns | 1.12 | 0.17 | False | ||||
| 26.86 ns | 31.59 ns | 1.18 | 0.19 | False | ||||
| 26.89 ns | 30.34 ns | 1.13 | 0.16 | False | ||||
| 20.66 ns | 23.04 ns | 1.11 | 0.21 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'Payloads
System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.383537652499253 > 28.435176409751723.
IsChangePoint: Marked as a change because one of 8/11/2023 2:10:11 AM, 9/21/2023 2:59:58 PM, 9/26/2023 10:37:55 AM falls between 9/14/2023 3:34:28 AM and 9/26/2023 10:37:55 AM.
IsRegressionStdDev: Marked as regression because -11.647751569056705 (T) = (0 -30.78837159224807) / Math.Sqrt((0.2889938127686523 / (10)) + (0.437199866498403 / (7))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (7) - 2, .025) and -0.12911050037706534 = (27.267810884733002 - 30.78837159224807) / 27.267810884733002 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.58777843320124 > 28.3029718981856.
IsChangePoint: Marked as a change because one of 8/11/2023 2:10:11 AM, 9/21/2023 2:59:58 PM, 9/26/2023 10:37:55 AM falls between 9/14/2023 3:34:28 AM and 9/26/2023 10:37:55 AM.
IsRegressionStdDev: Marked as regression because -15.351007875239846 (T) = (0 -30.751370237604675) / Math.Sqrt((0.11759462932962167 / (10)) + (0.30076020044386614 / (7))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (7) - 2, .025) and -0.13221992888317283 = (27.160244624856563 - 30.751370237604675) / 27.160244624856563 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.33995207145409 > 28.397674394292075.
IsChangePoint: Marked as a change because one of 8/11/2023 2:10:11 AM, 9/21/2023 2:59:58 PM, 9/26/2023 10:37:55 AM falls between 9/14/2023 3:34:28 AM and 9/26/2023 10:37:55 AM.
IsRegressionStdDev: Marked as regression because -12.886410778475025 (T) = (0 -30.54942087445162) / Math.Sqrt((0.34476016312751784 / (10)) + (0.21582843119739759 / (7))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (7) - 2, .025) and -0.1208235487870845 = (27.256226823134757 - 30.54942087445162) / 27.256226823134757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.036922947236878 > 21.69481572774904.
IsChangePoint: Marked as a change because one of 9/21/2023 2:59:58 PM, 9/26/2023 10:37:55 AM falls between 9/14/2023 3:34:28 AM and 9/26/2023 10:37:55 AM.
IsRegressionStdDev: Marked as regression because -25.57321509734295 (T) = (0 -23.22939316863335) / Math.Sqrt((0.05341585145566489 / (10)) + (0.03238681190947795 / (7))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (10) + (7) - 2, .025) and -0.12348835145689395 = (20.67613174494459 - 23.22939316863335) / 20.67613174494459 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository