[Perf] Linux/x64: 454 Regressions on 9/26/2023 5:24:05 PM #22385
Description
Run Information
| Name | Value |
|---|---|
| Architecture | x64 |
| OS | ubuntu 22.04 |
| Queue | TigerUbuntu |
| Baseline | fc5c29692fc1a92426b7d1ce8c501e7696062bb6 |
| Compare | 1be2db9f75035a01e1955a33b35743018b6d85a6 |
| Diff | Diff |
| Configs | AOT:true, CompilationMode:wasm, RunKind:micro |
Regressions in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
|---|---|---|---|---|---|---|---|---|
| 4.34 μs | 5.10 μs | 1.18 | 0.20 | False | ||||
| 241.36 μs | 258.09 μs | 1.07 | 0.04 | False | ||||
| 8.37 μs | 9.40 μs | 1.12 | 0.14 | False | ||||
| 8.57 μs | 9.84 μs | 1.15 | 0.11 | False | ||||
| 9.62 μs | 10.87 μs | 1.13 | 0.11 | False | ||||
| 5.49 μs | 6.69 μs | 1.22 | 0.13 | False | ||||
| 7.24 μs | 8.05 μs | 1.11 | 0.13 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>*'Payloads
System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.CachedJsonSerializerContext
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.100523353942944 > 4.4702239647536155.
IsChangePoint: Marked as a change because one of 8/21/2023 10:18:47 AM, 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM.
IsRegressionStdDev: Marked as regression because -13.601865147504437 (T) = (0 -5004.356264736374) / Math.Sqrt((13090.965074921933 / (51)) + (11369.307920302243 / (4))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (4) - 2, .025) and -0.17828377012597793 = (4247.157086956503 - 5004.356264736374) / 4247.157086956503 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCustomConverter
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.0935465286832 > 250.6117123201885.
IsChangePoint: Marked as a change because one of 8/10/2023 8:47:51 AM, 9/25/2023 10:28:34 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM.
IsRegressionStdDev: Marked as regression because -9.125609786452841 (T) = (0 -256062.12686409106) / Math.Sqrt((15271883.973650124 / (46)) + (28406660.998105086 / (8))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (8) - 2, .025) and -0.07552883728335538 = (238080.20574406016 - 256062.12686409106) / 238080.20574406016 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewDefaultOptions
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.39675822790876 > 8.753806682573117.
IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM.
IsRegressionStdDev: Marked as regression because -14.345817462796044 (T) = (0 -9644.008927213637) / Math.Sqrt((55536.523787042395 / (50)) + (27969.413307408977 / (4))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (4) - 2, .025) and -0.15460613477651597 = (8352.639603010874 - 9644.008927213637) / 8352.639603010874 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCustomizedOptions
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.844645757831696 > 9.011393570361816.
IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM.
IsRegressionStdDev: Marked as regression because -11.574256900191573 (T) = (0 -9947.613130381598) / Math.Sqrt((38030.29283586491 / (50)) + (43208.476843960874 / (4))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (4) - 2, .025) and -0.14300546605650022 = (8703.03198522926 - 9947.613130381598) / 8703.03198522926 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewCachedCustomConverter
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.874584178946192 > 9.858509310326363.
IsChangePoint: Marked as a change because one of 8/22/2023 4:48:44 PM, 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM.
IsRegressionStdDev: Marked as regression because -17.31964104446051 (T) = (0 -11006.718833541054) / Math.Sqrt((71108.33539221596 / (50)) + (18641.102206751137 / (3))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (3) - 2, .025) and -0.15942348460916644 = (9493.26883546036 - 11006.718833541054) / 9493.26883546036 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.CachedDefaultOptions
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.690192466813368 > 5.791846481442025.
IsChangePoint: Marked as a change because one of 9/26/2023 3:02:05 PM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM.
IsRegressionStdDev: Marked as regression because -16.3854792821721 (T) = (0 -6860.241281141385) / Math.Sqrt((22187.885913897528 / (51)) + (27715.5753048464 / (4))) is less than -2.005745995316835 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (4) - 2, .025) and -0.25780288484532726 = (5454.146562865447 - 6860.241281141385) / 5454.146562865447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.NewJsonSerializerContext
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.048988382841117 > 7.368679430049158.
IsChangePoint: Marked as a change because one of 8/22/2023 10:50:09 AM, 9/25/2023 2:54:53 AM, 9/27/2023 11:11:59 PM falls between 9/19/2023 8:08:41 AM and 9/27/2023 11:11:59 PM.
IsRegressionStdDev: Marked as regression because -9.786029631897627 (T) = (0 -7735.024634582908) / Math.Sqrt((27701.675285527694 / (42)) + (37432.59730869336 / (12))) is less than -2.0066468050606243 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (42) + (12) - 2, .025) and -0.08433201875563279 = (7133.446675732711 - 7735.024634582908) / 7133.446675732711 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository