Start-up performance degradation of.NET 8 in Amazon Linux 2023 over Amazon Linux 2 #96740
Description
Description
We are observing a start-up performance degradation of.NET 8 in Amazon Linux 2023 over Amazon Linux 2 for https calls. The performance is intact if it's a local API call (file read/write).
Configuration
- Which version of .NET is the code running on?
- .NET 8
- What OS version, and what distro if applicable?
- Amazon Linux 2023
- What is the architecture (x64, x86, ARM, ARM64)?
*x86_64 , ARM64 - If relevant, what are the specs of the machine?
Regression?
- Is this a regression from a previous build or release of .NET Core, or from .NET Framework?
- I have not tried the performance between.NET 6 and.NET 8 in Amazon Linux 2023, but if needed, I can get that information.
Data
Amazon Linux 2
// AfterActualRun
WorkloadResult 1: 1 op, 90633901.00 ns, 90.6339 ms/op <= ColdStart
WorkloadResult 2: 1 op, 1716177.00 ns, 1.7162 ms/op
WorkloadResult 3: 1 op, 1006949.00 ns, 1.0069 ms/op
WorkloadResult 4: 1 op, 894721.00 ns, 894.7210 us/op
WorkloadResult 5: 1 op, 997069.00 ns, 997.0690 us/op
// * Summary *
BenchmarkDotNet v0.13.12, Amazon Linux 2
Intel Xeon Platinum 8275CL CPU 3.00GHz, 1 CPU, 2 logical cores and 1 physical core
.NET SDK 8.0.101
[Host] : .NET 8.0.1 (8.0.123.58001), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
Job-LDEQOF : .NET 8.0.1 (8.0.123.58001), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
IterationCount=5 RunStrategy=ColdStart
| Method | Mean | Error | StdDev | Median | Min | Max |
|---|---|---|---|---|---|---|
| Foo | 19.95 ms | 161.52 ms | 41.95 ms | 1.330 ms | 0.8515 ms | 94.98 ms |
Amazon Linux 2023
// AfterActualRun
WorkloadResult 1: 1 op, 137960338.00 ns, 137.9603 ms/op <= ColdStart
WorkloadResult 2: 1 op, 2505819.00 ns, 2.5058 ms/op
WorkloadResult 3: 1 op, 1495447.00 ns, 1.4954 ms/op
WorkloadResult 4: 1 op, 1204715.00 ns, 1.2047 ms/op
WorkloadResult 5: 1 op, 1379885.00 ns, 1.3799 ms/op
// * Summary *
BenchmarkDotNet v0.13.12, Amazon Linux 2023
Intel Xeon Platinum 8275CL CPU 3.00GHz, 1 CPU, 2 logical cores and 1 physical core
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
Job-GQCBXH : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX-512F+CD+BW+DQ+VL
IterationCount=5 RunStrategy=ColdStart
| Method | Mean | Error | StdDev | Median | Min | Max |
|---|---|---|---|---|---|---|
| Foo | 28.91 ms | 234.75 ms | 60.96 ms | 1.495 ms | 1.205 ms | 138.0 ms |
Analysis
- Using dotnet-trace + perfview, we were able to pin point the problem in the Cryptography library (more details in the PerfView regression result info.zip) (thanks to @mconnew and @adamsitnik for helping with initial analysis). It seems (not sure) that the problem is with loading certificates from the local store. We would like to confirm ? and know the reason behind it. Along with that, what's the right way to resolve it?
- In the attachment I have added the sample app , the benchmark test and Perfview regression screenshot (base of the prefview regression is Amazon Linux 2). If you need full trace , let me know I can provide.