-
Notifications
You must be signed in to change notification settings - Fork 4
Description
Run Information
Name | Value |
---|---|
Architecture | x64 |
OS | ubuntu 22.04 |
Queue | TigerUbuntu |
Baseline | 7abea9ea6ac3cde23fe1a890338e22a5a58a51b6 |
Compare | 1d352cb36c06a16bc93e424809fb431d5795e741 |
Diff | Diff |
Configs | CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono |
Regressions in System.Memory.Span<Byte>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
---|---|---|---|---|---|---|---|---|
60.48 ns | 397.71 ns | 6.58 | 0.18 | True | ||||
256.92 ns | 3.14 ΞΌs | 12.23 | 0.08 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Repro Steps
Prerequisites (Build files either built locally or downloaded from payload above)
- Libraries build extracted to
runtime/artifacts
or build instructions: Libraries README args:-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
- CoreCLR product build extracted to
runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release
, build instructions: CoreCLR README args:-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
- Mono Runtime build extracted to
runtime/artifacts/bin/mono/$RunOS.$RunArch.Release
, build instructions: MONO README args:-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
- Dotnet SDK installed for dotnet commands
- Running commands from the runtime folder
Linux
# Set $RunDir to the runtime directory
RunDir=`pwd`
# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun
# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Byte>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun"
# Individual Commands:
# Restore
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span<Byte>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages
Windows
# Set $RunDir to the runtime directory
$RunDir="FullPathHere"
# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y
# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Byte>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe"
# Individual Commands:
# Restore
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span<Byte>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages
Payloads
System.Memory.Span<Byte>.LastIndexOfValue(Size: 33)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 397.71136071466543 > 64.33447055628001.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -189.2455527947637 (T) = (0 -403.05511784883623) / Math.Sqrt((3.0142742390094726 / (23)) + (65.5663766701062 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -5.530742921119259 = (61.716579984403886 - 403.05511784883623) / 61.716579984403886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Memory.Span<Byte>.LastIndexOfValue(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.1410916589103994 > 268.29530563977085.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -246.82954960363776 (T) = (0 -2938.594397480665) / Math.Sqrt((1.390864280518092 / (23)) + (2480.250044421032 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -10.504251481635073 = (255.43551461576791 - 2938.594397480665) / 255.43551461576791 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Name | Value |
---|---|
Architecture | x64 |
OS | ubuntu 22.04 |
Queue | TigerUbuntu |
Baseline | 7abea9ea6ac3cde23fe1a890338e22a5a58a51b6 |
Compare | 1d352cb36c06a16bc93e424809fb431d5795e741 |
Diff | Diff |
Configs | CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono |
Regressions in System.IO.Tests.Perf_Path
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
---|---|---|---|---|---|---|---|---|
237.71 ns | 388.59 ns | 1.63 | 0.13 | True | ||||
311.73 ns | 524.99 ns | 1.68 | 0.10 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Repro Steps
Prerequisites (Build files either built locally or downloaded from payload above)
- Libraries build extracted to
runtime/artifacts
or build instructions: Libraries README args:-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
- CoreCLR product build extracted to
runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release
, build instructions: CoreCLR README args:-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
- Mono Runtime build extracted to
runtime/artifacts/bin/mono/$RunOS.$RunArch.Release
, build instructions: MONO README args:-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
- Dotnet SDK installed for dotnet commands
- Running commands from the runtime folder
Linux
# Set $RunDir to the runtime directory
RunDir=`pwd`
# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun
# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Path*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun"
# Individual Commands:
# Restore
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Path*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages
Windows
# Set $RunDir to the runtime directory
$RunDir="FullPathHere"
# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y
# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.Perf_Path*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe"
# Individual Commands:
# Restore
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.IO.Tests.Perf_Path*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages
Payloads
System.IO.Tests.Perf_Path.GetFileName
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 388.59139044516826 > 252.25442797254843.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -56.87573249911704 (T) = (0 -387.95097047268945) / Math.Sqrt((22.407895948625338 / (23)) + (116.18055560608471 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.5973512315838152 = (242.87142539591997 - 387.95097047268945) / 242.87142539591997 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.IO.Tests.Perf_Path.GetFileNameWithoutExtension
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 524.9882787934918 > 327.1492849840699.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -44.45335062397346 (T) = (0 -541.2040081525383) / Math.Sqrt((64.71773905614282 / (23)) + (506.8376512978537 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.7433666769552156 = (310.43613217257854 - 541.2040081525383) / 310.43613217257854 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Name | Value |
---|---|
Architecture | x64 |
OS | ubuntu 22.04 |
Queue | TigerUbuntu |
Baseline | 7abea9ea6ac3cde23fe1a890338e22a5a58a51b6 |
Compare | 1d352cb36c06a16bc93e424809fb431d5795e741 |
Diff | Diff |
Configs | CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono |
Regressions in System.Xml.Linq.Perf_XName
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
---|---|---|---|---|---|---|---|---|
794.45 ns | 883.37 ns | 1.11 | 0.03 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Repro Steps
Prerequisites (Build files either built locally or downloaded from payload above)
- Libraries build extracted to
runtime/artifacts
or build instructions: Libraries README args:-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
- CoreCLR product build extracted to
runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release
, build instructions: CoreCLR README args:-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
- Mono Runtime build extracted to
runtime/artifacts/bin/mono/$RunOS.$RunArch.Release
, build instructions: MONO README args:-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
- Dotnet SDK installed for dotnet commands
- Running commands from the runtime folder
Linux
# Set $RunDir to the runtime directory
RunDir=`pwd`
# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun
# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Xml.Linq.Perf_XName*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun"
# Individual Commands:
# Restore
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Xml.Linq.Perf_XName*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages
Windows
# Set $RunDir to the runtime directory
$RunDir="FullPathHere"
# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y
# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Xml.Linq.Perf_XName*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe"
# Individual Commands:
# Restore
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Xml.Linq.Perf_XName*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages
Payloads
System.Xml.Linq.Perf_XName.CreateElementWithNamespaceImplicitOperator
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 883.372877953659 > 835.5046779884836.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -20.058625849679544 (T) = (0 -877.2965956603524) / Math.Sqrt((53.87609751788022 / (23)) + (274.2785734374012 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -0.09858120341579611 = (798.5723703742536 - 877.2965956603524) / 798.5723703742536 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
Name | Value |
---|---|
Architecture | x64 |
OS | ubuntu 22.04 |
Queue | TigerUbuntu |
Baseline | 7abea9ea6ac3cde23fe1a890338e22a5a58a51b6 |
Compare | 1d352cb36c06a16bc93e424809fb431d5795e741 |
Diff | Diff |
Configs | CompilationMode:tiered, LLVM:false, MonoAOT:false, MonoInterpreter:true, RunKind:micro_mono |
Regressions in System.Memory.Span<Int32>
Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio |
---|---|---|---|---|---|---|---|---|
944.16 ns | 5.07 ΞΌs | 5.37 | 0.06 | True | ||||
42.16 ns | 89.49 ns | 2.12 | 0.25 | True | ||||
107.76 ns | 423.09 ns | 3.93 | 0.17 | True |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Repro Steps
Prerequisites (Build files either built locally or downloaded from payload above)
- Libraries build extracted to
runtime/artifacts
or build instructions: Libraries README args:-subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
- CoreCLR product build extracted to
runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release
, build instructions: CoreCLR README args:-subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
- Mono Runtime build extracted to
runtime/artifacts/bin/mono/$RunOS.$RunArch.Release
, build instructions: MONO README args:-arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
- Dotnet SDK installed for dotnet commands
- Running commands from the runtime folder
Linux
# Set $RunDir to the runtime directory
RunDir=`pwd`
# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun
# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Int32>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun"
# Individual Commands:
# Restore
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span<Int32>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages
Windows
# Set $RunDir to the runtime directory
$RunDir="FullPathHere"
# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'
# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y
# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release
# Clone performance
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance
# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"
# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Int32>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe"
# Individual Commands:
# Restore
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Memory.Span<Int32>*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages
Payloads
System.Memory.Span<Int32>.LastIndexOfValue(Size: 512)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.0671871465847955 > 991.6764153986129.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -230.5613935734636 (T) = (0 -5146.390116649268) / Math.Sqrt((2628.030321060652 / (23)) + (4533.345099691136 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -4.376656518087421 = (957.172938114324 - 5146.390116649268) / 957.172938114324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Memory.Span<Int32>.LastIndexOfValue(Size: 4)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 89.48576259905823 > 43.54666237706443.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -53.13177922777004 (T) = (0 -94.46973619371741) / Math.Sqrt((0.7507153852309176 / (23)) + (20.332538496754676 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -1.2865599641728707 = (41.315223599609574 - 94.46973619371741) / 41.315223599609574 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
System.Memory.Span<Int32>.LastIndexOfValue(Size: 33)
ETL Files
Histogram
Description of detection logic
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 423.08919944974735 > 113.8718943951145.
IsChangePoint: Marked as a change because one of 10/3/2023 2:46:17 PM, 10/9/2023 8:09:24 PM falls between 10/1/2023 7:44:15 AM and 10/9/2023 8:09:24 PM.
IsRegressionStdDev: Marked as regression because -157.7362376288014 (T) = (0 -434.6710861962908) / Math.Sqrt((1.9254479914001583 / (23)) + (87.61342069650205 / (21))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (21) - 2, .025) and -2.9780146553750284 = (109.26834711605959 - 434.6710861962908) / 109.26834711605959 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
JIT Disasms
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository