Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

This is meant for benchmarking. No need to review until benchmarks show it is worth it.

This is meant for benchmarking.

@tmds is there any chance you could share your copy of System.Net.Sockets.dll with me? I am just lazy and would like to go the easy way without compiling your fork ;)

Adam, here it is: System.Net.Sockets.dll.tar.gz

BTW I was looking at the JSON profile today and we spent sth around 1.5% of total time in this particular lock

It looks like slight gains on x64, and mixed gain/loss on arm64.
The numbers aren't very consistent, and they deviate a lot from 1.5% spent in the lock for JSON.
If you run these benchmarks again, will the results be similar?

I've tried to run a few of them and the results seems similar (and still not super stable)

I've run the benchmarks one more time, the results look very similar:

(I've interrupted the ARM run as I am finishing work for today)

There's no reason these changes should regress performance.
I don't know what is going on on arm64. Do we care about figuring it out?
Adam, can you collect a perftrace on arm64 for jsonplatform with 128 connections before and after? Maybe it tells us something.

I don't know what is going on on arm64.

From what I've seen in the profiles so far is that all the Interlocked and Volatile operations which seem to be almost immediate on x64 are not so cheap on ARM64.

A good example is this simple micro-benchmark:

public class Perf_Volatile
{
    private double _location = 0;
    private double _newValue = 1;
    
    [Benchmark]
    public double Read_double() => Volatile.Read(ref _location);

    [Benchmark]
    public void Write_double() => Volatile.Write(ref _location, _newValue);
}

 BenchmarkDotNet=v0.12.1, OS=ubuntu 18.04
 ARMv8 Processor rev 1 (v8l), 4 logical cores
 .NET Core SDK=5.0.100-preview.4.20217.5
   [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.21702, CoreFX 5.0.20.21702), Arm64 RyuJIT
   Job-ULPSAR : .NET Core 5.0.0 (CoreCLR 5.0.20.21702, CoreFX 5.0.20.21702), Arm64 RyuJIT

 BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.778 (1909/November2018Update/19H2)
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.4.20217.2
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.20.21611, CoreFX 5.0.20.21611), X64 RyuJIT
  Job-OXOUZI : .NET Core 5.0.0 (CoreCLR 5.0.20.21611, CoreFX 5.0.20.21611), X64 RyuJIT

@kunalspathak is this expected? Can we do anything about it?

Adam, can you collect a perftrace on arm64 for jsonplatform with 128 connections before and after?

Sure, I can get them for you tomorrow. BTW when my VPN is not working I just close everything on my PC and run the benchmark server and wrk myself. I use the wrk arguments from this file: https://github.com/aspnet/Benchmarks/blob/master/src/WrkClient/wrk.yml and it typically can give me an answer whether my change is going to improve the perf or not. I know that it is far from perfect, but it should shorten your perf feedback loop.

From what I've seen in the profiles so far is that all the Interlocked and Volatile operations which seem to be almost immediate on x64 are not so cheap on ARM64

On x86/64 the architecture's memory model is strong enough that volatile operations end up serving just as a compiler barrier; the actual instruction output doesn't differ based on whether the read or write is volatile. You can see that here:
https://sharplab.io/#v2:EYLgxg9gTgpgtADwGwBYA0AXEBDAzgWwB8ABABgAJiBGAOgBUALWbAEwEsA7AcwG4BYAFDEAzJQBM5AMLkA3oPILKozhnIBZKgApYAM3Ir9ASnIBeAHzkAahAA22DGxswaAJRittMPW0P8BipX0OVTUxT29go1MLNj8AXyA=

On ARM, there's a weaker memory model, so volatile operations end up entailing actual barriers in the JIT'd instructions, e.g. dmb.

Echoing to what @stephentoub has said, one recommendation would be to perhaps see possibilities of reducing volatile variable access inside a loop. See #34225 for example.

Sure, I can get them for you tomorrow.

@adamsitnik let me make a few changes before you collect traces.

@adamsitnik I pushed the change. If you collect perftraces for arm64 for jsonplatform with 128 connections before and after, maybe we'll learn why this is regressing.

This is compiled System.Net.Sockets: System.Net.Sockets.dll.tar.gz

I got traces from Adam and took a look.
The variance between benchmarks makes it not possible to derive something meaningful.

This change is trying to replace something with something else, assuming it is cheaper. But I'm not really sure it is cheaper, and I don't have a good way to measure.
I'm giving up on this.

Thanks for trying, @tmds.