Allow stack size to be configured for native AOT #110455

keegan-caruso · 2024-12-05T21:57:56Z

If a native AOT compiled application can lower its allocated memory, more scenarios that are memory sensitive can use it.

The default stack size can be excessive for simple applications and can significantly contribute to the RSS.

For linux, there is the ulimit command, and this is respected, but it can be too broad.

With this trivial code:

int s = 0;

await Task.Run(() => Interlocked.Increment(ref s));

Console.ReadLine();

On linux with the default stack size of 8mb, RSS measured at 5916.

With setting the stack size to 1mb, the RSS measured at 3428.

Roughly a 40% decrease.

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs

jkotas · 2024-12-06T02:52:37Z

We have (undocumented) IlcDefaultStackSize build property to set the default stack size in Native AOT binaries. Would it be sufficient for what you are trying to do?

MichalStrehovsky · 2024-12-06T06:27:38Z

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs

+            const string stackSizeVar = "DefaultStackSize";
+            const string stackSizeEnvVar = "DOTNET_DefaultStackSize";


This shortcut (making the AppContext switch name same as the environment variable name) makes things inconsistent with how we normally name AppContext switches. See e.g. documented GC switches where DOTNET_gcServer environment variable is a synonym for System.GC.Server host config option (i.e. these are not prefixes and AppContext switches adhere to a naming convention).

It is a bit more work to do this correctly. Sample how it's done on the managed side (also notice the naming convention):

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.cs

Lines 66 to 70 in 8fca0a1

int timeoutMs =

AppContextConfigHelper.GetInt32Config(

"System.Threading.ThreadPool.ThreadTimeoutMs",

"DOTNET_ThreadPool_ThreadTimeoutMs",

DefaultThreadPoolThreadTimeoutMs);

On the native side we do this (it's what the GC uses to read both env variable settings and host config options):

runtime/src/coreclr/nativeaot/Runtime/gcenv.ee.cpp

Lines 770 to 784 in 8fca0a1

uint64_t uiValue;

if (g_pRhConfig->ReadConfigValue(privateKey, &uiValue))

{

*value = uiValue;

return true;

}

if (publicKey)

{

if (g_pRhConfig->ReadKnobUInt64Value(publicKey, &uiValue))

{

*value = uiValue;

return true;

}

}

Also, it would be nice to keep the configuration capabilities between NAOT and regular CoreCLR the same - make the configuration switch work the same way for both NAOT and CoreCLR.

@MichalStrehovsky @jkotas

I believe this feedback is addressed. Are there any other changes you would like to see?

This change would help one of the teams that I support.

I believe this feedback is addressed. Are there any other changes you would like to see?

I don't see this comment addressed:

The switch name should be something starting with System.Threading.Thread. Maybe System.Threading.Thread.DefaultStackSize? This is a naming convention respected across the entire framework.

If I'm reading it right, PalStartBackgroundWork will look at the environment variable, but will not look at RuntimeHostConfigurationOption (the environment variable settings and AppContext/RuntimeHostConfiguration settings are from disjoint namespaces). Is that expected?

Sorry for the confusion, I pushed 2975db0 awhile ago, but Github wasn't showing the commit in this diff. I followed jkotas's advice, rebasing and force pushed fixed the state.

keegan-caruso · 2024-12-06T23:14:12Z

We have (undocumented) IlcDefaultStackSize build property to set the default stack size in Native AOT binaries. Would it be sufficient for what you are trying to do?

Thanks. This works on Windows, it is not working for me on Linux.

On windows I checked against kernel32 GetCurrentThreadStackLimits. On linux I checked against strace.

jkotas · 2024-12-08T16:26:12Z

Thanks. This works on Windows, it is not working for me on Linux.

It works for musl libc (musl libc is used e.g. by Alpine Linux). glibc that is used by the most popular Linux distros like Ubuntu does not respect value set by -z,stack-size linker switch for some reason. That's unfortunate.

jkotas · 2025-02-10T03:41:08Z

Github status is stuck in "Checking for ability to merge automatically…".

Could you please rebase against main and force push? Hopefully, it is going to reset the status.

jkotas · 2025-02-10T03:45:58Z

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs

+
+        private static int StackSizeFromConfig
+        {
+            get => _stackSizeFromConfig ??= GetDefaultStackSize();


This pattern with Nullable<int> is not thread safe. Nullable is a struct with two fields. This pattern has a race condition caused by tearing.

I do not think you need to cache the value on the managed side. The value is cached on the unmanaged side. You can call every time.

MichalStrehovsky · 2025-02-10T07:41:07Z

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs

+            // the AppContext class.
+            // To have similar behavior as RhConfig, also read from environment variables, which have priority.
+
+            string? valueFromConfig = Environment.GetEnvironmentVariable(stackSizeEnvVar) ?? AppContext.GetData(stackSizeVar)?.ToString();


This can use AppContextHelper.GetInt32ComPlusOrDotNetConfig so that we only have one copy of the policy around parsing these.

Alternatively, this can call into the native runtime to reuse the value parsed there.

Use different keys from reading from env vs config

src/coreclr/nativeaot/Runtime/RhConfig.cpp

MichalStrehovsky · 2025-03-13T06:41:25Z

I think we should look at this in the context of #113379:

Here's a couple questions I have:

Are we happy with the name of the runtimeconfig.json name of this and DOTNET_ environment variable? (They look good to me.)
Are we happy with the radix? (I assume we want to keep it as base-16.)

If so, I can add the CoreCLR version so that we can also resolve #113379 and we can doc it.

Cc @kouvel @AaronRobinsonMSFT for opinion

am11 · 2025-03-13T12:46:12Z

The naming looks good as it's consistent with:

src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.cs:            AppContextConfigHelper.GetInt16ComPlusOrDotNetConfig("System.Threading.ThreadPool.MinThreads", "ThreadPool_ForceMinWorkerThreads", 0, false);
src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.cs:            AppContextConfigHelper.GetInt16ComPlusOrDotNetConfig("System.Threading.ThreadPool.MaxThreads", "ThreadPool_ForceMaxWorkerThreads", 0, false);
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.Unix.cs:            AppContextConfigHelper.GetBooleanConfig("System.Threading.ThreadPool.EnableWorkerTracking", "DOTNET_ThreadPool_EnableWorkerTracking");
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.Windows.cs:            AppContextConfigHelper.GetBooleanConfig("System.Threading.ThreadPool.UseWindowsThreadPool", "DOTNET_ThreadPool_UseWindowsThreadPool");

AaronRobinsonMSFT · 2025-03-13T15:54:30Z

@MichalStrehovsky See #113344 also.

Are we happy with the name of the runtimeconfig.json name of this and DOTNET_ environment variable? (They look good to me.)

I didn't see the DOTNET_ environment variable. Can you point me to it?

Are we happy with the radix? (I assume we want to keep it as base-16.)

This was a source of confusion on a partner team who had been assuming it was radix 10 for about 6 years. I think having it as radix 16 is fine, but we need to document this properly at https://learn.microsoft.com/dotnet/core/runtime-config/threading.

AaronRobinsonMSFT · 2025-03-13T15:55:52Z

DOTNET_DefaultStackSize

Seems reasonable to me.

am11 · 2025-03-13T16:22:03Z

I didn't see the DOTNET_ environment variable. Can you point me to it?

It's DOTNET_Threading_DefaultStackSize. Prefix is added later. It's consistent with existing convention DOTNET_ThreadPool_UseWindowsThreadPool, DOTNET_ThreadPool_EnableWorkerTracking etc.

kouvel · 2025-03-20T21:52:58Z

It's DOTNET_Threading_DefaultStackSize. Prefix is added later. It's consistent with existing convention DOTNET_ThreadPool_UseWindowsThreadPool, DOTNET_ThreadPool_EnableWorkerTracking etc.

CoreCLR already has a DOTNET_DefaultStackSize config setting. There are other thread-specific settings with DOTNET_Thread_*. Should we use the DOTNET_Thread_ prefix for consistency and add the new one to CoreCLR as well?

keegan-caruso requested a review from MichalStrehovsky as a code owner December 5, 2024 21:57

ghost added the area-NativeAOT-coreclr label Dec 5, 2024

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Dec 5, 2024

am11 reviewed Dec 5, 2024

View reviewed changes

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Threading/Thread.NativeAot.cs Outdated Show resolved Hide resolved

This was referenced Dec 6, 2024

iOS tests failing with WORKLOAD TIMED OUT - Killing user command. #108103

Open

iOS test fails with "App is not signed" #110395

Closed

[OSX]: AMDeviceSecureInstallApplicationBundle returned: 0xe800801c #110403

Open

MichalStrehovsky reviewed Dec 6, 2024

View reviewed changes

jkotas reviewed Feb 10, 2025

View reviewed changes

MichalStrehovsky reviewed Feb 10, 2025

View reviewed changes

Keegan Caruso added 3 commits February 10, 2025 13:09

Allow stack size to be configured for naot

50d14f5

Feedback:

2975db0

Use different keys from reading from env vs config

Use GetInt32ComPlusOrDotNetConfig and don't cache _stackSizeFromConfig

b6fd3f7

keegan-caruso force-pushed the configurable-stack-size-naot branch from 94b687c to b6fd3f7 Compare February 10, 2025 21:53

build-analysis bot mentioned this pull request Feb 11, 2025

System.Numerics.Tensors.Tests.ConvertTests.ConvertChecked failing with System.OverflowException #112286

Closed

agocke assigned MichalStrehovsky Mar 12, 2025

MichalStrehovsky reviewed Mar 13, 2025

View reviewed changes

src/coreclr/nativeaot/Runtime/RhConfig.cpp Outdated Show resolved Hide resolved

src/coreclr/nativeaot/Runtime/RhConfig.cpp Show resolved Hide resolved

Apply suggestions from code review

e0a3d1b

MichalStrehovsky added 2 commits March 18, 2025 23:58

Merge branch 'main' into configurable-stack-size-naot

6233767

Centralize stack size reading

67b1c4e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow stack size to be configured for native AOT #110455

Allow stack size to be configured for native AOT #110455

keegan-caruso commented Dec 5, 2024

jkotas commented Dec 6, 2024

MichalStrehovsky Dec 6, 2024

jkotas Dec 8, 2024

keegan-caruso Feb 8, 2025

MichalStrehovsky Feb 10, 2025

keegan-caruso Feb 10, 2025

keegan-caruso commented Dec 6, 2024

jkotas commented Dec 8, 2024

jkotas commented Feb 10, 2025

jkotas Feb 10, 2025

MichalStrehovsky Feb 10, 2025

jkotas Feb 10, 2025

MichalStrehovsky commented Mar 13, 2025

am11 commented Mar 13, 2025

AaronRobinsonMSFT commented Mar 13, 2025

AaronRobinsonMSFT commented Mar 13, 2025

am11 commented Mar 13, 2025 •

edited

Loading

kouvel commented Mar 20, 2025

		const string stackSizeVar = "DefaultStackSize";
		const string stackSizeEnvVar = "DOTNET_DefaultStackSize";

	int timeoutMs =
	AppContextConfigHelper.GetInt32Config(
	"System.Threading.ThreadPool.ThreadTimeoutMs",
	"DOTNET_ThreadPool_ThreadTimeoutMs",
	DefaultThreadPoolThreadTimeoutMs);

	uint64_t uiValue;
	if (g_pRhConfig->ReadConfigValue(privateKey, &uiValue))
	{
	*value = uiValue;
	return true;
	}

	if (publicKey)
	{
	if (g_pRhConfig->ReadKnobUInt64Value(publicKey, &uiValue))
	{
	*value = uiValue;
	return true;
	}
	}

Allow stack size to be configured for native AOT #110455

Are you sure you want to change the base?

Allow stack size to be configured for native AOT #110455

Conversation

keegan-caruso commented Dec 5, 2024

jkotas commented Dec 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keegan-caruso commented Dec 6, 2024

jkotas commented Dec 8, 2024

jkotas commented Feb 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichalStrehovsky commented Mar 13, 2025

am11 commented Mar 13, 2025

AaronRobinsonMSFT commented Mar 13, 2025

AaronRobinsonMSFT commented Mar 13, 2025

am11 commented Mar 13, 2025 • edited Loading

kouvel commented Mar 20, 2025

am11 commented Mar 13, 2025 •

edited

Loading