From @rolfbjarne on Fri, 15 Nov 2024 19:29:01 GMT

Could you get an MSBuild binlog and attach it here? Thanks!

From @bwinklesky on Fri, 15 Nov 2024 19:52:13 GMT

msbuild.binlog.zip

Let me know if this doesn't work

From @rolfbjarne on Mon, 18 Nov 2024 08:13:11 GMT

Unfortunately that didn't give me many clues.

Is there any chance we could get access to your project to compile it ourselves?

From @bwinklesky on Mon, 18 Nov 2024 19:06:13 GMT

Unfortunately that didn't give me many clues.

Is there any chance we could get access to your project to compile it ourselves?

@rolfbjarne I created a respository and gave you access. WasatchBackcountryAlliance is the app to build.

I ask that you destroy the source code after you've diagnosed the issue.

From @rolfbjarne on Tue, 19 Nov 2024 12:28:31 GMT

Unfortunately that didn't give me many clues.
Is there any chance we could get access to your project to compile it ourselves?

@rolfbjarne I created a respository and gave you access. WasatchBackcountryAlliance is the app to build.

I ask that you destroy the source code after you've diagnosed the issue.

I confirm I can reproduce the problem; I'll see if I can figure out where to route the bug and create a smaller test project that doesn't involve any of your code.

From @formerlymisterhenson on Tue, 19 Nov 2024 13:27:19 GMT

Well this error occured in my project as well, I went with:

<PropertyGroup Condition="'$(Configuration)|$(TargetFramework)|$(Platform)'=='Release|net9.0-ios|AnyCPU'">
    <EnableSGenConc>True</EnableSGenConc>
    <MtouchInterpreter>-all,System.Private.Windows.Core</MtouchInterpreter>
</PropertyGroup>

From @rolfbjarne on Tue, 19 Nov 2024 16:17:20 GMT

Smaller test case: aot-instances-1003365.zip

To repro: unzip & run ./bug.sh

Binlog: msbuild.binlog.zip

Crash report (mono-aot-cross): crash.txt

From @rolfbjarne on Tue, 19 Nov 2024 12:28:31 GMT

Unfortunately that didn't give me many clues.
Is there any chance we could get access to your project to compile it ourselves?

@rolfbjarne I created a respository and gave you access. WasatchBackcountryAlliance is the app to build.
I ask that you destroy the source code after you've diagnosed the issue.

I confirm I can reproduce the problem; I'll see if I can figure out where to route the bug and create a smaller test project that doesn't involve any of your code.

@rolfbjarne thanks for looking into this!

Just to let you know that I have this problem also - exactly as described above. I gave the solution from @formerlymisterhenson a shot and my build has now completed OK. However, not sure whether

True
-all,System.Private.Windows.Core

is going to have any adverse effects?

Tagging subscribers to 'os-ios': @vitek-karas, @kotlarmilos, @ivanpovazan, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

@BrzVlad could you please repro from Rolf's repro and get a real callstack - the one in the crash report is probably without symbols. That should help us triage this better.

Any news on this issue? we have the exact same

The fix is there, the release cycle will take a while. I would assume the new .NET9 version will be available sometime in January.

The backport to 9 is #110271.

Tried the suggestion above and others separate and together and still same issues.IOSBuild.zip

<PropertyGroup Condition="'$(Configuration)|$(TargetFramework)|$(Platform)'=='Release|net9.0-ios|AnyCPU'">
	<NoSymbolStrip Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">True</NoSymbolStrip>
	<CodesignKey>Apple Distribution: XXXXX (YYYYYY)</CodesignKey>
	<CodesignProvision>Fertilizer Calc Distribution</CodesignProvision>
	<ProvisioningType>manual</ProvisioningType>
	<ArchiveOnBuild>true</ArchiveOnBuild>
	<MtouchLink>None</MtouchLink>
	<!-- Disable code linking entirely -->
	<MtouchUseLlvm>True</MtouchUseLlvm>
	<!-- Disable LLVM compilation -->
	<EnableSGenConc>True</EnableSGenConc>
	<!-- Disable concurrent garbage collection -->
	<NoSymbolStrip>true</NoSymbolStrip>
	<MtouchNoSymbolStrip>True</MtouchNoSymbolStrip>
	<EmbedOnDemandResources>True</EmbedOnDemandResources>
	<SuppressTrimAnalysisWarnings>True</SuppressTrimAnalysisWarnings>
	<!-- Suppress warnings -->
	<EnableTrimAnalyzer>False</EnableTrimAnalyzer>
	<!-- Disable the trim analyzer -->
	<MtouchExtraArgs>--aot=verbose</MtouchExtraArgs>
	<MtouchExtraArgs>--aot-options=limit-llvm-stack=400MB</MtouchExtraArgs>
	<MtouchExtraArgs>--aot=threads=1</MtouchExtraArgs>
	<MtouchInterpreter>-all,System.Private.Windows.Core</MtouchInterpreter>
	<EnableSGenConc>True</EnableSGenConc>
</PropertyGroup>

1> With environment:
1> DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
1> Tool /Users/donaldjoyce/Library/Caches/Xamarin/XMA/SDKs/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/9.0.0/Sdk/../tools/mono-aot-cross execution finished (exit code = 139).
1> C:\Program Files\dotnet\packs\Microsoft.iOS.Sdk.net9.0_18.2\18.2.9170\targets\Xamarin.Shared.Sdk.targets(1279,3): error : Failed to AOT compile aot-instances.dll, the AOT compiler exited with code 139.
1> Tool /Users/donaldjoyce/Library/Caches/Xamarin/XMA/SDKs/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/9.0.0/Sdk/../tools/mono-aot-cross execution started with arguments: --path=/Users/donaldjoyce/Library/Caches/Xamarin/mtbs/builds/FertilizerCalcMobile/90be3bdb69a51e414ed28f356ba1bbe5e404546b43a276067601ad0a57db572c/obj/Release/net9.0-ios/ios-arm64/linked --aot=mtriple=arm64-ios,data-outfile=obj/Release/net9.0-ios/ios-arm64/nativelibraries/aot-output/arm64/FertilizerCalcMobile.aotdata,static,asmonly,dedup-skip,direct-icalls,threads=1,interp,full,readonly-value=ObjCRuntime.Runtime.Arch=i4/0,nodebug,dwarfdebug,llvm-path=/Users/donaldjoyce/Library/Caches/Xamarin/XMA/SDKs/dotnet/packs/Microsoft.NETCore.App.Runtime.AOT.osx-x64.Cross.ios-arm64/9.0.0/Sdk/../tools,outfile=obj/Release/net9.0-ios/ios-arm64/nativelibraries/aot-output/arm64/FertilizerCalcMobile.dll.s,llvm-outfile=obj/Release/net9.0-ios/ios-arm64/nativelibraries/aot-output/arm64/FertilizerCalcMobile.dll.llvm.o --debug --llvm -O=gsharedvt /Users/donaldjoyce/Library/Caches/Xamarin/mtbs/builds/FertilizerCalcMobile/90be3bdb69a51e414ed28f356ba1bbe5e404546b43a276067601ad0a57db572c/obj/Release/net9.0-ios/ios-arm64/linked/FertilizerCalcMobile.dll
1> Current directory: /Users/donaldjoyce/Library/Caches/Xamarin/mtbs/builds/FertilizerCalcMobile/90be3bdb69a51e414ed28f356ba1bbe5e404546b43a276067601ad0a57db572c

this worked for me I was able to do a release build. @rolfbjarne thank you

As of to date. Still not able to do a release, but have been able to upload a "Debug" version of the app for Test Flight until this is fixed.

As of to date. Still not able to do a release, but have been able to upload a "Debug" version of the app for Test Flight until this is fixed.

Not sure if you’re building against .NET 9 or not, but if you are… you can downgrade to version 8 and you should be able to publish a release build.

As of to date. Still not able to do a release, but have been able to upload a "Debug" version of the app for Test Flight until this is fixed.

Not sure if you’re building against .NET 9 or not, but if you are… you can downgrade to version 8 and you should be able to publish a release build.

I am developing for .Net 9. I did try switching to .Net 8 but many of my libraries also needed to be downgraded as well. Then some other issues came up that were fixed in the 9 libraries. So I will wait, this application is multi platform.

Removing the package SixLabors.ImageSharp from the project "resolved" the problem for me.

Any word on when this fix will be released? I too have the SizLablrs.ImageSharp nuget package and I'm getting this error. Unfortunately, I can't remove that package.

There’s no point raising an issue in the ImageSharp repository as the issue is here. Using the interpreter would lead to catastrophic performance also so I would not recommend that as a workaround.

I can confirm that this has a catastrophic performance hit. Unfortunately, the v9 release as a whole has been catastrophic for Web and iOS projects.

Also hitting this one. Fingers crossed for this Patch Tuesday! 🤞

Still present in 9.0.4 sadly

@jeremy-visionaid could you please create a separate issue with some more details about the failure you're seeing? It's possible it's yet another combination we haven't seen so far and the existing fixes don't cover.

@jeremy-visionaid could you please create a separate issue with some more details about the failure you're seeing? It's possible it's yet another combination we haven't seen so far and the existing fixes don't cover.

I could provide a project. But actually it is quite easy to reproduce it.
In my case:

• Create a new MAUI project targeting .Net 9.0
• Reference an affected NuGet package (e.g. SixLabors.ImageSharp)
• Use code from the NuGet package in the MainPage
• Try to compile the project for iOS (I work on a Mac)

I installed the latest .NET, SDK 9.0.203, installed the maui workload and I can't reproduce the original issue from this bug report. I also tried the repro steps from above, without reproducing any crashes.

On an arm64 mac I tried:

• dotnet new maui; left only net9.0-ios target framework
• dotnet add package SixLabors.ImageSharp, 3.1.7 version was added
• added var color = SixLabors.ImageSharp.Color.Aqua; in MainPage ctor
• dotnet build -c Release -> this was very expensive on memory/cpu, took around 16 mins due to llvm compilation
• disabled llvm via <MtouchUseLlvm>false</MtouchUseLlvm> and ran the build again, successfully finished in 2 mins this time.

I installed the latest .NET, SDK 9.0.203, installed the maui workload and I can't reproduce the original issue from this bug report. I also tried the repro steps from above, without reproducing any crashes.

On an arm64 mac I tried:

• dotnet new maui; left only net9.0-ios target framework
• dotnet add package SixLabors.ImageSharp, 3.1.7 version was added
• added var color = SixLabors.ImageSharp.Color.Aqua; in MainPage ctor
• dotnet build -c Release -> this was very expensive on memory/cpu, took around 16 mins due to llvm compilation
• disabled llvm via <MtouchUseLlvm>false</MtouchUseLlvm> and ran the build again, successfully finished in 2 mins this time.

I can confirm that adding <MtouchUseLlvm>false</MtouchUseLlvm> to the project file solves the problem for this test project.

For us it does not seem to work regardless MtouchUseLlvm true/false:

No repro for me referencing SixLabors.ImageSharp as above on M2 Pro, but it takes a very long time...

dotnet build -c:Release
Restore complete (0.5s)
  MauiTestApp net9.0-ios succeeded (2806.4s) → bin/Release/net9.0-ios/iossimulator-arm64/MauiTestApp.dll

Build succeeded in 2807.0s

Unfortunately, I can't share the project I can still repro on, but I'll have a go at cutting it down to something minimal to see which bit is causing the failure.

At the risk of going off topic, does anybody know why Catalyst builds so much faster that iOS? For the SixLabours example:

dotnet build -c:Release -f:net9.0-maccatalyst
  MauiTestApp net9.0-maccatalyst succeeded (11.6s) → bin/Release/net9.0-maccatalyst/maccatalyst-x64/MauiTestApp.dll
  MauiTestApp net9.0-maccatalyst succeeded (24.9s) → bin/Release/net9.0-maccatalyst/maccatalyst-arm64/MauiTestApp.dll

For the project I'm working on where I can repro, the iOS build fails after about 10 mins, but the Catalyst build fails after only 30 seconds.

OK. I have a minimal repro... It's just referencing types from System.Drawing.Common:

<PackageReference Include="System.Drawing.Common" Version="9.0.3" />

System.Diagnostics.Trace.WriteLine($"{System.Drawing.Size.Empty}");

Repo available at https://github.com/jeremy-visionaid/MauiTestApp/tree/aot-crash

Both iOS and Catalyst fail, but Catalyst fails much faster:
dotnet build -c:Release -f:net9.0-maccatalyst

Interestingly, the x64 build succeeds, but the arm64 build fails:

dotnet build -c:Release -f:net9.0-maccatalyst
  MauiTestApp net9.0-maccatalyst succeeded (11.6s) → bin/Release/net9.0-maccatalyst/maccatalyst-x64/MauiTestApp.dll
  MauiTestApp net9.0-maccatalyst failed with 2 error(s) (12.2s) → bin/Release/net9.0-maccatalyst/maccatalyst-arm64/MauiTestApp.dll
    /usr/local/share/dotnet/packs/Microsoft.MacCatalyst.Sdk.net9.0_18.4/18.4.9288/targets/Xamarin.Shared.Sdk.targets(1266,3): error : Failed to AOT compile System.Private.Windows.Core.dll, the AOT compiler exited with code 139.
    /usr/local/share/dotnet/packs/Microsoft.MacCatalyst.Sdk.net9.0_18.4/18.4.9288/targets/Xamarin.Shared.Sdk.targets(1266,3): error : Failed to AOT compile aot-instances.dll, the AOT compiler exited with code 139.

This is not an issue with the AOT compiler, rather a problem with packages and a missing dll. The errors reported during AOT compilation are Could not load file or assembly 'System.Formats.Nrbf. Adding a reference to System.Formats.Nrbf fixes compilation.

Having said this, you shouldn't use System.Drawing.Common in the first place, it was deprecated on non-windows platforms. https://github.com/dotnet/docs/blob/main/docs/core/compatibility/core-libraries/6.0/system-drawing-common-windows-only.md.

This particular problem was reported before, for example: dotnet/maui#27057

@BrzVlad Thanks for the reply. Sadly it's not so simple as just not using System.Drawing.Common since I'm investigating porting an old application from WinForms to MAUI. There would be a lot of code to port to new types and serialization of them in settings, documents etc. Sure I could at least handle it in code that I control (with some effort), but it's also possible a reference might be acquired transitively.

I think there are some other points for consideration here (I don't necessarily expect a reply since they not all compiler/runtime related, so maybe a bit off topic for this project):

1. The app builds and runs in Debug without giving any warnings (Why does the AOT compiler need the Nrfb reference but an interpreted build does not? Should this be covered by some AOT analyzer (or maybe the package owner)? Why are there no warnings by default for referencing the Drawing package if it's supposed to be attributed as Windows specific?)
2. AOT compilation works for maccatalyst-x64 so why not maccatalyst-arm64?
3. The cause of the build failure is not surfaced effectively. The messages regarding the missing types/reference aren't shown to the user in VS Code. The easiest way I can get to it at all is to build manually and turn off the terminal logger.

I'm grateful to have a workaround (it's certainly a lot less work than removing the dependency), but it does feel like the developer experience could be improved here - even if that's beyond the scope of the compiler/runtime.

Thank you @jeremy-visionaid for the effort of sharing more information with us and for your feedback!

I'll try to answer some of your questions:

1. • Why does it build in Debug and fails in Release. Debug builds use JIT or interpreter which only compile/interpret the absolute bare minimum of code necessary to run the app. For example, if you have a code like:

if (doIWantExceptions)
{
    ThrowMySpecialException();
}
else
{
    WriteItOutToLog();
}

If the doIWantExceptions is always false at runtime, the interpreter will never even look at ThrowMySpecialException. But AOT compiler doesn't know, it has to assume that the variable doIWantExceptions can be true and thus it has to compile ThrowMySpecialException method. And that method may bring in new dependencies one of which could cause a failure - like it not being there. Now the AOT compiler tries to be resilient to these problems, but it's mostly a "best effort" and if the failure occurs in the wrong spot, it will not handle it. ("handle it" means replacing the compiled code with "throw runtimeexception", to let it fail at runtime instead of compile time).
If it fails or not may depend on subtle changes in the input and configuration to the compiler, so for example it would not be surprising if AOT compiling the same code for Android would work while it fails for iOS.

2. • This is probably due to a subtle difference as mentioned above. We would need to debug the exact failure to be able to tell for sure. It's also very possible that changing the app slightly may suddenly make it fail on x64 as well, or it might just compile everything fine.
3. • Thanks for the feedback - this is definitely something we don't want to "hide" from you. If you have more data on this, please create a new issue and include information like: what is the exact error you see when building from command line, what does VS Code show you, how do to trigger the build from VS Code. We can then route this to the right team.

This is not an issue with the AOT compiler, rather a problem with packages and a missing dll. The errors reported during AOT compilation are Could not load file or assembly 'System.Formats.Nrbf. Adding a reference to System.Formats.Nrbf fixes compilation.

Having said this, you shouldn't use System.Drawing.Common in the first place, it was deprecated on non-windows platforms. https://github.com/dotnet/docs/blob/main/docs/core/compatibility/core-libraries/6.0/system-drawing-common-windows-only.md.

This particular problem was reported before, for example: dotnet/maui#27057

For us adding <PackageReference Include="System.Formats.Nrbf" Version="9.0.4" /> fixes publishing, thanks. We're using System.Drawing.Common and were not aware of the deprecation for other operating systems.

• AOT compilation works for maccatalyst-x64 so why not maccatalyst-arm64?

This is because there's no AOT compiler for x64, it'll always use the JIT/interpreter.

For me updating to .NET 9.0.4 (along with updating the iOS and MAUI workloads) fixed the build issue. (".NET 9 for iOS" project with ImageSharp)
The _AOTCompile step takes a bit longer with 9.0.4 than it did with .NET 8 (21 minutes vs 12 minutes), but it all compiles and runs.

Came across the same issue after adding a NotificationServiceExtension project for BlazorHybrid.
In BlazorHybrid I have:
<UseInterpreter>true</UseInterpreter>
When I added the same <UseInterpreter>true</UseInterpreter> to the NotificationServiceExtension project the error was gone.

This is because there's no AOT compiler for x64, it'll always use the JIT/interpreter.

Thanks @rolfbjarne! You're of course correct that _AOTCompile isn't being run, but I thought from the docs that it was supported:
https://learn.microsoft.com/en-us/dotnet/core/deploying/native-aot/?tabs=windows%2Cnet9plus

• Why does it build in Debug and fails in Release. Debug builds use JIT or interpreter which only compile/interpret the absolute bare minimum of code necessary to run the app.

@vitek-karas Thanks to you also for taking the time for such a detailed reply! With respect to point 1, I was more looking at it from the Debug/JIT side... It would have been nice to have gotten some kind of warning. e.g. If EnableAotAnalyzer is enabled then should the intent be for that to result in a warning that AOT will fail? And should referencing System.Drawing.Common in a non-Windows TFM give a warning?

should referencing System.Drawing.Common in a non-Windows TFM give a warning?

This I agree with - I'm guessing that when we deprecated the assembly on non-Windows we didn't really do work to correctly annotate the APIs (that's actually quite a lot of work). We can open a bug on it, not sure if it will be fixed though.

The analyzer would have a hard time with this - mostly due to technical limitations. It's a Roslyn analyzer, which only sees your source code and metadata about references. If the code you're calling directly is not in the references, then C# compiler itself will fail - no need to analyze for that. And if you're calling code which has the problem, the analyzer doesn't see that code. Ultimately there are a set of problems which the analyzer can't see, missing references in dependencies is one such set.

All good. It's working for me now, so I'm happy enough, and I'm grateful for the help. There are just some orthoganal issues that made it a bit harder to deal with than I would have liked. FWIW, if types like RotateFlipType and some others had been in System.Drawing.Primitives and not System.Drawing.Common then I might not have hit it directly, but at least I don't have to work around it now 👍

I am in the unfortunate position of having "upgraded" from VS2002 17.12.4 to 17.13.6 today. Since doing that, I can no longer build Debug builds of our main commercial project:

Failed to AOT compile Microsoft.Maui.Controls.HotReload.Forms.dll, the AOT compiler exited with code 139.
Failed to AOT compile aot-instances.dll, the AOT compiler exited with code 139.

Interestingly, a Release build seems to work fine. Obviously, I am stuck now. This is an iOS project that does not use MAUI. We are not referencing anything other than a couple of projects, which themselves do not reference anything.

How do I get myself unblocked?

Failed to AOT compile Microsoft.Maui.Controls.HotReload.Forms.dll, the AOT compiler exited with code 139.

Try disabling Hot Reload, it brings in assemblies that need MAUI, so it won't work if you're not using MAUI.

I guessed you meant Tools -> Options -> Debugging -> XAML Hot Reload. We've never used that feature. You were right, turning it off has fixed the problem. Thank you.

I really hope you are able to improve these error messages, and prioritise doing so. They are so unhelpful!

Anecdotic datapoint: I do not use hot reload (my project is native .NET, not MAUI) in Rider and the issue appeared when I finally migrated from an Intel Mac to an Apple Silicon Mac last week. Same setup, except maybe the minor version of .NET 9 is newer now, haven't checked that one (the new machine got the latest 9.0.203).

<UseInterpreter>true</UseInterpreter> fixed it for now

Tried without the interpreter and the app built on iOS azure pipeline but app crashes when run - trying to see what the error is... Added back interpreter=true and all good