[net9] System.Private.CoreLib occurs InvalidOperationException and InvalidCastException #114262
Comments
|
Tagging subscribers to this area: @mangod9 |
|
hey, i would love to look into this, could you please provide a minimal reproducible example? |
|
so far it's very difficult for me to understand where this is being reproduced: the project is very large, the stack trace is absolutely uninformative for me |
|
perhaps you can find breaking changes that occurred during the transition from net8 to net9 based on the stack trace |
|
i found similar issue |
|
Happens for us in random places in the iOS in release mode. This is a blocker to go for net9 |
|
TLDR: Just to add a me too to this issue though our situation we can only repeat in a release mode and we only get InvalidCastException This crash has only been recreated by us using a release-built iOS app with the default template AOT and trimming settings enabled. Attempting to create repro steps has been difficult as the crash is intermittent in that it seemingly can occur anywhere in the app however will occur multiple times within the span of a couple of hours for any given device (sometimes within 5 minutes of app start to around 1.5 hours of use). It seems to happen during periods of heavy activity such as background syncing and the UI reacting to data changes or during page navigation. Exception details: This issue does look very similar to the issue raised here: #81211 however this was resolved in .Net MAUI 8 RC2 and we are using .Net MAUI 9.0.50 SR5. We did have a theory that possibly the exception was being raised within an event handler that was using async void. To rule this out we have used the package: https://github.com/roubachof/Sharpnado.TaskMonitor to revert all event handlers back to returning void and if any tasks within these methods throw an exception this is now handled within a global error handler. Sadly, while this approach is definitely better moving forward it was not the cause of the unhandled exception we are experiencing on iOS. |
|
Well, add me to this, too. Happening in release mode .Net MAUI 9.0.10. It's very random (could happen while refreshing a ListView, closing a modal page, opening a modal page, toggling a Switch control, displaying an activity indicator etc.). That being said, what is the game plan here? |
We're in the process of triaging to determine what the problem is. If we're able to pin it down, I'm hopeful we'll be able to suggest workarounds better than falling back to the interpreter. If anyone can provide a simple repro, that would help speed everything up. |
|
Just commenting to say we are seeing the same issue. Only happens when using net9 on iOS. Also appears randomly and without any obvious pattern. |
|
We have the same issue when we run our Maui app on iOS without the VS debugger attached. When the debugger is attached, we never get the exception. Our project is quite complex and haven't tried to isolate this with a repro, but can provide some background in hopes it helps: Installed Workload Id Manifest Version Installation Sourceandroid 35.0.50/9.0.100 SDK 9.0.200, VS 17.13.35919.96 Project has Packages:
Process that causes the crashes: We have added try catch around everything, but the exception bubbles up to AppDomain.CurrentDomain.UnhandledException. Note that the same code works with dotnet 8. |
Not sure if this has been suggested, but can you set |
Thanks for the suggestion. We tried We also experience the same exceptions in a long running Parallel.ForEachAsync that unzips these files after downloading. It seems the more items we are putting thru the ForEachAsync, the higher probability we have for the exception to occur. |
Appreciate you trying it out and letting me know. I'll add |
|
We facing with the same problem. The app crashes randomly with Arg_InvalidCastException exception with the same stack as provided above. We just moved the app from Xamarin forms to MAUI9 and facing with this issue on release builds only. I was not able to reproduce it locally. |
|
Updating MAUI to 9.0.60 didn't help |
|
We are also facing the same problem on iOS, while using MAUI version Same situation as others, it occurs "at random", and on various pages. Definitely a blocker for us. If it can help, here is the IOS project <PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<CodesignKey>iPhone Distribution</CodesignKey>
<MtouchLink>SdkOnly</MtouchLink>
<UseInterpreter>true</UseInterpreter>
</PropertyGroup> |
|
@jeffhandley since you have moved this to a future milestone, does that mean it won't be fixed for dotnet 9? Has the issue been identified? Are there any workarounds? This is a major setback for our software if there is no resolution or workaround for dotnet 9. |
We are still investigating and will relay any kind of workarounds / fixes when we determine what the problem is. Rest assured, it has our full attention. |
|
OK, I have a repo app here: https://github.com/pme442/CVIssueApp I deployed this to my iPad mini 6th Gen (4GB of RAM) and ran in release mode. In this app, there are 3 tabs at the bottom: General Notes on how it works:
Here is what I did: After you do these things for about 15 to 20 minutes, go to the Error Log tab and see where you are with memory. Here is a screen shot of my Error Log:
Hope this helps! |
|
We're seeing the same here with a .NET MAUI app on iOS. Total blocker for moving to .NET9. |
Thanks - any repro or description of what you are doing in the lead up would be helpful. Are you seeing this on certain device types more often than others? |
As others have reported here, we are also seeing crashes at random basically. The crashes have been observed in very different situations, including ones where the app is almost doing nothing at all (no user interactive for a while).
Not really. We've seen it on various devices (including, for example, iPhone 14, iPhone 8, iPad Air 4th gen), and with various iOS versions (16 -18). Since the crash is occurring rarely and at random, it's hard to say if it occurs "more often" on a specific device. |
|
Trying to approach this issue from the source-code side, I've looked at the blame of Task.RunContinuations. There is only one commit that is recent enough (added between net8.0 and net9.0) and has potential for causing this issue: It adds the following line in Looking at the source of CollectionsMarshal.AsSpan, it makes the following assertions: Could it be that the crash is caused by one of these assertions not being fulfilled for some reason? The documentation of So maybe the issue is caused by some sort of race condition, where the |
I see that there is some locking code in front of the runtime/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs Lines 3505 to 3508 in 6a3747b However, the locking has also been modified in the commit above (from an acutal While I cannot prove that the commit mentioned above is the culprit that introduced this regression, I still think it's the most probable cause. [Note that I also blame-checked all the |
I'll note that, like @wgriep and @EmilienDup, we're setting Also, we use: |
This commit did not weaken any lock guarantees, all accesses to the continuations list are still done under lock (or after the last locking operation in case of |
True, but shouldn't the access to the list in I.e. instead of ... shouldn't it better be the following? Otherwise another locked access might be done simulteneously, while |
Plus, this would not only be a problem while to conversion to |
... and that is indeed a difference to the old .NET 8 implementation, where only the continuation count was determined up front, and could then be used safely in the subsequent loops. Converting to @stephentoub Could you possibly comment on this (as the reviewer of PR #93953)? |
|
The operations in |
Thanks for the comments, @pentp! So, if you are sure that PR #93953 is not what caused this issue, do you possibly have an idea what else could cause InvalidCastExceptions in |
That's what we're trying to figure out by surfacing the first exception, but we have been unsuccessful reproducing anything to this point. I do think it's something unique to the iOS runtime in .NET 9. |
Is there anything one can do to help debug this? Due to the ghostly nature of this issue, I think it will be hard to create a reliable reproducer (plus, my app is closed-source). Even worse: It only seems to occur in Release builds. So, if I manage to hit the exception by chance, what can I do to debug and pinpoint it? |
|
We also have this exact same problem and it´s driving us mad. All worked well before upgrading to .net9. We have a few background task that updates data every 15 min, some ui related updates every 15 seconds, and after a random number of times it just crashes. Some times it can take a pretty long time, but it feels pretty ransom. The app can be idle except for the background updates. I´ll just drop here all our findings and hope someone can get good use of it and figure out how to solve it. `System.InvalidCastException: ` 
`Incident Identifier: 28AD7BBA-770F-46EB-ABC8-8CEF07A29AC5 Date/Time: 2025-05-05 19:50:55.9033 +0200 Exception Type: EXC_CRASH (SIGABRT) Triggered by Thread: 20 Thread 0 name: Thread 1 name: Thread 2 name: Thread 3: Thread 4: Thread 5 name: Thread 6: Thread 7 name: Thread 8 name: Thread 9 name: Thread 10 name: Thread 11 name: Thread 12 name: Thread 13 name: Thread 14 name: Thread 15 name: Thread 16: Thread 17: Thread 18: Thread 19 name: Thread 20 name: Thread 21 name: Thread 22 name: Thread 23 name: Thread 24 name: Thread 25 name: Thread 26 name: Thread 20 crashed with ARM Thread State (64-bit): Binary Images: EOF The crash report indicates that Thread 20 crashed due to an EXC_CRASH (SIGABRT), which is caused by an explicit call to abort(). Below is an analysis of the crash and what can be inferred from the report. Key Findings from the Crash Report Exception Type: EXC_CRASH (SIGABRT) The crash occurred in mono_runtime_setup_stat_profiler (mini-posix.c:692). The stack trace shows that the crash occurred in the Mono runtime, specifically in mono_runtime_setup_stat_profiler. The crash occurred in the Mono runtime's thread management (start_wrapper in threads.c). Thread 20 appears to be executing application code via the Mono runtime. The crash occurred while handling an unhandled exception or setting up a profiler. The stack trace shows a call to mono_invoke_unhandled_exception_hook, indicating that an unhandled exception occurred in the application. The crash was caused by an explicit call to abort(), likely due to a critical error in the app's logic or runtime environment. The involvement of mini-posix.c and runtime.m suggests the issue might be related to the Mono runtime or its interaction with the app. Use the symbolication script to identify the exact function and line number where the crash occurred. Check for unhandled exceptions or invalid states in the identified function. Ensure the Mono runtime is properly configured and up to date. Run the app in debug mode with additional logging enabled to capture more details about the crash. If necessary, enable true for production builds to get more detailed stack traces. Let me know if you need help interpreting the symbolicated output or further investigating the crash! |
This looks very much like our setup |
If you are up for it, can you use this runtime pack built from a fork of our public 9.0 runtime branch? All you have to do within your project is add a nuget.config if you don't have one and add a local feed. We think this will surface an exception unique to iOS and may help us pinpoint the issue. Microsoft.NETCore.App.Runtime.Mono.ios-arm64.9.0.5.nupkg.zip |
Awesome, thanks for that nupkg! I hope that I'll be able to try it in the next days. 👍 |
@steveisok I just tried to include this nupkg into my app. In addition to your But then @steveisok Could you tell me what this error means and how to get rid of it? My project is a MAUI app with TFMs like |
|
I´m not 100% sure if we got it included correct, but delete all in my previously downloded nugets in the nuget folders to start fresh, so I do belive it worked. 0 FrontSystems 0x0000000105741af4 mono_interp_exec_method (in FrontSystems) (interp.c:6193) + 11033332 Full crash report `Incident Identifier: BB647D5A-72AD-461C-9DB7-16E91D4E882A Date/Time: 2025-05-06 13:25:22.6009 +0200 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Thread 0 name: Thread 1 name: Thread 2 name: Thread 3 name: Thread 4 name: Thread 5 name: Thread 6 name: Thread 7 name: Thread 8 name: Thread 9: Thread 10 name: Thread 11 name: Thread 12 name: Thread 13: Thread 14 name: Thread 11 crashed with ARM Thread State (64-bit): Binary Images: EOF |
|
Or maybe not, this is the other crash report we get, maybe this is the one related... Date/Time: 2025-05-06 13:26:37.5899 +0200 Exception Type: EXC_CRASH (SIGABRT) Triggered by Thread: 23 Thread 0 name: Thread 1 name: Thread 2 name: Thread 3: Thread 4 name: Thread 5 name: Thread 6 name: Thread 7 name: Thread 8 name: Thread 9 name: Thread 10 name: Thread 11 name: Thread 12 name: Thread 13: Thread 14 name: Thread 15: Thread 16: Thread 17: Thread 18 name: Thread 19 name: Thread 20 name: Thread 21 name: Thread 22 name: Thread 23 name: Thread 23 crashed with ARM Thread State (64-bit): Binary Images: EOF |
@janusw Try to add the following instead of the Ensure you have the provided Microsoft.NETCore.App.Ref.9.0.5.nupkg.zip |
@linnkrb With the patch applied, we expect it to throw a different managed exception, which will help us determine the root cause of the failure. Did you get any managed exception trace related to threading (but not the invalid cast, which should be already addressed in the provided NuGet packages)? |
Thanks for the hints, @kotlarmilos. With this I can build, but the results are the same as before. I still see So either your modifications have no influence on the observed behavior, or the nupkgs are still not picked up properly. I actually expected my @kotlarmilos Can you recommend a simple way to verify that my build uses the nupkgs you provided? |
|
Ok, sorry, I think I made a mistake, which caused the nupkgs not to be used. I fixed it, then saw errors like: I fixed that by removing the Android TFM, and only keeping iOS. I'm about to rebuild and retry ... |
We got the same, but re-build with just adding those pacgages to iOs, and still no luck getting anything that brings more light to the issue. All our crashlogs are now of this, just varies what thread that crashes depending on how long the app has been running `Incident Identifier: 67DBFC0B-B6EC-4EA9-8894-127D2BD91F80 Date/Time: 2025-05-06 16:42:54.7979 +0200 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Kernel Triage: Thread 0 name: Thread 1 name: Thread 2 name: Thread 3 name: Thread 4 name: Thread 5 name: Thread 6 name: Thread 7 name: Thread 8 name: Thread 9 name: Thread 10: Thread 11 name: Thread 12: Thread 13 name: Thread 14: Thread 15: Thread 9 crashed with ARM Thread State (64-bit): Binary Images: EOF |
|
I have used the fork and was able to get the following running on the iPhone. Hopefully this will help. As a refresher, we are running in a Parallel.ForEachAsync() TaskScheduler.UnobservedTaskException --> System.NullReferenceException: Object reference not set to an instance of an object. |
@wgriep and others, I appreciate you all being willing to drink from the firehose ;-). Please keep the info coming if you have it and we'll keep trying to narrow this down. |
Despite longer testing sessions on two different devices, I haven't been able to trigger any more crashes/exceptions with the modified runtime pkgs, which might indicate that these pkgs already fix the problem for our use case, or at least make it even more unlikely than before. I'll note that our use case is not as massively multi-threaded as the |
Description
reproduces absolutely randomly both in debug and in release mode, appeared after switching from net8-ios to net9-ios (It doesn't seem like it applies to iOS)
stacktrace of InvalidCastException:
System.InvalidCastException: Specified cast is not valid.
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.FinishContinuations()
at System.Threading.Tasks.Task.FinishStageThree()
at System.Threading.Tasks.Task.FinishStageTwo()
at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute)
at System.Threading.Tasks.Task.Finish(Boolean userDelegateExecute)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(Object workItem, Thread currentThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
stacktrace of InvalidOperationException (from Sentry):
System.InvalidOperationException: Arg_InvalidOperationException
?, in void ManualResetValueTaskSourceCore.SignalCompletion()
?, in void ManualResetValueTaskSourceCore.SetResult(long)
?, in void ThreadPoolValueTaskSource.ExecuteInternal()
?, in void ThreadPoolValueTaskSource.System.Threading.IThreadPoolWorkItem.Execute()
?, in void ThreadPoolWorkQueue.DispatchItemWithAutoreleasePool(object, Thread)
?, in bool ThreadPoolWorkQueue.Dispatch()
?, in void WorkerThread.WorkerThreadStart()
also screenshot from debug session (from AppDomain.CurrentDomain.UnhandledException):
Reproduction Steps
have not steps
Expected behavior
no crashes
Actual behavior
crash occurs
Regression?
works in net8-ios
Known Workarounds
No response
Configuration
net9, ios 18
Other information
No response
The text was updated successfully, but these errors were encountered: