I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/area-extensions-hosting
See info in area-owners.md if you want to be subscribed.

As a matter of fact, just to give a bit more of context, here is a test I have been doing:

lifetime.ApplicationStarted.Register(() => logger.LogInformation("ApplicationStarted Done...")); lifetime.ApplicationStopping.Register(() => logger.LogInformation("ApplicationStopping Done...")); lifetime.ApplicationStopped.Register(() => logger.LogInformation("ApplicationStopped Done..."));

ApplicationStarted appears in the event viewer perfectly well. But the other two are not. This when restaring or shutdown.

But if I stop gracefully the service manually, both messageds appear fine.

Looks similar to #62579 and #81274, @gkapellmann could you try the workaround mentioned and let us know if that helps your issue? The workaround is for unblocking you for short term, if you could let us know how it goes, it would be a useful data for to the team to find appropriate fix.

This issue has been marked needs-author-action and may be missing some important information.

This one is actually different than the error 1067 issue. In that case the service process was exiting before completing ServiceBase's OnStop method which is what notified SCM that the service cleanly stopped. In that case it would run all ApplicationStopping and ApplicationStopped.

In this case the machine is shutting down. I was looking at a similar issue internally and found that the problem was actually due to #41101. During a shutdown a service will first receive a CNTRL_SHUTDOWN_EVENT. The default is to do nothing for a service process per https://learn.microsoft.com/en-us/windows/console/handlerroutine#remarks, but that was changed for .NET processes in #41101.

I was able to workaround this by adding ConsoleControlHandler that cancels other handlers when seeing CNTRL_SHUTDOWN_EVENT - stopping the runtime's handler from shutting down the process and letting the service exit normally. In .NET 6.0 you can do this with a one-liner:

    using var registration = PosixSignalRegistration.Create(PosixSignal.SIGTERM, (context) => context.Cancel = true);

Just make sure that registration stays alive for the entire duration of the service - I put it in the Main() routine.

We should look into adding something like this to ServiceBase -- or having the runtime try to avoid handling CNTRL_SHUTDOWN_EVENT for services.

Tagging subscribers to this area: @dotnet/area-system-serviceprocess
See info in area-owners.md if you want to be subscribed.

Ideally we don't do this only in ServiceBase since that wouldn't fix cases where folks were implementing services themselves without using ServiceBase. Instead we should try to replicate whatever the OS does in the default control handler to distinguish between normal console processes and services.

Hello @ericstj, thank you for your answers!

I have not tested your workaround in .NET 6.0, but I did test wit with a small example in .NET 7.0 and is not working for me. As a matter of fact neither the StopAsync or StartAsync functions are getting triggered.

@gkapellmann if StartAsync is not triggered then you are having a problem with even starting up your service. Are you able to share your example?

@janvorli as FYI..

@ericstj Sure, no problem, I have created another repo with my very simple service example here.

It is based in .NET 7.0, when running you should be ableto see the events in the event manager, then shutting the PC down and then on again and I am not able to see the start async or stop async messages, but the main executeasyc events are there properly.

@gkapellmann I took a look at your repro. I was able to find out what was wrong by using Time-Travel debugging and examining the behavior at shutdown.

The PosixSignalRegistration workaround helped in that it did prevent the runtime from disabling exception handling, but the problem was there was still an unhandled exception in the logger:

(3/17/2023 9:24:59 AM) .ctor> Constructor...
(3/17/2023 9:24:59 AM) StartAsync> StartAsync...
(3/17/2023 9:24:59 AM) ExecuteAsync> Start of ExecuteAsync...
(3/17/2023 9:25:07 AM) StopAsync> StopAsync...
(3/17/2023 9:25:07 AM) StopAsync> System.AggregateException: An error occurred while writing to logger(s). (The interface is unknown.)
 ---> System.ComponentModel.Win32Exception (1717): The interface is unknown.
   at System.Diagnostics.EventLogInternal.InternalWriteEvent(UInt32 eventID, UInt16 category, EventLogEntryType type, String[] strings, Byte[] rawData, String currentMachineName)
   at System.Diagnostics.EventLogInternal.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
   at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
   at System.Diagnostics.EventLog.WriteEvent(EventInstance instance, Object[] values)
   at Microsoft.Extensions.Logging.EventLog.WindowsEventLog.WriteEntry(String message, EventLogEntryType type, Int32 eventID, Int16 category)
   at Microsoft.Extensions.Logging.EventLog.EventLogLogger.WriteMessage(String message, EventLogEntryType eventLogEntryType, Int32 eventId)
   at Microsoft.Extensions.Logging.EventLog.EventLogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.Logger.<Log>g__LoggerLog|13_0[TState](LogLevel logLevel, EventId eventId, ILogger logger, Exception exception, Func`3 formatter, List`1& exceptions, TState& state)
   --- End of inner exception stack trace ---
   at Microsoft.Extensions.Logging.Logger.ThrowLoggingError(List`1 exceptions)
   at Microsoft.Extensions.Logging.Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.Logger`1.Microsoft.Extensions.Logging.ILogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
   at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, String message, Object[] args)
   at Microsoft.Extensions.Logging.LoggerExtensions.LogInformation(ILogger logger, String message, Object[] args)
   at TestService.Worker.StopAsync(CancellationToken cancellationToken) in C:\scratch\SimpleWinService_dotnet\VeroTest-WinService\Worker.cs:line 45

This was preventing you from running your SaveCurrentLog method. I was able to workaround that by wrapping the call to _logger.LogInformation in your stopasync with a try/catch.

I'll take a deeper look at this to see why it's happening. Could be that this is normal during shutdown. Could be that a handle got disposed out from under the logger.

Looks like those unhandled exceptions can also occur during the ApplicationStopping and ApplicationStopped events where the logger is also called. There are also cases where the framework library will log:

I saw that the handle was still valid - so not a case of being disposed while still logging. The very first failure to write to the event log fails with RPC_S_SERVER_UNAVAILABLE at this stack:

00 0000003f`631bdd08 00007ffd`f7b9fe7c     ntdll!RtlRaiseException
01 0000003f`631bdd10 00007ffd`f8067c60     KERNELBASE!RaiseException+0x6c
02 0000003f`631bddf0 00007ffd`f8067c24     RPCRT4!RpcpRaiseException+0x34
03 0000003f`631bde20 00007ffd`f8067c06     RPCRT4!RpcRaiseException+0x14
04 0000003f`631bde50 00007ffd`f80f121a     RPCRT4!NdrSendReceive+0x66
05 0000003f`631bde80 00007ffd`f80f069b     RPCRT4!NdrpClientCall3+0x6aa
06 0000003f`631be1a0 00007ffd`f8d88cc0     RPCRT4!NdrClientCall3+0xeb
07 0000003f`631be530 00007ffd`f8d88991     ADVAPI32!ElfReportEventW+0x254
08 0000003f`631be6b0 00007ffd`47cfad54     ADVAPI32!ReportEventW+0x171
09 0000003f`631be760 00007ffd`47cfab9d     0x00007ffd`47cfad54
0a 0000003f`631be840 00007ffd`47cfa20b     System_Diagnostics_EventLog!Interop.Advapi32.ReportEvent+0xed [/_/src/libraries/System.Diagnostics.EventLog/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2239] 
0b 0000003f`631be8f0 00007ffd`47d01aba     System_Diagnostics_EventLog!System.Diagnostics.EventLogInternal.InternalWriteEvent+0x2cb [/_/src/libraries/System.Diagnostics.EventLog/src/System/Diagnostics/EventLogInternal.cs @ 1383] 
0c 0000003f`631be9b0 00007ffd`47d01816     System_Diagnostics_EventLog!System.Diagnostics.EventLogInternal.WriteEvent+0x27a [/_/src/libraries/System.Diagnostics.EventLog/src/System/Diagnostics/EventLogInternal.cs @ 1341] 
0d 0000003f`631bea90 00007ffd`47d017bb     System_Diagnostics_EventLog!System.Diagnostics.EventLog.WriteEvent+0x36 [/_/src/libraries/System.Diagnostics.EventLog/src/System/Diagnostics/EventLog.cs @ 1035] 
0e 0000003f`631beac0 00007ffd`47d0127d     System_Diagnostics_EventLog!System.Diagnostics.EventLog.WriteEvent+0x2b [/_/src/libraries/System.Diagnostics.EventLog/src/System/Diagnostics/EventLog.cs @ 1030] 
0f 0000003f`631beaf0 00007ffd`47d00f10     Microsoft_Extensions_Logging_EventLog!Microsoft.Extensions.Logging.EventLog.WindowsEventLog.WriteEntry+0xcd [/_/src/libraries/Microsoft.Extensions.Logging.EventLog/src/WindowsEventLog.cs @ 35] 
10 0000003f`631beb90 00007ffd`47d008c8     Microsoft_Extensions_Logging_EventLog!Microsoft.Extensions.Logging.EventLog.EventLogLogger.WriteMessage+0xe0 [/_/src/libraries/Microsoft.Extensions.Logging.EventLog/src/EventLogLogger.cs @ 178] 
11 0000003f`631bed20 00007ffd`47cfc3ab     Microsoft_Extensions_Logging_EventLog!Microsoft.Extensions.Logging.EventLog.EventLogLogger.Log<Microsoft.Extensions.Logging.FormattedLogValues>+0x418 [/_/src/libraries/Microsoft.Extensions.Logging.EventLog/src/EventLogLogger.cs @ 120] 

So it looks to me like the machine is just shutting down the other services that would be needed to write to the event log. I tried adding service dependencies on the EventLog and RpcSs services to the test service, but it didn't seem to help.

I tried retargeting this service to net6.0 and noticed the exact same problem, so I'm not so sure writing to the eventlog during shutdown has ever worked correctly here.

It is interesting, I am going to look back in my repo history.

In the end the real priority is to have SaveCurrentLog working, the event logger is just for debugging, and I guess that what you say regarding the services makes sense. I did get the saved logs before, could it be that the order of the services shutting down is semi-random?

For some reason when running as a service, the log txt file is not being written, but when running as a normal app (double click), the log txt file is written.

Double check the directory. Services run with a working directory of c:\windows\system32.

When the stopping is requested, StopAsync is triggered, but that shouldnt prevent ExecuteAsync to finish completely, right?

ExecuteAsync is called by StartAsync which receives a cancellationToken that will be cancelled when the application is stopped.

There is no guarantee that StopAsync waits for any completion of ExecuteAsync. It looks to me like your ExecuteAsync method is just polling the cancellation token - so no cancellation registration is guaranteed to run -- and nothing else would wait on ExecuteAsync exiting gracefully.

Right, so I think that things have gone really clearer for me with this.

Part of the problem is that I was trying to use the backgroundservice in a way its not meant to be used.

So:

• If ExecuteAsync has a loop, it not necessarily will be fully finished when the service is ending.
• StopAsync is called at the end, but whatever happens in it might already be unloaded or unavailable.
• StopAsync shouldn't have long tasks in it.

I have modified how I do the logging, and seems to be working fine now.

Thanks a lot @ericstj for your help through this.

If we all are ok, I am good to close this issue.

This issue did help us discover the shutdown problem with CoreCLR's CNTRL_SHUTDOWN_EVENT which I mentioned here: #83093 (comment).

We've seen other reports of this same problem, so I'd like to keep this issue open to track a fix for that in CoreCLR. @janvorli was looking into a fix for that.

@mangod9 this is a regression from a previous release and I don't think it should be moved to future. This bug prevents graceful service exit due to disabling exception handling and it should be addressed.

believe there is a workaround in place to help with this issue? Will mark it for 9 to determine the feasibility of the fix.

The workaround is required in the customer's application, and they would need to identify this problem (exceptions not being caught/stop not completing on shutdown) and understand what was going on to know they needed the workaround - which is a pretty big stretch given how hard it is to debug shutdown issues. We would still like to see a fix for this.

Ok, chatted with @janvorli on this to follow up if there is a good solution on the runtime side.

@janvorli @ericstj What do you expect the runtime fix to be?

If I am interpreting the comments above correctly, this is what's happening in general:

1. The OS starts sending shutdown callbacks to services (e.g. CONTROL_SHUTDOWN).
2. The service does not cleanup completely before returning from these callbacks. There is some ongoing async/background activity.
3. This background activity can encounter unhandled exceptions if it attempts to call into other services that has been shutdown already (e.g. log events using EventLog)

To avoid unhandled exceptions in the last step, I think the service either needs to shutdown completely in the response to the shutdown callback and/or the service needs to start ignoring unhandled exceptions at some point during shutdown. #42275 may help with the later.

The primary issue is that a service should not call ExitProcess when receiving the CTRL_SHUTDOWN_EVENT, as described in https://learn.microsoft.com/en-us/windows/console/handlerroutine#remarks:

When a console application is run as a service, it receives a modified default console control handler. This modified handler does not call ExitProcess when processing the CTRL_LOGOFF_EVENT and CTRL_SHUTDOWN_EVENT signals. This allows the service to continue running after the user logs off. If the service installs its own console control handler, this handler is called before the default handler. If the installed handler calls ExitProcess when processing the CTRL_LOGOFF_EVENT signal, the service exits when the user logs off.

The problem got introduced when .NET started to register the control handler in #55333.
To fix that, we should change the behavior of our handler when the .NET app is running as a service. In fact, I've discussed that with Windows developers in the past and they said that the MSDN doc is slightly misleading and this is true for system session, not just services.

I haven't looked into the details of our implementation of the code related to the handler, maybe it would be sufficient to return FALSE from the HandlerRoutine if we detect we are running in the system session at this point:

@janvorli has it right

The problem is the process is getting two different shutdown signals. Services should only be shutdown through SCM - and that's how it works in .NETFramework and earlier .NET versions. It broke with #55333. The Windows default control handler identifies a service process and avoids sending it CTRL_SHUTDOWN_EVENT - the CoreCLR should do the same since it's behaving like the OS and registering a handler for all application types. To behave most like the OS here we should try to identify if the process is a service process - we can't count on it using System.ServiceProcess - some .NET processes opt to PInvoke directly to SCM. If possible we should replicate the OS's detection mechanism.

It broke with #55333

#50527 and #55333 introduced a low-level signal handling API. I do not think that the introduction of these APIs alone broke anything. The break was introduced as side-effect of ConsoleLifeTime using these APIs in #56057.

It does not make sense for this low-level signal handling API to be opinionated and handle app-model specific concerns, like skipping the handler registration for Windows Services. Windows Services are not the only ones with this type of issue. For example, the Win32 GUI apps should be only shutdown through WM_QUIT message.

I think that the problem is caused by ConsoleLifeTime being used by Windows Services. Windows Services are not console apps, so it does not make sense for them to use Console app model. (And similarly, Win32 GUI apps are not console apps and so they should not use ConsoleLifeTime. Dtto for all other appmodels - WinUI, MAUI, Avalonia, ... .)

It is true that the #55333 on its own without any SIGTERM registered handler doesn't cause the issue and the problem is that the HandlePosixSignal sets context.Cancel=true which was introduced by the #56057. So, it seems the fix should be rather in the HandlePosixSignal - to not to set context.Cancel for processes running under system session.

Or I guess just not using the ConsoleLifetime. That's what you were trying to say, @jkotas, right?

Or I guess just not using the ConsoleLifetime. That's what you were trying to say, @jkotas, right?

Right. ConsoleLifetime implements console appmodel that is not compatible with Windows services.

Note that we have SystemdLifetime that handles same concern for Systemd processes ("Unix services"):

Folks aren't using ConsoleLifetime for windows services, they are using WindowsServiceLifetime which doesn't use Posix registration - but instead uses SCM for these events. I don't think there's anything wrong the user code or the hosting code is doing here. They are expecting for the process to continue running until they can complete the SCM request to shutdown. That's not happening because CoreCLR registered a console handler and the OS's default handler (which avoids shutting down services) is no longer used. It's not running because of some console lifetime registration - the coreCLR registers this for all applications. There is no way for an app to tell the coreCLR not do this, (other than the hack I shared which inserts another handler and avoids calling the coreCLR).

Makes sense. It is .NET Core -> .NET Framework regression introduced by dotnet/coreclr#17265.

dotnet/coreclr#17265 was an attempt to fix the console lifetime issues without introducing proper signal handling APIs. It did not work well, and we ended up introducing proper signal handling APIs anyway.

Should we revert dotnet/coreclr#17265 and go back to .NET Framework behavior so that we are not baking app-model specific concerns into runtime?

• We may need to treat it as a breaking change
• It would align the behavior between regular CoreCLR and NAOT. NAOT does not register the shutdown handler and nobody noticed so far.