When create Elsa.Workflows.Activities.NotFoundActivity in Elsa.Workflows.Serialization.Converters.ActivityJsonConverter the activityRegistry.Find() looks like it cannot find the ActivityDescriptor yielding these nullref exceptions

I'm wondering if handling these null values isn't going to hide the underlying problem of why the descriptor was null in the first place.

yes your right, I will add an exception in ActivityJsonConverter when
var notFoundActivityDescriptor = activityRegistry.Find()!; yields a null value (and remove the !)
So now the improvement will be that : The exception will be on what's gone wrong instead off a nullref exception.

I think we found how to reproduce this issue. Since ActivityRegistry is a singleton and RegisterAsync is only called when dealing with workflow graphs/builder. So when a workflow is started and the host process is restarted the registry is empty. resuming the workflow or quering the workflowdefinition will result in these exceptions.
So i think the trick would be to do the registry as an initialization task.

We frequently getting these issues after the container running the workflow-engine was restarted.

@sfmskywalker I've added a test to get this issue. MaterializeWorkflowPriorToPopulateRegistriesAsync
the commented code await host.PopulateRegistriesAsync(); fixes it. I see there is a startup task that should be called on startup

i think, i understand as to why this can happen.
activityRegistryPopulator.PopulateRegistryAsync loops thru all IEnumerable providers.
(thus also the WorkflowDefinitionActivityProvider).
This will attempt to load the definitions from the store. so there 2 ways that this can go wrong.

1. If the order of providers is returned in the wrong order (i think you can not rely on the order).
2. The first time because the _activitydescriptors are set at the end of the method (Interlocked.Exchange(ref _activityDescriptors, activityDescriptors); which shall not be reached since this is an exception when materializing the workflowdefinition.

So i changes the first step in PopulateRegistryAsync to populate TypedActivities only. In this case the NotFoundActivity is always available. then populate the workflowdefinitionstore and continue as it was.

@sfmskywalker do you think this is oke? (We tested it by replacing the activityRegistryPopulator and it seems to work)