Describe the bug

ScheduledTaskExecutor.ScheduleTask creates a TaskCompletionSource<object> without TaskCreationOptions.RunContinuationsAsynchronously. When StartProcessingAsync calls TrySetResult, the awaiting caller's continuation can run inline on the executor's single processing thread. If that continuation blocks (e.g. on a lock), the executor is stalled and any other caller doing a sync Execute() that triggers a state transition will deadlock waiting for the executor.

Expected behavior

TrySetResult should not run caller continuations inline on the executor's processing thread. The executor should return to its processing loop promptly so subsequent scheduled tasks aren't blocked by unrelated caller code.

Actual behavior

Inline continuations from TrySetResult can block the executor thread indefinitely, deadlocking any sync caller that needs the executor to process a state transition. (We hit this in production leading when a thread hung for 4 days until discovery).

Steps to reproduce

Reproduction repo: https://github.com/crnhrv/polly-deadlock-repro

Running this project prints out DEADLOCK DETECTED after we hit a task cancellation token set to 5 seconds.

Changing line 29 of ScheduledTaskExecutor.cs to the following fixes it:

var source = new TaskCompletionSource<object>(TaskCreationOptions.RunContinuationsAsynchronously);

Exception(s) (if any)

No response

Polly version

8.6.5

.NET Version

net9.0

Anything else?

As far as I'm aware this isn't some kind of design decision to discourage people from using Polly in a certain, incorrect, way that we have inadvertantly run into, but if it is then apologies for the mistaken report.

Full disclosure the repro was generated by an LLM, but I've verified that it works and just changing that line in Polly resolves the issue, which seems to match our own production issue.