Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Run different worker types in separate threads #803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dandavison opened this issue Mar 27, 2025 · 1 comment
Open

Run different worker types in separate threads #803

dandavison opened this issue Mar 27, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@dandavison
Copy link
Contributor

Currently, when workflow and activity workers are run in the same process, they share the the same thread (same event loop):

if self._activity_worker:
tasks.append(asyncio.create_task(self._activity_worker.run()))
if self._workflow_worker:
tasks.append(asyncio.create_task(self._workflow_worker.run()))

Running these (and Nexus workers when they arrive) in different threads would prevent an async activity that incorrectly does blocking I/O from blocking workflow progress. In addition to limiting blast radius it should make it easier to understand the failure modes.

@dandavison dandavison added the enhancement New feature or request label Mar 27, 2025
@cretz
Copy link
Member

cretz commented Mar 28, 2025

This is going to be tough to do. Notes:

  • We have to be able to use the primary/user-controlled event loop for running (some) activities/operations anyways, so maybe this is a workflow worker only thing?
  • Workflow workers will have to remove async and go threaded which will cost threads. There is the blocking call for polling (currently async, so not using Python thread). Then the code backgrounds the work when received which uses async today. And then that work uses a thread on a pool so that it can impose a deadlock timeout on it. This means every workflow activation uses 1 thread from the thread pool today. We either have to double the in-use thread count or run a single thread for the life of the workflow worker that runs its own asyncio event loop so it can background activations as received (not to be confused with the next point).
  • Workflow workers will always need to use primary/user-controlled event loop for user codecs anyways (e.g. via asyncio.run_coroutine_threadsafe) which means it is subject to the same blocking concerns. So any benefits are only for those not using codecs.

We will have to decide whether the benefit is worth the costs here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants