Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-128041: Add a terminate_workers method to ProcessPoolExecutor #128043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 36 commits into from
Mar 3, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
47b162a
gh-128041 - Add a terminate_workers method to ProcessPoolExecutor
csm10495 Dec 17, 2024
6ef8833
📜🤖 Added by blurb_it.
blurb-it[bot] Dec 17, 2024
61c9b14
Fix lint
csm10495 Dec 17, 2024
3bf5464
Swap to SIGTERM as the default
csm10495 Dec 17, 2024
4b285b8
Add some tests
csm10495 Dec 17, 2024
b4939fd
Update some docs
csm10495 Dec 17, 2024
ba6a4c0
Fix docs
csm10495 Dec 17, 2024
5d58e50
Fix docs
csm10495 Dec 17, 2024
0db381b
PR fixes/updates
csm10495 Dec 17, 2024
7ae1685
SIGKILL doesn't exist on windows
csm10495 Dec 17, 2024
f7ad96c
Update Lib/concurrent/futures/process.py
csm10495 Dec 17, 2024
2c0b578
Apply suggestions from code review
csm10495 Dec 18, 2024
a878221
Fix indenting from suggestions
csm10495 Dec 18, 2024
794ee25
Internally call shutdown to prevent a resource leak when calling term…
csm10495 Dec 20, 2024
64693a7
Change test to not validate calling of os.kill since shutdown may cal…
csm10495 Dec 20, 2024
926dff1
Commit to retrigger CI
csm10495 Dec 20, 2024
6d77c10
Merge branch 'python:main' into terminate_workers
csm10495 Feb 25, 2025
4429b2f
PR feedback. Split terminate_workers into terminate_workers and kill_…
csm10495 Feb 25, 2025
b8d6e5f
Remove un-needed imports
csm10495 Feb 25, 2025
f9a7714
lint
csm10495 Feb 25, 2025
7cfa42e
Harden a test a bit to ensure the correct type of kill/terminate was …
csm10495 Feb 25, 2025
2b31fab
rekick ci
csm10495 Feb 25, 2025
ad15ee5
Allow more time for queue to get data back
csm10495 Feb 25, 2025
0f57912
Use subTest to break up tests
csm10495 Feb 26, 2025
c16fde5
Merge branch 'main' into terminate_workers
csm10495 Feb 26, 2025
1bedb28
Apply suggestions from code review
csm10495 Feb 27, 2025
f1b0cf6
PR feedback: swap to dict with constants, better subtest parameteriza…
csm10495 Feb 27, 2025
cc5f359
swap to using context in the test
csm10495 Feb 27, 2025
b3cc8a2
trailing whitespace
csm10495 Feb 27, 2025
dbf9d32
Various pr feedbacks
csm10495 Mar 2, 2025
1e16da6
PR feedback: swap name of terminate_or_kill to force_shutdown
csm10495 Mar 2, 2025
52a5326
PR feedback: swap test names
csm10495 Mar 2, 2025
7f09586
PR feedback: use self.executor_type instead of ProcessPoolExecutor di…
csm10495 Mar 2, 2025
d5f7578
Add constants for terminate/kill methods
csm10495 Mar 2, 2025
0e42eca
feedback to get below 80 chars per pep8
csm10495 Mar 2, 2025
3f55347
Merge branch 'main' into terminate_workers
csm10495 Mar 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions Doc/library/concurrent.futures.rst
Original file line number Diff line number Diff line change
Expand Up @@ -415,6 +415,30 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
require the *fork* start method for :class:`ProcessPoolExecutor` you must
explicitly pass ``mp_context=multiprocessing.get_context("fork")``.

.. method:: terminate_workers()

Attempt to terminate all living worker processes immediately by calling
:meth:`Process.terminate <multiprocessing.Process.terminate>` on each of them.
Internally, it will also call :meth:`Executor.shutdown` to ensure that all
other resources associated with the executor are freed.

After calling this method the caller should no longer submit tasks to the
executor.

.. versionadded:: next

.. method:: kill_workers()

Attempt to kill all living worker processes immediately by calling
:meth:`Process.kill <multiprocessing.Process.kill>` on each of them.
Internally, it will also call :meth:`Executor.shutdown` to ensure that all
other resources associated with the executor are freed.

After calling this method the caller should no longer submit tasks to the
executor.

.. versionadded:: next

.. _processpoolexecutor-example:

ProcessPoolExecutor Example
Expand Down
5 changes: 5 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -444,6 +444,11 @@ contextvars
* Support context manager protocol by :class:`contextvars.Token`.
(Contributed by Andrew Svetlov in :gh:`129889`.)

* Add :meth:`concurrent.futures.ProcessPoolExecutor.terminate_workers` and
:meth:`concurrent.futures.ProcessPoolExecutor.kill_workers` as
ways to terminate or kill all living worker processes in the given pool.
(Contributed by Charles Machalow in :gh:`128043`.)


ctypes
------
Expand Down
71 changes: 71 additions & 0 deletions Lib/concurrent/futures/process.py
Original file line number Diff line number Diff line change
Expand Up @@ -626,6 +626,14 @@ class BrokenProcessPool(_base.BrokenExecutor):
while a future was in the running state.
"""

_TERMINATE = "terminate"
_KILL = "kill"

_SHUTDOWN_CALLBACK_OPERATION = {
_TERMINATE,
_KILL
}


class ProcessPoolExecutor(_base.Executor):
def __init__(self, max_workers=None, mp_context=None,
Expand Down Expand Up @@ -855,3 +863,66 @@ def shutdown(self, wait=True, *, cancel_futures=False):
self._executor_manager_thread_wakeup = None

shutdown.__doc__ = _base.Executor.shutdown.__doc__

def _force_shutdown(self, operation):
"""Attempts to terminate or kill the executor's workers based off the
given operation. Iterates through all of the current processes and
performs the relevant task if the process is still alive.

After terminating workers, the pool will be in a broken state
and no longer usable (for instance, new tasks should not be
submitted).
"""
if operation not in _SHUTDOWN_CALLBACK_OPERATION:
raise ValueError(f"Unsupported operation: {operation!r}")

processes = {}
if self._processes:
processes = self._processes.copy()

# shutdown will invalidate ._processes, so we copy it right before
# calling. If we waited here, we would deadlock if a process decides not
# to exit.
self.shutdown(wait=False, cancel_futures=True)

if not processes:
return

for proc in processes.values():
try:
if not proc.is_alive():
continue
except ValueError:
# The process is already exited/closed out.
continue

try:
if operation == _TERMINATE:
proc.terminate()
elif operation == _KILL:
proc.kill()
except ProcessLookupError:
# The process just ended before our signal
continue

def terminate_workers(self):
"""Attempts to terminate the executor's workers.
Iterates through all of the current worker processes and terminates
each one that is still alive.

After terminating workers, the pool will be in a broken state
and no longer usable (for instance, new tasks should not be
submitted).
"""
return self._force_shutdown(operation=_TERMINATE)

def kill_workers(self):
"""Attempts to kill the executor's workers.
Iterates through all of the current worker processes and kills
each one that is still alive.

After killing workers, the pool will be in a broken state
and no longer usable (for instance, new tasks should not be
submitted).
"""
return self._force_shutdown(operation=_KILL)
97 changes: 97 additions & 0 deletions Lib/test/test_concurrent_futures/test_process_pool.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
import os
import queue
import signal
import sys
import threading
import time
import unittest
import unittest.mock
from concurrent import futures
from concurrent.futures.process import BrokenProcessPool

from test import support
from test.support import hashlib_helper
from test.test_importlib.metadata.fixtures import parameterize

from .executor import ExecutorTest, mul
from .util import (
Expand All @@ -22,6 +26,19 @@ def __init__(self, mgr):
def __del__(self):
self.event.set()

TERMINATE_WORKERS = futures.ProcessPoolExecutor.terminate_workers.__name__
KILL_WORKERS = futures.ProcessPoolExecutor.kill_workers.__name__
FORCE_SHUTDOWN_PARAMS = [
dict(function_name=TERMINATE_WORKERS),
dict(function_name=KILL_WORKERS),
]

def _put_sleep_put(queue):
""" Used as part of test_terminate_workers """
queue.put('started')
time.sleep(2)
queue.put('finished')


class ProcessPoolExecutorTest(ExecutorTest):

Expand Down Expand Up @@ -218,6 +235,86 @@ def mock_start_new_thread(func, *args, **kwargs):
list(executor.map(mul, [(2, 3)] * 10))
executor.shutdown()

def test_terminate_workers(self):
mock_fn = unittest.mock.Mock()
with self.executor_type(max_workers=1) as executor:
executor._force_shutdown = mock_fn
executor.terminate_workers()

mock_fn.assert_called_once_with(operation=futures.process._TERMINATE)

def test_kill_workers(self):
mock_fn = unittest.mock.Mock()
with self.executor_type(max_workers=1) as executor:
executor._force_shutdown = mock_fn
executor.kill_workers()

mock_fn.assert_called_once_with(operation=futures.process._KILL)

def test_force_shutdown_workers_invalid_op(self):
with self.executor_type(max_workers=1) as executor:
self.assertRaises(ValueError,
executor._force_shutdown,
operation='invalid operation'),

@parameterize(*FORCE_SHUTDOWN_PARAMS)
def test_force_shutdown_workers(self, function_name):
manager = self.get_context().Manager()
q = manager.Queue()

with self.executor_type(max_workers=1) as executor:
executor.submit(_put_sleep_put, q)

# We should get started, but not finished since we'll terminate the
# workers just after
self.assertEqual(q.get(timeout=5), 'started')

worker_process = list(executor._processes.values())[0]
getattr(executor, function_name)()
worker_process.join()

if function_name == TERMINATE_WORKERS or \
sys.platform == 'win32':
# On windows, kill and terminate both send SIGTERM
self.assertEqual(worker_process.exitcode, -signal.SIGTERM)
elif function_name == KILL_WORKERS:
self.assertEqual(worker_process.exitcode, -signal.SIGKILL)
else:
self.fail(f"Unknown operation: {function_name}")

self.assertRaises(queue.Empty, q.get, timeout=1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do suspect we may see this come up as occasionally flaky in buildbot or CI systems as it is depend on the timing of the sleeps and kills which really can't be guaranteed on a loaded system. if so, _put_sleep_put can have its sleep increased. lets see how it goes first.


@parameterize(*FORCE_SHUTDOWN_PARAMS)
def test_force_shutdown_workers_dead_workers(self, function_name):
with self.executor_type(max_workers=1) as executor:
future = executor.submit(os._exit, 1)
self.assertRaises(BrokenProcessPool, future.result)

# even though the pool is broken, this shouldn't raise
getattr(executor, function_name)()

@parameterize(*FORCE_SHUTDOWN_PARAMS)
def test_force_shutdown_workers_not_started_yet(self, function_name):
ctx = self.get_context()
with unittest.mock.patch.object(ctx, 'Process') as mock_process:
with self.executor_type(max_workers=1, mp_context=ctx) as executor:
# The worker has not been started yet, terminate/kill_workers
# should basically no-op
getattr(executor, function_name)()

mock_process.return_value.kill.assert_not_called()
mock_process.return_value.terminate.assert_not_called()

@parameterize(*FORCE_SHUTDOWN_PARAMS)
def test_force_shutdown_workers_stops_pool(self, function_name):
with self.executor_type(max_workers=1) as executor:
task = executor.submit(time.sleep, 0)
self.assertIsNone(task.result())

getattr(executor, function_name)()

self.assertRaises(RuntimeError, executor.submit, time.sleep, 0)


create_executor_tests(globals(), ProcessPoolExecutorTest,
executor_mixins=(ProcessPoolForkMixin,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Add :meth:`concurrent.futures.ProcessPoolExecutor.terminate_workers` and
:meth:`concurrent.futures.ProcessPoolExecutor.kill_workers` as
ways to terminate or kill all living worker processes in the given pool.
(Contributed by Charles Machalow in :gh:`128043`.)
Loading