Closed
Description
Bug report
Bug description:
Below code demonstrates two problems that occur when
(a) worker process throws an exception, and
(b) shutdown(wait=False)
is used
- When max_workers < number of submitted tasks, raises
exception TypeError: object of type 'NoneType' has no len()
- When max_workers > number of submitted tasks, raises
exception TypeError: object of type 'NoneType' has no len()
AND then program hangs
There is no problem when shutdown(wait=True)
is used or if the worker processes exit cleanly.
import concurrent.futures
import multiprocessing as mp
import time
def task(n: int) -> int:
if n == 2:
raise Exception("Not gonna do it")
else:
time.sleep(n)
return n
def main() -> None:
# max_workers=2 results in raised exception and program exits
# max_workers=4 results in raised exception and program hangs
#
executor = concurrent.futures.ProcessPoolExecutor(
max_workers=2, mp_context=mp.get_context("forkserver"), max_tasks_per_child=1
)
f1 = executor.submit(task, 1)
f2 = executor.submit(task, 2)
f3 = executor.submit(task, 3)
result = 0
try:
result += f1.result()
result += f2.result()
result += f3.result()
print(f"Result = {result}")
except Exception as e:
print(f"Exception while getting result : {e}")
executor.shutdown(wait=False)
if __name__ == "__main__":
main()
CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Linked PRs
- [3.12] gh-132969: Fix exception/hang shutdown(wait=False) and a task exited abnormally #133220
- gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally #133222
- [3.13] gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222) #135343
- [3.14] gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222) #135344
Metadata
Metadata
Assignees
Labels
Projects
Status
Done