fix(logging): prevent log file race condition under parallel attempts#1827
fix(logging): prevent log file race condition under parallel attempts#1827nuthalapativarun wants to merge 4 commits into
Conversation
When multiprocessing.Pool forks worker processes they inherit the parent's open FileHandler file descriptors. Concurrent writes (or reentrant flushes triggered by third-party library destructors such as openai/httpcore closing connections in __del__) on those shared handles cause a RuntimeError in Python 3.13+. Add a _worker_logging_init pool initializer that closes all inherited handlers and re-opens a fresh FileHandler in each worker, giving every worker its own private file descriptor. The log file path is read from the GARAK_LOG_FILE env var that garak/__init__.py already sets before spawning any children. Fixes NVIDIA#1355 Signed-off-by: Varun Nuthalapati <[email protected]>
|
The fix description appears promising but can we obtain:
|
Adds tests/probes/test_probes_base_parallel_logging.py to reproduce and verify the fix for NVIDIA#1355. Four unit tests cover handler teardown, file handler creation from GARAK_LOG_FILE, absence of handler when env is unset, and idempotency. Two integration tests spawn a real Pool to confirm workers log concurrently without RuntimeError and that each worker opens its own private file descriptor rather than sharing the parent's inherited fd. Signed-off-by: Varun Nuthalapati <[email protected]> Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
Thanks for the clear feedback, @aishwaryap! I've added Unit tests for
Integration tests that exercise the actual Pool path:
All six pass locally. Let me know if you'd like any adjustments. |
jmartin-tech
left a comment
There was a problem hiding this comment.
The new tests added here validate expectations created by the new code, however they do not demonstrate why the code was needed. Is there a way to create a test that fails without the new initializer but passes with it to exercise the intent of the change?
|
Thanks for the feedback @jmartin-tech. I've added |
…ializer fix Signed-off-by: Varun Nuthalapati <[email protected]>
9843381 to
7aa5f86
Compare
|
Hi @jmartin-tech, following up on your review feedback ("the new tests... do not demonstrate why the code was needed"). I've added |
…ace fix Address review feedback from jmartin-tech on PR NVIDIA#1827: the existing tests only asserted that _worker_logging_init was wired into Pool(...), without exercising the actual fd-sharing bug from issue NVIDIA#1355. Add test_worker_inherits_parent_fd_without_initializer_but_not_with_it, which runs the same worker function through a fork-context Pool twice: once with no initializer (the pre-fix behavior, where a forked worker inherits the parent's exact FileHandler fd) and once with _worker_logging_init (the fix). The test asserts the worker's reported fd equals the parent's fd in the first case and differs in the second, so it fails if the initializer is removed or not passed to Pool. The test is skipped on platforms without a fork-capable multiprocessing context (e.g. Windows, which only supports spawn) since fd inheritance cannot be reproduced there; it runs for real on fork-based CI.
…ace fix Address review feedback from jmartin-tech on PR NVIDIA#1827: the existing tests only asserted that _worker_logging_init was wired into Pool(...), without exercising the actual fd-sharing bug from issue NVIDIA#1355. Add test_worker_inherits_parent_fd_without_initializer_but_not_with_it, which runs the same worker function through a fork-context Pool twice: once with no initializer (the pre-fix behavior, where a forked worker inherits the parent's exact FileHandler fd) and once with _worker_logging_init (the fix). The test asserts the worker's reported fd equals the parent's fd in the first case and differs in the second, so it fails if the initializer is removed or not passed to Pool. The test is skipped on platforms without a fork-capable multiprocessing context (e.g. Windows, which only supports spawn) since fd inheritance cannot be reproduced there; it runs for real on fork-based CI. Signed-off-by: Varun Nuthalapati <[email protected]>
4adbdf8 to
76d0003
Compare
|
Hi @jmartin-tech, I've pushed the regression test you asked for (added |
|
Hi @jmartin-tech and @aishwaryap — following up here. Pushed a regression test on 2026-06-26 that demonstrates the race condition directly: it fails without the fix and passes with it. DCO is clean. Would appreciate a re-review when you have a moment. |
|
Hi @jmartin-tech and @aishwaryap — just a gentle ping since it's been a couple of days. The requested changes have been addressed: a regression test was pushed on 2026-06-26 that fails without the fix and passes with it. Would appreciate a re-review when you get a chance. Thank you! |
|
Hi @jmartin-tech and @aishwaryap — following up again. The regression test was pushed on 2026-06-26: it fails without the fix and passes with it, demonstrating the race condition directly. Last bumped 2026-06-27 with no response yet. Would appreciate a re-review when you get a moment. Happy to make further adjustments if needed. Thanks! |
Summary
Fixes #1355.
When
multiprocessing.Poolforks worker processes they inherit the parent's openFileHandlerfile descriptor. Concurrent writes — or reentrant flushes triggered by third-party library destructors (e.g.openai/httpcoreclosing connections in__del__) — on those shared handles cause aRuntimeError: reentrant call inside <_io.BufferedWriter>in Python 3.13+.Fix: pass
initializer=_worker_logging_inittoPool(...)ingarak/probes/base.py. The initializer runs once in each worker process and:FileHandlervialogging.basicConfig, using theGARAK_LOG_FILEenv var thatgarak/__init__.pyalready sets before any children are spawned.Each worker then has its own private file descriptor pointing at the same log file, eliminating the shared-handle race.
Files changed
garak/probes/base.py— add_worker_logging_init()and pass it asinitializertoPoolTest plan
parallel_attempts: 2and a multi-probe run (e.g. the config from the issue comments) — confirm noRuntimeErrorappears in the log