Processor: replace loky with pebble to enforce worker timeouts #1345
Conversation
- `_page_worker`: remove `ThreadPool` mechanism introduced in 3cc4780 (which broke processors that are not threadsafe like TF/Keras) - since _no mechanisms_ work to stop computation in uniprocessing (as not even `_thread.interrupt_main()` or `signal.alarm()` would interrupt I/O or C library calls like libtesseract): drop - since neither stdlib's nor loky's ProcessPoolExecutor enforces timeouts on jobs: replace by pebble - apply `max_seconds` timeout iff in ProcessPool mode iff running with METS Server - make `test_run_output_timeout` xfail - add `test_run_output_metsserver_timeout`
|
I wonder whether we should still keep some mechanism in the page worker, though – for those cases where our timeout mechanism does work even in uniprocessing. Like interrupting I/O wait or CPU-bound Pythonic computation with If we don't do that, we at least still have to update documentation (i.e. TIMEOUT does not apply without METS Server). And, regardless, perhaps it would be better to have some actual test cases that cover the pathological case (simulating a long-lasting C library call like libtesseract). EDIT: BTW, it's the same with |
|
Just quickly on this point:
What use case beyond experimenting/developing OCR-D is there for non-METS-server deployment? If timeout and parallelization are relevant factors, users should use processing and METS server. |
For simplicity and backwards-compatibility, we still want to support isolated runs of processor CLIs. It would be a shame if v3 envvars do not work there, too. |
_page_worker: removeThreadPoolmechanism introduced in 3cc4780 (which broke processors that are not threadsafe like TF/Keras)_thread.interrupt_main()orsignal.alarm()would interrupt I/O or C library calls like libtesseract): dropmax_secondstimeout iff in ProcessPool mode iff running with METS Servertest_run_output_timeoutxfailtest_run_output_metsserver_timeoutsee OCR-D/ocrd_anybaseocr#115 (comment) for context (plus internal discussion)