fix(connectors): persistent event-loop bridge to stop Email Triage UI hang (#1579)#1589
Conversation
Failing tests that document the two bugs fixed in the follow-up commit: - AC2: repeated refresh must not raise cross-loop RuntimeError - AC3: stalled async op must surface ConnectorsError within timeout - AC4: agent-id contextvar must propagate through the bridge - Singleton/deadlock guard contracts for _loop.py
…ows hang (#1579) Email triage hung forever on Windows after ~21 Gmail HTTP requests because asyncio.run in get_access_token_sync created/destroyed a ProactorEventLoop on each call. The asyncio.Lock in the token cache was also bound to the first loop and raised RuntimeError when used in a subsequent loop (Python ≤ 3.11). Introduces _loop.py: a single daemon thread running one asyncio event loop for the process lifetime. All four sync→async worker-thread wrappers now submit coroutines via asyncio.run_coroutine_threadsafe with a 30s bounded wait. Contextvar propagation (agent-id) is preserved by capturing copy_context() at submit time and running the coroutine under it in _ctx_wrap. A deadlock guard raises RuntimeError if run_sync is called from the loop thread itself. Wrappers migrated: get_access_token_sync (api.py), get_or_refresh_sync (tokens.py), get_credential_sync (handler.py), emit_change else-branch (events.py). CLI asyncio.run calls (cli.py) are left untouched — they are one-shot entry points not subject to the teardown churn.
Preserve the original concurrent.futures.TimeoutError as the cause when raising ConnectorsError so the full traceback is available to debuggers and error reporters. Also removes an unused _var_token assignment in the test spotted during review.
Future.result(timeout=) raises concurrent.futures.TimeoutError, which is only an alias for builtins.TimeoutError on Python 3.11+. GAIA targets >=3.10, where it is a distinct class — the bare `except TimeoutError` would not catch it and a raw timeout would leak instead of the actionable ConnectorsError. Catch it by its canonical name so the bounded-wait guard fires on every supported Python.
Real-world validation (Windows)Drove the actual reported repro on Windows 11 / Python 3.12.10 ( No hang. Across runs the UI returned a clean answer in seconds — the infinite-"thinking" behavior is gone. Execution went worker-thread → the new The full 20-email fetch loop wasn't reached due to an orthogonal OAuth issue in the Agent UI (per-agent grant not conferred by the UI + a token-refresh 401) — tracked separately in #1592, not a regression of this change. Unit: |
SummarySolid, well-scoped fix that replaces the per-call The single most important thing: the implementation is correct and the test plan is refreshingly honest about its one gap — the production hang involves real keep-alive/TLS transports lingering at loop teardown, which the synthetic loopback harness can't reproduce, so the interactive real-Gmail Windows run remains the genuine confirming gate. Nothing blocking; the notes below are minor. Issues Found🟢 Timeout cancels the future but cannot stop an already-running coroutine ( On timeout, 🟢 Stopped-loop re-init orphans the old loop/thread ( If 🟢 Loop-start readiness uses a polling busy-wait (
Strengths
VerdictApprove with suggestions — no blocking issues. The three notes are minor (one best-effort-cancel comment, one latent re-init edge, one stylistic busy-wait) and none need to land before merge. Recommend merging once the interactive Windows real-Gmail gate in the description is confirmed, since that's the only check that exercises the actual reported hang. |
Address bot-review polish on the connectors loop bridge: - Replace the busy-wait startup spin with a threading.Event handshake signalled from inside the loop thread via call_soon(ready.set); drop the now-unused time import. Keeps the fail-loud RuntimeError on startup timeout. - Note that the timeout-path future.cancel() is best-effort: a coroutine already executing on the loop thread cannot be interrupted (bounded by the inner httpx 10s timeout). - Note that the re-init branch intentionally does not reclaim a prior loop (latent-only — the daemon loop only stops at process exit).
|
Thanks for the review — all three notes addressed in
On the confirming gate: validated on Windows 11 / Python 3.12 by driving the real repro ( |
amd#1591) ## Why this matters Before: a hung agent tool — e.g. the stuck connector token call behind amd#1579 — left the Agent UI stuck in "thinking" with no error, indefinitely. The base agent loop called `tool(**args)` with no timeout and `process_query` never observed any cancel signal, so the worker (`threading.Thread(target=_run_agent)`) leaked. The only backstop was the consumer-side 600s SSE timeout, which took a full 10 minutes *and* only broke the consumer loop, never the producer. After: every tool call is bounded (default 180s), so a hung tool surfaces an actionable error well under 600s instead of an infinite hang — and the producer thread is actually torn down rather than leaked. amd#1589 fixed the specific connector root cause; this closes the systemic gap so the next tool hang can't reproduce the same dead-end UX. Two composable, fail-loud guards live in the base `Agent`, so **all** agents benefit: - **Per-tool execution timeout** (`_execute_tool`): the tool body runs in a daemon worker joined with the resolved timeout; on timeout it raises `ToolExecutionTimeout` → an actionable error dict naming the tool and the `GAIA_AGENT_TOOL_TIMEOUT` knob. Default 180s via `tool_execution_timeout()` (mirrors the existing `default_max_steps()` — lazy env read, raises on an invalid value). New `@tool(timeout=...)` override; `generate_image` opts out at 900s because SD model download can legitimately take ~600s. - **Cooperative cancel** (`Agent._cancel_event`): checked at each step boundary in `_process_query_impl`. `_stream_chat_response` assigns a fresh event per request and sets it in `_cleanup_stream`, so the existing `producer.join(5s)` finally reaps the thread. The per-tool timeout keeps each step bounded, guaranteeing the cancel boundary is reached. Known, documented limit: Python can't kill the timed-out worker thread, so a hung tool's worker keeps running until the process exits — but it's a daemon and no longer blocks the agent loop or leaks the producer. Closes amd#1579. ## Test plan - [ ] `python -m pytest tests/unit/test_agent_tool_timeout.py -q` — 14 tests: blocking tool → bounded actionable error in <5s (not 30s/600s); per-tool override beats global; `tool_execution_timeout()` parsing incl. invalid-raises; SD `generate_image` 900s opt-out resolves through `_resolve_tool_timeout`; producer-style thread does not outlive the request on mid-run cancel (asserts `send_messages` called once + thread dead). - [ ] `python -m pytest tests/unit/test_tool_decorator.py tests/unit/test_tool_registry_isolation.py tests/unit/agents/test_null_tool_name.py tests/unit/agents/test_tool_not_found_error.py tests/unit/chat/ui/test_chat_helpers.py -q` — regression set, green. - [ ] `python util/lint.py --black --isort` — clean on the changed files.
Closes #1579
Why this matters
Before: in
gaia chat --uion Windows, "Triage my last 20 emails" hangs in the "thinking" phase forever — Lemonade emits one generation, then nothing, and no output ever appears. The Email Triage agent fetches Gmail before its next LLM call, and every Gmail request re-fetches an OAuth token throughget_access_token_sync→ a brand-newasyncio.run(...)per call (~21 per triage), from the agent's worker thread, all sharing one module-levelasyncio.Lock. Spinning up and tearing down a fresh event loop on every call is fragile: on Windows it can block inProactorEventLoopteardown (the reported hang), and on Python ≤3.11 the sharedasyncio.Lockraises a cross-loopRuntimeErroron the next refresh.After: all connector async work runs on a single persistent event loop for the process lifetime. The lock binds to one stable loop, there is no per-call loop teardown churn, and a stalled connector operation now raises an actionable error within a bounded timeout instead of hanging indefinitely.
Test plan
python -m pytest tests/unit/connectors/ -x -q— 509 passed (incl. 7 new bridge tests: repeated-refresh cross-loop safety, bounded-timeout raise, contextvar propagation, loop-thread deadlock guard)python util/lint.py --all— cleanWindowsProactorEventLoopPolicy): 25 rapidrun_synchttpx cycles on the persistent loop — the per-triage token-fetch pattern — complete with no hang; oldasyncio.run-per-call path timed alongside for comparisongaia chat --uireturns a triage and the UI leaves "thinking" (requires an interactive logon session — the connectors keyring is unreadable over SSH, so this can't be automated)The synthetic Windows harness verifies the persistent-loop bridge is stable but uses an instant-closing loopback endpoint, so it does not by itself reproduce the production hang (which involves real keep-alive/TLS transports lingering at loop teardown). The fix removes per-call loop teardown entirely; the interactive real-Gmail run is the confirming gate.