Fix SDK test completion races and timeout cleanup - #2642

Summary

Fix completion-subscription races across SDK tests, two Python fixture/cleanup defects, and missing pre-timeout diagnostics for intermittent CI hangs. SDK product sources, runtime pins, and snapshots are unchanged. Windows diagnostics add a CI-only pinned dotnet-stack tool manifest.

Current head: 77127f86dec3990e7bbfe809f177a3d94e63d6d6. SDK run 34787031867 is pending. The preceding run passed every Go, .NET, and Node check; its one Python failure now has a concrete stalled-operation trace and a verified fixture repair below. Full current-head CI is not yet green.

Completion-subscription repair

session.idle is ephemeral: subscribing after send returns can permanently miss completion, and durable history cannot recover idle. Starting a goroutine or scheduling an asyncio task is not itself a subscription fence.

Fake-RPC regressions force early terminal events before send replies, with idle omitted from durable history. Preserve prompts, session setup, output assertions, and nonblocking-send coverage. No forced already-idle flags, sleeps, timeout increases, or assertion weakening.

Python repairs

Abandoned test task after pytest-timeout

pytest-timeout can interrupt Runner.run outside the coroutine, leaving the timed-out test task alive and holding a session-disconnect lock. Module teardown then waits on that abandoned task, causing a second timeout.

The harness preserves diagnostics and cancels exactly the interrupted runner's test task. It does not cancel unrelated tasks, force-stop the runtime, or turn the initial timeout into success. Actual-plugin/module-fixture controls prove that old behavior reproduces the lock-only teardown stall, the fix removes it, and a runtime that continues withholding replies still causes teardown failure.

The original session-config test's initial 300-second timeout is not conclusively attributed to this cleanup defect. Its exact stalled await was absent from the original log. Six SDK RPC probes and six proxy probes confirmed the cleanup defect but did not identify that primary await. get_exchanges() uses independent HTTPX requests with inactivity timeouts, not a shared stdout response-future map.

Persisted-session fixture detached before completing its turn

Python Alpine ARM64/inprocess job 103801413979 failed test_should_list_find_and_inspect_persisted_session_state. The retained diagnostics identify session.disconnect() in the test's finally, awaiting a session.detach response for 299.922 seconds. This particular test creates an explicit stdio client even in the inprocess matrix cell; the shared fixture's FFI state is not the blocked connection.

The test enqueued an uncontrolled model turn merely to make the persisted-session listing nonempty, then inspected metadata and detached without observing turn completion. It now reuses the existing synthetic inference-response helpers, awaits send_and_wait with the same prompt and a 60-second completion budget, verifies the synthetic assistant response, and only then saves/lists metadata. All nonempty-list, exact-session, LocalSessionMetadataValue, and is_remote=False discriminator assertions remain. The obsolete persistence polling loop is removed. A nested finally also stops the per-test client if detach raises, without suppressing that detach failure.

Two controls invoke this exact E2E scenario with a controlled client: both fail against the original fixture and pass with the repair, proving completion-before-save and cleanup-on-detach-failure. The real pinned-runtime fixture also passed locally. This fixes the test's unobserved inference/cleanup overlap; it does not claim to identify or repair the native reason that the original detach reply was withheld.

Pre-timeout diagnostics

Python captures coroutine/async-generator await chains, pending RPC identifiers/methods, lock/transport state, Python thread stacks, and a bounded macOS native sample before cleanup destroys evidence. RPC payloads and arbitrary frame locals are excluded. An opt-in manual reproduce_timeout dispatch runs two full suites or ten session-config invocations with normal pytest/xdist options, first-failure exit, the existing 20-minute job budget, and per-invocation progress artifacts.

.NET macOS and Windows default/CAPI shard 1 wrap their unchanged test command with a watchdog retaining allowlisted provisioning/build/test/framework/shutdown progress, owned-process metadata, and bounded stacks. It observes descendants retaining output pipes after the root command exits. Windows uses a kill-on-close Job Object and retained process handles, plus a pinned dotnet-stack collector for up to four owned CoreCLR processes. Collectors have their own bounded Job Objects; cleanup never targets unrelated processes by executable name. Existing TRX/blame diagnostics remain enabled.

Go macOS/inprocess reuses this watchdog around the unchanged test.sh, retaining full selection, race detection, and the per-package timeout. Native sampling prioritizes the E2E process. An opt-in TestMain writes startup/completion state and all goroutine stacks one minute before the watchdog deadline, independently of buffered go test output.

The command deadline is the earlier of 15 minutes from launch or 16 minutes after checkout, leaving time within the unchanged 20-minute job limit for diagnostics, owned-process cleanup, and artifact upload. Timeout remains failure; an earlier nonzero result is preserved. Ordinary local Go invocations are unchanged. Artifacts exclude arbitrary console output, RPC payloads, environment dumps, heap/process dumps, and locals.

Limits: dotnet-stack does not capture .NET Framework/native CLI stacks; those are explicitly identified with thread metadata and existing blame/TRX evidence. Thread stacks do not reconstruct every suspended async state machine. Diagnostic validation and a subsequent non-reproduction do not prove a primary hang is repaired. See dotnet/ci/README.md and go/README.md.

Validation and preserved failure history

• Latest Python fixture fix: both shared-scenario controls failed against the original code and passed after the change. The real Windows stdio E2E passed in 14.08 seconds against pinned CLI 1.0.84-5. Ruff checks/formatting passed; no snapshot edits or full local suite run.
• Run 34785968413 at c9aee7d9: 49/50 SDK test jobs passed, including all Go/.NET/Node gates. Python Alpine reported 933 passed, nine existing skips, and the single 300-second detach timeout above. Diagnostic artifacts confirm Go macOS natural exit 0 in 127.533 seconds; Windows .NET passed 555 net8.0 plus 423 net472 tests with six existing skips and natural exit 0 in 473.532 seconds. No watchdog deadline intervention was needed.
• Diagnostic controls: 21 Node watchdog controls passed locally, with four existing POSIX-only controls inapplicable on Windows. These cover real managed waiting stacks, orphaned pipe holders, unrelated-process survival, supervisor termination, original failures, safe Go progress filtering, deadlines, and native-target prioritization. Windows helper build had zero warnings/errors; formatting and workflow checks passed. Go's seven focused test functions and go vet passed; an empty-selection E2E run verified TestMain integration, not full E2E behavior. Local cgo/race tooling was unavailable; real macOS CI subsequently passed the Go diagnostic controls with race detection enabled.
• Prior macOS .NET 103753104713 passed 555 tests with two existing skips and unchanged 452 method identities/case multiplicities. Natural exit 0 in 207.214 seconds; all 11 then-existing watchdog controls passed on macOS.
• Python timeout cleanup: 19 focused tests, Ruff, and SDK typechecking passed; all three controls passed real POSIX SIGALRM on macOS and Linux. Earlier diagnostic work passed 25 focused tests.
• Targeted reproduction 34765982698: all ten session-config invocations completed. Pre-cleanup stress 34764284304: four full suites passed 925 tests each; the original five-invocation configuration exhausted the job budget at the fifth start. The bounded scopes above correct that diagnostic-budget defect. These are non-reproduction evidence, not proof of a primary Python timeout repair.
• Run 34763599690 passed completely. Run 34765985408 passed 49/50 test jobs; macOS .NET shard 1 hit its 20-minute limit without retained evidence, motivating the original watchdog.
• Run 34768280584 ended with 48/50 test jobs passing. Two Node jobs initially failed before tests on a pinned CLI asset HTTP 500. After the identical URL recovered (HEAD 200, range GET 206), each received exactly one targeted retry: build/validation passed in attempt 2, Ubuntu/default in attempt 3, and the Node gate passed. Original Go/.NET hang results were carried forward, not rerun.
• That run's Go macOS/inprocess 103753104455 hit the annotated 20-minute job limit without a test/goroutine trace; the unchanged per-package 20-minute timer lost to the enclosing job deadline. Identical inputs/CLI/Go/runner passed 103746971240 in 158.794 seconds, and seven other Go configurations passed. The primary cause remains unattributed.
• Windows .NET shard 1 103753104714 also hit an annotated 20-minute limit. Metadata spans 25m01s, with the extra time unexplained. Logs return BlobNotFound, the archive lacks that shard, and no Windows artifact existed. Healthy 103746971275 passed 978 tests/six existing skips with unchanged inputs. Its primary cause likewise remains unattributed. Required aggregate failures correctly reflected these cancelled dependencies, not separate defects.
• Run 34761514877 passed 49/50 test jobs. Python macOS/inprocess reported 913 passed, one failed, nine existing skips, and one teardown error: system-message resume timed out at 300 seconds; module teardown timed out after the available-tools body passed. The original log lacked async/native diagnostics; identical Python code passed previously.
• First expanded run 34760106004 exposed five new Node fake-RPC fixture deadlocks. df3457bd added the existing sendStarted fence before completing the fake send; all Node configurations then passed. An earlier Go Windows/default timeout later passed unchanged without establishing attribution.

No test-failure reruns-to-green, manual CI cancellations, newly skipped existing SDK tests, or weakened checks/assertions were used. Only the two confirmed external download failures received targeted retries. Full-suite validation uses CI as requested; genuine pushes use existing PR concurrency behavior.

Origin

This follows github/copilot-agent-runtime#20254 and its original optional CAPI C# SDK failure, using #2635's abort-recovery fix as prior art. The original CI incident cannot conclusively be attributed to the reproduced subscription race because runtime logs were not retained.

Historical validation at fa8fa0d4: both new ordering cases failed with old code, then all 22 focused cases and all 16 permission E2Es passed; net472 build, formatting, and the .NET CI matrix passed. Local E2E used an isolated verified copy of the supplied runtime build, not the exact failing CI binary.

Generated by Copilot