feat: hard time limit cutoff + client disconnect cancellation#280
Merged
Conversation
Two-phase time limit in takeLimits: soft deadline (1s before) for
graceful return, hard deadline (1s after) via Promise.race to force-cut
blocked operations. AbortSignal threaded from HTTP handler through
engine, source connectors, and all the way to individual fetch() calls.
- EofPayload: add 'aborted' reason, cutoff ('soft'|'hard'), elapsed_ms
- takeLimits: manual iterator + Promise.race for hard deadline + signal
- ndjsonResponse: cancel() hook for Bun.serve() disconnect detection
- HTTP handlers: wireDisconnect() for Node outgoing.close + onCancel
- Stripe client: AbortSignal.any() combines pipeline + per-request timeout
- Subprocess source: child.kill() on signal abort
- Distinct log lines: SYNC_TIME_LIMIT_SOFT/HARD, SYNC_CLIENT_DISCONNECT, SYNC_ABORTED
- Unit tests for soft/hard cutoff, abort signal, elapsed_ms
- Black-box e2e test parameterized over Node/Bun/Docker runtimes
- CI: disconnect test steps for Node + Bun
Made-with: Cursor
Committed-By-Agent: cursor
Made-with: Cursor
Committed-By-Agent: cursor
Made-with: Cursor Committed-By-Agent: cursor
- Use syntactically valid postgres URL in pipeline header - Add response body logging on non-200 for CI debugging - Add process exit detection in waitForServer - Increase poll interval for server health checks Made-with: Cursor Committed-By-Agent: cursor
Made-with: Cursor Committed-By-Agent: cursor
Pino logger writes to stdout by default, not stderr. The test was only capturing stderr which was empty, causing all log assertions to fail. Made-with: Cursor Committed-By-Agent: cursor
The Stripe connector launches concurrent backfill requests, so the mock request count can be high even with a short time limit. Assert that no new requests arrive after the hard deadline fires instead. Made-with: Cursor Committed-By-Agent: cursor
Docker test builds an image from scratch (~2min+) which exceeds the beforeAll timeout in the main CI job. Only run when explicitly opted in. Made-with: Cursor Committed-By-Agent: cursor
…al, docker target 1. AbortError is no longer retryable in withHttpRetry — only TimeoutError is. Previously a pipeline abort would trigger up to 5 retries with exponential backoff, defeating the hard cutoff. 2. takeLimits abort listener is created once outside the loop instead of per-iteration, preventing O(messages) listener accumulation on the AbortSignal. 3. createRemoteEngine now forwards opts.signal to the underlying HTTP fetch so remote-engine callers get real cancellation. 4. Docker test helper uses --target engine (not the default service image) and accepts ENGINE_IMAGE env var for pre-built images. Made-with: Cursor Committed-By-Agent: cursor
Run the disconnect suite in the intended CI jobs only, wait for log lines instead of checking the buffer synchronously, and normalize the Docker mock URL for container reachability. The Docker job now runs the disconnect suite against the prebuilt engine image. Made-with: Cursor Committed-By-Agent: cursor
b4bf356 to
b317686
Compare
The Docker disconnect suite now uses host networking in CI so the engine container can reach the host mock server via localhost. Non-CI Docker runs still use bridge mode with host.docker.internal. Made-with: Cursor Committed-By-Agent: cursor
Rebased v2 added e2e/esbuild-binary-path.test.sh. Hook it back into the main CI test job so the workflow sanity check passes and the shell test actually runs in fresh-install CI. Made-with: Cursor Committed-By-Agent: cursor
AbortSignal is not serializable and doesn't belong in a Zod schema that could be used for JSON schema generation or wire validation. Keep it as a TypeScript-only type intersection on SourceReadOptions. Made-with: Cursor Committed-By-Agent: cursor
signal is a runtime-only cancellation concern, not a serializable option. It now lives as a separate positional argument on: - Engine.pipeline_read(pipeline, opts?, input?, signal?) - Engine.pipeline_sync(pipeline, opts?, input?, signal?) - Source.read(params, $stdin?, signal?) wireDisconnect renamed to createConnectionAbort — returns the controller instead of accepting startedAt and logging internally. The HTTP handler owns the logging via its own onDisconnect callback. Made-with: Cursor Committed-By-Agent: cursor
pipeline_write now respects client disconnect the same way pipeline_read and pipeline_sync do. Signal is threaded as a separate positional arg through Engine.pipeline_write, Destination.write, and the HTTP handler. Made-with: Cursor Committed-By-Agent: cursor
- channel: add return() and onReturn hook - split: propagate branch return() back to the source iterator - merge: call return() on child iterators when consumer stops - add direct teardown tests for channel, split, and merge Made-with: Cursor Committed-By-Agent: cursor
…test - createConnectionAbort now invokes a pure logging callback before aborting so Node close-driven disconnects emit SYNC_CLIENT_DISCONNECT again - update exec arity test for createSourceFromExec.read(params, stdin?, signal?) Made-with: Cursor Committed-By-Agent: cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
takeLimits: soft deadline (1s before) for graceful return, hard deadline (1s after) viaPromise.raceto force-cut blocked operationsAbortSignalthreaded all the way from HTTP handler → engine → source connector →fetch()callsoutgoing.close) and Bun (ReadableStream.cancel)SYNC_TIME_LIMIT_SOFT,SYNC_TIME_LIMIT_HARD,SYNC_CLIENT_DISCONNECT,SYNC_ABORTEDEofPayloadextended withcutoff('soft'|'hard'),elapsed_ms, and'aborted'reasonTest plan
Made with Cursor