🤖 fix: roll back a turn when shutdown lands in its pre-acceptance awaits - #4154
Conversation
sendMessage checked the shutdown latch once before awaiting snapshot materialization and the turn's history appends. A shutdown that began inside those awaits left the appended rows durable while the PREPARING gate refused the stream, so the next startup read the answer-less row as an interrupted turn and retried it. Re-check the latch at the rollback horizon, where the rows are still rollback-eligible, and refuse through the existing rollback path. Fixes #4073
This comment has been minimized.
This comment has been minimized.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c976c1cde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
…versible Shutdown refusals after truncateAfterMessage would otherwise leave neither the original tail nor the replacement. Gate every pre-persist shutdown refusal on the truncation not having happened; the PREPARING gate still refuses the turn with the row retained so startup recovery resumes the edit.
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
AgentSession.sendMessagechecked the shutdown latch once, before awaiting snapshot materialization and the turn's history appends. A shutdown that began inside those awaits left the appended rows durable while the turn was still refused, so the next startup read the answer-less user row as an interrupted turn and auto-retried it. The latch is now re-checked at the rollback horizon, where the rows are still rollback-eligible, and the refusal goes through the existing rollback path. Edits are the one exception: once their truncation has happened, shutdown lets the replacement row land instead of losing both versions of the input.Fixes #4073
Background
Codex flagged this on #4058 (round 23, P2). The pre-persist gate added there refuses shutdown before any row lands, but everything after it (
materializeAgentSkillSnapshots,materializeMcpPromptSnapshots, the snapshot/user-row appends) is awaited with no further latch check.isCurrentTurnstays true while the coordinator is merely closing, so the existing rollback-horizon checkpoint did not observe the shutdown either;coordinator.prepare()later rejected the turn withclosing, returningErrto the caller with the row already durable.Implementation
markRowsDurable()) that refuses throughrefuseBeforeAcceptance, the same helper the pre-persist gate uses: it rolls back this attempt's rows and marks them durable only if the rollback verifiably failed. Covering the horizon rather than only the two materialize awaits closes the append-I/O windows too, and every path (ordinary, pre-turn batch, on-send compaction row, token-budget batch) funnels through it.truncateAfterMessageruns before any of these checks and is irreversible. Every pre-persist shutdown refusal an edit can reach afterwards (the pre-persist gate, the two token-budget checks before the batch append, and the new horizon check) now goes throughshutdownRefusesBeforePersist(), which stays false once the truncation succeeded. The replacement row lands, the PREPARING gate refuses the turn with the row retained, and startup recovery resumes the edit. Before truncation, and for the missing-target no-op truncation, shutdown still refuses as before.scopedLifetimesjoin test previously pinned the retained row after a shutdown landing inside an ordinary send's append; its join assertions are unchanged and it now asserts the rolled-back outcome.Validation
startupAutoRetrycases run shutdown inside the stubbedmaterializeAgentSkillSnapshotsawait and inside the user row's own append (row durable, then removed). Both were red onorigin/mainat the history assertion (refusal returned, one extra durable row) and are green with the fix; seeded earlier rows survive the rollback.editMessageIdkeep[u0, a0, replacement]and start no provider; both fail when the truncation flag is not set.agentSession*.test.tssuites andmake static-checkpass locally.Risks
Low and confined to shutdown ordering. The checks only fire when
beginShutdown()has already run, and the turn was already going to be refused at the PREPARING gate; the change is that ordinary sends roll their rows back instead of retaining them, while post-truncation edits proceed to persist their replacement instead of refusing (previously the pre-persist gate could refuse a truncated edit and leave neither version). The emergency rollover path (rolloverAfterBudgetFailure) is separate and untouched.Generated with
xum• Model:anthropic:claude-fable-5-1• Thinking:xhigh• Cost:$0.00