Thanks to visit codestin.com
Credit goes to github.com

Skip to content

🤖 fix: roll back a turn when shutdown lands in its pre-acceptance awaits - #4154

Merged
ibetitsmike merged 2 commits into
mainfrom
mike/issue-4073-shutdown-latch
Sep 8, 2026
Merged

🤖 fix: roll back a turn when shutdown lands in its pre-acceptance awaits#4154
ibetitsmike merged 2 commits into
mainfrom
mike/issue-4073-shutdown-latch

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Summary

AgentSession.sendMessage checked the shutdown latch once, before awaiting snapshot materialization and the turn's history appends. A shutdown that began inside those awaits left the appended rows durable while the turn was still refused, so the next startup read the answer-less user row as an interrupted turn and auto-retried it. The latch is now re-checked at the rollback horizon, where the rows are still rollback-eligible, and the refusal goes through the existing rollback path. Edits are the one exception: once their truncation has happened, shutdown lets the replacement row land instead of losing both versions of the input.

Fixes #4073

Background

Codex flagged this on #4058 (round 23, P2). The pre-persist gate added there refuses shutdown before any row lands, but everything after it (materializeAgentSkillSnapshots, materializeMcpPromptSnapshots, the snapshot/user-row appends) is awaited with no further latch check. isCurrentTurn stays true while the coordinator is merely closing, so the existing rollback-horizon checkpoint did not observe the shutdown either; coordinator.prepare() later rejected the turn with closing, returning Err to the caller with the row already durable.

Implementation

  • One new check at the rollback horizon (after the caller-probe staleness checkpoint, before markRowsDurable()) that refuses through refuseBeforeAcceptance, the same helper the pre-persist gate uses: it rolls back this attempt's rows and marks them durable only if the rollback verifiably failed. Covering the horizon rather than only the two materialize awaits closes the append-I/O windows too, and every path (ordinary, pre-turn batch, on-send compaction row, token-budget batch) funnels through it.
  • Edits: truncateAfterMessage runs before any of these checks and is irreversible. Every pre-persist shutdown refusal an edit can reach afterwards (the pre-persist gate, the two token-budget checks before the batch append, and the new horizon check) now goes through shutdownRefusesBeforePersist(), which stays false once the truncation succeeded. The replacement row lands, the PREPARING gate refuses the turn with the row retained, and startup recovery resumes the edit. Before truncation, and for the missing-target no-op truncation, shutdown still refuses as before.
  • The scopedLifetimes join test previously pinned the retained row after a shutdown landing inside an ordinary send's append; its join assertions are unchanged and it now asserts the rolled-back outcome.

Validation

  • New startupAutoRetry cases run shutdown inside the stubbed materializeAgentSkillSnapshots await and inside the user row's own append (row durable, then removed). Both were red on origin/main at the history assertion (refusal returned, one extra durable row) and are green with the fix; seeded earlier rows survive the rollback.
  • Edit cases: the same two windows with editMessageId keep [u0, a0, replacement] and start no provider; both fail when the truncation flag is not set.
  • All 34 agentSession*.test.ts suites and make static-check pass locally.

Risks

Low and confined to shutdown ordering. The checks only fire when beginShutdown() has already run, and the turn was already going to be refused at the PREPARING gate; the change is that ordinary sends roll their rows back instead of retaining them, while post-truncation edits proceed to persist their replacement instead of refusing (previously the pre-persist gate could refuse a truncated edit and leave neither version). The emergency rollover path (rolloverAfterBudgetFailure) is separate and untouched.


Generated with xum • Model: anthropic:claude-fable-5-1 • Thinking: xhigh • Cost: $0.00

sendMessage checked the shutdown latch once before awaiting snapshot
materialization and the turn's history appends. A shutdown that began inside
those awaits left the appended rows durable while the PREPARING gate refused
the stream, so the next startup read the answer-less row as an interrupted turn
and retried it. Re-check the latch at the rollback horizon, where the rows are
still rollback-eligible, and refuse through the existing rollback path.

Fixes #4073
@chatgpt-codex-connector

This comment has been minimized.

@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8c976c1cde

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/agentSession.ts
@chatgpt-codex-connector

This comment has been minimized.

…versible

Shutdown refusals after truncateAfterMessage would otherwise leave neither the
original tail nor the replacement. Gate every pre-persist shutdown refusal on
the truncation not having happened; the PREPARING gate still refuses the turn
with the row retained so startup recovery resumes the edit.
@ibetitsmike

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: dcf172154f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

This comment has been minimized.

@ibetitsmike
ibetitsmike added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit 48dcc1b Sep 8, 2026
19 of 20 checks passed
@ibetitsmike
ibetitsmike deleted the mike/issue-4073-shutdown-latch branch September 8, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🤖 fix: sendMessage re-checks the shutdown latch only before its pre-persist awaits

1 participant