🤖 fix: acquire the event lock before the task-tree lock in task_send_message - #4161
Conversation
Resolve the lock inversion between reported-task cleanup (event then task-tree) and descendant sends (formerly task-tree then event). Keep the joint critical section intact and add a gated cleanup/send deadlock regression test. Fixes #4072
This comment has been minimized.
This comment has been minimized.
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: 938b1dc5c6
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
## Summary Version bump for the v0.28.5 patch release. Headline changes since v0.28.4: remote server connections in the desktop app (coder#4101), self-updating `xum server` under a restart supervisor (coder#4083, coder#4127), first-class GPT-6 Astra and Astra Pro support including Codex OAuth routing (coder#4064, coder#4094, coder#4106, coder#4124), token-budget context window rollovers (coder#4097), the workspace remembering model and mode on send (coder#3968), in-place plugin updates (coder#4164), the optional flat sidebar chat list (coder#3994), and copying selected chat text as Markdown (coder#4170). It also carries a long run of streaming, compaction, and task-lifecycle fixes (reconnect streaming coder#4123, message edits during active streams coder#4153, Codex OAuth prompt-cache routing coder#4159, compaction/history fencing coder#4133 through coder#4148, task lock ordering coder#4161) plus the Effect Wave 4 runtime refactors and deslop passes 1 through 3. ## Implementation Bumped with `node ./scripts/set-package-version.js 0.28.5` so the root `package.json` and the legacy `packages/mux-compat` forwarding package stay version-locked. `src/common/compat/productIdentity.test.ts` passes locally (8/8). After this PR merges, the `v0.28.5` tag will be applied to the squash commit and the GitHub Release published to trigger the desktop/npm/docker pipelines. --- _Generated with `xum` • Model: `anthropic:claude-fable-5-1` • Thinking: `xhigh` • Cost: `$1.64`_ <!-- mux-attribution: model=anthropic:claude-fable-5-1 thinking=xhigh costs=1.64 -->
Summary
task_send_messageto a descendant task nested the task-tree lifecycle lock outside the workspace event lock, while reported-task cleanup nests them the other way (event lock held, thenWorkspaceService.remove()takes the tree lock). When a send to a task interleaved with that task's cleanup, each side waited on the other's lock forever. The send path now acquires the event lock first, so every path that holds both locks uses one order, and a deterministic regression test pins the interleaving.Fixes #4072
Background
Surfaced by Codex on #4058, which added the live recheck inside
remove()and documented the inversion without changing it; the ordering predates that PR.Implementation
dispatchTrustedDescendantMessagenow nestsworkspaceEventLocks.withLock(taskId, () => withTaskTreeLifecycleLock(taskId, ...)). The joint critical section is unchanged (both locks are still held for the whole body); only the acquisition order flips.remove()out of the event-locked section because the event lock is the outermost lock everywhere else it appears: the stream-end, stream-abort and error listeners take it first, and cleanup rechecks, hard-timeout termination and workspace-turn finalization all nest inside it. Moving cleanup would have meant moving every tree-lock acquisition out from underhandleStreamEnd.workspaceEventLocksdeclaration now documents the order (event lock before task-tree lock) and the cleanup comment no longer describes an inversion.Validation
reported-task cleanup and task_send_message never deadlock on the event and task-tree locks: drives the realrequestReportedTaskCleanupRecheckon a reported workflow-owned child, parks asendMessageToDescendantAgentTaskon its lock acquisition inside the window before the (lock-mirroring)remove()takes the tree lock, then lets removal proceed. Red on the previous nesting (Expected "settled", Received "deadlocked"), green after the swap; wholetaskService.test.ts490/490.task_send_messageto a reported child reactivates it, to a live child queues, and stop-then-send reactivates, with no hang. The exact cleanup/send overlap window is covered by the deterministic test only; it is not observable by hand.Risks
Low. The send path holds the same two locks over the same body; only which one it waits for first changes. Other tree-lock holders (create, stop, retitle, remove, archive, unarchive) never take the event lock, so the new order adds no wait edge.
Generated with
xum• Model:anthropic:claude-fable-5-1• Thinking:xhigh• Cost:$36.01