feat: bill local tool execution time in chat agent runtime - #28211
feat: bill local tool execution time in chat agent runtime#28211jaaydenh wants to merge 22 commits into
Conversation
Local tool batches now persist their billable wall-clock window to chat_messages.runtime_ms, the source of truth for hb_agent_runtime_v1. Each batch bills one window, from the batch start to the last billed tool's completion, stored on that tool's message row. Tools in a batch run in parallel, so the window equals the union of billed execution intervals rather than a per-tool sum. Sub-agent orchestration tools (spawn_agent, wait_agent, message_agent, interrupt_agent, list_agents, list_subagent_models, and the deprecated close_agent alias) never extend the window: every chat, including children, bills its own runtime, so a parent's wait_agent would double count. Client-executed dynamic tools, external agents, parked time, and retry backoff remain unbilled. Interrupted batches bill the partial window: the buffer episode stamps the batch start, and the interrupt task places the window on the window-defining synthesized cancellation row. Batches without a live attempt (crash recovery, state promotion) bill nothing. The usage query already sums runtime_ms role-agnostically, so no schema, query, or cron changes are needed. Reported agent runtime increases from deploy forward.
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb26857eb3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…on times Address Codex review feedback: - Record each local tool call's completion instant on the message part buffer as the tool finishes (chatloop OnToolComplete), because tool results are published only after the whole batch completes. The interrupt task now reads these live stamps instead of inferring completions from buffered results, which never exist while a sibling tool is still running, so an interrupted batch no longer bills a finished tool through to the interrupt instant. - Assign the batch runtime to only the first tool row matching the window-defining tool call ID, so duplicate tool call IDs cannot multiply the billed sum. - Replace the agent-authored ARCHITECTURE.md buffer API entries with a TODO item for the PR author, per the chatd documentation rule.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a88b8a4d9a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…tched call Address Codex review feedback: - Compute the batch's billable window from completion instants aligned with the tool calls by occurrence instead of the ID-keyed timestamp map. Duplicate tool call IDs, which reach execution when lifecycle hooks are disabled, previously overwrote each other there, letting a short duplicate shrink the window a longer one should have defined. - Seed the buffer episode's tool batch with the dispatched call IDs so an interrupt can tell a call that was still running (seeded, no completion) from one rejected before execution (absent). Rejected calls previously looked like still-running work and billed the whole window up to the interrupt even when only unbilled tools actually ran.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 589bfce2a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ccurrence Address Codex review feedback: - Store the tool batch's dispatched calls on the buffer episode as an occurrence list instead of an ID-keyed map, stamping completions onto the first still-running occurrence of the reported ID. Duplicate tool call IDs, which reach execution when lifecycle hooks are disabled, previously collapsed into one shared state, so one duplicate finishing made its still-running twin look finished and ended the interrupted batch's window early. The interrupt path now consumes per-ID occurrence queues in the same dispatch order the unresolved history rows walk. - Close the buffer episode and snapshot its billing stamps in one critical section (CloseEpisodeForBilling) instead of reading ModelInvokedAt, ToolBatchStartedAt, and the completions before a separate CloseEpisode. Stamps recorded in those gaps went missing from the snapshot, billing a finished tool through to the interrupt or dropping a just-started batch's window entirely.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9ff5f50314
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…he interrupt instant Address Codex review feedback: - Thread the call's occurrence index through OnToolComplete and RecordToolCompletion so a completion stamps its own occurrence instead of the first still-running occurrence sharing its tool call ID. Same-ID calls with different billing classifications, such as an unbilled wait_agent finishing before a billed execute, previously ended the billed occurrence's window early. The interrupt path also consumes an occurrence for every dispatched call, including unbilled ones, to keep same-ID queues aligned with the history walk. - Pass the interrupt instant captured at episode close into the cancellation billing instead of taking a fresh clock reading inside machine.Update, so database contention or transaction retries no longer inflate a still-running call's billed window past the actual interrupt.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 68dee09773
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…osition Address Codex review feedback: - Seed each dispatched occurrence with its position in the step's unresolved tool-call order, the order interrupt reconstruction walks, and match occurrences to cancellation rows positionally instead of through per-ID queues. A call rejected before execution that shared an ID with a dispatched call previously consumed the dispatched occurrence and billed the whole window on a row that never ran. partitionAmbiguousToolCalls now reports each allowed call's input position so the seed does not re-derive it from IDs. - Skip starting the billing batch for exclusive-policy violations. chatloop synthesizes error results for the whole batch without dispatching any tool, so seeding those calls let an interrupt racing the synthetic results bill never-run work from the batch stamp to the interrupt.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f6e743565
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…er line Address Codex review feedback: the docs prose style guide requires each sentence on its own source line and the full text of a bullet item on a single line, reformatting the entire paragraph when any line in it is edited. The section kept fixed-column wrapping.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 45f962c4a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Address Codex review feedback: - Return the episode's first-close instant in the billing snapshot and use it as the interrupt instant. Each retried interrupt attempt previously took a fresh clock reading, billing still-running tools (and the model window) through every retry of a transient database failure; every billing input now comes from the episode's stable first-close state, so retries recompute identical windows. - Push the episode's eviction deadline out on every re-close, so a retry loop outlasting the buffer's retention window keeps its billing snapshot and buffered parts through a database outage instead of losing them to the cleanup loop; retry backoff (max 5s) stays well under the retention window (15s). The superseded heap item is skipped by the cleanup loop's existing identity check.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 167d9f8922
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… retries Address Codex review feedback: one interrupt attempt can outlive the buffer's closed-episode retention (a machine.Update stalled on a database outage runs up to the task timeout), and the eviction refresh only happens when an attempt begins. A retry after such a stall found a blank recreated episode and lost the interrupt instant, partial parts, and tool completions, underbilling the interrupted work and dropping the partial messages. The runner now attaches a snapshot holder to the interrupt task's input, which every retry attempt of the task instance shares. The first attempt to read the episode stores the billing state and buffered parts there, and later attempts reuse the carried snapshot instead of re-reading the evictable buffer.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fd2445eb5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d4760a5d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…e chat read Address Codex review feedback: the interrupt task read the chat row before snapshotting the buffer episode, so an initial read stalled past the buffer's retention let the cleanup loop evict the episode first, and the eventual snapshot found a blank recreated episode with no billing stamps or partial parts. StartInterrupt now captures the snapshot before its first database read, keyed by the task input's attempt number. Generation attempts only advance while the chat is running, so the pre-read key matches the row while the chat stays interrupting; if the post-read attempt number ever disagrees, the snapshot is retaken with the row's key. The close-and-read sequence moved into a closeInterruptEpisode helper shared by both paths.
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
# Conflicts: # coderd/x/chatd/chatloop/chatloop.go
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 458498e7ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Serial tool calls (SerialToolCalls) launch only after every concurrent sibling settles, so measuring them from the shared batch start charged the whole concurrent phase, including waits on unbilled sub-agent orchestration tools, and an interrupt treated a dispatched serial call that never launched as running since batch start. The batch window is now the union of the billed calls' execution intervals: chatloop stamps each call's start (concurrent calls at batch start, serial calls at their launch) and reports it through the new OnToolStart callback, the buffer records the marks per occurrence via RecordToolStart, and both the committed path and the interrupt path bill with the shared BilledIntervalsDuration union, so a span where only unbilled tools were running never bills and a dispatched call without a start mark bills nothing.
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Restore docs/ai-coder/usage-data-reporting.md to its state on main, removing this branch's edits to the agent runtime measurement section.
…n-for-agent-runtime
| // StartToolBatch stamps dispatch and seeds one entry per occurrence. A zero | ||
| // start is queued; a started entry with zero completion is running. Absent | ||
| // calls were not dispatched, and duplicate IDs remain distinct. | ||
| func (b *Buffer) StartToolBatch(key Key, calls []DispatchedToolCall) error { |
There was a problem hiding this comment.
why do we need this method? can't we call RecordToolStart multiple times?
| // start with the batch; serial calls may start later. If dispatchIndex does | ||
| // not match, the first unstarted same-ID occurrence is used. Unknown starts | ||
| // are dropped because they cannot correlate to unresolved calls. | ||
| func (b *Buffer) RecordToolStart(key Key, dispatchIndex int, toolCallID string, startedAt time.Time) error { |
There was a problem hiding this comment.
As far as I can tell dispatchIndex uniquely identifies a tool call. toolCallID is unnecessary. Please remove it here and in other methods.
| // the same critical section, preventing a read-close race. Unknown episodes | ||
| // close blank. Re-closing returns the original snapshot and refreshes | ||
| // retention so retries keep the same billing state and buffered parts. | ||
| func (b *Buffer) CloseEpisodeForBilling(key Key) (EpisodeBilling, error) { |
There was a problem hiding this comment.
Please remove this method. We already have a method for closing an episode, and introducing another one specifically for billing is confusing. As far as I can tell, the "read-close race" referenced in the comment has no practical impact.
| StopNudges *stopNudgeTracker | ||
| // InterruptSnapshot carries one interrupt task's first episode snapshot | ||
| // across retries. Nil re-reads the buffer. | ||
| InterruptSnapshot *interruptEpisodeSnapshot |
There was a problem hiding this comment.
I'm not convinced that the complexity introduced by adding InterruptSnapshot is justified by the problem that it solves (I guess making billing more accurate if interrupting a chat fails due to e.g. a database connection error?). Do we actually need it?
There was a problem hiding this comment.
Digging more into this, inflation of the billing numbers is very small and probably not worth the complexity. It seems that this may help only in the rare cases the DB has issues form greater than 15 seconds. I think its worth removing for now.
Documentation CheckUpdates Needed
Note The Automated review via Coder Agents |
Summary
Local tool execution time now counts toward Coder Agent runtime (
chat_messages.runtime_ms, the source of truth forhb_agent_runtime_v1), excluding sub-agent orchestration tools to avoid double counting, and without multiplying runtime for parallel tool calls.Closes CODAGT-928 (https://linear.app/codercom/issue/CODAGT-928/track-local-tool-execution-for-agent-runtime).
Problem
Agent runtime previously measured only model invocation wall clock (stream open to fully consumed). Time spent executing local tools between steps, including file operations, terminal commands, workspace provisioning, and MCP tools, was deliberately excluded, which undercounts the product definition of "actively processing a task".
Naive inclusion has two hazards: a batch of parallel tool calls would bill N windows for one wall-clock wait, and
wait_agentwould re-bill child agents that already bill their own model and tool time.Fix
Each local tool batch bills one window: from the batch start to the last billed tool's completion, persisted as
runtime_mson that tool's message row (ties broken by call order, all other rows stay NULL). Since all calls in a batch start together, this equals the union of the billed tools' execution intervals, so 5 parallel 10s reads bill 10s, not 50s.Sub-agent orchestration tools (
spawn_agent,wait_agent,message_agent,interrupt_agent,list_agents,list_subagent_models, plus the deprecatedclose_agentalias) never extend the window: every chat, including children, applies the same rules to its own runtime, so a parent'swait_agentwindow would double count. A lonewait_agentbills 0;execute10s in parallel withwait_agent60s bills 10s. A test pins the unbilled set to the registered sub-agent tool catalog so they cannot drift.Interrupted batches bill the partial window: the message part buffer stamps the batch start (mirroring
StartModelInvocation), and the interrupt task places the window on the window-defining synthesized cancellation row, using buffered result timestamps for tools that finished before the interrupt. Batches with no live attempt (crash recovery, state promotion) bill nothing, consistent with generation losing in-flight runtime on a crash.Client-executed dynamic tools, external agents, parked/idle time, and retry backoff remain unbilled.
GetTotalChatMessageRuntimeMsInRangealready sumsruntime_msrole-agnostically, so tool-role rows are picked up with no schema, query, or cron changes.NOTE: Reported agent runtime (
hb_agent_runtime_v1) increases from deploy forward. There is no backfill and no feature flag.Known gaps left out of scope: retry backoff and errored-attempt time stay unbilled (no persistence home for error turns), and the advisor tool's nested model call now bills as tool time by design.