feat(coderd): bill completed local tool batches in agent runtime - #28360
Conversation
832483d to
f3537d2
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Documentation CheckThis PR changes what counts toward billable Agent Time ( Updates Needed
Note This is part of stack #28359 → #28360 → #28361 → #28362. The billable-runtime behavior change lands in this PR, so the doc update belongs with it. If the docs are intentionally handled in a later PR of the stack, disregard. Automated review via Coder Agents |
f3537d2 to
82583bd
Compare
82583bd to
97fced6
Compare
97fced6 to
852caee
Compare
852caee to
1b99d6a
Compare
a4a03a1 to
defea90
Compare
6ecbee9 to
bb48de3
Compare
bb48de3 to
4b9c3ca
Compare
4b9c3ca to
e7d1bfb
Compare
e7d1bfb to
581fce2
Compare
…ocalTools (#28359) **Stack**: **#28359 (this PR)** → #28360 → #28361 → #28362 `chatloop.ToolExecutionOutcome` only wrapped a `PersistedStep`, so callers had to reach through `outcome.Step` for every field. `ExecuteLocalTools` now returns the `PersistedStep` directly and the wrapper type is deleted. No behavior change. First of a four-PR stack that makes local tool execution count toward Coder Agent runtime (splitting #28211). This refactor keeps the rename noise out of the feature PRs. Refs CODAGT-928 (https://linear.app/codercom/issue/CODAGT-928/track-local-tool-execution-for-agent-runtime).
Local tool execution between model steps previously billed nothing toward chat_messages.runtime_ms, the source of hb_agent_runtime_v1. Each completed local tool batch now bills one window: the union of its billed execution intervals, so parallel calls count once and serial calls count only from their own start. Sub-agent orchestration tools never extend the window because child chats bill their own runtime. The window persists as runtime_ms on the batch's first tool-result row, keeping the role-agnostic runtime sum query correct with at most one billed row per batch. Zero windows stay NULL. Interrupted batches still bill only the model window; a follow-up bills the partial tool window on cancellation rows.
…cord Usage sums runtime_ms across rows, so the batch window previously landed on the first tool-result row of the batch. Replace that arbitrary member stamping with one model-invisible tool-role row per billed batch carrying the runtime and an audit payload (billed window and interval count). Real tool results no longer carry batch-level runtime.
581fce2 to
c93e039
Compare
Stack: #28359 → #28360 (this PR) → #28361 → #28362
Summary
Completed local tool batches now count toward Coder Agent runtime (
chat_messages.runtime_ms, the source of truth forhb_agent_runtime_v1), excluding sub-agent orchestration tools to avoid double counting, and without multiplying runtime for parallel tool calls.Second of a four-PR stack splitting #28211. Stacked on #28359. Refs CODAGT-928 (https://linear.app/codercom/issue/CODAGT-928/track-local-tool-execution-for-agent-runtime).
Problem
Agent runtime previously measured only model invocation wall clock (stream open to fully consumed). Time spent executing local tools between steps, including file operations, terminal commands, workspace provisioning, and MCP tools, was deliberately excluded, which undercounts the product definition of "actively processing a task".
Naive inclusion has two hazards: a batch of parallel tool calls would bill N windows for one wall-clock wait, and
wait_agentwould re-bill child agents that already bill their own model and tool time.Fix
Each completed local tool batch bills one window: the union of the billed tools' execution intervals, persisted as
runtime_mson a dedicated usage record appended after the batch's tool-result rows. The record is a tool-role message withvisibility='model', so it never reaches the API, SSE, or clients, and prompt replay drops it because its single internaltool-batch-usagepart converts to no provider content. Its content carries an audit payload (billed_ms,billed_calls) so the billed window is inspectable after the fact; real tool-result rows never carry batch-level runtime. Concurrent calls all start at batch start, so 5 parallel 10s reads bill 10s, not 50s; serial calls (SerialToolCalls) count only from their own launch, so unbilled waits before them do not count.Sub-agent orchestration tools (
spawn_agent,wait_agent,message_agent,interrupt_agent,list_agents,list_subagent_models, plus the deprecatedclose_agentalias) never extend the window: every chat, including children, applies the same rules to its own runtime, so a parent'swait_agentwindow would double count. A lonewait_agentbills 0 and appends no usage record;execute10s in parallel withwait_agent60s bills 10s. A test pins the unbilled set to the registered sub-agent tool catalog so they cannot drift.GetTotalChatMessageRuntimeMsInRangealready sumsruntime_msrole-agnostically, so the usage record is picked up with no schema, query, or cron changes.Client-executed dynamic tools, external agents, parked/idle time, and retry backoff remain unbilled. Interrupted batches still bill only the model window; the rest of the stack adds partial-window billing via the same usage record.
ExecuteLocalToolsOptionsalso gains an optionalToolBillingRecorder(RecordStart/RecordComplete) observing per-occurrence dispatch-order execution stamps. This PR wires no recorder; the next PR in the stack connects it to the message part buffer for interrupt billing.NOTE: Reported agent runtime (
hb_agent_runtime_v1) increases from deploy forward. There is no backfill and no feature flag.