Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(coderd): bill completed local tool batches in agent runtime - #28360

Merged
jaaydenh merged 3 commits into
mainfrom
jaayden/codagt-928-stack-2-bill-completed-tool-batches
Aug 25, 2026
Merged

feat(coderd): bill completed local tool batches in agent runtime#28360
jaaydenh merged 3 commits into
mainfrom
jaayden/codagt-928-stack-2-bill-completed-tool-batches

Conversation

@jaaydenh

@jaaydenh jaaydenh commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Stack: #28359#28360 (this PR)#28361#28362

Summary

Completed local tool batches now count toward Coder Agent runtime (chat_messages.runtime_ms, the source of truth for hb_agent_runtime_v1), excluding sub-agent orchestration tools to avoid double counting, and without multiplying runtime for parallel tool calls.

Second of a four-PR stack splitting #28211. Stacked on #28359. Refs CODAGT-928 (https://linear.app/codercom/issue/CODAGT-928/track-local-tool-execution-for-agent-runtime).

Problem

Agent runtime previously measured only model invocation wall clock (stream open to fully consumed). Time spent executing local tools between steps, including file operations, terminal commands, workspace provisioning, and MCP tools, was deliberately excluded, which undercounts the product definition of "actively processing a task".

Naive inclusion has two hazards: a batch of parallel tool calls would bill N windows for one wall-clock wait, and wait_agent would re-bill child agents that already bill their own model and tool time.

Fix

Each completed local tool batch bills one window: the union of the billed tools' execution intervals, persisted as runtime_ms on a dedicated usage record appended after the batch's tool-result rows. The record is a tool-role message with visibility='model', so it never reaches the API, SSE, or clients, and prompt replay drops it because its single internal tool-batch-usage part converts to no provider content. Its content carries an audit payload (billed_ms, billed_calls) so the billed window is inspectable after the fact; real tool-result rows never carry batch-level runtime. Concurrent calls all start at batch start, so 5 parallel 10s reads bill 10s, not 50s; serial calls (SerialToolCalls) count only from their own launch, so unbilled waits before them do not count.

Sub-agent orchestration tools (spawn_agent, wait_agent, message_agent, interrupt_agent, list_agents, list_subagent_models, plus the deprecated close_agent alias) never extend the window: every chat, including children, applies the same rules to its own runtime, so a parent's wait_agent window would double count. A lone wait_agent bills 0 and appends no usage record; execute 10s in parallel with wait_agent 60s bills 10s. A test pins the unbilled set to the registered sub-agent tool catalog so they cannot drift.

GetTotalChatMessageRuntimeMsInRange already sums runtime_ms role-agnostically, so the usage record is picked up with no schema, query, or cron changes.

Client-executed dynamic tools, external agents, parked/idle time, and retry backoff remain unbilled. Interrupted batches still bill only the model window; the rest of the stack adds partial-window billing via the same usage record.

ExecuteLocalToolsOptions also gains an optional ToolBillingRecorder (RecordStart/RecordComplete) observing per-occurrence dispatch-order execution stamps. This PR wires no recorder; the next PR in the stack connects it to the message part buffer for interrupt billing.

NOTE: Reported agent runtime (hb_agent_runtime_v1) increases from deploy forward. There is no backfill and no feature flag.

@linear-code

linear-code Bot commented Aug 20, 2026

Copy link
Copy Markdown

CODAGT-928

@jaaydenh

Copy link
Copy Markdown
Contributor Author

@codex review

@jaaydenh
jaaydenh marked this pull request as ready for review August 20, 2026 09:19
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: f3537d262f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jaaydenh
jaaydenh requested a review from hugodutka August 20, 2026 09:20
@coderagents

coderagents Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Documentation Check

This PR changes what counts toward billable Agent Time (hb_agent_runtime_v1 / runtime_ms): completed local tool batches now count toward Coder Agent runtime, whereas they were previously excluded. That behavior is documented for administrators, so the doc needs to match.

Updates Needed

  • docs/ai-coder/agents/licensing-usage.md (Agent Time measurement section) - The opening definition still says "Agent Time is the cumulative duration of model invocations that produce Coder Agents chat messages." After this PR, Agent Time also includes local tool batch execution, so this is now inaccurate.
  • docs/ai-coder/agents/licensing-usage.md (Agent Time includes list) - Add local tool batch execution. Reflect the new billing model: a batch bills the union of billed tools' execution intervals, so parallel tool calls count once (not N times) and serial calls count only from their own start. Note that server-executed tools count even when the work runs in a connected workspace.
  • docs/ai-coder/agents/licensing-usage.md (Agent Time excludes list) - Add sub-agent orchestration tools (spawn_agent, wait_agent, message_agent, interrupt_agent, list_agents, list_subagent_models, and the deprecated close_agent alias), which never extend the runtime window, plus retry backoff. This aligns with the updated HBAgentRuntime doc comment in coderd/usage/usagetypes/events.go.

Note

This is part of stack #28359#28360#28361#28362. The billable-runtime behavior change lands in this PR, so the doc update belongs with it. If the docs are intentionally handled in a later PR of the stack, disregard.


Automated review via Coder Agents

@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from f3537d2 to 82583bd Compare August 20, 2026 11:28
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from 82583bd to 97fced6 Compare August 20, 2026 13:05
Comment thread coderd/x/chatd/chatloop/chatloop.go Outdated
Comment thread coderd/x/chatd/message_conversion.go Outdated
Comment thread coderd/x/chatd/chatloop/chatloop.go Outdated
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from 97fced6 to 852caee Compare August 21, 2026 09:25
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from 852caee to 1b99d6a Compare August 22, 2026 07:06
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from a4a03a1 to defea90 Compare August 24, 2026 09:52
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from 6ecbee9 to bb48de3 Compare August 24, 2026 15:24
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from bb48de3 to 4b9c3ca Compare August 24, 2026 15:49
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from 4b9c3ca to e7d1bfb Compare August 24, 2026 18:03
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from e7d1bfb to 581fce2 Compare August 25, 2026 09:09
Base automatically changed from jaayden/codagt-928-stack-1-refactor-execute-local-tools to main August 25, 2026 09:31
jaaydenh added a commit that referenced this pull request Aug 25, 2026
…ocalTools (#28359)

**Stack**: **#28359 (this PR)** → #28360#28361#28362

`chatloop.ToolExecutionOutcome` only wrapped a `PersistedStep`, so
callers had to reach through `outcome.Step` for every field.
`ExecuteLocalTools` now returns the `PersistedStep` directly and the
wrapper type is deleted. No behavior change.

First of a four-PR stack that makes local tool execution count toward
Coder Agent runtime (splitting #28211). This refactor keeps the rename
noise out of the feature PRs.

Refs CODAGT-928
(https://linear.app/codercom/issue/CODAGT-928/track-local-tool-execution-for-agent-runtime).
Local tool execution between model steps previously billed nothing
toward chat_messages.runtime_ms, the source of hb_agent_runtime_v1.
Each completed local tool batch now bills one window: the union of its
billed execution intervals, so parallel calls count once and serial
calls count only from their own start. Sub-agent orchestration tools
never extend the window because child chats bill their own runtime.

The window persists as runtime_ms on the batch's first tool-result row,
keeping the role-agnostic runtime sum query correct with at most one
billed row per batch. Zero windows stay NULL.

Interrupted batches still bill only the model window; a follow-up bills
the partial tool window on cancellation rows.
…cord

Usage sums runtime_ms across rows, so the batch window previously
landed on the first tool-result row of the batch. Replace that
arbitrary member stamping with one model-invisible tool-role row per
billed batch carrying the runtime and an audit payload (billed window
and interval count). Real tool results no longer carry batch-level
runtime.
@jaaydenh
jaaydenh force-pushed the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch from 581fce2 to c93e039 Compare August 25, 2026 09:31
@jaaydenh
jaaydenh merged commit b428c8f into main Aug 25, 2026
24 checks passed
@jaaydenh
jaaydenh deleted the jaayden/codagt-928-stack-2-bill-completed-tool-batches branch August 25, 2026 09:40
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 25, 2026
@jaaydenh jaaydenh added cherry-pick/v2.37 Cherry-pick PR targeting release/2.37 cherry-pick and removed cherry-pick/v2.37 Cherry-pick PR targeting release/2.37 labels Aug 26, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants