Instruments chatd with OpenTelemetry spans and Prometheus histograms covering the full lifecycle of an Agents chat turn, adds per-turn accounting that partitions each turn's wall time into disjoint categories, and adds a Grafana dashboard that renders the stage profile as a flamegraph and by stage tree level.

This is an experimental branch. It will be split into a stack of smaller PRs and closed; the description here is kept current so the branch can be reviewed as a whole in the meantime.

What's included

• chatloop.StageTracer: emits each lifecycle stage as both an OTel span and an observation on coderd_chatd_stage_duration_seconds{stage, scope, chat_kind, model, effort} from a single code path. Stages: chat_turn, queue_wait, capacity_wait, acquisition, generation_step, prepare, mcp_connect, provider_attempt, time_to_first_token, stream, thinking, tool_call, commit, compaction, retry_backoff.
• Per-turn accounting (chatloop.TurnAccumulator): when a turn finishes normally, it emits turn_stage_seconds, turn_stage_count, and stage_share_of_turn per stage, and turn_time_seconds and turn_time_share per category. Categories (scheduling, time_to_first_token, streaming, provider_error, retry_backoff, tool_execution, compaction, chatd_overhead, unattributed) are disjoint and sum to the turn. Nested stages report their time to their parent so the parent's category receives only its own time.
• One chat_turn span per prompt: the runner opens a turn span on the first generation task and replaces it when a finish transition promotes a queued message (anchored at the moment that message was queued) or when a new prompt starts a task after the previous turn finished. Turns close in two steps: the finishing transition marks the turn complete from inside the generation step, and the span settles once that step's own stage has ended. Each task carries a turn token so a task that outlives its turn cannot complete or invalidate the one that replaced it.
• Scope and chat kind: detached background work (title, summary, status label) is scope="background", starts its own trace roots, and never reports to the turn accumulator. chat_kind (root/subagent) is on every span and metric.
• Model and effort: the resolved model and effective reasoning effort are span attributes and histogram labels, threaded to provider_attempt via the transport.
• coderd_chatd_stage_anomalies_total{reason}: counts stage observations dropped for inverted clocks, finished turns dropped for a non-positive duration, and turns whose categories summed to more than their duration.
• Grafana dashboard at examples/monitoring/dashboards/grafana/chatd-lifecycle/: stage flamegraph and hierarchy table with a $stat selector, the turn time partition, and one row per stage tree level (duration per occurrence, seconds per turn, occurrences per turn, share of turn), filterable by $chat_kind/$model/$effort. The README documents every panel.
• capacity_wait is a per-occurrence stage only: it is measured by the acquisition loop before the turn exists and its window lies inside acquisition, so it is absent from the category map and the per-turn panels.

Verification

Verified end to end against a live dev deployment with Tempo + Prometheus + Grafana on the initial instrumentation: 100+ real chat turns across two models, exact count agreement between spans and histogram observations for every stage, clean span nesting, HTTP >= 400 provider attempts marked as span errors, dashboard rendering in Grafana 11.4.

The turn accounting, rotation, and the review fixes below are covered by unit tests in coderd/x/chatd and coderd/x/chatd/chatloop (race detector clean) and have not yet been re-verified against a live deployment.

Not sampled in the test environment: capacity_wait (premium license disables the concurrent-agent cap), mcp_connect (no MCP servers configured), compaction (context never neared the threshold).

Review fixes

A four-persona panel review (concurrency, lifecycle, observability, hidden behavior change) found no request-path regression but several defects in the numbers. Fixed in the last five commits:

• The turn's accounting was emitted from inside the final generation step, before that step's stage ended, so the finishing step was lost on every turn that promoted a queued message; the span also closed at runner teardown instead of turn end, inflating every share's denominator.
• Invalidate after a promotion hit the newly opened turn rather than the failed one, and a retried task opened a second chat_turn root with a duplicate acquisition.
• Background work derived its context from the turn's and inherited the accumulator, so title/summary generation was counted as the user's turn.
• capacity_wait could never reach the accumulator, so three per-turn panels were permanently empty; its acquisition-loop bookkeeping also reset under load and kept stale entries across skips.
• Over-attributed turns were emitted silently; duration buckets started at 10ms while several stages are sub-10ms; share buckets stopped at 1 while stages can exceed the turn; stage count buckets stopped at 128 against a 1200 step limit.

Notes and known limitations

• chat_turn spans are standalone trace roots: the turn executes asynchronously on a worker (possibly another replica) and no trace context is persisted, so linking to the originating HTTP request span is not possible.
• The turn's start comes from the trigger message's database timestamp while its end comes from the worker's clock, so clock skew between Postgres and a replica lands in scheduling or drops the sample (counted in stage_anomalies_total).
• A requires_action wait stays inside the turn span with no category, so human-in-the-loop turns report large unattributed time.
• No error dimension on the new metrics; turns that fail are excluded from the per-turn rollup, so turn percentiles describe successful turns only.
• Roughly 20k to 100k new series per replica depending on distinct model/effort pairs. Operator sizing note still to be written.
• Follow-ups not addressed here: OTel semconv names for the HTTP span attributes, reasoning_effort vs effort label naming, the flamegraph mixing per-occurrence and per-turn units, the provider transport wrapper being installed unconditionally, and the Anthropic quickgen 400 (temperature with thinking) that is now visible as scope="background" error spans.

chat_turn
├── queue_wait          chat_queued_messages insert -> promotion
├── capacity_wait       capacity.go limiter acquire (per-occurrence profile only)
├── acquisition         trigger message insert -> Acquire() applied
└── generation_step (per step, repeats)
    ├── prepare         prompt build, model resolution, context hydration
    │   └── mcp_connect mcpclient connect
    ├── provider_attempt  one HTTP round trip (per attempt)
    ├── stream          stream open -> stream close
    │   └── time_to_first_token   stream open -> first part
    ├── retry_backoff   wait between provider attempts
    ├── thinking        reasoning part created_at -> completed_at
    ├── tool_call       per local tool call start -> completion
    ├── commit          CommitStep DB txn
    └── compaction      auxiliary compaction LLM call

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.