feat: account chatd turn time by stage and category - #28974

When a turn finishes normally, its stages are rolled up and emitted once as five histograms labelled {stage|category, chat_kind, model} (reasoning effort is a span attribute, not a metric label):

• coderd_chatd_turn_stage_seconds, turn_stage_count, stage_share_of_turn: per stage, the total seconds, occurrences, and fraction of the turn. Stages overlap, so shares can exceed 1; the share buckets extend to 10 and the count buckets past the 1200 step limit.
• coderd_chatd_turn_time_seconds, turn_time_share: an exclusive partition of the turn's wall time into scheduling, time_to_first_token, streaming, provider_error, retry_backoff, tool_execution, compaction, preparation, persistence, chatd_overhead, and unattributed. All eleven are emitted every turn.

turn_time_seconds is registered at the basic and full stage metrics levels; the other four families are registered only at --chat-stage-metrics=full. The accumulator runs at every level, so the partition is available at basic even for stages that the level leaves span-only. The per-turn histograms use an 11-bucket ladder from 1s to 1h, since sub-second resolution carries nothing for a value summed over a turn.

How the partition is built

Stages nest, and a nested stage's time is also its parent's time, so summing stage durations over-counts. That is why the per-stage totals above are a profile rather than a partition. The partition instead assigns each stage's own time (its duration minus its attributing children) to one category. A TurnAccumulator rides on the turn context; each attributing stage reports its full duration to its parent when it ends, and the parent's category receives only what is left.

chat_turn                                        own time -> category
├── acquisition | queue_wait                     full     -> scheduling
└── generation_step (repeats per step)           own      -> chatd_overhead, or
    │                                                        tool_execution when the
    │                                                        step ran local tools
    ├── prepare                                  own      -> preparation
    │   └── mcp_connect                          own      -> preparation
    ├── retry_backoff                            own      -> retry_backoff
    ├── stream                                   own      -> streaming, or
    │   │                                                    provider_error on failure
    │   ├── time_to_first_token                  own      -> time_to_first_token, or
    │   │                                                    provider_error on failure
    │   └── provider_attempt  (overlaps TTFT)    not categorized
    ├── thinking              (inside stream)    not categorized
    ├── tool_call             (inside step)      not categorized
    ├── commit                                   own      -> persistence
    └── compaction                               own      -> compaction
                                                 remainder -> unattributed

provider_attempt, thinking, and tool_call are recorded in the per-stage totals but never claim time from their parent or land in a category: each lies entirely inside a stage that is already categorized, so counting them would double-count. capacity_wait is excluded for the same reason (its window lies inside acquisition). Only turn-scoped stages started by a live tracer report to the accumulator, so background work never lands in a turn. Errored or interrupted turns are invalidated and emit nothing.

A single 10s step, drawn to scale:

t (s)  0    1    2    3    4    5    6    7    8    9    10
generation_step  |=================================================|
prepare          |========|
stream                    |==================================|
  time_to_first_token     |========|
  provider_attempt        |=====|
commit                                                       |===|

durations   step 10   prepare 2   stream 7   ttft 2   attempt 1.5   commit 1
own time    prepare 2, ttft 2, stream 7-2 = 5, commit 1, step 10-(2+7+1) = 0
categories  preparation 2, persistence 1, chatd_overhead 0, time_to_first_token 2, streaming 5
            total 10 = step duration; the attempt's 1.5s appears in the per-stage totals, never in a category

In practice the step's own time is not zero: the gaps between phases (decision logic, buffer bookkeeping, tool dispatch) are what chatd_overhead from the step itself captures.

A turn whose categories sum to more than its duration is emitted as measured and counted in coderd_chatd_stage_anomalies_total{reason="overattributed"}; a finished turn with a non-positive duration is counted as nonpositive_turn. capacity_wait is not categorized because its window lies inside acquisition.

Part of the chatd lifecycle observability stack: 1 extract step · 2 stage tracer · 3 stream stages · 4 provider attempts · 5 turn span · 6 capacity wait · 7 turn accounting · 8 dashboard. Supersedes the experimental branch in #28741.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.