feat: add chatd lifecycle stage tracer - #28969
Draft
jscottmiller wants to merge 1 commit into
Draft
Conversation
Contributor
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
This was referenced Sep 4, 2026
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
from
September 9, 2026 23:19
95c1126 to
c2a8273
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
from
September 11, 2026 02:25
c2a8273 to
f5bfc09
Compare
Add StageTracer, which starts one OpenTelemetry span and records one coderd_chatd_stage_duration_seconds observation per chat lifecycle stage from the same call, so span and histogram durations cannot diverge. Stages carry a scope, chat kind, and model identity read from the context or set on the span, and stages reconstructed from persisted timestamps are emitted through Record and RecordAs. Windows with inverted or negative durations are dropped and counted on coderd_chatd_stage_anomalies_total by reason. The histogram is labelled by stage, scope, chat kind, and model. Reasoning effort is a span attribute only, since it multiplies series per model without changing how the wait and connect stages behave. Buckets are 16 round boundaries from 50ms to 1h so alert thresholds land on bucket edges. Add --chat-stage-metrics (CODER_CHAT_STAGE_METRICS), off|basic|full, default basic. Basic observes the wait, connect, and model-call stages only; full observes every stage; off registers none of the stage families. Families disabled by the level are still constructed against no registerer so every recorder is callable at every level. A coderd_chatd_stage_metrics_level gauge reports the configured level.
jscottmiller
force-pushed
the
scott/chatd-stages/2-stage-tracer
branch
from
September 11, 2026 19:17
f5bfc09 to
0643175
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
chatloop.StageTracer, which emits each chat lifecycle stage as both an OpenTelemetry span and an observation on a newcoderd_chatd_stage_duration_seconds{stage, scope, chat_kind, model}histogram from a single call, so trace and metric durations cannot disagree. Nothing calls it yet; the following PRs instrument the chat loop, the provider transport, and the runner.StageModel, and context carriers for scope (turn/background) and chat kind (root/subagent). Scope and kind are plain context values so they survive a no-op tracer provider.Start,StartRoot,StartRootAtfor live spans;Record/RecordAsfor stages reconstructed from timestamps.Endis idempotent;EndWithoutObservationcloses the span without a histogram sample.*StageTracer, nil provider, or nil metrics is usable and discards.coderd_chatd_stage_anomalies_total{reason}counts stage observations dropped for a negative duration or an inverted window instead of dropping them silently.0.05 0.1 0.25 0.5 1 2.5 5 10 20 30 60 120 300 600 1800 3600) so alert thresholds land on bucket edges. Reasoning effort is a span attribute (reasoning_effort) but not a metric label: it multiplies series per model while changing only the model-call stages.--chat-stage-metrics/CODER_CHAT_STAGE_METRICS(off,basic,full; defaultoff) selects how much is exposed as metrics.offregisters none of the stage families;basicobserves the wait, connect, and model-call stages (chat_turn,queue_wait,capacity_wait,acquisition,mcp_connect,stream,time_to_first_token,provider_attempt,tool_call,commit,retry_backoff);fullobserves every stage. Empty or unrecognized values resolve tooff. Spans are emitted at every level. Families disabled by the level are still constructed against no registerer so every recorder is callable, andcoderd_chatd_stage_metrics_level{level}reports the configured level.Part of the chatd lifecycle observability stack: 1 extract step · 2 stage tracer · 3 stream stages · 4 provider attempts · 5 turn span · 6 capacity wait · 7 turn accounting · 8 dashboard. Supersedes the experimental branch in #28741.
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.