docs(examples/monitoring): add chatd lifecycle Grafana dashboard - #28975
Draft
jscottmiller wants to merge 1 commit into
Draft
docs(examples/monitoring): add chatd lifecycle Grafana dashboard#28975jscottmiller wants to merge 1 commit into
jscottmiller wants to merge 1 commit into
Conversation
This was referenced Sep 4, 2026
jscottmiller
force-pushed
the
scott/chatd-stages/7-turn-accounting
branch
from
September 9, 2026 23:19
422889e to
fbf8b65
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
3 times, most recently
from
September 9, 2026 23:52
7d306ba to
b4c4da0
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/7-turn-accounting
branch
from
September 11, 2026 02:25
fbf8b65 to
2eec66e
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 11, 2026 02:25
b4c4da0 to
098e14d
Compare
Add a Grafana dashboard that plots the chatd stage histogram and the turn-end rollups one tree level at a time with a selectable summary statistic, plus summary panels for the chat pipeline and the per-turn partition of turn wall time. The README documents the stage hierarchy by tree level, the dashboard dimensions, each panel group, and setup. Rows that need only the basic stage metrics level are expanded by default; the per-turn distribution panels and the stages observed only at the full level sit in collapsed rows named for the level. A stat panel reports the level from coderd_chatd_stage_metrics_level. The README adds a metric levels section with series cost, example alert rules on the round bucket edges, and a scrape-time relabel snippet for alert-only deployments.
jscottmiller
force-pushed
the
scott/chatd-stages/7-turn-accounting
branch
from
September 11, 2026 19:17
2eec66e to
948b3e7
Compare
jscottmiller
force-pushed
the
scott/chatd-stages/8-dashboard
branch
from
September 11, 2026 19:17
098e14d to
d52393b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grafana dashboard for the chatd lifecycle metrics at
examples/monitoring/dashboards/grafana/chatd-lifecycle/, with a README documenting every panel.Rows, each with a
$statselector (mean, p50, p90, p95, p99): a stat panel showing the--chat-stage-metricslevel fromcoderd_chatd_stage_metrics_level; the turn time partition (per-model mix, seconds per turn by category, unattributed time); one row per stage tree level with duration per occurrence for the stages observed atbasic(the server default isoff, so the dashboard needs--chat-stage-metrics=basicorfullon the coderd replicas); throughput, TTFT, and background provider calls. Each level also has a collapsed row titled(full level only: --chat-stage-metrics=full)holding the per-turn panels (seconds per turn, occurrences per turn, share of turn), the per-turn category share, and the stages observed only atfull(generation_step,prepare,thinking,compaction). The label schema is the same at every level, so those panels read "No data" rather than erroring atbasic. Variables:$datasource,$stat,$chat_kind,$model. Every division guards its denominator and everyhistogram_quantilekeepsle.The README documents the levels with an annotated stage tree, the series cost per level, example alert rules on the round bucket edges, and a scrape-time relabel snippet for alert-only deployments.
An earlier revision had an aggregate flamegraph and hierarchy table row built from a 15-stage
label_replaceexpression over[$__range]. It was removed: the query loaded every bucket series for the whole dashboard range on each refresh, and because Prometheus cannot compute self time for overlapping stages the flamegraph shape was misleading. The per-level rows carry the same per-stage data with bounded lookback.Verified against a live dev deployment (Grafana 12.1, Prometheus, Tempo) at both
basicandfull: every expanded panel populates atbasicwith 326 series after three turns on one model, and the collapsed rows populate atfull(1,132 series after six turns).Part of the chatd lifecycle observability stack: 1 extract step · 2 stage tracer · 3 stream stages · 4 provider attempts · 5 turn span · 6 capacity wait · 7 turn accounting · 8 dashboard. Supersedes the experimental branch in #28741.
🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.