Thanks to visit codestin.com
Credit goes to github.com

Skip to content

docs(examples/monitoring): add chatd lifecycle Grafana dashboard - #28975

Draft
jscottmiller wants to merge 1 commit into
scott/chatd-stages/7-turn-accountingfrom
scott/chatd-stages/8-dashboard
Draft

jscottmiller wants to merge 1 commit into
scott/chatd-stages/7-turn-accountingfrom
scott/chatd-stages/8-dashboard

Conversation

@jscottmiller

@jscottmiller jscottmiller commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Grafana dashboard for the chatd lifecycle metrics at examples/monitoring/dashboards/grafana/chatd-lifecycle/, with a README documenting every panel.

Rows, sharing one dashboard-level $stat selector (mean, p50, p90, p95, p99): a stat panel showing the --chat-stage-metrics level from coderd_chatd_stage_metrics_level; the turn time partition (per-model mix, seconds per turn by category, unattributed time), read from the turn_time_seconds_total and turns_total counters so it works at basic; one row per stage tree level with duration per occurrence for the stages observed at the default basic level; throughput, TTFT, and background provider calls. Each level also has a collapsed row titled (full level only: --chat-stage-metrics=full) holding the per-turn panels (seconds per turn, occurrences per turn, share of turn), the per-turn category share, and the stages observed only at full (generation_step, prepare, thinking, compaction). The label schema is the same at every level, so those panels read "No data" rather than erroring at basic. Variables: $datasource, $stat, $chat_kind, $organization, $model; $organization is applied wherever $chat_kind is. Every division guards its denominator and every histogram_quantile keeps le.

The README documents the levels with an annotated stage tree, the series cost per level, example alert rules on the round bucket edges, and a scrape-time relabel snippet for alert-only deployments.

An earlier revision had an aggregate flamegraph and hierarchy table row built from a 15-stage label_replace expression over [$__range]. It was removed: the query loaded every bucket series for the whole dashboard range on each refresh, and because Prometheus cannot compute self time for overlapping stages the flamegraph shape was misleading. The per-level rows carry the same per-stage data with bounded lookback.

Verified against a live dev deployment (Grafana 12.1, Prometheus, Tempo) at both basic and full: every expanded panel populates at basic with 148 series after three root turns on one model (326 before the 12-bucket ladder and the counter-based partition), and the collapsed rows populate at full (1,132 series after six turns).


Part of the chatd lifecycle observability stack: 1 extract step · 2 stage tracer · 3 stream stages · 4 provider attempts · 5 turn span · 6 capacity wait · 7 turn accounting · 8 dashboard. Supersedes the experimental branch in #28741.

🤖 This PR was generated by Coder Agents on behalf of @jscottmiller.

@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch 3 times, most recently from 7d306ba to b4c4da0 Compare September 9, 2026 23:52
@jscottmiller jscottmiller added the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Sep 11, 2026
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from b4c4da0 to 098e14d Compare September 11, 2026 02:25
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 098e14d to d52393b Compare September 11, 2026 19:17
@jscottmiller jscottmiller removed the experimental Changes that might not necessarily be merged, until its approved to proceed with. label Sep 14, 2026
@jscottmiller
jscottmiller added this pull request to stack #29305 September 14, 2026 20:01
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from d52393b to 33ab94d Compare September 14, 2026 20:16
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 33ab94d to 099519e Compare September 14, 2026 20:34
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from 099519e to fe4d8df Compare September 14, 2026 22:06
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from fe4d8df to aa86059 Compare September 15, 2026 21:39
Add a Grafana dashboard that plots the chatd stage histogram and the
turn-end rollups one tree level at a time with a selectable summary
statistic, plus summary panels for the chat pipeline and the per-turn
partition of turn wall time. The README documents the stage hierarchy
by tree level, the dashboard dimensions, each panel group, and setup.

Rows that need only the basic stage metrics level are expanded by
default; the per-turn distribution panels and the stages observed only
at the full level sit in collapsed rows named for the level. A stat
panel reports the level from coderd_chatd_stage_metrics_level. The
README adds a metric levels section with series cost, example alert
rules on the round bucket edges, and a scrape-time relabel snippet for
alert-only deployments.
@jscottmiller
jscottmiller force-pushed the scott/chatd-stages/8-dashboard branch from aa86059 to 0b20a69 Compare September 16, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant