coder · jscottmiller · Sep 4, 2026
diff --git a/coderd/x/chatd/ARCHITECTURE.md b/coderd/x/chatd/ARCHITECTURE.md
@@ -778,6 +778,52 @@ State updates processed by the loop come from:
 
 The runner is responsible for subscribing to the `chat:update:{chat_id}` pubsub channel. During bootstrap, it must first subscribe to the channel and then fetch the initial state of the chat from the database to avoid missing any updates.
 
+### Lifecycle tracing
+
+The runner owns a `chat_turn` trace span for the turn it is running, implemented by `runnerTurnSpan` in `turn_trace.go`. The span and the stages inside it are emitted through `chatloop.StageTracer`, which produces an OpenTelemetry span and an observation on the `coderd_chatd_stage_duration_seconds{stage, scope, chat_kind, model}` histogram from a single `End` call, so wherever both exist the trace and metric durations cannot disagree. Tracing is enabled by the `TracerProvider` server option. A nil provider disables spans without disabling the histogram. The `--chat-stage-metrics` option (`off`, `basic`, `full`; default `off`) filters the histogram without affecting spans: `off` registers none of the stage families, `basic` observes only the wait, connect, and model-call stages, and `full` observes every stage. `coderd_chatd_stage_metrics_level` reports the configured level.
+
+`chat_turn` is a standalone trace root. The HTTP request that triggered the turn ran on a different goroutine, and often a different replica, from the worker that runs it, and no trace context is persisted with the message, so there is nothing to parent the span to.
+
+#### Turn span lifecycle
+
+One `chat_turn` span covers one prompt, not one runner. A runner keeps ownership of a chat across queued-message promotions, and runners are also spawned for abandon, interrupt, and timeout tasks that run no turn, so the span is started lazily by the first generation task and replaced when the turn finishes.
+
+- Start: every iteration of the generation loop calls `Ensure`, which returns a context parented to the open turn and a `turnToken` identifying it. When no turn is open, `Ensure` starts one with its start timestamp backdated to the trigger message's `created_at` and records an `acquisition` stage from that instant to now, covering the time between the message landing in history and a worker picking the chat up.
+- Complete: after the `FinishTurn` transition commits, the generation step calls `Complete`. This marks the turn finished but leaves the span open. If the transition promoted a queued message, `FinishTurnResult.PromotedQueuedAt` carries that message's `created_at` to `Complete`.
+- Settle: when the step returns to the generation loop, after the step's own `generation_step` stage has ended, the loop calls `Settle`, which closes the span. Closing in two steps ensures the finishing step is counted inside the turn. If `Complete` recorded a promotion, `Settle` immediately opens the next turn anchored at the moment the promoted message was queued and records a `queue_wait` stage from that instant to the promotion. A turn opened this way records no `acquisition` stage, since the two windows would overlap.
+- Next prompt: if a new prompt starts a generation task while a finished turn is still open, `Ensure` settles the old turn first and then opens a new one.
+- Runner exit: the runner ends whatever span is still open when it shuts down.
+
+The runner cancels the active task and spawns its replacement without waiting for the old goroutine to exit (see [Event processing](#event-processing)), so an old task can still be unwinding while the new one calls `Ensure` and rotates the turn. Each task carries the `turnToken` returned by its own `Ensure` call, and `Complete` and `Settle` do nothing when the token does not identify the open turn. A stale task therefore cannot close the turn that replaced its own.
+
+Queued messages can also be promoted outside a generation step, through `PromoteQueued`. That path records `queue_wait` as a standalone stage, since no turn exists yet to attach it to.
+
+Work detached from the turn, such as title, summary, and status label generation, runs on a context with the span context stripped and `scope=background`, so its stages start their own trace roots and are separable from turn-scoped stages in the histogram.
+
+#### Stages
+
+Every stage carries `scope` (`turn` or `background`) and `chat_kind` (`root` or `subagent`), and once the model is resolved, `model`. Spans additionally carry `reasoning_effort`; it is not a metric label because it multiplies series per model. Stages that are not tied to a model call (`acquisition`, `queue_wait`, `mcp_connect`, `retry_backoff`, `commit`) carry an empty `model` label. `prepare` is stamped with the model once preparation resolves it.
+
+At `--chat-stage-metrics=basic` the histogram observes `chat_turn`, `queue_wait`, `acquisition`, `mcp_connect`, `stream`, `time_to_first_token`, `provider_attempt`, `tool_call`, `commit`, and `retry_backoff`. `generation_step`, `prepare`, `thinking`, and `compaction` are span-only at that level.
+
+Live stages wrap a section of code and end when it returns:
+
+- `generation_step`: one iteration of the generation loop, from loading state to applying a transition. It carries `generation_attempt` and `generation_action`.
+- `prepare`: generation preparation, including model resolution and tool assembly.
+- `mcp_connect`: connecting to the configured MCP servers, inside `prepare`.
+- `provider_attempt`: one HTTP round trip to the model provider, emitted by the transport, so a retried request produces one stage per attempt. It ends when response headers arrive and is marked errored for HTTP status 400 and above.
+- `stream`: the provider stream, from opening the request to consuming the last part.
+- `time_to_first_token`: nested in `stream`, from opening the request to the first streamed part. A stream that ends before any part arrives closes this span with an error and records no histogram observation.
+- `retry_backoff`: the wait before retrying a failed LLM API call.
+- `commit`: the `CommitStep` transaction.
+- `compaction`: a compaction pass.
+
+Reconstructed stages are recorded after the fact from timestamps captured elsewhere:
+
+- `acquisition` and `queue_wait`: described above.
+- `thinking`: one per reasoning part, from the part's start to its completion timestamp in the persisted step.
+- `tool_call`: one per local tool call, from the tool billing recorder's start and completion stamps.
+
 ### Event shape
 
 Every event that the runner loop processes has the following shape:

diff --git a/coderd/x/chatd/chatd.go b/coderd/x/chatd/chatd.go
@@ -21,6 +21,7 @@ import (
 	"github.com/google/uuid"
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/sqlc-dev/pqtype"
+	"go.opentelemetry.io/otel/attribute"
 	"go.opentelemetry.io/otel/trace"
 	"golang.org/x/sync/errgroup"
 	"golang.org/x/xerrors"
@@ -2145,9 +2146,10 @@ func (p *Server) PromoteQueued(
 	}
 
 	var (
-		result      PromoteQueuedResult
-		refreshChat database.Chat
-		refreshedOK bool
+		result           PromoteQueuedResult
+		refreshChat      database.Chat
+		refreshedOK      bool
+		promotedQueuedAt time.Time
 	)
 	machine := p.newChatMachine(opts.ChatID)
 	updateErr := machine.Update(ctx, func(tx *chatstate.Tx, store database.Store) error {
@@ -2167,6 +2169,7 @@ func (p *Server) PromoteQueued(
 		}
 		if promoteResult.InsertedMessage != nil {
 			result.PromotedMessage = *promoteResult.InsertedMessage
+			promotedQueuedAt = promoteResult.QueuedMessage.CreatedAt
 		}
 		// Capture the chat inside the transaction so the watch event
 		// published below uses the snapshot bump and status change
@@ -2186,6 +2189,13 @@ func (p *Server) PromoteQueued(
 	if refreshedOK {
 		p.publishChatPubsubEvent(refreshChat, codersdk.ChatWatchEventKindStatusChange, nil)
 	}
+	if !promotedQueuedAt.IsZero() {
+		var chatKind string
+		if refreshedOK {
+			chatKind = chatKindAttr(refreshChat)
+		}
+		p.recordQueueWait(ctx, opts.ChatID, chatKind, promotedQueuedAt, p.stages.Now())
+	}
 	return result, nil
 }
 
@@ -4612,7 +4622,7 @@ func (p *Server) finalizeSuccessfulTurnStatusLabelWithAfterFunc(
 	logger slog.Logger,
 	afterFinalize func(context.Context, string),
 ) {
-	finalizeCtx, stopFinalizeCtx := p.inflightContext(ctx)
+	finalizeCtx, stopFinalizeCtx := p.inflightChatContext(ctx, chat)
 	if err := p.goInflight(func() {
 		defer stopFinalizeCtx()
 		statusLabel := p.generateFinalTurnStatusLabel(finalizeCtx, chat, status, runResult, logger)
@@ -4698,7 +4708,7 @@ func (p *Server) setLastTurnSummaryAsync(
 	if chat.LastTurnSummary.Valid && strings.TrimSpace(chat.LastTurnSummary.String) == summary {
 		return
 	}
-	updateCtx, stopUpdateCtx := p.inflightContext(ctx)
+	updateCtx, stopUpdateCtx := p.inflightChatContext(ctx, chat)
 	if err := p.goInflight(func() {
 		defer stopUpdateCtx()
 		p.updateLastTurnSummary(updateCtx, chat, chat.HistoryVersion, summary, logger)
@@ -4718,7 +4728,7 @@ func (p *Server) clearLastTurnSummaryAsync(
 	chat database.Chat,
 	logger slog.Logger,
 ) {
-	clearCtx, stopClearCtx := p.inflightContext(ctx)
+	clearCtx, stopClearCtx := p.inflightChatContext(ctx, chat)
 	if err := p.goInflight(func() {
 		defer stopClearCtx()
 		p.updateLastTurnSummary(clearCtx, chat, chat.HistoryVersion, "", logger)
@@ -4813,7 +4823,7 @@ func (p *Server) maybeGenerateChatSummaryAsync(
 	if chat.ParentChatID.Valid {
 		return
 	}
-	ctx, cancel := p.inflightContext(ctx)
+	ctx, cancel := p.inflightChatContext(ctx, chat)
 	if err := p.goInflight(func() {
 		defer cancel()
 		p.generateAndStoreChatSummary(ctx, logger, chat)
@@ -4994,7 +5004,7 @@ func (p *Server) storeSubagentReportSummaryAsync(
 	chat database.Chat,
 	logger slog.Logger,
 ) {
-	summaryCtx, stopSummaryCtx := p.inflightContext(ctx)
+	summaryCtx, stopSummaryCtx := p.inflightChatContext(ctx, chat)
 	if err := p.goInflight(func() {
 		defer stopSummaryCtx()
 		p.storeSubagentReportSummary(summaryCtx, chat, logger)
@@ -5090,14 +5100,43 @@ func (p *Server) Close() error {
 // must be called once the work completes to release the shutdown hook.
 // The caller is responsible for providing their own timeout.
 func (p *Server) inflightContext(reqCtx context.Context) (context.Context, func()) {
-	ctx, cancel := context.WithCancel(context.WithoutCancel(reqCtx))
+	// Inflight work outlives the caller, so the caller's span and stage
+	// scope are stripped from the context: spans started on this context
+	// become their own roots instead of children that end after their
+	// parent, and their stages are recorded as background work.
+	detached := trace.ContextWithSpanContext(context.WithoutCancel(reqCtx), trace.SpanContext{})
+	detached = chatloop.ContextWithScope(detached, chatloop.ScopeBackground)
+	ctx, cancel := context.WithCancel(detached)
 	stop := context.AfterFunc(p.ctx, cancel)
 	return ctx, func() {
 		stop()
 		cancel()
 	}
 }
 
+// recordQueueWait emits the queue_wait stage for a message that sat
+// queued from queuedAt until promotedAt. The span context is stripped
+// from ctx so the stage is a standalone span rather than a child of
+// the span in ctx, and the scope and chat kind are set explicitly
+// because ctx does not carry the turn's. An empty chatKind records the
+// stage without one.
+func (p *Server) recordQueueWait(ctx context.Context, chatID uuid.UUID, chatKind string, queuedAt, promotedAt time.Time) {
+	standalone := trace.ContextWithSpanContext(ctx, trace.SpanContext{})
+	standalone = chatloop.ContextWithChatKind(standalone, chatKind)
+	p.stages.RecordAs(standalone, chatloop.StageQueueWait, chatloop.ScopeTurn,
+		chatloop.StageModel{}, queuedAt, promotedAt, nil,
+		attribute.String(chatloop.AttrChatID, chatID.String()),
+	)
+}
+
+// inflightChatContext is inflightContext for work that belongs to a
+// known chat. The chat kind is set on the returned context so the
+// stages of the detached work carry it.
+func (p *Server) inflightChatContext(reqCtx context.Context, chat database.Chat) (context.Context, func()) {
+	ctx, stop := p.inflightContext(reqCtx)
+	return chatloop.ContextWithChatKind(ctx, chatKindAttr(chat)), stop
+}
+
 func (p *Server) goInflight(f func()) error {
 	if p.inflightClosed.Load() {
 		return errInflightClosed

diff --git a/coderd/x/chatd/chatstate/transitions.go b/coderd/x/chatd/chatstate/transitions.go
@@ -1448,6 +1448,9 @@ type FinishTurnInput struct{}
 type FinishTurnResult struct {
 	Chat            database.Chat
 	PromotedMessage *database.ChatMessage
+	// PromotedQueuedAt is the queued row's creation time when this
+	// transition promoted a queue head, and the zero time otherwise.
+	PromotedQueuedAt time.Time
 }
 
 // FinishTurn completes a running turn.
@@ -1509,8 +1512,9 @@ func (tx *Tx) FinishTurn(_ FinishTurnInput) (FinishTurnResult, error) {
 		promoted = &inserted[len(inserted)-1]
 	}
 	return FinishTurnResult{
-		Chat:            updated,
-		PromotedMessage: promoted,
+		Chat:             updated,
+		PromotedMessage:  promoted,
+		PromotedQueuedAt: head.CreatedAt,
 	}, nil
 }