feat: add hourly hb_agent_runtime_v1 usage events for Coder Agent runtime - #27312
Conversation
…main # Conflicts: # coderd/x/chatd/chattool/createworkspace.go
Replace the lexicographic template-ranking comparator in list_templates with a frecency score (frequency discounted by recency), per reviewer feedback. - Add GetTemplateRankingSignalsByOwnerID, returning the user's recent active and recently-deleted workspace counts, last usage, and the count of distinct active developers in the org. Recently-deleted workspaces now contribute (recovering history the deleted=false filter discarded), scoped to a lookback window, and the prebuilds system user is excluded from the org popularity count. Replaces GetWorkspaceUsageGroupedByTemplateIDByOwnerID. - Compute the affinity score in Go (Wp*(active + Wd*deleted)*0.5^(age/half_life) + Wo*ln(1+org_devs)) because sqlc cannot reliably compile the parameterized decay expression; the query returns the raw signals. Weights, half-life, and lookback are explicit constants. - Recommendation confidence is now a single score comparison: a decisive query match recommends on its own, otherwise the top score must clear a floor derived from the active-developer minimum and lead the runner-up by a derived margin. Stale-but-frequent usage no longer recommends. - Replace the AsSystemRestricted call for the cross-user org count with a narrow dbauthz wrapper checking workspace-owner read plus a template-metadata read. - Clarify list_templates/read_template guidance in the detached prompt.
Address the latest PR review feedback for frecency-based template ranking: - Authorize template ranking signals against the exact candidate template IDs using the same prepared-filter semantics as GetTemplatesWithFilter, so users with template ACL access keep ranking signals without broad org-wide template read. - Split the dbauthz mock coverage for org-scoped and any-organization calls; normalize duplicate subtest suffixes in the dbauthz method-test harness. - Surface deleted-only personal usage evidence in list_templates output with a recently-deleted count and last-used timestamp. - Assert the raw SQL query returns the maximum last_used_at value. - Clarify detached prompt guidance for user_selection_required and conditional read_template usage. - Document ListTemplatesOptions requirements and defaults.
Address CRF-26 by adding list_templates operation context to the user-facing asOwner authorization error response.
Keep TestChatSystemPrompt's detached workspace awareness expectation in sync with the updated list_templates guidance.
…gent runtime Adds a new heartbeat usage event type, hb_agent_runtime_v1, that measures the total agent-loop runtime (in milliseconds) of Coder Agents (chats) per UTC hour, summed from chat_messages.runtime_ms. Events are produced by a new lock-free hourly reconciler (enterprise/coderd/usage.Generator) that scans a trailing 7-day window for missing hourly buckets and fills them with deterministic IDs (hb_agent_runtime_v1:<bucket start>) and created_at set to the bucket start. Idle hours are zero-filled, hours missed during downtime are backfilled, and ON CONFLICT (id) DO NOTHING makes concurrent replicas safe without locking. Events flow to Tallyman through the existing publisher; the publish_usage_data license flag gates egress only. The aggregate_usage_event() trigger sums runtime_ms per day into usage_events_daily for the new type. Also fixes two latent bugs found along the way: - The ai-seats usage cron job never registered because the CronJob was missing its EventType and the Register error was discarded. - subjectUsagePublisher lacked usage_event create permission, so any heartbeat insert under AsUsagePublisher would fail authz once the cron actually ran. InsertHeartbeatUsageEvent now takes an explicit createdAt so generators can backfill historical buckets; the cron passes clock.Now() to preserve its existing behavior. Do not include this in a release until Tallyman accepts hb_agent_runtime_v1: permanent rejections are marked done-forever locally and those buckets would never be re-sent.
Rebuilds the branch content as origin/main plus the usage-events feature: the branch was originally cut from a feature branch rather than main, and main's migration numbering advanced, so the migration is renumbered from 000517 to 000546 and generated artifacts are refreshed against main.
created_at on usage_events is the event occurrence time rather than the row insertion time: it drives the daily rollup day and is sent to the usage collector as the event timestamp. Backfilled heartbeat events are the first producer where the two diverge, so make the semantics discoverable at the schema level like the table's other columns.
There was a problem hiding this comment.
Round 2, first panel pass on the feature (18 reviewers). The mandatory split from round 1 is resolved cleanly: slice 1 (the ai-seats cron fix + usage-publisher create permission) moved to base PR #27508, and this PR now carries only the feature plus its two per-query authz checks. Churn guard: PROCEED.
The design earns broad praise. The deterministic-ID + created_at = bucket start + ON CONFLICT (id) DO NOTHING trio makes concurrent replicas and re-runs idempotent with no locking, and because trigger_aggregate_usage_event is AFTER INSERT it does not fire on conflict-skipped rows, so the daily rollup cannot double-count (verified independently by Knuckle, Komugi, Knov, Hisoka, Meruem). Perf is a non-issue: bounded 168-bucket window, index-backed range scans (Killua). Authz scoping is sound and the gate is tighter than a chat-read (Kurapika). Enumerations are fully paired across every switch/constraint/trigger site (Melody). Tests are genuine and dense (Bisky: "Oh, these are lovely stones. Real ones.").
Severity count: 1 P2, 2 P3, 3 Nit, 4 Note.
The one finding that needs a human decision (CRF-2): the generator's completeness signal is "a row exists for the bucket," but a permanently-rejected event also leaves a row, so its hour is seen complete forever and its runtime is silently and unrecoverably dropped. The PR's WARNING correctly covers the pre-release window (do not ship before Tallyman accepts the type), but the same path fires for any permanent rejection after acceptance (a transient misclassified as permanent, schema drift, or an event aging past the publish cutoff), with no metric, no alert, and no in-band recovery. Neither the code nor a reviewer can accept that as permanent: this needs a human decision, either add a signal + recovery path, or file a ticket that explicitly accepts the gap. The current control is entirely procedural.
CRF-3 and CRF-4 are cheap robustness fixes: one bad bucket should not return and stall the rest of the window, and the concurrency test should fail on a swallowed Warn so it can actually catch an idempotency regression.
Process note (not an inline comment): the feature commit body (206c738) still claims it "fixes two latent bugs" (ai-seats cron, create permission) that now live in #27508. The PR description is correct ("Depends on #27508"), so a squash-merge discards the stale claim and nothing is hurt; a rebase-merge would ship a commit to main crediting a fix that landed elsewhere. Worth a reword if this is rebase-merged.
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 446d77cacd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Round 3: review blocked. The only change since round 2 is a rebase onto the updated base PR #27508 (it reverts the ai-seats cron block on this branch, now carried by the base, and rewords one dbauthz comment). No finding-location file changed, and none of the 10 findings from round 2 have a fix, an acknowledgement, a contest, or a linked ticket. Re-triggering the review is not a response to the findings.
Further panel review is paused until you either push fixes or reply on the threads. Re-running 18 reviewers against unchanged code would reproduce the same 10 findings.
Open and unaddressed (all still Unresolved):
P2
- CRF-2 (generator.go:151) needs a human decision, not code silence: a permanently-rejected event leaves a row, so the generator sees that hour as complete forever and its billable runtime is silently, unrecoverably dropped. The PR WARNING covers only the pre-release window; the same path fires for any permanent rejection after Tallyman accepts the type. Either add a metric/alert plus a recovery runbook, or file a ticket that explicitly accepts the post-acceptance silent-loss gap. A rebase does not resolve this.
P3
- CRF-3 (generator.go:168): one bad bucket aborts the whole ascending pass and stalls later buckets for up to 7 days; fix is
continuenotreturn. - CRF-4 (generator_test.go:332): the concurrency test cannot fail on a lost-idempotency regression because insert errors are swallowed to
Warnand slogtest ignoresWarn.
Nit
- CRF-1 (generator.go:97): startup jitter hardcodes
4*time.Minuteinstead ofagentRuntimeJitter. - CRF-5 (server.go:165): the generator goroutine's logger name does not match its pprof label.
- CRF-6 (inserter_test.go:84): test comment implies a
clock.Now()risk the blank-receiver function cannot have.
Note
- CRF-7 (chats.sql:2422): redundant
AND cm.runtime_ms IS NOT NULL; its test case cannot fail. - CRF-8 (dbauthz.go:4956): cross-chat aggregate gated on
usage_event:create; add a negative role test so the invariant is mechanical. - CRF-9 (000552 up.sql:5): the "Backfilled"
created_atcomment misdescribes this type; every event trails insertion by ~65m. - CRF-10 (generator.go:31): the 5-minute eligibility lag rests on an unenforced commit-latency assumption; document it.
Separately, the @coder-tasks doc check on this PR flags that docs/ai-coder/usage-data-reporting.md enumerates what leaves the deployment for Tallyman and does not yet list hb_agent_runtime_v1; that transparency page matters for air-gapped and legally-restricted deployments since the generator writes these events to the local ledger unconditionally in enterprise builds. That is its finding, not a panel finding, but it is also open.
Reply on any thread or push a fix and the panel resumes next round.
🤖 This review was automatically generated with Coder Agents.
- Schedule ticks against the lag-shifted clock so a bucket whose eligibility instant is still pending in the current hour is picked up minutes after startup instead of an hour later. - Skip only a failed bucket instead of aborting the whole pass, so one poison bucket (e.g. a negative runtime sum) cannot stall every later bucket until it ages out of the window. - Document that an existing row marks its bucket complete regardless of publish outcome, and the commit-latency assumption behind the eligibility lag. - Reference the jitter constant in the startup delay, align the logger name with the pprof service label, and reword the created_at column comment (every hb_agent_runtime_v1 event carries the bucket start, not just backfilled ones). - Tests: fail the concurrent-replica race on any Warn-level log so a lost ON CONFLICT dedup is detected, cover the poison-bucket skip, and assert owner/member roles cannot call the runtime-sum query.
|
/coder-agents-review |
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
Round 4 (14 reviewers). Strong round: the author engaged every finding, and the fixes are genuine and at the root cause, not the symptom. Verified fixed and closed: CRF-1 (jitter constant), CRF-3 (per-bucket continue, with TestGeneratorPoisonBucket proving a poison bucket no longer stalls later ones), CRF-4 (warnSink now fails the concurrency test on a swallowed idempotency error), CRF-5 (logger renamed to match the pprof label), CRF-6 (misleading clock setup removed), CRF-8 (mechanical owner/member denial test), CRF-9 (created_at comment fixed in all three mirror sites), CRF-10 (commit-latency assumption documented). CRF-7 acknowledged and left as-is, consistent with the finding's own guidance. The unrequested tick-scheduling change (addressing Codex's P2, wake against the lag-shifted clock) was independently verified correct by Takumi, Komugi, Mafu-san, Bisky, and Knov. Bisky: "Oh, this test suite is lovely, and this time the shine is real."
One finding is not resolved, and there are new findings the fixes surfaced.
CRF-2 (P2) is re-raised. The panel verified the coder-side of your defense: the local ledger is preserved regardless of publish outcome, and the published_at=NULL re-arm genuinely re-queues rows. For a fast permanent rejection caught by Tallyman's 1-minute alert, re-arm within the window works. But the defense addresses only the permanent-rejection path, and two verified non-permanent paths reach the same silent, un-self-healing loss without ever firing that alert: publishOnce returns 0,nil when the license disables publishing, and network/response-omission failures set Permanent:false and loop. In both, rows sit published_at IS NULL, the generator's row-existence dedup marks those buckets complete forever, and once they cross the publisher's 30-day created_at cutoff they are dropped with no coderd-side signal. The root is that dedup keys off row existence, not publish success. The clean fix is to treat an un-published, failed row as incomplete so backfill self-heals inside the 7-day window, which closes the whole class (permanent and temporary). If you'd rather accept the residual, that is a human decision and needs a linked ticket, not a PR-body note; the tallyman-side arguments (single permanent cause, the alert) are real but live in another repo and cannot be verified here, so the coupling deserves a cross-repo tracking issue.
New this round: CRF-11 (P3) the recovery path you documented is contradicted by the shipped code comment and silently expires ~30 days after the bucket; CRF-12 (P3) first-deployment backfill emits up to a week of retroactive, backdated agent-runtime charges for usage that predates the feature, which is a product/finance call; CRF-13 (P3) no test guards the daily-rollup keystone (AFTER INSERT + ON CONFLICT DO NOTHING) that prevents double-billing; plus two nits and two notes.
Still open from another bot, not a panel finding: the @coder-tasks doc check flags that docs/ai-coder/usage-data-reporting.md, the transparency list of what leaves the deployment for Tallyman, does not yet mention hb_agent_runtime_v1.
🤖 This review was automatically generated with Coder Agents.
| now := g.clock.Now().UTC() | ||
| // Bucket [H, H+1) becomes eligible at H + interval + lag. | ||
| latestEligible := now.Add(-AgentRuntimeInterval - AgentRuntimeEligibilityLag).Truncate(AgentRuntimeInterval) | ||
| earliest := now.Truncate(AgentRuntimeInterval).Add(-AgentRuntimeWindow) |
There was a problem hiding this comment.
P3 [CRF-12] On the feature's first deployment, the 7-day backfill bills up to 168 hours of pre-feature agent runtime, dated to before the mechanism existed. (Hisoka)
earliest := now.Truncate(hour).Add(-AgentRuntimeWindow)scans the trailing 7 days unconditionally. There is no start-time floor ... On the very first boot after this ships, every bucket in the window has nousage_eventsrow, sogenerateBucketsumschat_messages.runtime_msfor each hour and inserts all 168 ... they are real, previously-unbilled usage from before the billing path existed.
runtime_ms has been populated since #27451, and each event's created_at is the bucket start (within Metronome's 34-day dedup window), so Tallyman bills them: the first upgrade for any customer with prior-week agent activity produces a retroactive lump of backdated charges. "Self-healing backfill" reads as gap-repair, but on first run the whole window is the gap. Whether to bill retroactively at rollout is a product/finance decision, not one the code should make silently: either a human signs off, or add a first-run floor (do not backfill before the generator's first observed boot).
🤖
There was a problem hiding this comment.
Acknowledged, not fixed here, because the call is not the code's to make.
The mechanic checks out: earliest is now.Truncate(hour).Add(-AgentRuntimeWindow) with no first-boot floor, so on the first pass after an upgrade no bucket in the window has a usage_events row, all 168 get filled, and each carries created_at at its bucket start, well inside Metronome's 34-day dedup window. Any deployment with prior-week agent activity therefore produces a retroactive lump on upgrade.
Whether to bill pre-feature usage at rollout is a product/finance decision, so it is left for an explicit human sign-off before merge rather than silently encoded either way. If the answer is "do not bill it", the fix is a first-run floor (record the generator's first observed boot and clamp earliest to it), not shrinking AgentRuntimeWindow, which would also weaken the intended gap repair.
Reply generated with Coder Agents on behalf of @jaaydenh.
- Suppress the per-bucket generation warning when the pass is aborted by context cancellation, so shutdown mid-insert is not logged as a bucket failure (CRF-17). - Correct the HBAgentRuntime doc: the measured step ends when the model stream finishes, so local tool execution between steps is excluded (CRF-16). - Rename cronDateFormat to usageEventIDTimeFormat, name the startup delay floor, and use stableID in the generator to match the cron (CRF-15). - Widen the authz denial test to every built-in site and organization role, and describe why org-scoped usage_event grants cannot satisfy the site-scoped check (CRF-14). - Assert the daily rollup is unchanged when a bucket ID is inserted twice (CRF-13). - Document the publish re-arm SQL and its 30-day created_at bound at the dedup site (CRF-11). - List hb_agent_runtime_v1 in the usage data reporting doc.
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
Round 4 responsePushed Fixed, with a reply on each thread: CRF-11 (re-arm SQL and its 30-day bound now in the code comment), CRF-13 (duplicate-bucket insert asserted not to move the daily rollup, mutation-checked), CRF-14 (denial test widened to every built-in site and org role, comment corrected to the site-scope reason), CRF-15 ( Docs: Acknowledged, pending human sign-off, not silently decided: CRF-12 (first-deployment retroactive backfill) and CRF-2 (dedup keys off row existence). On CRF-2, one correction worth having on the record before choosing: publish-state-aware dedup is not just a filter change.
|
…e migration to 000556 main added 000552 through 000555 after this branch was cut, so the merged tree had two 000552 migrations and every migration-dependent CI job failed with "duplicate migration file".
…000560 The agent runtime usage events migration was numbered 000556, which collides with 000556_user_secrets_enabled from main. golang-migrate rejects duplicate versions, so every test that opens a database failed with "duplicate migration file: 000556_user_secrets_enabled.down.sql". Renumber the migration and its fixture to 000560, the next free version after main's 000559.
There was a problem hiding this comment.
Reminder: check migration number before merge
…000561 Main added 000560_chat_summary_boundary_index_by_id, which collides with this branch's 000560_usage_events_agent_runtime. golang-migrate rejects duplicate version numbers, so every job that runs migrations panics with "duplicate migration file". Renumber to 000561 to restore a unique sequence.
closes CODAGT-839
closes CODAGT-843
closes CODAGT-773
Summary
Adds a new heartbeat usage event type,
hb_agent_runtime_v1, measuring the total agent-loop runtime of Coder Agents (chats) per UTC hour, plus a reconciler that generates one event per hour with self-healing backfill over a trailing 7-day window. Events flow to Tallyman through the existing publisher unchanged. This measures the new Coder Agents (thechatstables), not the deprecated Tasks counted bydc_managed_agents_v1.Independent of #27508, which fixes the dead ai-seats cron registration. Both PRs carry the identical
usage_eventcreate permission hunk for the usage-publisher subject (this feature's generator and the ai-seats cron each need it for heartbeat inserts), so they can land in either order and the overlap merges cleanly.Warning
Do not include this in a release until Tallyman accepts
hb_agent_runtime_v1. The publisher marks permanently rejected events as done-forever, and the generator then sees those buckets as complete locally, so their usage would be silently and permanently lost.Details
Each event's payload is
{"runtime_ms": N}: the sum ofchat_messages.runtime_msfor messages created in the hour bucket[H, H+1), across all chats (sub-agents, API-created, archived, and soft-deleted messages included). Events use deterministic IDs (hb_agent_runtime_v1:<bucket start>) withcreated_atset to the bucket start, so concurrent replicas race safely viaON CONFLICT (id) DO NOTHINGwithout locking, and daily rollups attribute backfilled hours to the correct day. Idle hours produce zero-valued events. A bucket becomes eligible 5 minutes after it closes; hours missing for longer than the 7-day window are forfeited, which can only undercount.Note that this makes
usage_events.created_atexplicitly the event occurrence time rather than the row insertion time; the two only diverge for backfilled events. It already behaved as the occurrence timestamp (it drives the daily rollup day and is shipped to Tallyman/Metronome as the event timestamp), and the migration now documents this with aCOMMENT ON COLUMN, which also surfaces as a Go doc comment onUsageEvent.CreatedAt.The new
usage.Generatorruns unconditionally in enterprise builds; thepublish_usage_datalicense flag continues to gate egress only, so air-gapped deployments still fill their local ledger. Theaggregate_usage_event()trigger sumsruntime_msper day intousage_events_daily(unlikehb_ai_seats_v1, which takes the daily max).InsertHeartbeatUsageEventnow takes an explicitcreatedAtso generators can backfill historical buckets; the cron passesclock.Now()to preserve its existing behavior.Tallyman follow-up
Prompt for the Tallyman-repo agent