Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
1195de3
feat: automate weekly AI model price book refresh
evgeniy-scherbina Aug 13, 2026
76b0a4b
refactor(scripts/aibridgepricesdiff): list models only in the summary
evgeniy-scherbina Aug 13, 2026
131b1c1
refactor(scripts/aibridgepricesdiff): count changed models, not price…
evgeniy-scherbina Aug 13, 2026
2eb4944
refactor(scripts/aibridgepricesdiff): compare models instead of price…
evgeniy-scherbina Aug 16, 2026
34ea737
refactor: minor changes in comments
evgeniy-scherbina Aug 16, 2026
6a2d79d
test(scripts/aibridgepricesdiff): split multi-model fixtures onto the…
evgeniy-scherbina Aug 16, 2026
29aa769
test(scripts/aibridgepricesdiff): one field and one row per line in f…
evgeniy-scherbina Aug 16, 2026
9f5d1df
refactor: minor changes
evgeniy-scherbina Aug 16, 2026
f553306
test(scripts/aibridgepricesdiff): assert the rendered document and er…
evgeniy-scherbina Aug 16, 2026
9d17c89
refactor: minor changes
evgeniy-scherbina Aug 17, 2026
e756f5b
fix(.github/workflows): attribute refresh commits to github-actions[bot]
evgeniy-scherbina Aug 17, 2026
4b198c1
docs: say AI Gateway in prose, keep aibridge identifiers
evgeniy-scherbina Aug 17, 2026
a9cc639
chore(CODEOWNERS): own only the generated price artifacts
evgeniy-scherbina Aug 17, 2026
b32746f
refactor(.github/workflows): name the refresh workflow aigateway
evgeniy-scherbina Aug 17, 2026
f06344e
refactor(.github/workflows): rename the webhook secret to AIGATEWAY_P…
evgeniy-scherbina Aug 17, 2026
ff7cf11
TEMPORARY: run the price refresh on push to this branch
evgeniy-scherbina Aug 17, 2026
8ee6bd9
fix(.github/workflows): send a real newline in the Slack alert
evgeniy-scherbina Aug 17, 2026
b890e7d
fix(.github/workflows): fail when the Slack webhook is missing
evgeniy-scherbina Aug 17, 2026
c8f49c4
refactor: minor changes
evgeniy-scherbina Aug 17, 2026
5ae4a94
fix(.github/workflows): manage the refresh PR through the REST API
evgeniy-scherbina Aug 17, 2026
4cca010
chore(CODEOWNERS): add @ssncferreira and @johnstcn to the price artif…
evgeniy-scherbina Aug 17, 2026
64a3b55
chore(.github/workflows): refresh AI Gateway prices on Thursdays
evgeniy-scherbina Aug 18, 2026
5284552
test(scripts/aibridgepricesdiff): cover a newly populated price
evgeniy-scherbina Aug 18, 2026
127ce34
test(scripts/aibridgepricesdiff): cover a missing price becoming zero
evgeniy-scherbina Aug 18, 2026
16d0d65
feat(scripts/aibridgepricesdiff): collapse model lists in refresh PRs
evgeniy-scherbina Aug 19, 2026
5697690
style(.github/workflows): keep PR review-note paragraphs on single lines
evgeniy-scherbina Aug 19, 2026
915c61b
feat(.github/workflows): send a weekly no-change Slack heartbeat
evgeniy-scherbina Aug 19, 2026
b279d2e
refactor(.github/workflows): consolidate Slack notifications
evgeniy-scherbina Aug 19, 2026
1d6c51a
refactor(.github/workflows): name the notification status after its c…
evgeniy-scherbina Aug 19, 2026
71197af
chore(.github/workflows): remove the temporary branch trigger
evgeniy-scherbina Aug 19, 2026
eecf329
Merge remote-tracking branch 'origin/main' into yevhenii/aigov-578-au…
evgeniy-scherbina Aug 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 178 additions & 0 deletions .github/workflows/aigateway-prices-refresh.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# Refreshes the AI Gateway price book from live upstream data (models.dev)
# once a week and opens a pull request when the generated artifacts change.
#
# The price book seeds customer-visible cost numbers, so the refresh is never
# merged automatically. The workflow only ever proposes a change; a human
# reviews and merges it.
#
# Behavior:
# - Runs every Thursday. If regeneration produces no diff, the run ends

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-12] Header Behavior block does not mention the no-change Slack heartbeat. (Leorio)

The Behavior list says "the run ends without opening anything" and "Failures are announced in Slack." Since commit 915c61be9c the workflow also posts a Slack heartbeat when nothing changed (line 170: :white_check_mark: *AI Gateway price book refresh completed.*). A reader working only from the header would think Slack fires only on failure, then wonder on the first quiet Thursday whether the heartbeat is a bug. Add a bullet:

#   - Announces successful no-change runs to Slack as a heartbeat, so a
#     silently missed schedule is caught.

🤖

# without opening anything.
# - Reuses a single branch and pull request, force-pushing each week, so at
# most one refresh PR is open and it always carries the newest snapshot.
# - Fails loudly when the generator refuses to run, which it does by design
# when upstream drops a model pinned in overrides.jq or curated in
# curation.json. Failures are announced in Slack.
name: aigateway-prices-refresh

on:
schedule:
# 09:00 UTC every Thursday, leaving three business days before Tuesday releases.
- cron: "0 9 * * 4"
workflow_dispatch: # allows manual runs for testing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-10] workflow_dispatch: # allows manual runs for testing narrows the intent without evidence. (Gon)

workflow_dispatch is the GitHub Actions manual-trigger keyword; anyone editing this file already knows what it does. Manual runs are useful for reasons beyond testing (ad-hoc refresh, unsticking a failed run), so "for testing" is not the whole story. Delete the trailing comment.

🤖


permissions: {}

concurrency:
group: aigateway-prices-refresh

env:
REFRESH_BRANCH: bot/aigateway-prices-refresh
PRICES_FILE: coderd/aibridge/prices/data/prices.json
CATALOG_FILE: site/src/pages/AgentsPage/components/ChatModelAdminPanel/knownModels/knownModelsGenerated.json

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-13] PRICES_FILE and CATALOG_FILE duplicate the Makefile target paths at Makefile:1057 and Makefile:1066; there is no static coupling forcing the two lists to agree. (Mafuuu)

make gen/aibridge-prices writes whatever the Makefile targets say. The workflow detects changes via git diff --quiet -- "${PRICES_FILE}" "${CATALOG_FILE}" and commits via git add -- "${PRICES_FILE}" "${CATALOG_FILE}". If someone renames the frontend catalog target in the Makefile without updating this workflow, the diff check on the stale path returns "no diff" (the old path is gone from HEAD, nothing to compare), so the run reports no changes and no PR is opened while a real catalog diff sits in the working tree unstaged.

Options: derive the paths from the Makefile (e.g., print them from a helper target), or add a guard step that fails when either env path is not tracked in HEAD.

🤖


jobs:
refresh:
name: Refresh price book
runs-on: ubuntu-latest
permissions: {}
steps:
- name: Harden Runner
uses: step-security/harden-runner@b09bb98e06d4d774595224525879c09bc6e98c40 # v2.20.1
with:
egress-policy: audit

- name: Checkout

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-6] Checkout has no ref:, so a workflow_dispatch from a non-main branch smuggles that branch's history into the shared refresh PR. (Hisoka P3, Ryosuke Note)

Checkout has no ref:, so it defaults to github.ref. On schedule that is fine. On workflow_dispatch GitHub lets any actor with write pick a branch, and github.ref becomes that branch. Line 123 then git checkout -b "${REFRESH_BRANCH}" off that HEAD, git add only the two artifact paths, git commit, and git push --force refs/heads/bot/aigateway-prices-refresh. The remote refresh branch is now that topic branch's history plus one price commit. Line 142 creates the PR with base=main, so the diff shown to reviewers is main..<topic> for every file the topic branch touched, not just prices.

The implementation plan explicitly says workflow_dispatch "only fire from the default branch." It does not. Only maintainers can dispatch, so this is unlikely by accident, but the round-1 P1 (a temporary push: trigger for a personal branch, added because the author wanted to test from a branch) shows the pattern where testing from a non-main branch is real. A successor dispatching to test a change would open a refresh PR shipping unrelated commits. Reviewer opens a many-file diff titled "chore: refresh AI model price book" and either closes it in confusion or, worst case, merges it.

Two mechanical fixes, either sufficient:

  1. Pin the checkout ref: add ref: main under with: on the Checkout step, or git fetch origin main && git checkout -b "${REFRESH_BRANCH}" origin/main before staging.
  2. Guard the job: jobs.refresh.if: github.ref == 'refs/heads/main'. Costs a manual main re-dispatch to test.

🤖

uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Set up mise tools
uses: ./.github/actions/setup-mise
with:
install-args: "go node pnpm"

# Needed by catalog generation, which formats its output with biome.
- name: Install pnpm dependencies
uses: ./.github/actions/pnpm-install

- name: Snapshot the current price book
run: cp "${PRICES_FILE}" "${RUNNER_TEMP}/prices-before.json"

- name: Regenerate price book and model catalog
run: make gen/aibridge-prices

- name: Detect changes
id: detect
run: |
set -euo pipefail
if git diff --quiet -- "${PRICES_FILE}" "${CATALOG_FILE}"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-3] Catalog-only refresh opens a PR titled "chore: refresh AI model price book" whose body reads "No price changes." (Hisoka P3, Kite P3, Meruem P3, Ryosuke P3, plus Mafuuu Nit, Pariston Note, Razor Note, Zoro Note)

Detect changes flips changed=true when either ${PRICES_FILE} or ${CATALOG_FILE} differs, yet Build pull request body runs aibridgepricesdiff against the two prices.json snapshots only. knownModelsGenerated.json carries contextLimit, maxOutputTokens, reasoningEffort, displayName, and aliases, none of which are in prices.json; when upstream changes one of those without touching prices, a PR opens whose body reads ## Price book changes\n\nNo price changes. The review note two lines further down claims "The summary above lists what moved", which is now false.

The class of bug is that "did anything change" lives in two places (the git-diff gate and the tool's price-only rendering) and the two can disagree. Two fixes eliminate the gap rather than paper over it:

  1. Snapshot ${CATALOG_FILE} too and teach aibridgepricesdiff to name catalog moves alongside prices; rename the heading (## Snapshot changes or similar) so the label does not promise a price-only view.
  2. Gate the PR-open decision on PRICES_FILE changing alone, letting a catalog-only drift wait for the next price move. Cheaper but delays catalog updates.

Option 1 is the more honest fix. Reviewers on this PR are the same for both artifacts, so the routing is fine; the mismatch is cognitive but predictable and repeated weekly.

🤖

# exit 0 => NO differences => nothing to propose
echo "Price book already matches upstream; nothing to propose."
echo "changed=false" >> "$GITHUB_OUTPUT"
else
# exit 1 => differences found
git diff --stat -- "${PRICES_FILE}" "${CATALOG_FILE}"
echo "changed=true" >> "$GITHUB_OUTPUT"
fi

- name: Build pull request body
if: steps.detect.outputs.changed == 'true'
run: |
set -euo pipefail
go run ./scripts/aibridgepricesdiff \
-old "${RUNNER_TEMP}/prices-before.json" \
-new "${PRICES_FILE}" > "${RUNNER_TEMP}/summary.md"

{
cat "${RUNNER_TEMP}/summary.md"
echo
echo "## Review notes"
echo
echo "Regenerated by \`make gen/aibridge-prices\` from the live [models.dev](https://models.dev) catalog. Both artifacts come from one snapshot, so they ship together:"
echo
echo "- \`${PRICES_FILE}\`"
echo "- \`${CATALOG_FILE}\`"
echo
echo "These are customer-visible cost numbers taken from upstream data, so this PR is never merged automatically. The summary above lists what moved; check the diff for exact figures before approving."
echo
echo "Opened automatically by the [aigateway-prices-refresh workflow](${RUN_URL})."
} > "${RUNNER_TEMP}/body.md"

cat "${RUNNER_TEMP}/body.md"
env:
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

- name: Open or update the refresh pull request
if: steps.detect.outputs.changed == 'true'
env:
# Use cdrci's token instead of the default GITHUB_TOKEN: PRs opened
# with GITHUB_TOKEN do not trigger workflow runs, so the refresh
# would arrive without CI signal.
GH_TOKEN: ${{ secrets.CDRCI_GITHUB_TOKEN }}
PR_TITLE: "chore: refresh AI model price book"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-5] Prose name drift: "AI model price book" here and in the Slack failure text (line 168) versus "AI Gateway price book" in the workflow header (line 1) and the heartbeat text (line 170). (Chopper, Gon, Leorio, Ryosuke)

The PR description states the naming policy explicitly: identifiers stay aibridge, prose says "AI Gateway". The file header at line 1 follows that ("Refreshes the AI Gateway price book"), and the no-change Slack text at line 170 follows it too.

PR_TITLE: "chore: refresh AI model price book" produces the weekly commit message and PR title, so every future refresh commit lands in main's history spelling the object differently from the workflow that produced it. Line 168 has the same drift right next to the compliant heartbeat two lines down, which also fragments Slack search ("AI Gateway price book" misses the failure alerts).

Change both to "AI Gateway price book".

🤖

run: |
set -euo pipefail

# persist-credentials is disabled on checkout, so authenticate the
# push explicitly.
git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"

git checkout -b "${REFRESH_BRANCH}"
git add -- "${PRICES_FILE}" "${CATALOG_FILE}"
git commit -m "${PR_TITLE}"

# Force-push: the branch is regenerated from the newest upstream
# snapshot each week, so the previous contents are always stale.
git push --force origin "refs/heads/${REFRESH_BRANCH}"
Comment thread
evgeniy-scherbina marked this conversation as resolved.

# REST, not `gh pr`: those go through GraphQL, which needs a read:org
# scope that cdrci's token lacks.
owner="${GITHUB_REPOSITORY%%/*}"
pr_number="$(gh api "repos/${GITHUB_REPOSITORY}/pulls?state=open&base=main&head=${owner}:${REFRESH_BRANCH}" --jq '.[0].number // empty')"
if [ -n "${pr_number}" ]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/pulls/${pr_number}" \
-f title="${PR_TITLE}" \
-f body="$(cat "${RUNNER_TEMP}/body.md")" \
--silent
echo "Updated existing PR #${pr_number}."
else
gh api --method POST "repos/${GITHUB_REPOSITORY}/pulls" \
-f title="${PR_TITLE}" \
-f head="${REFRESH_BRANCH}" \
-f base=main \
-f body="$(cat "${RUNNER_TEMP}/body.md")" \
--jq '.html_url'
fi

- name: Send Slack notification

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-14] Cancelled runs go silent because the notification step gates on failure() || changed == 'false', and failure() does not fire on cancellation. (Mafuuu)

If a scheduled run is manually cancelled or superseded by the concurrency group, no Slack signal is emitted. The workflow's stated intent is "Failures are announced in Slack"; cancellation is not a failure, but for a weekly cron whose whole point is to not be silent, on-call may want to know a run was killed mid-flight instead of finding out at the next heartbeat. Consider if: always() && steps.detect.outputs.changed != 'true' with a cancelled branch, or accept the gap and document it in the header comment.

🤖

if: failure() || steps.detect.outputs.changed == 'false'
env:
JOB_STATUS: ${{ job.status }}
PRICE_BOOK_CHANGED: ${{ steps.detect.outputs.changed }}
SLACK_WEBHOOK: ${{ secrets.AIGATEWAY_PRICES_SLACK_WEBHOOK }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
if [ -z "${SLACK_WEBHOOK}" ]; then
echo "::error::AIGATEWAY_PRICES_SLACK_WEBHOOK is not set; the notification could not be sent."
exit 1
fi

if [ "${JOB_STATUS}" = "failure" ]; then
# printf, not a double-quoted literal: bash leaves \n as two
# characters, and jq --arg then escapes the backslash, so Slack
# would print \n as text instead of breaking the line.
text="$(printf ':warning: *AI model price book refresh failed.*\nThe generator fails by design when upstream drops a model pinned in scripts/aibridgepricesgen/overrides.jq or curated in curation.json. Logs: %s' "${RUN_URL}")"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-4] Slack failure alert asserts one specific root cause for every failure mode in the job. (Leorio P2, Chopper P3, Mafuuu P3, Meruem P3, Pariston Note, Razor Nit)

At 2 AM:

⚠️ AI model price book refresh failed.
The generator fails by design when upstream drops a model pinned in scripts/aibridgepricesgen/overrides.jq or curated in curation.json. Logs: <run_url>

The step's if: is failure() || steps.detect.outputs.changed == 'false', so this branch runs when anything failed: Harden Runner, Checkout, Set up mise tools, Install pnpm dependencies, make gen/aibridge-prices for reasons unrelated to model drops, git push, the gh api calls (the PR description itself records that the verification runs hit HTTP 503 from api.github.com/graphql). The on-call reader is anchored to overrides.jq / curation.json before opening the run log, burns time confirming those files are fine, and only then reads the actual failure.

Diagnose, don't guess. Name the operation and run URL first; if a leading hypothesis is worth keeping, present it as one common cause, not the cause:

:warning: *AI Gateway price book refresh failed.* Logs: %s
A common cause is upstream dropping a model pinned in scripts/aibridgepricesgen/overrides.jq or scripts/aibridgepricesgen/curation.json. If the logs point elsewhere, that is the real cause; treat this hint as a starting point, not a diagnosis.

🤖

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-15] Slack failure text spells one referenced file with its full path and the other without. (Meruem)

scripts/aibridgepricesgen/overrides.jq is spelled with the directory; curation.json is not, even though it lives in the same directory. When a responder greps for the referenced file, one path resolves and the other does not. Either add the prefix to curation.json or drop it from overrides.jq.

🤖

elif [ "${PRICE_BOOK_CHANGED}" = "false" ]; then
text=":white_check_mark: *AI Gateway price book refresh completed.* No generated changes were found. Run: ${RUN_URL}"
else
echo "::error::Unexpected notification state: status=${JOB_STATUS}, changed=${PRICE_BOOK_CHANGED}"
exit 1
fi

payload="$(jq -nc --arg text "${text}" '{text: $text}')"
curl -fsSL -X POST -H 'Content-type: application/json' -d "${payload}" "${SLACK_WEBHOOK}"
echo "Sent Slack notification"
7 changes: 7 additions & 0 deletions CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,10 @@ coderd/database/queries/ai_provider_keys.sql @ibetitsmike @johnstcn
coderd/database/queries/aicostcontrol.sql @ibetitsmike @johnstcn
codersdk/aiproviders.go @ibetitsmike @johnstcn
codersdk/aiproviders_bedrock.go @ibetitsmike @johnstcn

# Generated price book and frontend model catalog. The
# aigateway-prices-refresh workflow proposes updates to both from live
# models.dev data, and they carry customer-visible cost numbers, so every
# refresh needs a review from someone who owns them.
coderd/aibridge/prices/data/prices.json @evgeniy-scherbina @ssncferreira @johnstcn
site/src/pages/AgentsPage/components/ChatModelAdminPanel/knownModels/knownModelsGenerated.json @evgeniy-scherbina @ssncferreira @johnstcn

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-7] New rule silently displaces @DanielleMaywood as an owner of knownModelsGenerated.json. (Netero)

CODEOWNERS uses last-match-wins. Line 31 (site/src/pages/AgentsPage/ @DanielleMaywood, unchanged by this PR) previously covered the frontend catalog. The new rule at line 49 lists only @evgeniy-scherbina @ssncferreira @johnstcn, so any change to that file (workflow or manual) now requests review only from those three. The PR body discusses ownership of curation.json and overrides.jq but not this narrowing.

Likely intentional (the file is generated and the new owners are the pricing-data reviewers), but the surrounding comment does not state the intent and the displacement is invisible in the diff. Confirm this is the intent, and add a line to the comment noting that the frontend owner is deliberately excluded from generated catalog updates.

🤖

198 changes: 198 additions & 0 deletions scripts/aibridgepricesdiff/main.go
Comment thread
evgeniy-scherbina marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
// aibridgepricesdiff renders a human-readable Markdown summary of the
// difference between two AI Gateway price seed files (the prices.json produced
// by aibridgepricesgen).
//
// The price refresh workflow uses it to fill the pull request body, so
// a reviewer sees at a glance which models appeared, which disappeared, and
// which repriced. Exact figures are deliberately left to the pull request
// diff, which is the source of truth.
//
// Usage:
//
// aibridgepricesdiff -old <path> -new <path>
package main

import (
"encoding/json"
"flag"
"fmt"
"io"
"os"
"sort"
"strings"

"golang.org/x/xerrors"
)

// priceRow mirrors the seed file schema written by aibridgepricesgen. Pointer
// fields preserve the distinction between "not populated by upstream" (null)
// and "explicitly zero" (0).
type priceRow struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably move this to a shared package, as it is used by aibridgepricesgen and https://github.com/coder/coder/blob/main/coderd/aibridge/prices/prices.go#L26C6-L26C13

And possibly other structs/method. Can be done in a follow-up cleanup PR 👍

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I can fix it in a follow-up PR

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-2] Fourth verbatim copy of the price-seed schema; two adjacent sync-note comments still say the schema lives in three places. (Netero P2, Robin P2, Zoro P2, Ryosuke P3, Razor P3, Hisoka P4)

coderd/aibridge/prices/prices.go:23-25 and scripts/aibridgepricesgen/main.go:68-71 both declare that "the JSON contract for the price seed lives in three places that must stay in sync": the seeder struct, the generator struct, and the batch SQL upsert in coderd/database/queries/aicostcontrol.sql. This PR adds priceRow here as a silent fourth. json.Unmarshal ignores unknown fields, so when a new price column lands upstream (say cache_hit_price), readRows will drop it, samePrices will not compare it, and compare will report 0 changed for a real repricing.

Consequence: the weekly refresh renders "No price changes." over a real reprice, and the summary the reviewer is asked to trust is quietly narrower than the diff below it. The tool exists precisely so reviewers see what moved; this bug hazard defeats its purpose on the first schema extension.

Two fixes:

  1. Minimal, matches existing pattern: update the two sibling NOTE comments to say four places and enumerate the diff tool.
  2. Structural: extract the row struct into a leaf schema package (e.g., coderd/aibridge/prices/priceseed) that owns only the type and JSON tags, import it from both scripts, and delete the drift-warning comments. go build then catches the drift the comment was asking humans to catch.

Ryosuke's framing: three-comments-in-sync is a rule; one imported type is a fact. @ssncferreira's earlier thread suggested the same structural move as a follow-up; the panel is elevating this to P2 because the consequence (silent "No price changes" on future column additions) is real and customer-visible, and there is no ticket linked to the follow-up commitment. Needs a fix here, a linked ticket, or an explicit human decision to accept the gap.

🤖

Provider string `json:"provider"`
Model string `json:"model"`
InputPrice *int64 `json:"input_price"`
OutputPrice *int64 `json:"output_price"`
CacheReadPrice *int64 `json:"cache_read_price"`
CacheWritePrice *int64 `json:"cache_write_price"`
}

// modelKey identifies a model across the two snapshots.
type modelKey struct {
provider string
model string
}

func (r priceRow) key() modelKey {
return modelKey{provider: r.Provider, model: r.Model}
}

func (k modelKey) String() string {
return k.provider + "/" + k.model
}

func less(a, b modelKey) bool {
if a.provider != b.provider {
return a.provider < b.provider
}
return a.model < b.model
}
Comment thread
evgeniy-scherbina marked this conversation as resolved.

// diff is the full comparison between two snapshots

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-9] diff type doc is missing a terminating period. (Gon, Leorio)

// diff is the full comparison between two snapshots diverges from the sibling docs in the same file (priceRow at 27-29, modelKey at 39). Add the period.

🤖

type diff struct {
added []modelKey
removed []modelKey
changed []modelKey
}

func (d diff) empty() bool {
return len(d.added) == 0 && len(d.removed) == 0 && len(d.changed) == 0
}

func main() {
oldPath := flag.String("old", "", "path to the previous prices.json (required)")
newPath := flag.String("new", "", "path to the refreshed prices.json (required)")
flag.Parse()
if err := run(*oldPath, *newPath, os.Stdout); err != nil {
_, _ = fmt.Fprintf(os.Stderr, "aibridgepricesdiff: %v\n", err)
os.Exit(1)
}
}

func run(oldPath, newPath string, w io.Writer) error {
if oldPath == "" || newPath == "" {
return xerrors.New("-old and -new are both required")
}
oldRows, err := readRows(oldPath)
if err != nil {
return xerrors.Errorf("read %s: %w", oldPath, err)
}
newRows, err := readRows(newPath)
if err != nil {
return xerrors.Errorf("read %s: %w", newPath, err)
}
_, err = io.WriteString(w, render(compare(oldRows, newRows)))
return err
}

func readRows(path string) ([]priceRow, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var rows []priceRow
if err := json.Unmarshal(data, &rows); err != nil {
return nil, xerrors.Errorf("parse: %w", err)
}
return rows, nil
}

// compare classifies every model as added, removed, or changed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-11] compare doc omits the deterministic-sort contract. (Leorio)

The comment says "classifies every model as added, removed, or changed." The function also sorts each output slice by (provider, model), and TestCompareSortsDeterministically locks that behavior in. render iterates each slice in order and depends on it for stable PR bodies. A reader working only from the doc would think the output order is iteration-dependent. Extend:

// compare classifies every model as added, removed, or changed, and sorts
// each category by (provider, model) so the rendered summary is stable.

🤖

func compare(oldRows, newRows []priceRow) diff {
oldByKey := make(map[modelKey]priceRow, len(oldRows))
for _, r := range oldRows {
oldByKey[r.key()] = r
}

var d diff
seen := make(map[modelKey]struct{}, len(newRows))
for _, r := range newRows {
seen[r.key()] = struct{}{}
prev, ok := oldByKey[r.key()]
switch {
case !ok:
d.added = append(d.added, r.key())
case !samePrices(prev, r):
d.changed = append(d.changed, r.key())
}
}
for _, r := range oldRows {
if _, ok := seen[r.key()]; !ok {
d.removed = append(d.removed, r.key())
}
}

for _, keys := range [][]modelKey{d.added, d.removed, d.changed} {
sort.Slice(keys, func(i, j int) bool { return less(keys[i], keys[j]) })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-8] sort.Slice is superseded by slices.SortFunc since Go 1.21; .claude/docs/GO.md names slices.SortFunc as the modern replacement. (Ging-go, Meruem)

go.mod is on go 1.26.5. Drop the less helper and the sort import:

slices.SortFunc(keys, func(a, b modelKey) int {
    return cmp.Or(cmp.Compare(a.provider, b.provider), cmp.Compare(a.model, b.model))
})

🤖

}
return d
}

// samePrices reports whether two rows for the same model carry identical
// prices. A price moving to or from null counts as a change.
func samePrices(a, b priceRow) bool {
return equalPrice(a.InputPrice, b.InputPrice) &&
equalPrice(a.OutputPrice, b.OutputPrice) &&
equalPrice(a.CacheReadPrice, b.CacheReadPrice) &&
equalPrice(a.CacheWritePrice, b.CacheWritePrice)
}

func equalPrice(a, b *int64) bool {
if a == nil || b == nil {
return a == nil && b == nil
}
return *a == *b
}

// render writes the Markdown summary: counts, then the models in each
// category. Empty categories are omitted.
func render(d diff) string {
var b strings.Builder
write := func(format string, args ...any) {
_, _ = fmt.Fprintf(&b, format, args...)
}

write("## Price book changes\n\n")
if d.empty() {
write("No price changes.\n")
return b.String()
}

write("%s added, %s removed, %s changed.\n",
plural(len(d.added), "model"),
plural(len(d.removed), "model"),
plural(len(d.changed), "model"),
)

renderList := func(heading string, keys []modelKey) {
if len(keys) == 0 {
return
}
write("\n<details>\n<summary>%s</summary>\n\n", heading)
for _, k := range keys {
write("- %s\n", k)
}
write("</details>\n")
}
renderList("Added", d.added)
renderList("Removed", d.removed)
renderList("Changed", d.changed)
Comment thread
ssncferreira marked this conversation as resolved.

return b.String()
}

func plural(n int, noun string) string {
if n == 1 {
return fmt.Sprintf("%d %s", n, noun)
}
return fmt.Sprintf("%d %ss", n, noun)
}
Loading
Loading