Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(hub): R2 bucket structure + Cloudflare Worker publish API (#1095)#1409

Merged
kovtcharov-amd merged 1 commit into
mainfrom
claudia/task-864911fe
Jun 4, 2026
Merged

feat(hub): R2 bucket structure + Cloudflare Worker publish API (#1095)#1409
kovtcharov-amd merged 1 commit into
mainfrom
claudia/task-864911fe

Conversation

@kovtcharov-amd

Copy link
Copy Markdown
Collaborator

Why this matters

Until now the Agent Hub had a manifest format (gaia-agent.yaml, #1091) but nowhere to publish agents to or download them from. This adds the cloud distribution layer: a Cloudflare Worker fronting an R2 bucket that lets vetted publishers upload an agent's manifest + artifact and serves a lightweight catalog the gaia agent CLI and Hub UI can read. Publishing is gated — bad tokens, out-of-scope authors, and attempts to overwrite a released version are all rejected loudly, and the artifact checksum is generated server-side so it can never be spoofed by the uploader.

Implements Phase 3A–3C of the Agent Hub plan (#1095). Isolated infra — everything lives under workers/agent-hub/; no src/gaia code is touched and there is no dependency on the in-flight #1102 restructure. JSON field names mirror src/gaia/hub/manifest.py so the catalog is consumable by the same code that reads source manifests.

R2 bucket layout

index.json                                # lightweight catalog (all agents)
agents/<id>/manifest.json                 # per-agent aggregate (all versions)
agents/<id>/<version>/gaia-agent.yaml      # exact manifest uploaded for this version
agents/<id>/<version>/<filename>           # the artifact (wheel or binary)

Wiring secrets for deploy (maintainer)

Deploy needs Cloudflare resources not in the repo:

  1. npx wrangler r2 bucket create gaia-agent-hub (name matches wrangler.toml)
  2. npx wrangler secret put PUBLISH_TOKENS — JSON map of token → publisher, e.g.
    {"<amd-token>":{"publisher":"AMD","authors":["AMD"]},"<admin-token>":{"publisher":"Hub Admin","authors":["*"]}}. The authors list bounds which author values a token may publish under; "*" is reserved for hub admins.
  3. npx wrangler deploy (optionally uncomment the routes line to bind hub.amd-gaia.ai/*).

Full instructions and the JSON schemas are in workers/agent-hub/README.md and workers/agent-hub/schemas/.

Test plan

  • cd workers/agent-hub && npm install && npm test — 32 tests pass (auth rejection, publisher-scope + ownership, version immutability, server-side checksum, index rebuild, download routes, manifest validation, semver ordering)
  • npm run typecheck — clean (tsc --noEmit, strict)
  • npm run deploy:dry-run (wrangler deploy --dry-run) — config valid, bundle builds, R2 binding + vars resolved
  • Maintainer: provision R2 bucket + PUBLISH_TOKENS secret, then wrangler deploy and smoke-test POST /publish + GET /index.json

Refs #1095 · spec: docs/spec/agent-hub-restructure.mdx (Phase 3, R2 distribution). No dependency on #1102.

Cloud distribution layer for the Agent Hub (Phase 3A-3C). A Cloudflare
Worker fronting an R2 bucket: POST /publish validates an uploaded
gaia-agent.yaml + artifact, enforces per-publisher auth, publisher scope,
and version immutability, generates a server-side SHA-256, then rebuilds a
lightweight index.json. GET routes serve the catalog, per-agent manifests,
and artifact downloads.

Isolated infra under workers/ only — no src/gaia changes, no dependency on
the #1102 restructure. Field names mirror src/gaia/hub/manifest.py so the
catalog is consumable by the same code that reads source manifests.

Runnable locally with vitest (in-memory R2 fake) and wrangler dev
(Miniflare). Deploy needs an R2 bucket + PUBLISH_TOKENS secret the
maintainer provisions; see README.
@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Solid, well-isolated foundation for the Agent Hub distribution layer — the design is sound and the security posture (server-side checksum, version immutability, publisher scoping, fail-loud auth) is exactly right. Code is clean, typed strict, and the in-memory R2 fake makes the 32 handler-level tests genuinely meaningful. The one thing worth fixing before this goes live: the Worker's manifest validation drifts from the canonical src/gaia/hub/manifest.py on reserved agent IDs, which can let through agents that no GAIA client can install. A couple of operational gaps (in-memory size handling, CORS, CI) are also worth a look.

Issues

🟡 Important

1. Reserved-ID check missing — diverges from the canonical validator (src/manifest.ts:136)

parseManifest mirrors manifest.py's id regex, languages, tiers, and platforms — but not its reserved-ID block. manifest.py:393 rejects ids like chat, code, doc, file, etc. (the built-in agent names); the Worker accepts them. The consequence is concrete: a publisher can upload an agent at agents/chat/, but any client that later reads its gaia-agent.yaml runs it through manifest.py, which rejects the id on parse — so the published agent is un-installable. This also masks itself in the suite, where the sample fixture uses id: chat (test/fake-r2.ts:178), so every test publishes a reserved id without noticing.

Suggested direction: mirror the reserved set in manifest.ts and reject in parseManifest, then change the test fixture to a non-reserved id (e.g. my-chat). Since the README stresses the field names mirror manifest.py, the validation rules should too — otherwise the two will keep drifting silently.

2. 250 MiB default vs. the Worker memory limit; size check happens after full buffering (src/publish.ts:25,124-135)

The artifact is fully materialized in memory (artifactFile.arrayBuffer()Uint8Array → SHA-256) before the size check at line 129. Cloudflare Workers cap at ~128 MB memory, so DEFAULT_MAX_BYTES = 250 MiB is above what the runtime can actually hold — a large native-binary artifact (the PR explicitly supports binaries, not just wheels) will OOM with an opaque 500 rather than the clean 413, and the limit never protects memory because the bytes are already buffered. Recommend rejecting via Content-Length up front, dropping the default well below the memory ceiling, and/or streaming to R2 (createMultipartUpload) for large artifacts. At minimum, lower the default and document the ceiling.

3. No CORS headers on the catalog routes (src/index.ts:20-29)

The PR positions index.json / per-agent manifests for consumption by "the Hub UI". If that UI is served from a different origin than hub.amd-gaia.ai, browser fetch will be blocked — there are no Access-Control-Allow-Origin headers on serveObject/json. Is the UI same-origin? If not, the GET routes need CORS. Worth confirming before the UI work lands on top of this.

4. Nothing runs these 32 tests in CI

No workflow under .github/workflows/ covers workers/agent-hub/ (the repo's CI is Python-only), so npm test / npm run typecheck only ever run locally. For isolated infra this is the moment to add a small Node workflow gated on workers/agent-hub/** — otherwise the suite silently rots. Reasonable as a fast follow-up, but call it out so it isn't forgotten.

🟢 Minor

  • Immutable artifacts get no Cache-Control (src/index.ts:25-28) — versioned artifact/manifest keys are immutable and could carry a long max-age; index.json a short one. Cheap CDN win.
  • eslint-disable comments with no ESLint config (test/fake-r2.ts:56 et al.) — the disables are inert; either add a flat ESLint config + lint script or drop the directives.
  • Token comparison isn't constant-time (src/auth.ts:59) — object-key lookup on the bearer token. Low risk here (network jitter dominates, tokens are high-entropy), noting only for completeness.

Strengths

  • Security model is the highlight — server-side SHA-256 that's never trusted from input and is also handed to R2's put integrity check; version immutability enforced both via the manifest (source of truth) and an object-level head() as defense-in-depth; ownership check that blocks even *-scoped admin tokens from hijacking another author's id (test/publish.test.ts:68). The auth deliberately returns a generic message so callers can't distinguish token-vs-scope failure.
  • Fail-loudly throughout — matches CLAUDE.md exactly: unset/malformed PUBLISH_TOKENS is a loud 500, never allow-all; every guard raises a structured, actionable HttpError instead of degrading.
  • Test approach is excellent — the in-memory FakeR2 honors the optional sha256 integrity check, so the suite exercises real handler flow end-to-end (auth, scope, immutability, checksum, index rebuild, downloads) without Miniflare, and even asserts the original artifact is byte-for-byte untouched after a rejected republish.
  • Genuinely isolated — no src/gaia coupling, no dependency on the in-flight Agent Hub: Restructure — move production agents to hub/agents/ #1102 restructure, clean module boundaries.

Verdict

Request changes — the design is right and most of this is mergeable as-is, but issue #1 (reserved-ID divergence making published agents un-installable, hidden by the id: chat fixture) should be resolved before merge, and #2#4 are worth addressing or consciously deferring with a tracked follow-up.

Note: I reviewed the source and tests directly but did not run npm test / npm run typecheck in this environment; the PR reports both green.

@kovtcharov-amd kovtcharov-amd merged commit 8b206e7 into main Jun 4, 2026
22 checks passed
@kovtcharov-amd kovtcharov-amd deleted the claudia/task-864911fe branch June 4, 2026 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant