feat(typescript): persist findings and embeddings through the API - #660

Summary

Implement part 2 of the findings service: persist complete findings and embeddings through the existing HTTP endpoints and list stored findings with pagination.

Stacked on #653, with dev/kyleb/findings-service as the base so this diff contains only part 2. After #653 merges, rebase this commit onto main and retarget the PR.

Changes

• Merge the updated feat(typescript): add findings service plumbing #653 base and carry all findings database edits into the canonical plugins/codex-security/scripts/ source. The SDK bundle is generated by build:plugin and remains untracked. Preserve the existing findings operations and migration 33, update the checkout example to the canonical path, and make the extracted helper follow the neighboring scripts' isolated-import and help behavior.
• Remove the unused provisional dedupe endpoint, result type, stub service, startup wiring, dedicated mock test, and package entry. Bulk insertion stays on its direct path; unknown requests do not call the embedder.
• Wire POST /v1/bulk/findings to validate the existing Finding model, generate embeddings, and atomically store findings plus vectors. Return finding IDs in request order.
• Implement GET /v1/findings with limit/offset pagination, a default page size of 50, and nextOffset.
• Keep embedding generation and storage behind separate interfaces. Reuse the workbench database, append migration 33, share the finding upsert with CLI indexing, and invalidate vectors when stored documents change. Existing scan occurrences are preserved.
• Add an OpenAI embedder using text-embedding-3-large with 1,536 dimensions, token-aware batching, lossless chunking, and normalized pooling. Add .env.example and document Compose credential setup and the API contracts. No CLI commands or flags are added.
• Extend package allowlists, focused tests, and the Docker CI smoke test for the installed server, SQLite persistence, and clean shutdown.
• Extract Docker orchestration and HTTP assertions into an SDK-typed TypeScript smoke runner explicitly included in tsconfig.json, with a small mounted TypeScript helper using Node SQLite for storage checks. Both files are explicitly type-checked. YAML only sets up Bun and invokes the runner; helpers are excluded from npm and the production image.

Testing

Canonical plugin source update (cead95c4)

• build:plugin and check:plugin-source passed on this branch: 119 generated plugin files, including workbench_findings.py, and no tracked files under sdk/typescript/_bundled_plugin.
• Focused service, embedding, and plugin-build tests: 19 passed.
• Both full SDK suites passed: seeds 12345 and 1547932008, each with 1,941 passed, 29 skipped, and zero failures.
• pnpm run types (including the canonical MCP app), pnpm run format, and git diff --check passed.
• Compared the relocated findings helper and indexing functions structurally with the previous branch head; their function bodies and migration 33 are unchanged.
• Updated the existing canonical Python schema expectations for migration 33 and the embeddings table. All 61 setup/migration tests and both affected scan/coordinator tests passed on this branch.
• Container image build, the 319-entry npm archive, and installed-package smoke passed, including SDK imports, NodeNext types, CLI, 119 bundled plugin files, MCP initialization, and bundled Codex. The subsequent service smoke could not bind its existing fixed host port 3000; it cleaned up its test container, volume, and network. No production port or Compose default was changed.
• The first container CI attempt exposed a fixture read from the not-yet-generated host bundle. The smoke runner now reads the canonical source example directly. Types and formatting passed after that fixture-only correction; production code is unchanged from the full SDK runs above.

Earlier verification

Verified at 2e7d40b7:

• Focused service/embedding suites: 13 passed. The unsupported bulk-dedupe path returns HTTP 404 without calling the embedder; bulk insertion, pagination, identity conflicts, and embedding-failure rollback remain covered.
• Full SDK suites: 1,904 passed, 28 skipped, 0 failed in each run (seeds 12345 and 3562923079).
• pnpm run types, pnpm run format, and git diff --check: passed.
• pnpm pack and pnpm run check:package: passed with 314 archive entries. Inspected the archive: live server modules are present; the removed dedupe module and smoke fixtures are absent. Installed public imports, NodeNext types, CLI, 114 bundled plugin files, bundled Codex, and nested-worker checks passed.
• CI passed, including Linux, macOS, Windows, and container checks for insertion, embeddings, pagination, shutdown, and restart persistence. The local Docker smoke was not rerun because its fixed port 3000 was occupied; container CI exercised the updated script. Provider responses in checks are synthetic.

Risk and rollout

• This preview has no API authentication. Compose continues to publish only on the host loopback interface; use an authenticated proxy before sharing access.
• Nonempty inserts send the supplied finding JSON to the OpenAI embeddings API and require OPENAI_API_KEY or CODEX_API_KEY. Automated verification uses synthetic provider responses, not a live embedding API call.
• Database migration 33 applies automatically. Back up existing state before upgrading. Imports update existing finding IDs; identity conflicts roll back the entire batch. Historical findings are not automatically embedded.
• Candidate retrieval and model-based duplicate/group review remain in the next PR; this API does not expose a provisional deduplication route.

Public disclosure review

• No customer, partner, prospect, or user identities, data, or identifying details are included.
• No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
• I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.