feat: add venue-aware storage schema #53

terwey · 2025-11-01T18:49:21Z

Summary

add venues and bot assignment tables to the storage schema and update hyperliquid tables to use composite identifiers
refresh sqlc queries to accept venue-aware arguments and regenerate models for the new schema

Testing

go generate ./gen/storage

https://chatgpt.com/codex/tasks/task_e_690653d47af08326926a79832136ffb6

chatgpt-codex-connector

💡 Codex Review

recomma/storage/order_scalers.go

Lines 378 to 382 in cc77d12

    
           MultiplierUpdatedBy: params.MultiplierUpdatedBy, 
        
           CreatedAtUtc:        createdAt.UTC().UnixMilli(), 
        
           SubmittedOrderID:    params.SubmittedOrderID, 
        
           Skipped:             boolToInt(params.Skipped), 
        
           SkipReason:          params.SkipReason,

Update scaled-order insert to match new composite key API

The schema changes replaced the auto‑increment scaled_orders.id with the (venue_id, wallet, order_id) primary key and removed submitted_order_id, so sqlcgen.InsertScaledOrderParams no longer exposes SubmittedOrderID and the generated method now returns only error. InsertScaledOrderAudit still sets SubmittedOrderID on the params struct and assigns the result of InsertScaledOrder to id, which no longer compiles. The function needs to supply the new venue/wallet/order identifiers and drop the removed field and return value.

recomma/storage/order_scalers.go

Lines 446 to 450 in cc77d12

    
           func (s *Storage) ListScaledOrdersByMetadata(ctx context.Context, md metadata.Metadata) ([]ScaledOrderAudit, error) { 
        
           	s.mu.Lock() 
        
           	defer s.mu.Unlock() 
        
           	rows, err := s.queries.ListScaledOrdersByMetadata(ctx, md.Hex())

Adjust scaled-order listing calls for new sqlc signatures

ListScaledOrdersByMetadata still invokes the generated query with a single md string, but the updated SQL now requires a struct containing both venue_id and md. This call (and the similar one for deals) will not compile until the caller is changed to pass the composite key.

recomma/storage/storage.go

Lines 329 to 333 in cc77d12

    
           params := sqlcgen.InsertHyperliquidStatusParams{ 
        
           	Md:            md.Hex(), 
        
           	Status:        raw, 
        
           	RecordedAtUtc: time.Now().UTC().UnixMilli(), 
        
           }

Hyperliquid persistence lacks venue and payload fields

The new schema expects InsertHyperliquidStatusParams (and the create/modify/cancel upserts) to carry venue_id, wallet, order_id, payload_type, and payload_blob, but the storage layer still populates only Md and the old Status blob. The generated structs no longer have a Status field, so this code will fail to compile and would violate the NOT NULL/foreign key constraints even if it did. The caller needs to supply the new composite identifiers and payload metadata for each hyperliquid submission/status write.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

terwey · 2025-11-01T19:14:42Z

@codex review, for context see specs/multi_venue_emission.adoc

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-11-01T19:17:57Z

storage/sqlc/schema.sql

+CREATE TABLE IF NOT EXISTS venues (
+    id            TEXT PRIMARY KEY,
+    type          TEXT NOT NULL,
+    display_name  TEXT NOT NULL,
+    wallet        TEXT NOT NULL,
+    flags         JSON NOT NULL DEFAULT (CAST('{}' AS BLOB)),
+    CHECK (json_valid(flags))
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS idx_venues_type_wallet
+    ON venues(type, wallet);
+
+CREATE TABLE IF NOT EXISTS bot_venue_assignments (
+    bot_id          INTEGER NOT NULL,
+    venue_id        TEXT    NOT NULL,
+    is_primary      INTEGER NOT NULL DEFAULT 0 CHECK(is_primary IN (0, 1)),
+    assigned_at_utc INTEGER NOT NULL DEFAULT (CAST(unixepoch('now','subsec') * 1000 AS INTEGER)),
+    PRIMARY KEY (bot_id, venue_id),
+    FOREIGN KEY(bot_id) REFERENCES threecommas_bots(bot_id) ON DELETE CASCADE,
+    FOREIGN KEY(venue_id) REFERENCES venues(id) ON DELETE CASCADE
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS idx_bot_venue_assignments_primary
+    ON bot_venue_assignments(bot_id)
+    WHERE is_primary = 1;
+
 CREATE TABLE IF NOT EXISTS hyperliquid_submissions (
-  md              TEXT PRIMARY KEY,
-  action_kind     TEXT NOT NULL CHECK(action_kind IN ('create','modify','cancel')),
-  create_payload  JSON,
-  modify_payloads JSON NOT NULL DEFAULT (CAST('[]' AS BLOB)),
-  cancel_payload  JSON,
-  updated_at_utc  INTEGER NOT NULL DEFAULT(unixepoch('now','subsec') * 1000),
-  botevent_row_id INTEGER NOT NULL,
-  CHECK (create_payload  IS NULL OR json_valid(create_payload)),
-  CHECK (modify_payloads IS NULL OR json_valid(modify_payloads)),
-  CHECK (cancel_payload  IS NULL OR json_valid(cancel_payload))
+    venue_id        TEXT NOT NULL,
+    wallet          TEXT NOT NULL,
+    order_id        TEXT NOT NULL,
+    md              TEXT NOT NULL,
+    action_kind     TEXT NOT NULL CHECK(action_kind IN ('create','modify','cancel')),
+    create_payload  JSON,
+    modify_payloads JSON NOT NULL DEFAULT (CAST('[]' AS BLOB)),
+    cancel_payload  JSON,
+    payload_type    TEXT NOT NULL,
+    payload_blob    BLOB NOT NULL,
+    updated_at_utc  INTEGER NOT NULL DEFAULT(unixepoch('now','subsec') * 1000),
+    botevent_row_id INTEGER NOT NULL,
+    PRIMARY KEY (venue_id, wallet, order_id),
+    FOREIGN KEY(venue_id) REFERENCES venues(id) ON DELETE CASCADE,
+    CHECK (create_payload  IS NULL OR json_valid(create_payload)),
+    CHECK (modify_payloads IS NULL OR json_valid(modify_payloads)),
+    CHECK (cancel_payload  IS NULL OR json_valid(cancel_payload))
 );

+CREATE INDEX IF NOT EXISTS idx_hyperliquid_submissions_md
+    ON hyperliquid_submissions(md);
+
 CREATE TABLE IF NOT EXISTS hyperliquid_status_history (
-    id              INTEGER PRIMARY KEY AUTOINCREMENT,
-    md              TEXT    NOT NULL,
-    status          BLOB    NOT NULL,
-    recorded_at_utc INTEGER NOT NULL DEFAULT(unixepoch('now','subsec') * 1000)
+    venue_id        TEXT NOT NULL,
+    wallet          TEXT NOT NULL,
+    order_id        TEXT NOT NULL,
+    md              TEXT NOT NULL,
+    payload_type    TEXT NOT NULL,
+    payload_blob    BLOB NOT NULL,
+    recorded_at_utc INTEGER NOT NULL DEFAULT(unixepoch('now','subsec') * 1000),
+    PRIMARY KEY (venue_id, wallet, order_id, recorded_at_utc),
+    FOREIGN KEY(venue_id, wallet, order_id) REFERENCES hyperliquid_submissions(venue_id, wallet, order_id) ON DELETE CASCADE


Add migration for new venue columns

The schema now expects hyperliquid_submissions, hyperliquid_status_history, and scaled_orders to include venue_id, wallet, order_id, and typed payload columns, but the DDL still uses CREATE TABLE IF NOT EXISTS. On any existing database these statements will no-op and leave the old layout in place, while the regenerated queries immediately reference the new columns (UpsertHyperliquidCreate, InsertScaledOrder, etc.). Upgrading without manually dropping the tables will therefore cause runtime “no such column” errors. Consider adding an explicit migration/alter or recreating the tables so the new schema is applied automatically.

Useful? React with 👍 / 👎.

terwey · 2025-11-02T17:04:57Z

@codex based on the spec in specs/multi_venue_emission.adoc and the rename of metadata to orderid (git history shows this) fix the merge conflicts for merging codex/update-database-schema-and-queries-for-venues into codex/investigate-multi-wallet-support-for-hyperliquid

chatgpt-codex-connector · 2025-11-02T17:10:17Z

Summary

Dropped existing Hyperliquid submission/status tables before recreating them so the venue-aware schema (including payload metadata) is applied during upgrades. storage/sqlc/schema.sqlL75-L111
Dropped and recreated the scaled_orders table to ensure migrations pick up the composite venue/wallet/order identifiers required by the new schema. storage/sqlc/schema.sqlL173-L198

Testing

⚠️ go test ./... (skipped: command terminated because required module downloads are unavailable in this environment)

View task →

terwey · 2025-11-02T21:45:50Z

@codex please investigate why the go tests are failing and fix the root cause, I suspect the changes conflict. Check the ./specs dir first if you have questions.

chatgpt-codex-connector · 2025-11-02T21:56:38Z

Summary

Reworked scaled_orders to use a (venue_id, wallet, order_id) primary key and enforced a foreign key back to hyperliquid_submissions, aligning persistence with venue-aware identifiers. storage/sqlc/schema.sqlL173-L195
Regenerated sqlc definitions and helpers to drop the legacy auto-increment ID, consume order ID lists, and keep scaled-order conversions aligned with the composite key data coming back from the database. storage/sqlc/queries.sqlL557-L866 storage/sqlcgen/models.goL58-L75 storage/order_scalers.goL39-L54
Updated storage naming to talk about order identifiers instead of metadata when listing Hyperliquid submissions and deal histories. storage/storage.goL510-L557

Testing

✅ go generate ./gen/storage
⚠️ go test ./storage -run Test -count=0 (terminated because Go module downloads hang in this environment)

View task →

terwey · 2025-11-02T22:21:46Z

Summary

Ensured scaled order audits always persist the original 3Commas order identifier and added shared converters that map the new sqlc row wrappers (including legacy “#”-suffixed records) back into storage models.

Reintroduced an autoincrement primary key on scaled_orders and regenerated the sqlc query/result types that feed storage readers and audit lookups.

Updated the order log assembly to reuse the shared converter when emitting scaled order audit entries through the API.

terwey · 2025-11-02T22:22:29Z

@codex please provide a conventional commit + summary for all changes in this PR for the merge commit

chatgpt-codex-connector · 2025-11-02T22:24:21Z

Conventional Commit
feat(storage): add venue-aware hyperliquid persistence

Summary

Added venue registry and bot assignment tables while rebuilding Hyperliquid submission/status and scaled order tables around (venue_id, wallet, order_id) identifiers and payload metadata so upgrades apply the new schema automatically. storage/sqlc/schema.sqlL49-L199
Regenerated sqlc queries and models to accept venue-scoped arguments, emit payload metadata, and return composite scaled-order rows for audits and deal views. storage/sqlc/queries.sqlL527-L888 storage/sqlcgen/models.goL11-L139
Updated storage logic to populate default venue and wallet identifiers, persist typed payload blobs for Hyperliquid submissions/statuses, and translate the new scaled-order results for streaming and API responses. storage/storage.goL328-L607 storage/order_scalers.goL330-L648

Testing

Not run (not requested)

View task →

terwey added the codex label Nov 1, 2025 — with ChatGPT Codex Connector

chatgpt-codex-connector bot reviewed Nov 1, 2025

View reviewed changes

terwey force-pushed the codex/investigate-multi-wallet-support-for-hyperliquid branch from 752edc0 to f628030 Compare November 2, 2025 16:59

terwey added 4 commits November 2, 2025 21:49

feat(storage): add venue-aware schema changes

8cb9475

fix(storage): align hyperliquid data with venues

df38a12

fix(storage): recreate hyperliquid tables for venue keys

38d681a

refactor: replace metadata identifiers with orderid

e9f6c00

terwey force-pushed the codex/update-database-schema-and-queries-for-venues branch from ea385fd to e9f6c00 Compare November 2, 2025 21:40

terwey added 2 commits November 2, 2025 23:00

fix(storage): align scaled orders with venue keys

4fc33c9

fix(storage): restore scaled order audit retrieval

b13be88

terwey merged commit b28522e into codex/investigate-multi-wallet-support-for-hyperliquid Nov 2, 2025
1 check passed

terwey deleted the codex/update-database-schema-and-queries-for-venues branch November 2, 2025 22:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add venue-aware storage schema #53

feat: add venue-aware storage schema #53

Uh oh!

terwey commented Nov 1, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

terwey commented Nov 1, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Nov 1, 2025

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

chatgpt-codex-connector bot commented Nov 2, 2025

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

chatgpt-codex-connector bot commented Nov 2, 2025

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

chatgpt-codex-connector bot commented Nov 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	MultiplierUpdatedBy: params.MultiplierUpdatedBy,
	CreatedAtUtc: createdAt.UTC().UnixMilli(),
	SubmittedOrderID: params.SubmittedOrderID,
	Skipped: boolToInt(params.Skipped),
	SkipReason: params.SkipReason,

	func (s *Storage) ListScaledOrdersByMetadata(ctx context.Context, md metadata.Metadata) ([]ScaledOrderAudit, error) {
	s.mu.Lock()
	defer s.mu.Unlock()

	rows, err := s.queries.ListScaledOrdersByMetadata(ctx, md.Hex())

	params := sqlcgen.InsertHyperliquidStatusParams{
	Md: md.Hex(),
	Status: raw,
	RecordedAtUtc: time.Now().UTC().UnixMilli(),
	}

feat: add venue-aware storage schema #53

feat: add venue-aware storage schema #53

Uh oh!

Conversation

terwey commented Nov 1, 2025

Summary

Testing

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

terwey commented Nov 1, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Nov 1, 2025

Choose a reason for hiding this comment

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

chatgpt-codex-connector bot commented Nov 2, 2025

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

chatgpt-codex-connector bot commented Nov 2, 2025

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

terwey commented Nov 2, 2025

Uh oh!

chatgpt-codex-connector bot commented Nov 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants