-
Notifications
You must be signed in to change notification settings - Fork 0
fix: resolve multi-venue reconciliation and memory issues #78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: resolve multi-venue reconciliation and memory issues #78
Conversation
This commit addresses three bugs related to multi-venue support: Bug #2 (Medium): Take Profit Reconciliation Doesn't Fan Out - Changed DealSnapshot.ActiveTakeProfit from *OrderSnapshot to []OrderSnapshot - Updated snapshot building to collect ALL active take-profits across venues - Modified ReconcileTakeProfits to iterate and reconcile each venue independently - Added cancelTakeProfitBySnapshot and reconcileActiveTakeProfitBySnapshot helpers - Location: filltracker/service.go:156-282 Bug #3 (Medium): Fill Tracker Unbounded Memory Growth - Added CleanupStaleDeals method to remove completed/inactive deals - Integrated periodic cleanup (every 10 minutes) in main event loop - Removes deals with no activity for >1 hour and all orders complete - Location: filltracker/service.go + cmd/recomma/main.go Bug #4 (Low): Replay Logic Edge Case - Documented edge case where venue additions mid-deal may miss replays - Added inline comment suggesting periodic reconciliation for production - Location: engine/engine.go:286-292 The take-profit fan-out fix ensures that when a deal has orders across multiple venues, all venues' take-profits are reconciled correctly, preventing stale orders on non-primary venues. The memory cleanup prevents unbounded growth of the fill tracker's in-memory state by periodically removing fully completed deals.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Orders []OrderSnapshot | ||
|
|
||
| ActiveTakeProfit *OrderSnapshot | ||
| ActiveTakeProfits []OrderSnapshot | ||
| LastTakeProfitEvent *tc.BotEvent |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update engine references after ActiveTakeProfit rename
The struct now exposes ActiveTakeProfits []OrderSnapshot, but callers such as engine.adjustActionWithTracker still access snapshot.ActiveTakeProfit. Building ./engine fails with snapshot.ActiveTakeProfit undefined, so the rename here needs corresponding changes wherever the snapshot is consumed.
Useful? React with 👍 / 👎.
filltracker/service.go
Outdated
| if len(snapshot.ActiveTakeProfits) > 0 { | ||
| for _, tp := range snapshot.ActiveTakeProfits { | ||
| s.reconcileActiveTakeProfitBySnapshot(ctx, submitter, snapshot, tp, desiredQty) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ReconcileTakeProfits resizes each venue to total net qty
Inside ReconcileTakeProfits the loop passes the same desiredQty (the aggregate deal net size across all venues) to every reconcileActiveTakeProfitBySnapshot call. With multiple active take profits—one per venue—each order is resized to the full aggregate position rather than its venue’s portion, so a two-venue deal holding 1 unit each will try to sell 2 units on every venue. This duplicates the take‑profit size and can over-liquidate positions.
Useful? React with 👍 / 👎.
Updated all test assertions to use the new ActiveTakeProfits slice instead of the single ActiveTakeProfit pointer. Changes include: - Replace NotNil checks with Len(slice, 1) assertions - Replace Nil checks with Empty(slice) assertions - Access first element via [0] where needed
Updated adjustActionWithTracker to iterate through all active take-profits when checking if a take-profit already matches the desired position. This ensures proper handling of multi-venue scenarios where multiple take-profits may exist.
|
@codex review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Codex Review
recomma/filltracker/service.go
Lines 232 to 234 in 08680ac
| modify := adapter.ToModifyOrderRequest(snapshot.Currency, recomma.BotEvent{BotEvent: *evt}, oid) | |
| modify.Order.Size = desiredQty | |
| modify.Order.ReduceOnly = true |
Each active take-profit is reconciled by calling reconcileActiveTakeProfitBySnapshot with snapshot.Position.NetQty, which is the aggregate net quantity across all venues. Inside that helper the modify request always sets modify.Order.Size = desiredQty. When a deal spans multiple venues, this causes every venue’s reduce-only order to be resized to the entire net position rather than that venue’s share, doubling or tripling the exit size and overshooting the held inventory. The size should be derived from the specific tp/venue rather than the global net.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
filltracker/service.go
Outdated
| // Reconcile each venue's take-profit independently | ||
| if len(snapshot.ActiveTakeProfits) > 0 { | ||
| for _, tp := range snapshot.ActiveTakeProfits { | ||
| s.reconcileActiveTakeProfitBySnapshot(ctx, submitter, snapshot, tp, desiredQty) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing take-profit creation for venues without an active order
The reconciliation loop only invokes ensureTakeProfit when len(snapshot.ActiveTakeProfits) == 0. If a deal has at least one active take-profit but some venues are missing theirs, this branch never runs and no new order is created for the missing venues, so the original multi‑venue fan-out bug persists. The logic should detect venues lacking an active take-profit and call ensureTakeProfit for them instead of skipping whenever one venue already has a live order.
Useful? React with 👍 / 👎.
Critical fixes for multi-venue take-profit reconciliation: Issue #1 (P1): Each venue's take-profit was being sized to the GLOBAL net position instead of that specific venue's position. This caused oversized exit orders (e.g., if venue A had 60 and venue B had 40, both would try to exit 100, totaling 200 and overshooting inventory). Fix: Added calculateVenuePositions() to compute net position per venue by summing filled buy/sell quantities for each venue identifier. Each venue's take-profit is now correctly sized to match only its position. Issue #2 (P1): ensureTakeProfit was only called when NO active take-profits existed. If one venue had an active TP, other venues without TPs would never get them created, leaving the multi-venue fan-out incomplete. Fix: Changed logic to iterate through ALL venue positions and reconcile or create take-profits for each venue independently. Now checks per-venue: - If venue has position but no TP: create one - If venue has position and TP: reconcile size - If venue has no position but TP exists: cancel it Location: filltracker/service.go:155-215
Critical fix for position calculation: Problem: calculateVenuePositions was using full OrderIdentifier (including OrderId with specific BotEventID) as map key. This created separate entries for each order even if they were from the same venue. For example: - baseOrder (BotEventID=1) at venue A: +5 - closeOrder (BotEventID=3) at venue A: -5 Result: Two separate map entries instead of one aggregated position of 0 Also, activeTPs map couldn't match against venuePositions because the OrderIds differed (TP has its own BotEventID). Fix: 1. Added venueKey struct to key positions by venue+wallet only 2. Modified calculateVenuePositions to aggregate all orders per venue 3. Changed activeTPs lookup to use venueKey instead of full OrderIdentifier 4. Added cleanup loop to cancel TPs for venues with no position This ensures: - Orders from same venue aggregate correctly (5 buy + 5 sell = 0 net) - TPs can be matched to their venue's position for reconciliation - TPs without positions get canceled - Each venue's TP is sized to that venue's specific holdings Location: filltracker/service.go:155-247
Test failures showed that when creating TPs for venues, the code was using base order OrderIds instead of the take-profit's OrderId. Problem: In venuePositions map, identifiers come from non-TP orders (base orders, safety orders, etc). When creating a TP for a venue without one, passing this identifier meant using the wrong OrderId (e.g., base order's BotEventID=1 instead of TP's BotEventID=2). Fix: Added buildTakeProfitIdentifier() helper that: 1. Searches snapshot for any TP order to get the correct TP OrderId 2. Falls back to loading TP metadata from storage 3. Constructs identifier with target venue+wallet but TP's OrderId 4. Returns nil if no TP metadata exists (lets ensureTakeProfit handle lookup) This ensures: - Created TPs use the correct take-profit OrderId - Each venue gets a TP with matching OrderId but venue-specific venue+wallet - Tests expecting specific TP OrderIds now pass Fixes failing tests: - TestUpdateStatusIgnoresOlderTimestamps/recreates_missing_take_profit - TestEnsureTakeProfitRecreatesAfterStaleSubmission Location: filltracker/service.go:211-238
|
@codex review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // calculateVenuePositions computes the net position per venue from filled orders. | ||
| // Returns a map keyed by venue+wallet (not including OrderId) to properly aggregate | ||
| // all orders for the same venue. | ||
| func (s *Service) calculateVenuePositions(snapshot DealSnapshot) map[recomma.OrderIdentifier]float64 { | ||
| // First, aggregate by venue+wallet only | ||
| venuePositions := make(map[venueKey]float64) | ||
| venueIdentifiers := make(map[venueKey]recomma.OrderIdentifier) | ||
|
|
||
| for _, order := range snapshot.Orders { | ||
| if order.ReduceOnly { | ||
| continue // Skip take-profits themselves | ||
| } | ||
|
|
||
| key := venueKey{venue: order.Identifier.VenueID, wallet: order.Identifier.Wallet} | ||
|
|
||
| // Track one identifier per venue for result map (use any order's identifier) | ||
| if _, exists := venueIdentifiers[key]; !exists { | ||
| venueIdentifiers[key] = order.Identifier | ||
| } | ||
|
|
||
| if order.Side == "B" || strings.EqualFold(order.Side, "BUY") { | ||
| venuePositions[key] += order.FilledQty | ||
| } else { | ||
| venuePositions[key] -= order.FilledQty | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Include reduce-only fills when computing venue net positions
The new calculateVenuePositions helper skips every order with ReduceOnly set, so per-venue net quantities are calculated using only buy/sell orders that are not take-profits. Filled reduce-only orders are exactly the sells that flatten a long position. When they are ignored, a venue that has already exited via a filled take-profit will still report a positive net quantity (e.g. buy 100, TP sell 100 → venueNetQty remains 100). ReconcileTakeProfits will treat that venue as still long and recreate/correct take-profit orders indefinitely even though the position is flat. Previously the logic relied on snapshot.Position.NetQty, which already included reduce-only fills, so this regression can cause repeated unnecessary order submissions after a deal closes.
Useful? React with 👍 / 👎.
Critical bug: calculateVenuePositions was skipping ALL reduce-only orders, including filled take-profits. This caused venues with fully-exited positions to still report positive net quantities, leading to infinite TP recreation. Example of the bug: Buy order: +100 units (filled) TP order: -100 units (filled, reduce-only) WRONG (before): Skip TP because reduce-only → venueNetQty = +100 ReconcileTakeProfits sees +100, creates new TP Next cycle: still +100, creates another TP (infinite loop!) CORRECT (now): Include TP filled qty → venueNetQty = 0 ReconcileTakeProfits sees 0, cancels any active TPs The confusion was thinking we should exclude TPs from position calculation, but we should only exclude UNFILLED quantities. Using FilledQty automatically handles this: - Filled TPs: FilledQty > 0, correctly reduces position - Active TPs: FilledQty = 0 or partial, doesn't affect net calculation This matches the original behavior where snapshot.Position.NetQty included all fills (including reduce-only). Location: filltracker/service.go:246-278
|
@codex review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return recomma.Action{Type: recomma.ActionNone, Reason: "skip take-profit: position flat"}, false | ||
| } | ||
|
|
||
| active := snapshot.ActiveTakeProfit | ||
| if !skipExisting && active != nil && active.ReduceOnly && nearlyEqual(active.RemainingQty, desiredQty) { | ||
| logger.Debug("take profit already matches position", | ||
| slog.Float64("desired_qty", desiredQty), | ||
| slog.Float64("existing_qty", active.RemainingQty), | ||
| ) | ||
| return recomma.Action{Type: recomma.ActionNone, Reason: "take-profit already matches position"}, false | ||
| // Check if any active take-profit already matches the desired position | ||
| if !skipExisting && len(snapshot.ActiveTakeProfits) > 0 { | ||
| for _, active := range snapshot.ActiveTakeProfits { | ||
| if active.ReduceOnly && nearlyEqual(active.RemainingQty, desiredQty) { | ||
| logger.Debug("take profit already matches position", | ||
| slog.Float64("desired_qty", desiredQty), | ||
| slog.Float64("existing_qty", active.RemainingQty), | ||
| slog.String("venue", active.Identifier.Venue()), | ||
| ) | ||
| return recomma.Action{Type: recomma.ActionNone, Reason: "take-profit already matches position"}, false | ||
| } | ||
| } | ||
| } | ||
|
|
||
| switch action.Type { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Compare take-profits against per-venue quantities
After changing DealSnapshot.ActiveTakeProfit to the slice ActiveTakeProfits, adjustActionWithTracker still derives desiredQty from the global snapshot.Position.NetQty and only checks whether any active take‑profit has a remaining quantity equal to that global total. In a multi‑venue deal where, for example, venue A holds 60 units and venue B holds 40 units, each venue now correctly tracks its own 60/40 take profits. However none of those entries will ever equal the aggregate 100, so this method will always fall through and emit a new order sized to the entire deal when any venue’s take‑profit event is processed, oversizing that venue’s exit order until reconciliation later cancels it. The skip check should compare against the venue-specific position (or otherwise aggregate per venue) instead of the global net.
Useful? React with 👍 / 👎.
Critical P1 bug: adjustActionWithTracker was comparing individual venue
take-profits against the global Position.NetQty, causing mismatches in
multi-venue scenarios.
Problem:
Venue A: 60 units position, TP remaining = 60
Venue B: 40 units position, TP remaining = 40
Global: 100 units net
adjustActionWithTracker checks:
Does 60 == 100? No
Does 40 == 100? No
Falls through to sizing logic → tries to set TP.Size = 100
Oversizes venue TPs until ReconcileTakeProfits corrects them
Root cause: adjustActionWithTracker is called BEFORE fanning out to venues,
so it doesn't have per-venue position data. It only sees global net qty.
Solution: Detect multi-venue scenario (multiple active TPs) and skip the
fast-path sizing logic. Let ReconcileTakeProfits handle all multi-venue
sizing correctly with its per-venue position calculation.
Behavior:
- Single venue (0-1 active TPs): Fast path applies global sizing ✓
- Multi-venue (2+ active TPs): Fast path defers to reconciliation ✓
- ReconcileTakeProfits: Always uses correct per-venue sizing ✓
This prevents the engine from oversizing venue TPs while still allowing
the fast path to work for single-venue deals.
Location: engine/engine.go:524-604
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
bb4da9f
into
codex/extend-cmd/recomma-to-handle-multiple-venues
This commit addresses three bugs related to multi-venue support:
Bug #2 (Medium): Take Profit Reconciliation Doesn't Fan Out
Bug #3 (Medium): Fill Tracker Unbounded Memory Growth
Bug #4 (Low): Replay Logic Edge Case
The take-profit fan-out fix ensures that when a deal has orders across multiple venues, all venues' take-profits are reconciled correctly, preventing stale orders on non-primary venues.
The memory cleanup prevents unbounded growth of the fill tracker's in-memory state by periodically removing fully completed deals.