Initial version of Tx Queue#53
Conversation
There was a problem hiding this comment.
cubic analysis
1 issue found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/allora_sdk/rpc_client/tx_queue.py">
<violation number="1" location="src/allora_sdk/rpc_client/tx_queue.py:260">
P1: FIFO tiebreaking is broken: `_sequence_counter` is tracked but never included in the sort key, so items with identical deadline/priority/`created_at` have non-deterministic ordering. The `sequence_no` should be appended to the sort-key tuple as the final tiebreaker.
According to linked Linear issue ENGN-5614, the required ordering is `deadline_at -> priority -> FIFO`.</violation>
</file>
Linked issue analysis
Linked issue: ENGN-5614: SDK py: create submission queue component
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | priority/deadline/FIFO ordering | Sort key and scheduling implement deadline>priority>FIFO |
| ✅ | per-account strict sequencing | Single account worker ensures one-consumer sequencing |
| ✅ | concurrent progress across different accounts | Different account workers run concurrently and tested |
| ✅ | bounded retries with backoff/jitter | Backoff calc + jitter and retry loop present |
| ✅ | sequence invalidation + refetch on mismatch | SequenceState.invalidate + adapter.fetch_sequence used on mismatch |
| ❌ | TxManager exposes queue-backed enqueue path without breaking existing submit_transaction(...) | No TxManager changes or integration code in diffs |
| ❌ | At least worker/reputer path can submit through queue with priority/deadline | No worker/reputer integration present in PR |
| ✅ | Stop semantics are defined and tested (cancel_pending=True/False) | stop(cancel_pending) implemented and both behaviors tested |
| ✅ | Unit tests cover success/failure paths and typed exceptions | Comprehensive unit tests for ordering, retries, mismatch, timeouts |
| ❌ | Docs include usage guidance and migration notes | No documentation files or docs changes present |
| ✅ | Typed adapter contract (Protocol) for fetch_sequence/submit/classify_error | TxQueueAdapter Protocol defines required adapter API |
| ✅ | Awaitable pending handle compatible with current async call patterns | PendingQueueTx implements awaitable wait and tests await it |
| ✅ | Starvation mitigation via age-based priority boost | compute_effective_priority implements age-based boosts |
| ✅ | Scheduler + per-account workers implemented | Scheduler loop and per-account worker tasks exist |
| ✅ | Sequence cache with atomic invalidate/advance operations | SequenceState provides current_or_fetch, invalidate, advance |
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
2 issues found across 6 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/allora_sdk/__init__.py">
<violation number="1" location="src/allora_sdk/__init__.py:34">
P2: Module `__getattr__` doesn't cache resolved attributes. Every access to a lazy-loaded name (e.g., `allora_sdk.AlloraRPCClient`) re-enters `__getattr__`, repeating string comparisons and `import_module`/`getattr` calls. The standard PEP 562 pattern stores the result in `globals()` so `__getattr__` is bypassed on subsequent lookups.</violation>
</file>
<file name="src/allora_sdk/rpc_client/__init__.py">
<violation number="1" location="src/allora_sdk/rpc_client/__init__.py:24">
P2: Module-level `__getattr__` does not cache resolved attributes in `globals()`. Per the standard PEP 562 lazy-loading pattern, after resolving an attribute you should store it with `globals()[name] = val` so subsequent lookups go through normal attribute access and bypass `__getattr__`. Without this, repeated attribute access via `rpc_client.AlloraRPCClient` re-invokes `__getattr__` every time.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
3 issues found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/allora_sdk/rpc_client/tx_manager.py">
<violation number="1" location="src/allora_sdk/rpc_client/tx_manager.py:498">
P1: `_submit_via_queue` does not invalidate the gas price cache or escalate fees on `InsufficientFeesError`, unlike the old retry path which clears `_cached_gas_price`/`_gas_price_cache_time` and applies `fee_multiplier = 1.0 + attempt * 0.5`. Queue retries will repeatedly submit with the same stale gas price and base fee multiplier.</violation>
<violation number="2" location="src/allora_sdk/rpc_client/tx_manager.py:504">
P1: Queue-backed retries for `OutOfGasError` are ineffective: `gas_multiplier` is hardcoded to `1.0` with no escalation across attempts. The old `_attempt_submissions` path increases gas by 30% per attempt (`1.0 + attempt * 0.3`), but the queue adapter has no mechanism to escalate. Consider tracking the attempt number in the payload or the adapter, or passing it via the submission context.</violation>
</file>
<file name="src/allora_sdk/rpc_client/client_emissions.py">
<violation number="1" location="src/allora_sdk/rpc_client/client_emissions.py:45">
P2: Return type `Union[PendingTx, int]` is incorrect when `use_queue=True` — `submit_transaction` returns a `PendingQueueTx` in that case, which is a separate class with a different interface. The annotation should be updated to `Union[PendingTx, PendingQueueTx, int]` (or a shared protocol) for all three methods that now accept `use_queue`.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Code Review SummaryMulti-agent review — 5 reviewers + adversarial challenge Linear project: https://linear.app/alloralabs/project/pr-review-allora-networkallora-sdk-py-53-initial-version-of-tx-queue-c10b17cdd10d 🔴 Must Fix
🟡 Should Fix
🔵 Consider
💬 Additional Inline CommentsThese reference lines outside the diff and could not be posted as inline review comments.
|
| self.adapter.submit(item.payload, sequence), | ||
| timeout=item.timeout.total_seconds(), | ||
| ) | ||
| await self._sequence_state.advance(item.account_id) |
There was a problem hiding this comment.
🔴 On-chain errors (OutOfGas, InsufficientFees) consume the on-chain sequence but the queue doesn't advance or invalidate the cache. When _raise_for_status raises after wait_for_tx confirms block inclusion, the sequence was consumed on-chain. But _process_item only calls advance() after a fully successful submit() return — the exception propagates before advance is reached. The error is classified as RETRYABLE (not SEQUENCE_MISMATCH), so the cache is never invalidated. The next retry reuses the stale cached sequence, which immediately fails with AccountSequenceMismatchError, wasting an entire retry attempt.
Fix: advance the sequence cache (or invalidate it) for on-chain errors that consumed the sequence before retrying.
| async def close(self): | ||
| """Close client and cleanup resources.""" | ||
| logger.debug("Closing Allora client") | ||
| if self.tx_manager is not None: | ||
| await self.tx_manager.close(cancel_pending_queue=False) | ||
| if self.events: |
There was a problem hiding this comment.
🔴 self.events and self.grpc_client have no class-level None defaults. self.events is only set when websocket_url is provided; self.grpc_client is only set in the gRPC branch. Calling close() on a REST-only or no-websocket client will raise AttributeError. The PR adds AlloraWorker._cleanup → AlloraRPCClient.close(), making this a concrete production crash path.
[architecture, adversarial-a, adversarial-b — both verifiers confirmed as must-fix]
| async def close(self): | |
| """Close client and cleanup resources.""" | |
| logger.debug("Closing Allora client") | |
| if self.tx_manager is not None: | |
| await self.tx_manager.close(cancel_pending_queue=False) | |
| if self.events: | |
| async def close(self): | |
| """Close client and cleanup resources.""" | |
| logger.debug("Closing Allora client") | |
| if self.tx_manager is not None: | |
| await self.tx_manager.close(cancel_pending_queue=False) | |
| if getattr(self, 'events', None): | |
| await self.events.stop() | |
| if getattr(self, 'grpc_client', None): | |
| self.grpc_client.close() |
| else: | ||
| result = await asyncio.wait_for( | ||
| self.adapter.submit(item.payload, sequence), | ||
| timeout=item.timeout.total_seconds(), | ||
| ) | ||
| await self._sequence_state.advance(item.account_id) | ||
| if not item.handle._final_future.done(): | ||
| item.handle._final_future.set_result(result) |
There was a problem hiding this comment.
🔴 Successful submit followed by advance() failure silently loses the success result. If adapter.submit() (line 350) returns successfully but advance() (line 355) raises, the exception falls to except Exception, gets classified as FATAL, and set_exception is called on the future. The caller sees a failure for a transaction that was already included on-chain. The success is irrecoverably lost.
Fix: set the future's result immediately after submit() succeeds, before calling advance(). Wrap advance() in its own try/except that logs the error but does not overwrite the already-resolved future.
| else: | |
| result = await asyncio.wait_for( | |
| self.adapter.submit(item.payload, sequence), | |
| timeout=item.timeout.total_seconds(), | |
| ) | |
| await self._sequence_state.advance(item.account_id) | |
| if not item.handle._final_future.done(): | |
| item.handle._final_future.set_result(result) | |
| if item.timeout is None: | |
| result = await self.adapter.submit(item.payload, sequence) | |
| else: | |
| result = await asyncio.wait_for( | |
| self.adapter.submit(item.payload, sequence), | |
| timeout=item.timeout.total_seconds(), | |
| ) | |
| if not item.handle._final_future.done(): | |
| item.handle._final_future.set_result(result) | |
| try: | |
| await self._sequence_state.advance(item.account_id) | |
| except Exception: | |
| logger.warning(f"Failed to advance sequence cache for {item.account_id} after successful submit", exc_info=True) | |
| return |
| info = resp.info | ||
| sequence_number = info.sequence if sequence is None else sequence | ||
| logger.debug(f"Account info: seq={info.sequence}, num={info.account_number}") | ||
|
|
There was a problem hiding this comment.
🟡 Redundant account_info RPC on every broadcast + misleading debug log. account_number is stable for a wallet's lifetime — re-fetching it via RPC on every broadcast wastes a network round-trip per submission. Additionally, this log prints info.sequence (the chain's live value) rather than sequence_number (the value actually used for signing). After sequence-mismatch recovery these differ, actively misleading debugging.
[architecture, correctness, performance — all confirmed by both adversarial agents]
| return self._tx_queue | ||
|
|
||
| async def _submit_via_queue(self, payload: _QueuedSubmission, sequence: Optional[int]) -> TxResponse: | ||
| await self._pre_flight_checks() |
There was a problem hiding this comment.
🟡 _pre_flight_checks() is called unconditionally on every attempt including retries. This makes 2 RPC calls (auth_client.account() + bank_client.balance()) per attempt. Combined with the unconditional account_info call in _build_and_broadcast, each queue submission attempt makes at least 3 overhead RPCs. For N items with R retries each, this is N*(R+1)*3 extra calls.
[adversarial-a — both verifiers confirmed as should-fix]
| fee_tier=fee_tier, | ||
| ) | ||
| queue_handle = await tx_queue.enqueue( | ||
| request_id=f"tx-{self.parent_tx_id}", |
There was a problem hiding this comment.
🟡 Race condition: parent_tx_id read/increment spans two await points. Between reading at line 291 and incrementing at line 299, _ensure_tx_queue_started() and enqueue() both yield, allowing interleaving. Two concurrent callers can get the same request_id.
|
|
||
| @dataclass | ||
| class TxResponse: | ||
| code: int = 0 |
There was a problem hiding this comment.
🔵 Add a docstring to the fallback TxResponse. The fallback _RequestMessage above has a docstring explaining its purpose, but this TxResponse does not. Both exist for the same reason (test-environment compatibility without generated protos).
| code: int = 0 | |
| @dataclass | |
| class TxResponse: | |
| """Minimal TxResponse used when generated protos are unavailable.""" | |
| code: int = 0 | |
| raw_log: str = "" | |
| txhash: str = "" | |
| codespace: str = "" |
| @@ -195,7 +214,10 @@ async def delegate_stake( | |||
| fee_tier: FeeTier = FeeTier.STANDARD, | |||
There was a problem hiding this comment.
🔵 delegate_stake has no docstring despite receiving the same three queue parameters and return-type change as register and insert_worker_payload. The new parameters are completely undocumented here.
|
|
||
| def __init__(self) -> None: | ||
| self._sequences: Dict[str, int] = {} | ||
| self._locks: Dict[str, asyncio.Lock] = defaultdict(asyncio.Lock) |
There was a problem hiding this comment.
🔵 Add a comment explaining the Python 3.10+ safety of defaultdict(asyncio.Lock). This pattern diverges from the rest of the SDK (explicit asyncio.Lock() in __init__). It is correct for Python ≥3.10 since Lock() no longer binds to an event loop at construction time — but this is non-obvious and could be 'fixed' incorrectly by a future maintainer.
| self._locks: Dict[str, asyncio.Lock] = defaultdict(asyncio.Lock) | |
| # asyncio.Lock() is safe to construct lazily since Python 3.10 (no longer binds to a loop at construction time). | |
| self._locks: Dict[str, asyncio.Lock] = defaultdict(asyncio.Lock) |
|
|
||
| class PendingQueueTx(Generic[TResult]): | ||
| def __init__(self) -> None: | ||
| self.created_at = datetime.now() |
There was a problem hiding this comment.
🔵 self.created_at is assigned but never read anywhere in the codebase. QueueItem already tracks created_at using the injected now_fn. Remove this dead state or document its intended external use.
| self.created_at = datetime.now() | |
| self._final_future: asyncio.Future[TResult] = asyncio.get_running_loop().create_future() |
Purpose
Adds a Generic Transaction Submission Queue Component for the Python SDK.
This branch introduces a reusable queue with priority/deadline scheduling, per-account sequential processing, retry handling, and sequence-mismatch recovery (invalidate + refetch). It also adds focused unit tests for ordering, retries, deadline behavior, stop/drain semantics, and same-account vs cross-account execution.
Goal
Provide a foundation for queue-backed transaction submission while preserving backward compatibility for existing submission paths.
Properties
Priority-aware scheduling
Optional deadline-based dispatch/fail-fast behavior
Strict per-account sequencing
Sequence cache invalidation + refetch on mismatch
Retry/backoff for retryable failures
Typed queue errors (deadline exceeded, retries exhausted, queue stopped)
Async awaitable result handle
Configurable queue bounds (priority range, backoff, starvation boost)