Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix: grpclib error handling — wrong exception type + no transport recovery (DEVOP-595)#66

Draft
gh-allora wants to merge 6 commits into
devfrom
fix/grpc-error-handling
Draft

fix: grpclib error handling — wrong exception type + no transport recovery (DEVOP-595)#66
gh-allora wants to merge 6 commits into
devfrom
fix/grpc-error-handling

Conversation

@gh-allora

@gh-allora gh-allora commented May 13, 2026

Copy link
Copy Markdown

Fix grpclib error handling — wrong exception type + no transport recovery (DEVOP-595)

Parent diagnostic: DEVOP-593
Fixes: DEVOP-595
Sibling PR: see allora-offchain-node fix/rpc-silent-stall (DEVOP-594)

What this fixes

Same class of silent-stall problem the Go offchain-node had, but worse: the Python SDK never had a working RPC error-handling path to begin with.

Bug 1: except grpc.RpcError is dead code

tx_manager.py imported grpc (the grpcio library) and caught grpc.RpcError in two places — _estimate_gas and _get_tx. But the gRPC channel underneath is grpclib.client.Channel from the unrelated grpclib library. grpclib.GRPCError and grpclib.StreamTerminatedError inherit directly from Exception, NOT from grpc.RpcError.

Result: both except grpc.RpcError blocks were silent dead code for the entire lifetime of the SDK.

  • _estimate_gas: account-sequence-mismatch detection during gas simulation never fired. Real grpclib errors fell into the bare except Exception below and re-raised without being converted to AccountSequenceMismatchError. The retry loop never refreshed the sequence number for the next attempt.
  • _get_tx: TxNotFoundError detection "worked" only by accident — the bare except Exception fallback's substring match on str(e) caught it because str(GRPCError) happens to contain both "tx" and "not found". Load-bearing accident.

Bonus latent bug: if anything had actually entered the grpc.RpcError block, e.details() would have crashed with TypeError. grpclib's GRPCError.details is an attribute (defaulting to None), not a method.

Bug 2: No retry on transport failure

The retry loop in _attempt_submissions catches specific cosmos-level errors but the bare except Exception on line 340 sets the future to the error and returns. One grpclib.StreamTerminatedError or GRPCError(UNAVAILABLE) mid-broadcast kills the whole tx with zero retries and no channel re-dial.

Bug 3: grpclib.Channel has no keepalive, no reconnect, no pool

Verified empirically against grpclib==0.4.8:

Channel public methods: close, request
contains 'keepalive': False / 'reconnect': False / 'pool': False

When Cloudflare half-closes the HTTP/2 stream, the Channel object is dead and there's no recovery — unless the caller dials a new one.

Pre-existing build break (prep commit)

pyproject.toml on dev was unbuildable: dependabot PR #64 introduced "requests *" which is invalid PEP 508 (a bare * is not a version specifier). uv sync / pip install -e . both failed. The same dep was also listed correctly on a different line, so just removing the broken entry restores the build. Isolated as its own commit so it can be cherry-picked or reverted independently.

Changes (5 commits)

2fa9f9b fix(deps): remove invalid 'requests *' PEP 508 specifier from pyproject.toml
a90eaff fix(tx_manager): catch grpclib.GRPCError instead of grpc.RpcError (dead code)
55532b9 feat(tx_manager): GRPCTransportError + retry-loop handler for transport failures
bd4cd0a feat(client): wire AlloraRPCClient._reconnect_grpc as the TxManager recovery hook
13401c3 fix(tx_manager): align log levels with retry semantics + surface grpc_status
  1. Prep: fix pyproject.toml so make dev works.
  2. Dead-code fix: replace import grpc with from grpclib.exceptions import GRPCError. Fix e.details()e.message. Add regression tests pinning down GRPCError ∉ grpc.RpcError and GRPCError.message is a string attribute (not callable).
  3. GRPCTransportError + retry-loop handler:
    • New exception class for transport failures: StreamTerminatedError, GRPCError(UNAVAILABLE), or GRPCError whose message contains "connection reset by peer", "connection timed out", or "unexpected EOF".
    • New _is_grpclib_transport_error(exc) classifier — single source of truth for what counts as transport-class. Mirrors lib/errors.go isGRPCTransportError() from the Go-side fix.
    • _attempt_submissions retry loop gains an except GRPCTransportError branch that retries within the existing retry budget, calling _on_transport_failure() between attempts.
    • Defensive net for raw GRPCError/StreamTerminatedError that escapes from broadcast_tx or account_info paths.
  4. Reconnect wiring: AlloraRPCClient now has _reconnect_grpc() that closes the dead grpclib.Channel, dials a fresh one, and re-binds all stub references — including TxManager.tx_client, auth_client, bank_client, feemarket_client — in place so the existing instances keep working. TxManager.__init__ accepts an optional reconnect_callback. The reconnect path goes through a factored-out _build_clients() method so the initial-setup path and reconnect path can't drift.
  5. Log levels & grpc_status surface: retry-attempt logs promoted from debug to warning (so dashboards see the retries, not just the final outcome). grpc_status=<Status.name> added to simulation error logs so dashboards can group by class without substring-matching.

Verification done locally

uv run pytest tests/ --ignore=tests/test_api_client_integration.py
======================== 16 passed in 0.48s ========================

3 pre-existing tests + 13 new tests across:

  • _get_tx conversion paths (NOT_FOUND → TxNotFoundError; UNAVAILABLE → GRPCTransportError; connection reset by peerGRPCTransportError; StreamTerminatedErrorGRPCTransportError; non-transport non-not-found propagates untouched; happy path)
  • _is_grpclib_transport_error classifier (4 positive + 5 negative cases)
  • _on_transport_failure hook (invokes callback exactly once; clean no-op without callback; swallows callback failures)
  • Regression assertions: GRPCError ∉ grpc.RpcError and GRPCError.message is str not callable

Integration tests not run (require testnet API access).

Stub reference caveat

Callers that captured a direct Stub reference BEFORE a reconnect (e.g. my_stub = client.tx_manager.tx_client; ...; await my_stub.broadcast_tx(...)) will still hit the dead channel. New invocations through client.tx, client.bank, etc. always hit the latest channel because the Client* wrappers reference TxManager / query_client by attribute and the reconnect path mutates those in place. Documented in AlloraRPCClient._reconnect_grpc docstring.

Concurrency note

Stub references on TxManager are reassigned without a lock. Safe under the current design because submit_transaction is sequential per TxManager instance (no parallel _attempt_submissions loops on the same manager). If that invariant ever changes, the stub swap will need to move under a lock or be made copy-on-write.

Out of scope (tracked in parent DEVOP-593)

  • Multi-endpoint URL list in AlloraNetworkConfig (deferred — cosmpy.NetworkConfig only accepts a single URL)
  • Switching from grpclib to grpcio (deferred — would require regenerating all betterproto-based client stubs; the proto pipeline is fragile)
  • Broader pyproject.toml cleanup (triple-listed requests, discontinued requests-async dep)

Recommended merge order

Merge the Go-side fix first (DEVOP-594). Validate on a testnet canary pod for 24h. If the diagnosis is correct, the same pattern works for this SDK and we can merge with confidence.


Summary by cubic

Fixes gRPC error handling in the Python SDK by catching grpclib exceptions, classifying transport failures, and auto‑reconnecting the channel to retry broadcasts, preventing silent stalls and mid‑broadcast failures (DEVOP-595). Also removes an invalid requests * dependency and updates uv.lock to restore installs and CI.

  • Bug Fixes

    • Catch grpclib.GRPCError/StreamTerminatedError instead of grpc.RpcError; use .message (not .details()).
    • Add GRPCTransportError and _is_grpclib_transport_error(...) to treat UNAVAILABLE, stream resets, timeouts, and unexpected EOF as transport failures.
    • Retry transport failures in _attempt_submissions and call AlloraRPCClient._reconnect_grpc() to redial grpclib.Channel and rebind stubs in place.
    • Promote retry logs to warning and include grpc_status=<Status> in error logs for easier triage.
    • Convert tx lookup NOT_FOUND to TxNotFoundError and restore sequence‑mismatch handling during simulation.
    • Regenerate uv.lock after removing the invalid dependency to keep uv sync --locked green.
  • Migration

    • Do not cache direct gRPC stub instances across reconnects; use client.tx, client.bank, etc., which always point to the current channel.

Written for commit 11a3c40. Summary will update on new commits. Review in cubic

gh-allora added 6 commits May 13, 2026 17:22
…ct.toml

Pre-fix: 'dev' branch was unbuildable. `uv sync` / `pip install -e .` both
fail with:

  ValueError: Dependency #7 of field `project.dependencies` is invalid:
  Expected semicolon (after name with no version specifier) or end
      requests *
               ^

The bad entry was introduced by dependabot PR #64 ('Bump requests from
2.32.4 to 2.33.0'); the merge ended up with TWO 'requests' entries — line
32 'requests *' (invalid PEP 508 — a bare asterisk is not a valid version
specifier) and line 39 'requests' (well-formed and equivalent to what was
intended). Removing the broken line 32 leaves the well-formed one in
place; no behaviour change.

This prep commit is bundled into the fix/grpc-error-handling branch
because the editable install needs to succeed before any test in this
PR can be run. It's deliberately isolated as its own commit so it can be
cherry-picked or reverted independently of the rest of the gRPC error
handling work.

NOTE: 'requests' is still also referenced transitively via 'requests-async'
on line 42, but 'requests-async' itself is a discontinued package (no
release since 2019). A broader cleanup of the requests/requests-async/
aiohttp dependency situation should be tracked separately.
…ad code)

tx_manager.py imported 'grpc' (the grpcio library) and caught grpc.RpcError
in two places — _estimate_gas and _get_tx. But the gRPC channel underneath
is grpclib.client.Channel, from the unrelated 'grpclib' library. grpclib's
exception types (GRPCError, StreamTerminatedError, ProtocolError) all
inherit directly from Exception and are NOT subclasses of grpc.RpcError.

Result: both except blocks were silent dead code for the entire lifetime
of the SDK.

Practical impact:
  - _estimate_gas: 'account sequence mismatch' detection during gas
    simulation never fired. Real grpclib errors fell into the bare
    'except Exception' below it and re-raised without being converted to
    AccountSequenceMismatchError. The retry loop in _attempt_submissions
    then never refreshed the sequence number for the next attempt.
  - _get_tx: TxNotFoundError detection 'worked' only by accident — the
    bare 'except Exception' fallback's substring match on str(e) caught it
    because str(GRPCError) happens to contain both 'tx' and 'not found'.
    A genuinely tx-not-found-only check (e.g. 'tx' in details and 'not
    found' in details) was load-bearing on this accident.

Bonus latent bug: if anything had actually entered the grpc.RpcError
block, 'e.details()' would have crashed with TypeError. grpclib's
GRPCError.details is an attribute (defaulting to None), not a method.
The fix uses .message instead, which is the correct accessor for the
textual payload from the server.

Tests added: tests/test_tx_manager_grpclib_errors.py
  - 3 real-path tests for _get_tx (not-found conversion, propagation,
    happy path) using AsyncMock with grpclib.GRPCError side effects
  - 2 regression assertions pinning down grpclib semantics so a future
    'simplify the import' change doesn't accidentally re-introduce the
    bug:
      * GRPCError is NOT a subclass of grpc.RpcError
      * GRPCError.message is an attribute (str), not a callable
…rt failures

Adds explicit handling for grpclib transport-layer failures in the
_attempt_submissions retry loop. Pre-fix: any grpclib.StreamTerminatedError
or grpclib.GRPCError(Status.UNAVAILABLE) mid-broadcast fell through to the
bare 'except Exception' branch, which set the future to the error and
returned — zero retries, no recovery.

Changes:

1. New GRPCTransportError exception class. Raised when we observe one of
   the four transport-class failures (stream-terminated, UNAVAILABLE,
   'connection reset by peer', 'unexpected EOF'). Same scenarios that
   trigger node-switching in the Go offchain-node fix.

2. New _is_grpclib_transport_error(exc) helper. Classifies an arbitrary
   exception as a transport failure (positive: StreamTerminatedError,
   UNAVAILABLE, the substring set above; negative: NOT_FOUND, sequence
   mismatch, permission errors, RuntimeError, etc.). Single source of
   truth for what counts as transport-class.

3. _estimate_gas and _get_tx now wrap grpclib transport errors as
   GRPCTransportError before re-raising. Non-transport grpclib errors
   keep their existing semantics (NOT_FOUND → TxNotFoundError, account
   sequence mismatch → AccountSequenceMismatchError, others propagate).

4. _attempt_submissions retry loop now has two new branches:
   - 'except GRPCTransportError': retry within the existing retry budget,
     call _on_transport_failure() between attempts (the actual reconnect
     wiring lands in the next commit).
   - 'except (GRPCError, StreamTerminatedError)' defensive net: catches
     grpclib errors that escaped from the broadcast/auth_info path
     without going through the GRPCTransportError wrapper. Re-classifies
     via _is_grpclib_transport_error and routes to the same retry
     behaviour, or falls through to the existing fail-the-future
     handling for non-transport flavours.

5. _on_transport_failure() hook added as a no-op stub. Commit 3 will
   replace this with an actual channel close-and-redial. The no-op emits
   a WARNING-level log so the dead-channel-retry situation is at least
   visible until the wiring lands.

Tests added (5 new in tests/test_tx_manager_grpclib_errors.py):
  - get_tx converts UNAVAILABLE to GRPCTransportError
  - get_tx converts 'connection reset by peer' to GRPCTransportError
  - get_tx converts StreamTerminatedError to GRPCTransportError
  - get_tx still re-raises non-transport non-not-found errors untouched
    (test now uses PERMISSION_DENIED instead of UNAVAILABLE)
  - _is_grpclib_transport_error classification: 4 positive cases (incl.
    StreamTerminatedError and all 3 substring forms) and 5 negative
    cases pinning the boundary (NOT_FOUND, sequence mismatch, permission
    denied, RuntimeError, plain Exception)

All 13 existing+new unit tests pass. Integration tests not run (require
testnet API access).
…ecovery hook

Replaces the no-op stub from the previous commit with the actual
close-and-redial logic. When a transport-class failure is detected in
the retry loop, TxManager now invokes a callback that:

  1. Closes the current grpclib.Channel (best-effort; closing a dead
     channel can itself raise — caught and debug-logged so it doesn't
     mask the original failure)
  2. Dials a fresh Channel against the same host/port/ssl settings
  3. Constructs fresh QueryStub instances bound to the new channel
  4. Mutates the TxManager's stub references in place
     (tx_client, auth_client, bank_client, feemarket_client) so the
     existing TxManager instance keeps working without callers needing
     to know anything reconnected

Implementation:

  * client.py: factored the stub-construction logic out of __init__
    into _build_clients(). __init__ now calls it once; _reconnect_grpc()
    calls it again on demand. Splitting the logic keeps the initial-
    setup path and the reconnect path on a single code path so they
    can't drift.

  * client.py: new async _reconnect_grpc() method. REST-protocol-aware
    (no-op when the configured URL is REST-only — REST uses pooled HTTP
    and doesn't have the same dead-channel problem).

  * client.py: AlloraRPCClient passes self._reconnect_grpc as the
    reconnect_callback when constructing TxManager.

  * tx_manager.py: __init__ now accepts reconnect_callback. The hook
    swallows exceptions from the callback itself — a failed reconnect
    should not cascade into a failed submission, since the retry loop
    is going to try again anyway.

Concurrency note: TxManager's stub references are reassigned without a
lock. This is safe under the current design because submit_transaction
is sequential per TxManager instance (no parallel _attempt_submissions
loops on the same manager). If that invariant ever changes, the stub
swap will need to move under a lock or be made copy-on-write.

Stub reference caveat: callers that captured a direct Stub reference
BEFORE a reconnect (e.g. `my_stub = client.tx_manager.tx_client` then
later `await my_stub.broadcast_tx(...)`) will still hit the dead
channel. New invocations through self.tx / self.bank / etc. always hit
the latest channel because the Client* wrappers reference TxManager /
query_client by attribute, and we mutate those in place. Documented
in the _reconnect_grpc docstring.

Tests added (3 new):
  - _on_transport_failure invokes the configured callback exactly once
  - _on_transport_failure is a clean no-op when no callback is configured
  - _on_transport_failure swallows callback failures (does not propagate
    a reconnect-failed exception to the retry loop)

All 16 unit tests pass.
…_status

Two related changes:

1. Promote retry-attempt logs from debug to warning.

   The retry loop in _attempt_submissions logs at debug when retrying
   on OutOfGasError, InsufficientFeesError, AccountSequenceMismatchError,
   and TxTimeoutError. At debug, these were effectively invisible in
   production — alerting systems and operators couldn't see when a
   transaction was struggling and retrying. They could only see the
   final outcome (success or final error).

   Warning is the right level: we are not yet at error (there are
   retries remaining and they may succeed) but we are also not in the
   happy path. This mirrors the Go-side fix where the per-retry log
   in QueryDataWithRetry went info → warning.

   The final-attempt logs (already at error level) remain unchanged.
   The GRPCTransportError retry path added in the previous commit
   already logs at warning, so this brings the rest of the retry
   branches into consistency.

2. Surface grpclib.Status name in simulation error logs.

   When grpclib raises GRPCError, the .status field carries the
   semantic class of the failure (NOT_FOUND, UNAVAILABLE, INVALID_ARGUMENT,
   PERMISSION_DENIED, etc.). Including the status name as a structured-ish
   field (grpc_status=<name>) lets dashboards group failures by class
   without having to substring-match against the freeform error message.

   Mirrors what the Go-side fix does with HTTP status codes in
   triageHTTPStatusError — same intent (make the failure self-describing
   so alerting rules don't have to be clever).

All 16 unit tests still pass.
The earlier prep commit (`fix(deps): remove invalid 'requests *' PEP 508
specifier`) edited pyproject.toml but didn't refresh uv.lock. CI on PR #66
runs `uv sync --locked` which refused with:

  The lockfile at `uv.lock` needs to be updated, but --locked was provided.
  To update the lockfile, run `uv lock`.

This commit is the result of running `uv lock` against the updated
pyproject.toml — a single-line removal of the duplicate `requests`
entry from uv.lock's requires-dist section. No behavioral change.

Caught by self-review of PR #66 — would have failed CI on every push
otherwise.

@gh-allora gh-allora left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of fix/grpc-error-handling

Reviewed by Hermes Agent at author's request. Verdict: conditional approve — one CI-breaking bug just pushed as a fix commit (11a3c40), and one concurrency concern below worth addressing before merge. Other findings are non-blocking.

🔧 Just fixed (was blocking CI)

  • uv.lock out of sync with pyproject.toml — the prep commit 2fa9f9b removed the "requests *" line from pyproject.toml but didn't run uv lock. CI runs uv sync --locked which refused with The lockfile at uv.lock needs to be updated. All 8 matrix jobs failed at the install step before any test ran. Pushed 11a3c40 fix(deps): regenerate uv.lock after pyproject.toml change to address. Self-review caught this — would have failed CI on every push otherwise.

⚠️ Warnings

  • src/allora_sdk/rpc_client/client.py:147_reconnect_grpc has no lock around it. If two coroutines simultaneously hit transport failures and both invoke reconnect_callback (e.g. two concurrent submit_transaction calls or a stale wait_for_tx poll racing with a new submission), both will try to close() the same self.grpc_client and then both call _build_clients(). The try/except around old.close() will swallow the double-close, but _build_clients will run twice and the second run will replace the freshly-dialed channel from the first. The result is a dangling channel that's never closed. Suggested fix: wrap _reconnect_grpc body in an asyncio.Lock (or use a sentinel like self._reconnecting = True with an asyncio.Event).

  • src/allora_sdk/rpc_client/client.py:181 (docstring) — "callers that captured client.tx, client.bank etc. will continue holding the new stub" is not quite right. The TxManager swap works because we mutate attributes on the existing instance, but the AuthClient, BankClient, etc wrappers in self.auth / self.bank / etc are replaced wholesale each time _build_clients() runs. Anyone who did b = client.bank before reconnect holds the old BankClient wrapper, which holds the old bank_query stub bound to the dead channel. The docstring should distinguish: "new invocations through client.tx.broadcast_tx(...) work because TxManager attributes are rebound; new invocations through client.bank.balance(...) work because we replace client.bank; callers that captured a wrapper or stub reference before the reconnect will hit the dead channel." Or — better — make the non-Tx clients also mutate-in-place by extracting the same tx_manager-style attribute-rebind pattern for them.

💡 Suggestions

  • src/allora_sdk/rpc_client/tx_manager.py:421 — The defensive except (GRPCError, StreamTerminatedError) branch in _attempt_submissions is wider than the GRPCTransportError it complements. If broadcast_tx ever raises a GRPCError(Status.NOT_FOUND) or GRPCError(Status.INVALID_ARGUMENT) directly (not a transport class), the current code falls to _is_grpclib_transport_error(err) == False, fails the future, and returns. That's the same behavior as the pre-fix except Exception so no regression — but it's worth a comment that the non-transport grpclib path here is the intended terminal state, not an oversight.

  • src/allora_sdk/rpc_client/tx_manager.py:_pre_flight_checks — Not changed in this PR, but worth flagging adjacent-to-the-fix: _pre_flight_checks swallows Exception at debug level (logger.debug(f"Pre-flight check warning: {e}")). When the channel is dead, the pre-flight call will hit the same transport error and silently log-and-continue. The actual transport-failure trigger then happens in _build_and_broadcast and goes through the retry path correctly — but the silent-swallow in pre-flight is an unrelated landmine. Worth a follow-up to either re-raise transport errors out of pre-flight or at minimum bump the log level on transport failures.

  • src/allora_sdk/rpc_client/tx_manager.py:139(exc.message or "").lower()exc.message is typed as str in grpclib (not Optional[str]), so the or "" defense is redundant. Either trust the type and write exc.message.lower(), or update the comment to note we're being defensive against None returns from older grpclib versions.

  • tests/test_tx_manager_grpclib_errors.py:test_grpclib_error_does_not_inherit_from_grpc_rpcerror — Importing grpc (the grpcio library) in this test pulls it into the test environment even though the production code doesn't use it. grpcio is currently listed as a dep via cosmpy (transitively) so the import happens to work, but a future cleanup of cosmpy/grpcio could break this test silently. Worth a pytest.importorskip("grpc") guard or a comment explaining why we deliberately need both grpc libraries available for this regression assertion.

✅ Looks good

  • The dead-code fix itself is clean. Replacing except grpc.RpcError with except grpclib.GRPCError is a one-line semantic correction. .details().message is the correct API per grpclib's source. Two regression tests (test_grpclib_error_does_not_inherit_from_grpc_rpcerror and test_grpclib_error_message_attribute_is_string_not_method) pin down the library boundary so a future refactor can't silently re-introduce the bug.
  • _is_grpclib_transport_error has both positive AND negative test cases. Notably the "Unavailable alone is NOT a transport error" case asserts the boundary against parseable HTTP-status errors — same shape as the Go-side review.
  • Reconnect callback indirection is correct. TxManager doesn't own the channel; injecting the callback at construction keeps the dependency direction sane and means tests can use a real TxManager with a fake reconnect (test_on_transport_failure_invokes_reconnect_callback, test_on_transport_failure_swallows_reconnect_errors).
  • _on_transport_failure swallows callback errors. A failed reconnect (DNS/TCP/TLS) returning to the retry loop with a likely-still-dead channel is correct — the retry loop will retry, and if the network is genuinely down the next attempt will fail too and the retry budget will exhaust naturally. Tested.
  • Log-level uplift is consistent with the Go-side fix: retry-attempt logs went debug → warning, final failures stay error. Existing alerting keyed on level=error now sees forecaster transport failures.
  • pyproject.toml prep commit is correctly isolated as its own commit so it can be cherry-picked or reverted independently.

Self-review at author's request. Recommended action: address the asyncio.Lock concurrency concern in _reconnect_grpc and tighten the _reconnect_grpc docstring re: which captured references survive a reconnect. Other suggestions are non-blocking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant