Clement Issue 3: TX Simulation Catch-All Hides Guaranteed Failures

Verdict

Clement's issue 3 is still valid in the current tree.

The current TxManager.submit_transaction() treats every simulation-stage exception as if it were merely a gas-estimation failure. That is too broad. Some simulation failures mean "we could not estimate gas", but others mean "the transaction message is invalid and should not be broadcast." The current code does not distinguish those cases.

There is also a second, smaller bug in the same block: fee preview is wrapped in the same try as simulation, so a fee-preview failure can discard a successful gas simulation and force a fallback to default gas.

Current Problem Code

Current code in src/allora_sdk/rpc_client/tx_manager.py:

async def submit_transaction(
    self,
    type_url: str,
    msgs: list[Any],
    fee_tier: FeeTier = FeeTier.STANDARD,
    max_retries: int = 2,
    timeout: Optional[timedelta] = None,
    account_seq: Optional[int] = None,
) -> "PendingTx":
    if self.wallet is None:
        raise Exception('No wallet configured. Initialize client with private key or mnemonic.')

    estimated_gas_limit: Optional[int] = None
    try:
        estimated_gas_limit = await self.simulate_transaction(type_url, msgs)
        logger.debug(f"Simulated gas requirement for {type_url}: {estimated_gas_limit}")
        fee_preview = await self._calculate_optimal_fee(
            estimated_gas_limit,
            self._fee_multipliers[fee_tier],
        )
        logger.debug(f"Estimated fee for {type_url}: {fee_preview.amount} {fee_preview.denom}")
    except Exception as e:
        logger.debug(f"Unable to simulate transaction for gas estimate, falling back to defaults: {e}")
        estimated_gas_limit = None

    async with self._parent_tx_id_lock:
        next_parent_tx_id = self.parent_tx_id
        self.parent_tx_id += 1

    pending = PendingTx(
        manager=self,
        parent_tx_id=next_parent_tx_id,
        type_url=type_url,
        msgs=msgs,
        fee_tier=fee_tier,
        max_retries=max_retries,
        timeout=timeout,
    )

    asyncio.create_task(
        self._attempt_submissions(pending, estimated_gas_limit, account_seq=account_seq)
    )

    return pending

The problematic part is:

except Exception as e:
    logger.debug(f"Unable to simulate transaction for gas estimate, falling back to defaults: {e}")
    estimated_gas_limit = None

This catches all of the following as equivalent:

• Deterministic chain rejection, represented as TxError.
• Sequence errors, represented as AccountSequenceMismatchError.
• Simulation out-of-gas after retry exhaustion, represented as OutOfGasError.
• Fee errors, represented as InsufficientFeesError.
• Real simulation infrastructure failures, such as a transient RPC problem.
• Fee-preview failures from _calculate_optimal_fee(), even after simulation succeeded.

Those cases should not have the same behavior.

Why This Is A Real Bug

simulate_transaction() already classifies errors into meaningful exception types:

def _exception_from_simulation_error(self, error: grpc.RpcError) -> Exception:
    error_details = error.details() if hasattr(error, 'details') else str(error)
    error_msg = error_details or str(error)
    error_class = self._classify_error_from_message(error_msg)

    if error_class == OutOfGasError:
        return OutOfGasError(f"Simulation ran out of gas: {error_msg}")
    elif error_class == AccountSequenceMismatchError:
        return AccountSequenceMismatchError(f"Sequence mismatch during simulation: {error_msg}")
    elif error_class == InsufficientFeesError:
        return InsufficientFeesError(f"Insufficient fees during simulation: {error_msg}")
    else:
        code = error.code() if hasattr(error, 'code') else None
        try:
            code_value = code.value[0] if isinstance(code.value, tuple) else int(code.value)
        except (TypeError, IndexError, AttributeError):
            code_value = 1

        return TxError(
            codespace="simulation",
            code=code_value,
            message=error_msg,
            tx_hash=None
        )

But submit_transaction() immediately flattens those classified errors back into estimated_gas_limit = None.

That defeats the purpose of simulation classification. A simulation TxError such as "unauthorized", "already submitted", "account not found", or a malformed message is not a gas estimation problem. Broadcasting after that is at best noisy and at worst fee-wasting.

Current Test Codifies The Bad Behavior

The current test suite explicitly asserts the problematic fallback:

@pytest.mark.asyncio
async def test_submit_transaction_falls_back_when_simulation_raises_tx_error(
    monkeypatch: pytest.MonkeyPatch,
) -> None:
    manager = _make_manager()
    manager.simulate_transaction = AsyncMock(
        side_effect=TxError(codespace="emissions", code=13, message="unauthorized")
    )
    manager._attempt_submissions = AsyncMock(return_value=None)

    # ...

    pending = await manager.submit_transaction(
        type_url="/emissions.v9.InsertWorkerPayloadRequest",
        msgs=[Mock()],
        fee_tier=FeeTier.STANDARD,
    )

    # Current behavior: simulation errors are swallowed and gas falls back to defaults.
    assert pending.type_url == "/emissions.v9.InsertWorkerPayloadRequest"
    manager._attempt_submissions.assert_called_once()
    args, kwargs = manager._attempt_submissions.call_args
    assert args[1] is None  # gas_limit
    assert kwargs["account_seq"] is None

That test should be inverted. A deterministic simulation TxError should stop submission before _attempt_submissions() is scheduled.

Concrete Failure Mode

Example flow with current code:

1. Reputer/inferer calls an emissions transaction method.
2. TxManager.submit_transaction() calls simulate_transaction().
3. The chain simulation rejects the message as unauthorized.
4. simulate_transaction() raises TxError(codespace="simulation", code=..., message="unauthorized").
5. submit_transaction() catches it with except Exception.
6. The SDK logs only at debug level.
7. The SDK creates a PendingTx.
8. The SDK schedules _attempt_submissions(..., gas_limit=None).
9. _build_and_broadcast() falls back to _estimate_gas(type_url).
10. The SDK broadcasts a transaction that simulation had already rejected.

The user now sees a later broadcast-time or inclusion-time failure, not the original simulation failure. In production, this makes debugging harder and can waste fees.

Extra Problem: Fee Preview Can Discard A Good Simulation

This block also wraps _calculate_optimal_fee():

estimated_gas_limit = await self.simulate_transaction(type_url, msgs)
logger.debug(f"Simulated gas requirement for {type_url}: {estimated_gas_limit}")
fee_preview = await self._calculate_optimal_fee(
    estimated_gas_limit,
    self._fee_multipliers[fee_tier],
)
logger.debug(f"Estimated fee for {type_url}: {fee_preview.amount} {fee_preview.denom}")

If simulation succeeds but fee preview raises, the catch-all sets:

estimated_gas_limit = None

That throws away a valid simulated gas estimate. Fee preview is only used for logging. It should not be able to force a default-gas fallback after simulation succeeded.

Integration Perspective

This differs from allora-offchain-node semantics.

In allora-offchain-node, transaction construction calls simulation during BuildAndSignTransaction(). If simulation fails, transaction construction returns an error. Retry handling then happens in the transaction send path, where sequence, fees, and gas are handled as classified cases.

The current SDK behavior is weaker:

• It collapses simulation rejection into "no gas estimate".
• It proceeds to broadcast after deterministic simulation errors.
• It logs the original simulation failure at debug level only.
• It lets a fee-preview/logging failure destroy a successful gas estimate.

For a tool intended to replace allora-offchain-node, this should be tightened.

Suggested Fix

Design Goal

Do not treat every simulation failure as recoverable.

Recommended behavior:

• If simulation returns a deterministic TxError, fail fast and do not broadcast.
• If simulation exhausts out-of-gas retries, fail fast. Falling back to the default gas limit is likely worse than the already-failed larger simulation limit.
• If simulation hits account-sequence mismatch, fail fast or retry simulation with fresh account info. Do not blindly broadcast.
• If simulation hits a fee-specific artifact that is known to be caused by simulation fee settings, either retry simulation with a realistic fee or fall back with a warning.
• If simulation infrastructure is unavailable, fallback can be acceptable, but it should be logged at warning level.
• Fee-preview failure should never erase a successful simulation result.

Minimal Code Change

This is the smallest useful change:

estimated_gas_limit: Optional[int] = None

try:
    estimated_gas_limit = await self.simulate_transaction(type_url, msgs)
except TxError:
    logger.error(
        "Transaction simulation rejected %s; not broadcasting",
        type_url,
        exc_info=True,
    )
    raise
except (OutOfGasError, AccountSequenceMismatchError) as err:
    logger.error(
        "Transaction simulation failed for %s; not broadcasting: %s",
        type_url,
        err,
        exc_info=True,
    )
    raise
except InsufficientFeesError as err:
    logger.warning(
        "Transaction simulation failed fee check for %s; falling back to default gas: %s",
        type_url,
        err,
        exc_info=True,
    )
    estimated_gas_limit = None
except Exception as err:
    logger.warning(
        "Unable to simulate transaction for %s; falling back to default gas: %s",
        type_url,
        err,
        exc_info=True,
    )
    estimated_gas_limit = None

if estimated_gas_limit is not None:
    logger.debug("Simulated gas requirement for %s: %s", type_url, estimated_gas_limit)
    try:
        fee_preview = await self._calculate_optimal_fee(
            estimated_gas_limit,
            self._fee_multipliers[fee_tier],
        )
        logger.debug(
            "Estimated fee for %s: %s %s",
            type_url,
            fee_preview.amount,
            fee_preview.denom,
        )
    except Exception as err:
        logger.debug("Unable to preview fee for %s: %s", type_url, err)

This preserves fallback for truly unclassified simulation infrastructure failures, but prevents known deterministic transaction failures from being broadcast.

Better Long-Term Fix

The cleaner version is to introduce a dedicated exception for simulation availability failures and keep classified chain rejections separate:

class GasSimulationUnavailableError(Exception):
    """Raised when gas simulation could not run due to transport or service availability."""

Then simulate_transaction() should only raise GasSimulationUnavailableError for cases where the SDK genuinely could not ask the chain for an estimate. It should continue raising TxError, OutOfGasError, AccountSequenceMismatchError, and InsufficientFeesError for chain-evaluated transaction failures.

Then submit_transaction() becomes simpler:

try:
    estimated_gas_limit = await self.simulate_transaction(type_url, msgs)
except GasSimulationUnavailableError as err:
    logger.warning(
        "Gas simulation unavailable for %s; falling back to default gas: %s",
        type_url,
        err,
        exc_info=True,
    )
    estimated_gas_limit = None
except Exception:
    logger.error(
        "Transaction simulation failed for %s; not broadcasting",
        type_url,
        exc_info=True,
    )
    raise

That makes the policy explicit:

• "Could not simulate" can fallback.
• "Simulated and failed" does not broadcast.

Suggested Test Changes

Replace The Current TxError Fallback Test

Current test name:

test_submit_transaction_falls_back_when_simulation_raises_tx_error

Suggested replacement:

@pytest.mark.asyncio
async def test_submit_transaction_does_not_broadcast_when_simulation_rejects_tx(
    monkeypatch: pytest.MonkeyPatch,
) -> None:
    manager = _make_manager()
    manager.simulate_transaction = AsyncMock(
        side_effect=TxError(codespace="emissions", code=13, message="unauthorized")
    )
    manager._attempt_submissions = AsyncMock(return_value=None)

    with pytest.raises(TxError, match="unauthorized"):
        await manager.submit_transaction(
            type_url="/emissions.v9.InsertWorkerPayloadRequest",
            msgs=[Mock()],
            fee_tier=FeeTier.STANDARD,
        )

    manager._attempt_submissions.assert_not_called()

Add A Test For Fee Preview Failure

This captures the second bug:

@pytest.mark.asyncio
async def test_submit_transaction_keeps_simulated_gas_when_fee_preview_fails(
    monkeypatch: pytest.MonkeyPatch,
) -> None:
    manager = _make_manager()
    manager.simulate_transaction = AsyncMock(return_value=345678)
    manager._calculate_optimal_fee = AsyncMock(side_effect=RuntimeError("fee preview failed"))
    manager._attempt_submissions = AsyncMock(return_value=None)

    scheduled: list[asyncio.Task] = []

    def _capture_task(coro):
        task = asyncio.get_running_loop().create_task(coro)
        scheduled.append(task)
        return task

    monkeypatch.setattr(asyncio, "create_task", _capture_task)

    await manager.submit_transaction(
        type_url="/emissions.v9.InsertWorkerPayloadRequest",
        msgs=[Mock()],
        fee_tier=FeeTier.STANDARD,
    )

    manager._attempt_submissions.assert_called_once()
    args, kwargs = manager._attempt_submissions.call_args
    assert args[1] == 345678

    for task in scheduled:
        await task

Optional Fallback Test

If the SDK intentionally keeps fallback for simulation infrastructure outages, add a narrow test for only that class of error. If no dedicated exception exists yet, this test should wait until the exception is introduced; otherwise it risks keeping the current broad fallback alive.

Risk Of The Suggested Change

This is a behavior change. Some callers may currently rely on submit_transaction() returning a PendingTx even when simulation fails. After the minimal fix, deterministic simulation failures would raise before a PendingTx is created.

That behavior is still preferable for production transaction safety, but if API compatibility is important, there is an alternative:

1. Create the PendingTx before simulation.
2. If simulation returns deterministic failure, set pending._final_future.set_exception(err).
3. Return the failed PendingTx without scheduling _attempt_submissions().

That preserves the "caller awaits pending.wait()" shape while still preventing broadcast.

Suggested compatibility variant:

pending = PendingTx(...)

try:
    estimated_gas_limit = await self.simulate_transaction(type_url, msgs)
except TxError as err:
    logger.error("Transaction simulation rejected %s; not broadcasting", type_url, exc_info=True)
    pending._final_future.set_exception(err)
    return pending

I would only choose this variant if existing SDK users strongly depend on submit_transaction() never raising simulation errors directly.

Final Recommendation

Fix this before positioning the SDK as the replacement runtime for production offchain-node users.

The minimum acceptable fix is:

1. Stop swallowing TxError, OutOfGasError, and AccountSequenceMismatchError from simulation.
2. Separate fee-preview logging from simulation so preview failures do not discard valid gas estimates.
3. Change the current simulation test so deterministic simulation rejection prevents _attempt_submissions().
4. Keep any fallback path narrow and explicit, ideally behind a dedicated "simulation unavailable" exception.

Summary by cubic

Hardened transaction simulation, gas, and fee handling to stop broadcasting guaranteed failures, keep valid gas estimates, and make fallbacks explicit. Also added first‑class worker modules (Inferer, Forecaster, Reputer), default loss methods, autostake, staking queries, examples, and better logging.

• Bug Fixes

  • TxManager.submit_transaction: fail fast on TxError, OutOfGasError, AccountSequenceMismatchError, and InsufficientFeesError; fallback only when RPC is unavailable via new GasSimulationUnavailableError; keep simulated gas when fee preview fails.
  • Simulation now accepts and uses account_seq so the same sequence is used for simulation and broadcast; keeps parent tx ID safety and serialized submissions.
  • Error classification: map gRPC UNAVAILABLE/DEADLINE_EXCEEDED to GasSimulationUnavailableError; raise WalletNotConfiguredError instead of a generic exception.
  • Forecaster submissions: allow forecast‑only payloads and require at least one of inference or forecast.
  • REST template: pass x-cosmos-block-height, ignore unknown fields, and avoid per‑request session churn.
  • Cache effective dynamic gas price with multiplier.

• Migration

  • Deterministic simulation failures now raise (including fee errors); callers should handle exceptions instead of expecting a PendingTx on guaranteed failures. Fallback to default gas happens only when simulation is unavailable.
  • Gas simulation is required for all transactions; fee‑preview failures no longer discard simulated gas.
  • Worker callbacks now receive RunContext; update run functions to async def run(ctx: RunContext) -> .... New helpers: make_reputer_function, get_network_inference, get_block_time.
  • Reputer loss safety: non‑finite inputs raise; ztae/zptae require explicit loss configuration. Default loss selection available via allora_sdk.loss_methods.
  • Dependencies: add runtime grpcio and protobuf. AlloraNetworkConfig adds dynamic_gas_price_default_multiplier and query_timeout_secs; testnet faucet URL updated.

^{Written for commit 7b8f197. Summary will update on new commits. Review in cubic}