Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Automated budget circuit breakers #85

Description

@harrymunro

Summary

Implement configurable circuit breakers that automatically trigger damage control procedures when resource thresholds are crossed, without waiting for the admiral's quarterdeck checkpoint.

Motivation

Nelson currently tracks token budget as a manual concern — the admiral checks burn rate at each checkpoint. But checkpoints can be 2+ task completions apart, and a runaway ship can consume significant budget between checks. The OWASP Agentic Top 10 (2026) identifies uncontrolled resource consumption as a key risk category.

Detailed Design

Circuit Breaker Thresholds

Configurable in sailing-orders.json or mission config:

Threshold Default Action
Context window > 80% on any ship 80% Auto-file damage report, signal admiral
Mission duration > configured limit None Surface warning to Admiralty
Token spend > 70% budget with < 40% tasks complete 70%/40% Elevate to Station 2, surface budget alarm
Cost per task > 3x estimated rate 3x Trigger crew-overrun damage control
Consecutive task failures > 2 2 Trigger scuttle-and-reform evaluation
Ship idle > 10 minutes with incomplete task 10m Trigger man-overboard evaluation

Implementation

Circuit breakers are implemented as hooks:

  1. PostToolUse hook on nelson-data.py checkpoint: After each checkpoint write, evaluate all thresholds against fleet-status.json. If any threshold is crossed, emit an OTel event and invoke the appropriate damage control procedure.

  2. TeammateIdle hook: Check if the idle ship has been idle longer than the configured threshold. If so, trigger man-overboard evaluation.

  3. Budget tracking enhancement: Extend fleet-status.json with burn_rate_per_task and projected_budget_at_completion fields. Compute at each checkpoint.

OTel Integration

Each circuit breaker firing emits an OTel span with:

  • nelson.circuit_breaker.type: which threshold was crossed
  • nelson.circuit_breaker.value: the actual value that triggered it
  • nelson.circuit_breaker.threshold: the configured threshold
  • nelson.circuit_breaker.action: what damage control procedure was invoked

Configuration

# In sailing-orders.yaml or mission config
circuit_breakers:
  hull_integrity_threshold: 80
  budget_alarm_ratio: 0.7  # 70% budget with < 40% tasks
  cost_per_task_multiplier: 3
  consecutive_failures: 2
  idle_timeout_minutes: 10
  enabled: true

Graceful Degradation

Circuit breakers are advisory by default — they surface alarms and invoke procedures but don't kill ships. A strict mode option would auto-trigger relief-on-station for hull integrity breaches and auto-abort for budget overruns.

Rationale

  • HMS Audacious rated automated budget circuit breakers as High impact / Low effort
  • HMS Astute identified hull integrity as self-reported and unreliable — circuit breakers provide automated enforcement
  • HMS Daring identified calibrated autonomy as a cross-cutting pattern
  • OWASP Agentic Top 10 (2026) identifies uncontrolled resource consumption as a key risk

Effort Estimate

Medium

Impact

High — closes a real safety gap and makes hull integrity meaningful

Dependencies

Benefits from: OTel span emission (#3) for circuit breaker event visibility

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions