feat(routing): make the three router-comparison arms declarative - #366

Summary

Makes three equivalent aggregate SGLang deployments expressible as srt-slurm YAML, runnable through both normal Slurm submission and srtctl apply --bash. All three can serve the same model over the same two aggregate TP4 workers on one 8×H100 host under the same workload; only routing differs, and every difference is a declared field.

This adds the schema and renderer support only — no sample recipes are shipped. docs/config-reference.md documents both new surfaces with worked YAML.

What the three arms need

1. frontend.type: sgl-router — new adapter for SGLang's experimental Rust router (experimental/sgl-router) over a static aggregate worker pool.

• Builds the --worker-urls list from the allocated topology and supplies the required --model-id / --tokenizer-path (a bare model directory is rejected by the router's tokenizer loader, so this points at tokenizer.json).
• Readiness gates on every advertised worker's /health before the router's /readyz, because /readyz turns green after a single registration. /readyz returns a bare status code, so wait_for_model gained a status-only path rather than parsing an empty body.
• Static discovery has no prefill/decode split (that mode is Kubernetes-selector driven), so a P/D topology is rejected at load time.
• Workers switch to sglang.launch_server; the router discovers each worker's ZMQ KV-event publisher from /server_info, so backend.kv_events_config is the entire event wiring — no NATS, etcd, or Dynamo runtime.
• frontend.binary / frontend.source select the executable or cargo build --release it first.

2. Native sidecars in the direct-host renderer — apply --bash previously ignored dynamo.sidecar and always launched dynamo.sglang. It now emits the same coupled engine/sidecar supervisor the Slurm path uses: the sidecar starts only once the engine's gRPC port binds, either process exiting stops the other, and co-located workers get deterministic gRPC port offsets. build_sidecar_launch_command was split so the string-producing half is shared rather than reimplemented.

3. dynamo.policy_catalog — declares an external worker-selection policy catalog crate. Dynamo links exactly one catalog through the optional dynamo-worker-selection-policy-catalog dependency behind the custom-policy cargo feature; this makes that link declarative. The build:

1. materializes the crate at an immutable rev;
2. repoints its dynamo-kv-router dependency at lib/kv-router in the checkout being built — without this, cargo resolves a second copy of that crate and the plugin's WorkerSelectionPolicy types no longer unify with the ones the bindings compiled against;
3. replaces Dynamo's catalog dependency alias;
4. builds the wheel with --features custom-policy; and
5. publishes the crate's YAML next to the wheel, which srtctl passes to Dynamo's normal --router-policy-config.

The catalog identity is folded into the wheel cache key so patched and unpatched builds of the same commit never collide. Requires dynamo.hash — top_of_tree has no stable revision to build against. Not a prebuilt binary: both Slurm and direct build from source, sharing one definition of the linking contract in the stdlib-only direct_stages/common.py.

Fields that express the difference

Holding model, agg_nodes: 1 / agg_workers: 2 / gpus_per_agg: 4, engine flags, kv_events_config.aggregated: true, telemetry, and workload constant, these are the only fields that vary:

Arms 2 and 3 render byte-identical worker commands — asserted by a test.

Drive-by fix

expand_observability was injecting Dynamo DYN_REQUEST_TRACE / span env into non-Dynamo frontends and their raw engines, advertising trace artifacts nothing writes. Now gated on frontend.type: dynamo.

Validation

• srtctl dry-run, srtctl apply --bash, and bash -n pass for all three arms; the rendered direct-host script is syntax-checked inside the tests.
• tests/test_router_comparison.py — 24 tests covering raw-router command/topology rendering, direct-host sidecar rendering and lifecycle coupling, and policy-catalog source-build/linking plus --router-policy-config plumbing. Each arm is built from an inline config fixture.
• Full suite: 1531 passed. ruff check/format clean; ty check shows no new diagnostics against baseline.
• Not run: no GPU host available here, so no smoke benchmark against a live stack.

Three pre-existing failures on main are untouched: two are macOS-only (os.sched_getaffinity), and one is a local untracked recipe using a removed observability.scrape_metrics key.