feat(glm5): add validated GB300 OCI-AGA recipes - #382

Review: srtctl install workflow + GLM5/Kimi2.6 recipes

Reviewed locally off refs/pull/382/head (4 commits, 37 files, +5119/−2). I ran tests/test_install.py (27 passed), validated all 25 new recipes through validate_config_file (all clean), rendered the generated sbatch and bash -n'd it (clean), and diffed lint against main.

Heads-up on scope: 25 of 37 files are recipes, but the other 12 add a whole new user-facing subcommand, a new srtctl.install package, a new hard runtime dependency, and a new top-level recipe tree. Summary of the non-recipe surface:

A. Real issues

A1. The srtslurm.yaml write in the generated sbatch is unlocked 🔴

In build_sbatch_script, the with open(lock_path) block closes right after y.load(f), so flock is released before the mutation and os.replace:

with open(lock_path, 'a+') as lockf:
    fcntl.flock(lockf, fcntl.LOCK_EX)
    with open(yml_path) as f:
        doc = y.load(f) or CommentedMap()

# ← lock released here; everything below runs unlocked
model_paths[...] = ...
os.replace(tmp_path, yml_path)

Two installs finishing together can each read the old file and write over each other — one alias silently disappears from the user's cluster config. srtslurm_yaml_writer.register_aliases gets this right; the inline copy lost an indent level. One-line fix.

A2. srtctl install glm5 doesn't work 🔴

registry.py documents itself as "hardcoded registry of installable models," but the CLI marks --hf-repo-id, --model-alias, and --container-image as required=True and builds the spec purely from flags. REGISTRY, get_spec, and available_models are reachable only from tests. Either wire the registry into _resolve_spec (flags become optional overrides) or delete it — the docstrings currently describe a feature that isn't there.

A3. registry.glm5.default_recipe points at a nonexistent path 🔴

recipes/GLM5/disagg/trtllm_dynamo/gb300_nvfp4/ISL8K_OSL1K/STP/... doesn't exist — the GLM5 recipes in this PR live under CrossCluster_Recipes/. It's echoed as the "Next step" line in the job log. Currently unreachable because of A2, which is how it survived.

A4. --hf-revision doesn't isolate the download directory 🔴

model_storage_dirname() keys on repo id alone. Install revision A, then revision B: same directory, snapshot_download overwrites what it fetches and prunes nothing, so the alias ends up pointing at a mixture. This directly undercuts the "Support pinned model revisions" commit. Fold a short SHA into the dir name, or refuse to reuse a dir recorded at a different revision.

A5. CI lint is red 🔴

ruff format --check src/srtctl/ is a CI step and fails on cli/install.py, install/container.py, install/slurm.py, install/srtslurm_yaml_writer.py. Also I001 in two of them, ISC004 at container.py:42, and the new submit.py:62 import sits outside the sorted block. Mechanical.

A6. install/ isn't gitignored 🟡

Default install base is <srtctl_root>/install — weights, .sqsh files, sbatch scripts, logs. Hundreds of GB show up as untracked in git status.

A7. The 14 GB300 sglang recipes reference an alias nothing registers 🟡

They use model.path: glm5-fp8; the install flow only ever writes nvidia/GLM5-NVFP4. Those recipes need a manual srtslurm.yaml edit, while the new docs say alias editing is unnecessary after install. Either the docs need a carve-out or the recipes should use the registered alias.

A8. Minor but real 🟡

• Enroot scratch dir /tmp/$USER/srtctl_enroot_$$ leaks on failure — set -e skips the rm -rf; use a trap.
• Fixed install_<model>.sbatch filename: a re-submit rewrites a running job's script.
• srtctl_root_from_package()'s parents[3] is correct only for editable installs; a plain pip install resolves inside site-packages. Consider a guard or a SRTCTL_ROOT override.

B. Big changes — implications and whether they're justified

B1. srtctl grows an artifact-provisioning role — justified

It submitted benchmark jobs; now it downloads weights, pulls containers, and edits cluster config. Defensible: the manual path in installation.md was genuinely painful and make setup already set the precedent of srtctl touching srtslurm.yaml. The consequence is that srtctl now owns a mutation path over a file users hand-edit — which is why A1 matters more than its one-line size suggests.

B2. Logic duplicated between Python modules and the generated script — cost already visible

The compute-node work is a self-contained bash file with inlined Python; install/container.py, install/model.py, install/setup.py, and srtslurm_yaml_writer.py re-implement the same steps and are exercised only by tests. The stated rationale (reads top-to-bottom, no srtctl importable on the compute node) is reasonable — login-node debuggability matters. But the two copies have already diverged (A1), and the 27 passing tests certify code that never executes in production. Either drop the unused modules and test the rendered script (against a fake sbatch/enroot), or have the script call python -m srtctl.install.run and keep one implementation.

B3. huggingface_hub as a hard dependency — weakly justified

Every srtctl apply user now pulls it even if they never install a model. Cheap alternative: an [install] extra, with the existing ImportError message pointing at pip install srtctl[install] — the code already handles the missing-import case gracefully.

B4. SLURM-only, always sbatch — decision justified, resourcing isn't

Sensible for large downloads where login nodes are throttled. But you can't provision from a workstation or inside an existing allocation (hard-errors on SLURM_JOB_ID), the job inherits --gpus-per-node and --exclusive from cluster config (a pure data-movement job occupying GPU nodes), and --time comes from default_time_limit — which recipes set to values like 01:30:00. For a GLM-5-scale download plus enroot import, timeout is the likely first-run experience. --cpus-per-task=8/--mem=128G are hardcoded. Please add at least --time and --partition overrides.

B5. New top-level CrossCluster_Recipes/ — not justified as-is

Nothing references it — no doc, Makefile, or CI job — and validate-recipes globs Path('recipes') only, so these 25 files go unvalidated on every future PR. Internal naming diverges too (Disagg vs disagg, ISL8K_OSL1K vs 8k_1k). If the split is intentional (cross-cluster-portable vs cluster-tuned), it needs a README and a CI glob; otherwise these belong under recipes/. This is the change most likely to rot quietly.

B6. Docs repointed to github.com/NVIDIA/srt-slurm — correct

Overdue; just noting it rode along in a recipes PR.

B7. "Validated" and derived recipes are mixed — needs a marker

The commit says "validated GB300 OCI-AGA recipes," but the six 16k_512 files are headed # Derived for 16K/512 from ... while their 8k_1k twins say # Expanded from .... Extrapolated configs sitting next to measured ones, distinguished only by a comment, invites someone quoting derived numbers as validated. A validated:/derived: field or a directory README would settle it. Coverage is also lopsided — 1k_1k has one recipe per mode vs six elsewhere.

C. What's good here

• The problem is real and the shape is right. Download weights → import container → register aliases → print the next command is exactly the manual sequence users were running by hand out of installation.md.
• No hardcoded cluster values. default_account/default_partition/default_time_limit are read from srtslurm.yaml at submit time, and the module docstring says so explicitly. use_gpus_per_node_directive even gets a positive-integer check with a message naming the offending key.
• Injection-conscious codegen. Five distinct regexes validate every user-controlled field before interpolation into a generated shell script — the right instinct, with tight per-field character sets rather than one lazy catch-all.
• Atomic config writes. mkstemp + os.replace + finally cleanup in both copies, comment-preserving ruamel round-trip, and _set_alias warns on overwrite instead of silently clobbering a user's alias. The intent behind A1 was clearly there.
• Failure messages tell you what to do. Missing HF_TOKEN names the settings URL and the exact export; missing huggingface_hub prints the srun-shell pip install line; missing sbatch asks whether you're on a SLURM cluster. The generated script prints huggingface_hub.__file__ with a comment saying it's there to catch "works on my machine."
• unset PYTHONPATH before venv activation, with a comment explaining the inherited-PYTHONPATH failure it prevents. Clearly earned.
• Idempotence where it counts. Container import skips an existing .sqsh, is_bootstrapped checks executability rather than mere existence, _set_alias returns unchanged when the value already matches.
• Credential preflight warns by default, with --strict-auth-preflight to opt into hard failure — right default, since enroot can source credentials from places the check doesn't know about, and the comment says as much.
• 27 tests, covering alias add/overwrite/unchanged/section-creation, the validation regexes, and sbatch directive rendering (see B2 for the caveat about coverage).
• All 25 recipes validate clean, carry identity.model.revision SHAs for reproducibility, and the 16k_512 variants show coherent retuning rather than copy-paste — context length, chunked-prefill, max-running-requests, and isl/osl all move together.
• Docs updated in the same PR, including flow ordering (make setup → review defaults → install) and an explicit section on when manual alias edits are still needed.

Bottom line: A1, A2/A3, A4, and A5 are blocking. B2 and B5 are the structural calls worth settling before this lands — both get more expensive with time.

stop merging recipes to srtslurm

Closing with the 2.0 merge (#407): recipe trees no longer live in this repository. Recipes live in the downstream InferenceMAX tree (benchmarks/multi_node/srt-slurm-recipes/), and the in-repo corpus is the curated examples/ matrix. srtctl migrate -f <recipe> --in-place rewrites a v1 recipe to the 2.0 layout. Please reopen these downstream if they are still current.