feat(runtime): reuse digest-pinned containers across SRT steps - #344

Motivation

An SRT job can launch multiple Slurm steps with the same container image. When each step materializes the same digest-pinned image independently, startup time and registry work are repeated unnecessarily.

Summary

• Materialize a digest-pinned registry image once through Pyxis before service fan-out, then reuse the resulting SquashFS file across applicable steps.
• Discover a per-user cache automatically from the resolved output directory, while allowing an explicit cluster path.
• Add auto, required, and native policies; preserve container_cache_path as a backward-compatible explicit-path shorthand.
• Record declared, resolved, and effective image identity plus cache outcome in the run lockfile.
• Preserve registry URI prefixes and redact embedded credentials from provenance and failure messages.

Configuration

container_cache:
  mode: auto
  # path: /shared/container-cache
  lock_timeout_seconds: 600

With no cache configuration, auto uses <output-dir>/.srtctl/container-cache/<uid>. Mutable tags continue through native Pyxis handling. required fails instead of falling back, and native disables reusable materialization.

Safety

• Validate cache directory ownership and permissions, reject symlinked paths and entries, and require regular SquashFS files.
• Use a bounded, no-follow per-entry lock so concurrent callers import once without waiting forever.
• Publish via a private temporary file, file and directory synchronization, and atomic replacement.
• Verify cached images are readable from every allocated node before using them.
• Fall back explicitly in auto mode when safe reuse is unavailable; never silently weaken required mode.

Validation

End-to-end cache reuse

The full SRT-Slurm lifecycle was run twice on the same compute node and Slurm allocation with the same configuration and a digest-pinned 30.06 GB image containing 104 registry layers. The first run started with an empty explicit cache; the second reused the populated cache.

Reusing the populated cache reduced end-to-end elapsed time by 2m53s (52.1%), a 2.09x speedup for this large-image case. This is a cold-to-warm reusable-cache comparison, not a parent-to-PR percentage.

The exact parent revision was also tested twice with the same image. Both attempts exceeded the existing fixed 300-second infrastructure-readiness window during the first native registry-reference import, so there is no finite parent completion time from which to calculate a percentage. This result is treated as a startup-reliability signal rather than folded into the speedup above.

Automated validation

• Source lint and formatting checks pass.
• Focused cache, configuration, dry-run, and lockfile suite: 317 passed.
• Linux CI repository suite: 1,545 passed, 2 skipped, 6 deselected.
• Focused cache helper coverage: 100% statement coverage.