Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(sweep): filter invalid vLLM GPU topologies - #375

Open
dipeshbabu wants to merge 1 commit into
NVIDIA:mainfrom
dipeshbabu:agent/filter-sweep-gpu-topologies
Open

fix(sweep): filter invalid vLLM GPU topologies#375
dipeshbabu wants to merge 1 commit into
NVIDIA:mainfrom
dipeshbabu:agent/filter-sweep-gpu-topologies

Conversation

@dipeshbabu

Copy link
Copy Markdown

Summary

  • filter vLLM sweep variants whose TP × PP × PCP × DP world cannot run within the GPUs allocated to each worker
  • preserve valid mixed topologies such as TP4/DP2 on an eight-GPU worker
  • keep intentional TP-only configurations that use fewer GPUs than the reserved node
  • apply the guard before full schema validation so one oversized vLLM Router variant does not abort the entire sweep
  • leave SGLang and TensorRT-LLM expansion unchanged because their parallel dimensions have different semantics

Motivation

Cartesian topology sweeps in srt-slurm-recipes currently expand combinations such as TP8/PP8/DP8 on an eight-GPU node. Those jobs cannot run, but constraining the recipes to TP8/PP1 also removes useful coverage such as TP4/DP2.

This addresses the root cause requested in NVIDIA/srt-slurm-recipes#9 by filtering incompatible vLLM variants during sweep generation.

Validation

  • pytest tests/ -q on Linux: 1551 passed, 2 skipped, 6 deselected
  • pytest tests/test_sweep.py -q on Linux and Windows: 25 passed
  • ruff check src/srtctl/
  • ruff format --check src/srtctl/
  • ty check src/srtctl/core/sweep.py
  • expanded all six recipes from srt-slurm-recipes PR Add Kimi K2.5 disagg STP and MTP recipes for GB200 NVfp4 (ISL8K_OSL1K and ISL1K_OSL1K) #9:
    • DeepSeek-R1 and Nemotron retain TP4/DP2 and cap retained worlds at eight GPUs
    • MiniMax retains its allocation-compatible DP8 variant

Checklist

  • Added regression coverage for oversized TP/PP, mixed TP/DP, vLLM Router, non-vLLM isolation, and fully invalid sweeps
  • Commit includes a DCO sign-off

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant