Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Power] support dedicated infrastructure with measured GPU power - #424

Draft
edwingao28 wants to merge 2 commits into
NVIDIA:mainfrom
edwingao28:fix/dedicated-infra-power
Draft

[Power] support dedicated infrastructure with measured GPU power#424
edwingao28 wants to merge 2 commits into
NVIDIA:mainfrom
edwingao28:fix/dedicated-infra-power

Conversation

@edwingao28

Copy link
Copy Markdown
Contributor

Power telemetry currently rejects a dedicated etcd/NATS node because its placement moves the benchmark away from the collector's clock. Keep the actual Slurm batch host as head and benchmark host, and reserve another node for infrastructure. This supports homogeneous and heterogeneous allocations while excluding the infrastructure node from serving GPUs.

Configuration validation rejects dedicated frontend/client nodes and srun_options node-placement overrides when power is enabled. Dry-run output and the configuration reference describe the placement. This uses the existing collector and applies to fixed-sequence power workloads; it does not add AgentX collection support.

Testing

  • python -m pytest tests/test_configs.py tests/test_telemetry.py tests/test_dry_run.py: 369 passed, covering shared/dedicated infrastructure, both allocation types, batch-host placement, worker exclusion, invalid overrides, and dry-run output.
  • ruff check src, source and changed-test formatting checks, and git diff --check: passed.
  • Earlier make check: 1,950 passed and four failures. Two new dry-run assertions were corrected for Rich table line wrapping and pass in the final focused run. The other two failures reproduce on untouched base ed72cbef: the mock submission assertion in test_apply_mock.py and the CPU-affinity test in test_fingerprint.py (os.sched_getaffinity is absent on macOS). The repository's non-blocking type check emitted diagnostics. The full suite was not repeated after the assertion fix.
  • CPU-only validation; no Slurm/GPU run or CI dispatch.

…cture

启用功耗测量时将 benchmark 和 head 固定在 Slurm batch 主机,保留共享时钟;将独立 etcd/NATS 放到其他节点并从 serving GPU 范围排除。兼容普通和异构分配,同时拒绝会改变测量主机的客户端、前端和 srun 节点覆盖配置。

Signed-off-by: Wenyao Gao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant