[Power] support dedicated infrastructure with measured GPU power - #424
Draft
edwingao28 wants to merge 2 commits into
Draft
[Power] support dedicated infrastructure with measured GPU power#424edwingao28 wants to merge 2 commits into
edwingao28 wants to merge 2 commits into
Conversation
…cture 启用功耗测量时将 benchmark 和 head 固定在 Slurm batch 主机,保留共享时钟;将独立 etcd/NATS 放到其他节点并从 serving GPU 范围排除。兼容普通和异构分配,同时拒绝会改变测量主机的客户端、前端和 srun 节点覆盖配置。 Signed-off-by: Wenyao Gao <[email protected]>
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Power telemetry currently rejects a dedicated etcd/NATS node because its placement moves the benchmark away from the collector's clock. Keep the actual Slurm batch host as head and benchmark host, and reserve another node for infrastructure. This supports homogeneous and heterogeneous allocations while excluding the infrastructure node from serving GPUs.
Configuration validation rejects dedicated frontend/client nodes and
srun_optionsnode-placement overrides when power is enabled. Dry-run output and the configuration reference describe the placement. This uses the existing collector and applies to fixed-sequence power workloads; it does not add AgentX collection support.Testing
python -m pytest tests/test_configs.py tests/test_telemetry.py tests/test_dry_run.py: 369 passed, covering shared/dedicated infrastructure, both allocation types, batch-host placement, worker exclusion, invalid overrides, and dry-run output.ruff check src, source and changed-test formatting checks, andgit diff --check: passed.make check: 1,950 passed and four failures. Two new dry-run assertions were corrected for Rich table line wrapping and pass in the final focused run. The other two failures reproduce on untouched baseed72cbef: the mock submission assertion intest_apply_mock.pyand the CPU-affinity test intest_fingerprint.py(os.sched_getaffinityis absent on macOS). The repository's non-blocking type check emitted diagnostics. The full suite was not repeated after the assertion fix.