fix(profiling): support Dynamo Nsight capture - #411

Defaulting capture_scope to "selected" (with worker_index=0/worker_rank=0) is a silent behavior change for existing type: nsys (iteration-triggered) recipes on multi-worker / multi-rank vLLM/SGLang. Before this PR, worker_stage wrapped every process (if profiling.is_nsys:) and the benchmark received all leader endpoints via _logical_worker_endpoints(). After this PR, an existing recipe with only start_step/stop_step set will, with no config change, wrap only worker 0 / rank 0 and send only that one control endpoint to the benchmark.

Two options:

• Default capture_scope: "all" so existing recipes keep wrapping everything, and opt into targeted capture explicitly; or
• Keep "selected" but call this out loudly in the PR description / a changelog note with the one-line migration (capture_scope: all).

I'd lean toward defaulting to "all" for backward compat — targeted capture is the new capability, so it's the one that should opt in. If "selected" stays the default, please add a test pinning the default-on-multi-worker behavior so the change is intentional and visible.

Fixed in 7ac824c: capture_scope now defaults to all, preserving existing all-worker wrapping. Targeted capture requires explicit selected. Added a regression that runs the real allocator for two logical workers across four nodes, with both per_gpu and per_node layouts, and verifies all physical ranks are wrapped and every allocated DYN_SYSTEM_PORT reaches the helper.

nsys_trace and capture_range_end are scoped to non-TRT-LLM only — _get_nsys_prefix_trtllm still hardcodes cuda,nvtx,ucx / stop / --sample=none / --cuda-graph-trace=node, so setting either field on a TRT-LLM run is silently ignored.

To be clear, this isn't a regression: the TRT-LLM hardcoding is preexisting, and extra_nsys_args (which is applied on the TRT-LLM path) still works as before. It's just that the two new fields don't apply there.

This will be addressed in #412, which routes all of these flags through shared helpers (_nsys_trace_domain, _nsys_cuda_graph_trace_mode, _nsys_sample_arg) across all four get_nsys_prefix_* branches, so the override surface is uniform across backends and there's no silent non-TRT-LLM-only scope. No action needed in this PR — flagging so reviewers know the gap is closed in the follow-up.

Agreed; keeping this PR scoped to Dynamo capture. The schema/docs/PR description explicitly mark nsys_trace and capture_range_end as non-TRT-LLM. No TRT-LLM behavior change here.

This hunk rewraps get_env_vars from a one-line signature to multi-line and deletes the blank line between the Args block and Returns: in the docstring. It's unrelated to the PR's goal, adds diff noise, and removing the Args/Returns blank line breaks Google-style docstring section separation (and may trip a docstring linter). Please revert this hunk.

Fixed in 7ac824c: restored the single-line get_env_vars signature and the blank line between Args and Returns.

Building a synthetic Endpoint with fake node names and delegating to the real backend.endpoints_to_processes is a neat way to derive the valid rank set without duplicating per-backend topology logic — and it works because Endpoint.gpu_indices is per-node and the per_gpu DP branch keys node_rank off dp_rank driven by the gpu_indices count.

Two small asks:

1. Add a one-line comment noting the assumption it relies on — e.g. # mirrors Endpoint's uniform-per-node gpu_indices assumption; gpus_per_worker must be a multiple of gpus_per_node for multi-node workers — so the next reader doesn't have to re-derive it.
2. It runs a full endpoints_to_processes (including port allocation) during validation just to read node_rank. Cheap, and the fake node names isolate any shared port allocator, but worth a brief note that this is validation-time work.

Addressed in 7ac824c. The comment now explains that this mirrors allocate_endpoints: multi-node workers get a uniform full per-node GPU set and partial-node workers a contiguous subset. A second comment notes that validation expansion uses a fresh port allocator, without reserving live ports or consuming runtime allocations. Added tests comparing the validation rank set with real allocated per_gpu/per_node topologies and their system ports.

show_extensions now gates on or config.profiling.enabled (line 416), so the inner if profiling.enabled: on line 437 is redundant. Drop one of the two checks.

I retained both guards after checking the surrounding condition: show_extensions is an OR of custom benchmark/container, observability, telemetry, Mooncake, and profiling. A custom benchmark can make the outer condition true while profiling is disabled, so removing the inner check would display disabled profiling. Removing profiling from the outer condition would hide profiling-only configurations. Added a clarifying comment and regression assertions for both cases in 7ac824c.

nsys_trace and extra_nsys_args: ["--trace=..."] both target the same nsys -t flag. A user setting both (e.g. nsys_trace: "cuda,nvtx,osrt" plus extra_nsys_args: ["--trace=osrt"]) will pass two -t/--trace flags and rely on nsys's last-wins behavior. Worth a one-line note here recommending nsys_trace over --trace= in extra_nsys_args when the dedicated field exists.

Added the note in 7ac824c: for non-TRT-LLM workers prefer nsys_trace and do not also pass --trace through extra_nsys_args; otherwise duplicate options depend on Nsight argument parsing.