feat(observability): combined host-attribution coverage (#356 + #413 + #415) + scraper label fix - #417

…idation matrix

Replaces the draft rubric with measured thresholds from 7 validation runs
(2x baseline, 2x pinned, 3x dedicated-frontend at DSV4 c1010 on GB300):
clean-node run-delay floor 0.00-0.01 ms/s; shared-node asymmetry 140-350x
(persisting at 190x under pinning, collapsing to 1x in all three
dedicated-frontend runs); affinity_ncpus as the direct pinning observable
(full-width vs CPUs-per-GPU), unchanged by placement. Adds the operator
three-step decision flow and expected-gain guidance (+4.0% pinning on
288-CPU nodes / +1.1% on 144-CPU; +0.5-0.85% placement).

Signed-off-by: Yuewei Na <[email protected]>

…ault

The Prometheus surface Tachometer captures says nothing about the processes
underneath it: Dynamo publishes no process_* or thread metrics and node_exporter
only sees the machine. The AgentX/VR200 frontend spin (fastokens 0.3.1 sizing its
BPE rayon pool to all 352 CPUs -> 352 idle workers burning ~110 cores) was
invisible in every scraped family and was found with sacct plus an out-of-band
per-thread /proc sampler.

Add ncabatoff/process-exporter (pinned multi-arch image, port 9256) as a third
built-in exporter, launched on every node that hosts a backend rank OR a frontend
replica -- the frontend node is the one the per-node exporters skip when the
frontend is head-placed or dedicated, and it is where frontend CPU pathologies
live. Groups come from <log_dir>/process-exporter.yml written at launch
(frontend, dynamo_trtllm/sglang/vllm handlers + ranks, the MPI launcher, the
benchmark client, infra daemons; first match wins, no catch-all). -threads=true
publishes per-thread-name CPU and thread counts, so a runaway pool shows up as a
step in thread_count and a CPU cluster on one thread name. Endpoints are scraped
unfiltered so groupname/threadname/mode labels pass through to the parquet and
to server_metrics_export.jsonl unchanged.

node_exporter additionally gets --collector.processes (host-wide thread total,
process states, procs_running/blocked).

An explicit process_exporter block overrides the image/port/command, and
default_exporters: false still disables all built-ins. Dry-run lists the new
exporter; docs/config-reference.md documents it.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Yuewei Na <[email protected]>

The tachometer node_exporter launch enabled only cpu/infiniband/meminfo, and
the host sampler read /proc/stat + /proc/meminfo but no PSI. The result: none
of the scheduler-pressure signal the retired steady_probe.sh sampler carried
(PSI stall time, procs_running/blocked, context-switch and page-fault/reclaim
counters, per-NUMA free memory) was collected anywhere -- exactly the class of
signal that distinguishes a busy box from one whose work is blocked waiting.

Two complementary changes, both cheap procfs/sysfs reads with no measured
decode-latency cost (unlike dense NVML sampling):

- node_exporter (all worker nodes): add the stat, vmstat, pressure, and
  meminfo_numa collectors. The vendored NodeExporterFilter passes the new
  families through its default arm, so no scraper change is needed. Extracted
  into tachometer_node_exporter_command_template() mirroring the DCGM template;
  an explicit recipe node_exporter.command still wins.
- host_sampler (orchestrator/head node, which gets no exporter): add a
  /proc/pressure/{cpu,memory,io} PSI read to each sample. This is that node's
  only PSI source and covers the frontend node the exporters never reach.

Tests: node-exporter template asserts the four new collectors and
explicit-command override; host_sampler PSI parse + CONFIG_PSI-absent cases.

Signed-off-by: Yuewei Na <[email protected]>

…-setup binary

The upstream ncabatoff/process-exporter image is FROM scratch: no shell,
no /root. pyxis/enroot on hecate refuses to start it
(`enroot-switchroot: failed to change directory: /root`, then
`/bin/sh: No such file or directory` once the home is mounted), so the
first run from this branch (hecate 565854) had to relaunch the exporter
by hand as a bare binary. The exporter is a static Go executable that
needs neither a container nor privileges, so ship it the way nats-server
and etcd are shipped: `make setup ARCH=<compute_arch>` downloads the
release tarball into configs/process-exporter and start_tachometer runs
it under plain srun with host paths for the binary and its group file.

- TelemetryExporterConfig.binary: host-native launch mode. Relative
  paths resolve against SRTCTL_SOURCE_DIR / the checkout root, where
  make setup installs host binaries; container_image is ignored.
- DEFAULT_PROCESS_EXPORTER now points at configs/process-exporter with
  an empty container_image. Validation requires binary or
  container_image for every exporter, process_exporter included.
- Missing binary (checkout whose make setup predates this) skips the
  process-exporter leg with a warning instead of failing the run;
  `srtctl apply` warns at submit time as well.
- An explicit process_exporter.container_image with binary unset keeps
  the container launch (group file via the /logs mount).
- Makefile setup step (PROCESS_EXPORTER_VERSION ?= 0.8.7), .gitignore,
  dry-run row shows "host binary configs/process-exporter :9256",
  docs/config-reference.md, tests for both launch modes, the skip path,
  binary resolution and validation.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Yuewei Na <[email protected]>

# Conflicts:
#	src/srtctl/cli/mixins/telemetry_stage.py

…xporterFilter

Two node_exporter families enabled by the combined observability branch lost
their only distinguishing label in the vendored scraper filter, so every series
of the family collapsed into one metric name in the parquet:

- node_memory_numa_*{node="N"} (--collector.meminfo_numa, NVIDIA#415): the generic
  `memory_` arm emitted the bare metric name, folding all NUMA nodes together.
  Now `memory_numa_<x>{numa_node=N}`; host-wide meminfo stays label-free.
- node_processes_state{state="R"|"S"|"D"|...} and
  node_processes_threads_state{thread_state=...} (--collector.processes, NVIDIA#413):
  the default arm keeps only a fixed label whitelist that had neither key.
  `state` and `thread_state` are added to the whitelist.

Label-free stat/vmstat/pressure families were already passed through unchanged;
a test pins that too.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: Yuewei Na <[email protected]>