feat(power): validate multinode GPU energy metrics / 校验多机 GPU 能耗指标

[edwingao28]

Stacked on #2323 — base is wenyao/validated-single-node-power. After #2323 merges this will rebase and retarget to main. Only the last 3 commits are new.

What this does

Consumes the srt-slurm dcgm-power artifact package (v1 wire contract: power/{manifest.json, samples.csv, windows/*.json}) for disaggregated multinode runs, re-validates it independently, and patches energy metrics into the per-concurrency agg JSON:

• the 8 whole-deployment metrics from feat(power): validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323 (power_valid, avg_power_w, avg_total_gpu_power_w, total_gpu_energy_j, joules_per_successful_query, joules_per_input_token, joules_per_output_token, joules_per_total_token)
• 4 role metrics: prefill_gpu_energy_j, decode_gpu_energy_j, prefill_joules_per_input_token, decode_joules_per_output_token. Role energy is board-level energy over the full serving window. It is not kernel-phase energy.

Trust model: the producer's stored publication_valid is never trusted. The consumer recomputes validity from the persisted bytes, mirroring srt-slurm's own offline validator (identity sextet, strict sample parsing, per-device UUID 1:1, timestamp monotonicity, window status invariants, fixed 3.0 s max sample gap, window bracketing, stored-evidence equality). It then cross-checks the stored verdict; any disagreement is producer_verdict_mismatch and power_valid=0. The producer git commit must equal the POWER_PRODUCER_SHA pin in every mode. The workspace result is bound to its measurement window by canonical-JSON SHA256 against the original at LOGS/<result_path> — the launcher renames results when copying, so names can't be compared.

Default mode is best-effort: invalid telemetry publishes power_valid=0 plus a validation sidecar and never fails the benchmark. REQUIRE_POWER=1 fails the job after the audit artifacts exist.

Validated against a real GB200 artifact

Canary run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30587402515 (Qwen3.5-397B FP8, 1P1D TP4, concurrency 4, producer pin 6609d46a4c74ed66fc2f7014d7e790efe1d23bde).

• srt-slurm's offline validator at the pinned commit: exit 0, publication_valid: True recomputed.
• This consumer replayed offline on the same package: power_valid=1, avg 343.72 W per GPU (2749.74 W across 8), total_gpu_energy_j=189898.14 over the 69.06 s window, prefill 73214.34 J + decode 116683.80 J (sums exactly).
• An independent stdlib-only trapezoid recompute from raw samples.csv agrees to 2e-10 relative error.
• The replay caught one real bug before merge: GB200 1P1D launches as a plain 2-node Slurm job, so every device reports het_group=None. The topology gate now enforces het-group distinctness only when het groups are present. Het-job deployments (e.g. GB300) keep the strict check.

Known deltas (documented, left unchanged)

• _load_benchmark_data (feat(power): validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323) bases its duration tolerance on wall time; the producer uses the reported duration. The divergence is reject-side only: a ~1% clock-skew band on >50 s runs where the consumer withholds metrics the producer would accept. Not touched here to keep feat(power): validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323's reviewed code intact.
• The single-node run() shares the raise-before-write pattern this PR closes in the multinode path (non-finite metrics now flip the verdict to invalid instead of dying mid-patch). The single-node path is untouched.

Tests

• utils/test_aggregate_power_multinode.py — 22 tests. A self-consistent package builder, then every contract gate tampered one at a time; each rejection asserted in both modes.
• utils/test_process_result.py::TestMultinodePower — 5 wiring tests through the real subprocess path.
• .github/workflows/test-process-result.yml now collects the new suite and triggers on the new files.

---

中文：

堆叠在 #2323 上——base 为 wenyao/validated-single-node-power，#2323 合并后会 rebase 并改回 main，只需 review 最后 3 个 commit。

本 PR 消费 srt-slurm 多机 dcgm-power artifact（v1 wire contract），向 agg JSON 写入 8 个整机指标和 4 个 role 指标。prefill/decode 能耗是整个服务窗口的板级能耗，不是算子阶段能耗。

信任模型：不信任 producer 存储的 publication_valid。消费端从持久化字节独立复算（与 srt-slurm 离线校验器同一套检查），再与 stored verdict 交叉校验，不一致即 producer_verdict_mismatch。producer SHA 必须命中 POWER_PRODUCER_SHA pin。workspace 结果与 window 通过 canonical JSON SHA256 绑定（launcher 拷贝时会改名，不能比文件名）。默认 best-effort，无效遥测只记 power_valid=0 不失败；REQUIRE_POWER=1 才严格失败。

已用真实 GB200 canary artifact 离线验证（数据见上），并借此在合并前修掉一个真实 bug：GB200 1P1D 是普通两节点 Slurm 作业，所有设备 het_group=None，拓扑门限现在只在 het group 存在时强制互异（GB300 het 作业仍严格）。

单元测试 22 个（逐一破坏每个 contract gate，双模式断言）+ process_result 接线测试 5 个；test-process-result workflow 已接入新文件。