Add multi-GPU training smoke tests with cross-socket xfail

[hujc7]

Why

The multi-GPU workflow parametrizes each test over one device per shard, which cannot express a job that owns two GPUs at once. Nothing in CI exercised multi-GPU rendering, and NVBUG#6565122 — a heap corruption in Kit's Isaac Sim RTX renderer that only fires when the two GPUs span a cross-socket boundary — shipped uncaught.

What

Smoke tests that launch real two-rank training runs and choose their GPU pair by interconnect class, parsed from nvidia-smi topo -m. On a two-socket 8-GPU host the default cuda:0,cuda:1 is a same-switch pair, so a fixed pick would never exercise the failing path.

Case	Runs when	Verdict
physics-only	≥2 visible CUDA devices	strict — never vacuous
camera × 3 stacks, same-switch	host has a PIX/NVL pair	strict — the regression guard
camera × 3 stacks, cross-socket	host has a SYS pair	xfail for the two Kit-renderer stacks; strict for kitless newton_mjwarp,ovrtx

The unmeasured classes (PXB, PHB, NODE) skip with a reason rather than assert either verdict — every measurement behind this is at the two extremes.

Design notes

• The kitless stack stays strict cross-socket. It passes today; if it ever fails, the conclusion that the fault is the Isaac Sim RTX renderer is wrong.
• The xfail is applied only after the documented signature is confirmed (cudainterop + exit 139), so an OOM or an argument error on a cross-socket pair is not recorded as the known crash.
• A pass requires Training time: in the output, not just exit 0 — an OOM exits 0 through the launcher.
• Silence is the hang signal: 90 s without output kills the run, 600 s hard backstop. The child runs in its own process group and is killed with killpg, since killing only the wrapper leaves rank workers holding GPU memory.
• gpu_pairs_by_topology() fails closed — missing nvidia-smi, non-zero exit, unparsable or truncated matrix, a row outside the header, or a MIG host all yield "skip", never "no boundary present".

Testing

• test_gpu_topology.py — 11 unit tests, fixtures are verbatim output from three real hosts (2-socket 4v4, 1v3 with a NIC column, single-socket PHB/NODE, NVLink). Runs anywhere, including single-GPU CI.
• Reviewed with codex; 8 findings applied, including a missing --entrypoint bash that would have made the CI step run zero tests while reporting green.

⚠️ The smoke tests themselves have not been executed on real GPUs yet — this PR is opened partly so CI runs them. Expect ~35 min on a full two-socket runner (Kit boot dominates at ~4 min per case).