Standardize srt-slurm on v1.0.36

[cquil11]

Summary

• add configs/srt-slurm-version.txt as the single srt-slurm version pin
• add a shared checkout helper and route all six multinode SRT launchers through it
• clone only NVIDIA srt-slurm v1.0.36, then overlay InferenceX-owned recipes and setup scripts
• keep the 49 active recipe files referenced by configs/nvidia-master.yaml that are not included in the v1.0.36 release
• declare all 326 checked-in throughput recipes as benchmark.type: custom
• run those custom benchmarks through srt_bench_serving_entrypoint.sh and InferenceX's benchmark_lib.sh

Why the recipe files are present

The 49 added recipes are active CONFIG_FILE targets that do not exist in the v1.0.36 release. They are recipe data, not alternate srt-slurm code. Without the overlay, those matrix entries would fail before submission because their selected YAML files would be missing.

The checked-in throughput recipes now explicitly select the InferenceX custom benchmark command. A central hydrator resolves the selected recipe or override at launch time and bakes its benchmark and topology values into explicit, shell-quoted command arguments. No benchmark environment block is passed to the entrypoint.

Recipes supplied by the immutable upstream v1.0.36 checkout still use upstream's sa-bench type. The same hydrator converts only those selected upstream recipes to the InferenceX custom command in an ephemeral config before srtctl apply.

InferenceX benchmark integration

The custom wrapper preserves the existing throughput contract:

• concurrency sweeps, finite or infinite request rates, prompt and warmup multipliers
• random ISL/OSL ranges and chat-template behavior
• DeepSeek-V4 prompt encoding without importing sa_bench_tokenizers
• served-model and tokenizer paths
• aggregated and disaggregated GPU metadata
• /logs/inferencex-bench_isl_<ISL>_osl_<OSL>/results_concurrency_*.json

The entrypoint hardcodes the InferenceX workspace, frontend address and port, backend, trust behavior, profiling mode, and eval mode. It calls run_benchmark_serving from benchmark_lib.sh directly.

The 14 existing agentic custom recipes retain their own commands and environments.

Compatibility audit

The pinned tag resolves to NVIDIA/srt-slurm commit cb651682117df129162cd276cf567ec70dc57b36.

I compared the previously selected tags, commits, NVIDIA feature branches, cquil11/srt-slurm-nv branches, and the InferenceX chore/upstream-srt-slurm prototype against v1.0.36. The released tag contains the runtime behavior the launchers rely on, including hash-pinned Dynamo source builds, cargo/maturin bootstrap support, --no-preflight, custom benchmark commands and environments, agentic client/orchestrator placement, multi-node health handling, and the node-IP sanitization fix.

The side-branch benchmark prototype depended on an InferenceX custom wrapper. This PR brings that wrapper into the target branch, extends the shared benchmark helper with host, request-rate, and warmup pass-throughs, and adds the DeepSeek-V4 fast-tokenizer fallback to the upstream InferenceX benchmark client. No launcher executes code from a side branch after this change.

Recipe schema migrations:

• replace six legacy benchmark.tokenizer_mode values with the v1.0.36 benchmark.custom_tokenizer import path
• remove benchmark.warmup_req_rate, which is not part of the released schema
• remove benchmark.aiperf_server_metrics from two custom agentic recipes because that opt-in exists only on the fork; the InferenceX agentic runner already treats worker metrics URLs as optional

Validation

• checked out the shared helper and confirmed HEAD == cb651682117df129162cd276cf567ec70dc57b36
• validated all 340 checked-in recipe files against v1.0.36: 326 throughput custom recipes hydrated and 14 agentic custom recipes preserved
• verified there are no remaining sa-bench declarations or references in the checked-in recipe tree
• validated 386 active SRT CONFIG_FILE assignments from configs/nvidia-master.yaml, resolving to 388 valid custom variants; 367 assignments required hydration
• verified every generated InferenceX custom variant has 15 explicit arguments and no benchmark.env
• exercised the entrypoint with conflicting external environment values and confirmed they do not affect the benchmark command
• verified every active recipe setup script exists in the resulting checkout
• generated the fixed-sequence NVIDIA matrix successfully (1,132 entries)
• 224 passed in the matrix logic test suite
• exercised the InferenceX benchmark helper's host, request-rate, warmup, tokenizer, and DSV4 argument propagation
• exercised the DeepSeek-V4 fast-tokenizer fallback and non-DSV4 error path
• bash -n passed for the custom wrapper, shared helper, and all six launchers
• shellcheck passed for the custom wrapper and shared helper
• Python compilation and git diff --check passed

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档