fix(serve): stop hardcoding vLLM GPU memory utilization (EAI-7353)

[rominf]

Problem

Serving a 0.5B model on a large-VRAM machine reserved roughly 200 GB of VRAM.

The vLLM adapter always passed --gpu-memory-utilization 0.80, and vLLM reserves that fraction of total device VRAM for its KV cache regardless of how small the model is. There was no way for a user to override it.

Changes

• Stop overriding vLLM's default. When the user does not ask for a value, no --gpu-memory-utilization argument is passed at all, so vLLM's own default applies. Chosen over hardcoding vLLM's current 0.9 so the value cannot drift out of sync with upstream.
• Add rocm serve --gpu-memory-utilization <FRACTION> so the value can be set explicitly. The flag is vLLM-scoped and follows the existing --tool-call-parser precedent: same warn-and-ignore behavior on other engines, routed through EngineRecipeHint.required_flags, so the engine protocol is unchanged.
• Validate 0 < value <= 1 with an error that names the flag.
• Extract a pure vllm_serve_args helper so the launch argv is unit-testable.
• Update docs/vllm.md: drop the now-stale "0.80 by default for display/WSL headroom" rationale, document the flag, and state that it is a fraction of total — not free — VRAM.
• Update docs/testing.md: point the vLLM TheRock acceptance runbook at the renamed test so the documented command runs.

Why an engine-scoped flag

An engine-agnostic knob was considered and rejected: llama.cpp/lemonade has no fractional-VRAM-reservation concept, and multi-engine wrappers in the wider ecosystem (Xinference, LocalAI, RamaLama, KubeAI, Jan, llama-swap) all decline to unify this for the same reason. gpu-memory-utilization is also the near-universal spelling (vLLM, TGI's equivalent, Ray Serve, KServe, KubeAI, NIM).

Behavior change

On machines where the old 0.80 was providing display or WSL headroom, the effective reservation now rises to vLLM's default. Passing --gpu-memory-utilization 0.8 restores the previous behavior.

Related: the e2e free-VRAM floor in tests/e2e-cucumber/tests/e2e/serving_steps.rs caps at 90% of total, which now sits exactly at vLLM's default instead of having slack. That is a tight tolerance on small cards and may deserve a deliberate margin in a follow-up.

Test plan

• 7 new unit tests plus one rewritten (resolve_model_surfaces_conservative_vram_default → resolve_model_omits_gpu_memory_utilization_default) across engines/vllm and apps/rocm: flag absent (no argument emitted), flag present (value reaches argv), and invalid input (0, 1.5, -0.2, abc, NaN, inf, empty).
• Fails-before verified by temporarily restoring the hardcoded flag/value pair.
• No expectations.toml rows reference this ticket, so none needed narrowing.

Risk: low for the code path; medium for users who relied on the implicit 0.80 headroom, which the docs now cover.

[juhovainio]

Review: fix(serve): stop hardcoding vLLM GPU memory utilization (EAI-7353)

Verification performed: checked out the branch locally, confirmed the diff scope, ran the full affected test suites (rocm, rocm-engine-vllm), and ran clippy on both crates. Everything compiles, all 8 new/changed tests pass, clippy is clean.

What it does

Removes rocm-cli's hardcoded --gpu-memory-utilization 0.80 for vLLM (which was reserving that fraction of total VRAM regardless of model size). Now vLLM's own default applies unless the user passes the new rocm serve --gpu-memory-utilization <FRACTION>, threaded through the existing EngineRecipeHint.required_flags channel (same mechanism --tool-call-parser already uses), so no engine protocol change is needed.

Code quality

• Clean, minimal diff that reuses an existing extension point rather than inventing a new one.
• vllm_serve_args extraction into a pure, unit-testable function is a solid improvement — argv construction was previously only exercisable via a live process spawn.
• Validation (parse_gpu_memory_utilization) happens before anything launches, giving a fast, actionable error instead of a buried vLLM argparse failure.
• The note-plumbing (gpu_memory_utilization_note stored without a note: prefix, added by each rendering path) is a little subtle but I confirmed both call sites (plan path and serve_summary::render_summary) apply the prefix exactly once — no duplication.

Issues / things worth a second look

• allow_hyphen_values = true on the new flag is a real, self-acknowledged tradeoff: --gpu-memory-utilization with no value will swallow the next argument (e.g. --gpu-memory-utilization --verbose eats --verbose as the value, then fails with an error naming the flag but showing --verbose as the bad input). Documented in a comment — worth an explicit nod that this is accepted, not an oversight.
• Behavior change is real and user-facing: deployments relying on the old 0.80 ceiling now reserve whatever vLLM's own (typically higher) default is. Well documented in both the PR description and docs/vllm.md, with the exact flag to restore old behavior — good practice for a breaking-ish change.
• Self-identified follow-up risk (not a blocker): the e2e free-VRAM floor (MAX_FREE_VRAM_FLOOR_MIB) now sits right at vLLM's default utilization instead of having slack. Worth a tracking ticket if one doesn't exist.
• Minor: format!("{value}") for a whole-number fraction like 1.0 emits "1" rather than "1.0" — cosmetic only, Python's argparse float type accepts both.

Test coverage

Thorough — 7 new tests + 1 rename, covering: flag absent (no default injected), explicit value reaching argv, replacement of a recipe-authored value while preserving other flags, non-vLLM engines left untouched, the full invalid-input matrix (0, 1.5, -0.2, abc, empty, NaN, inf), and that the "ignored for this engine" note reaches the interactive summary path (not just the plan path).

Security

No concerns — local CLI argument validation and Command::arg construction, no shell injection surface.

Verdict

Solid, well-tested, well-documented change. No blocking issues found. The allow_hyphen_values edge case and the e2e VRAM-floor tolerance are worth author awareness but shouldn't hold up merging.