fix(local-ai): use CUDA for allocatable GPU memory

[joelagnel]

Summary

• query the CUDA driver directly for NVIDIA device identity and total/free allocatable memory
• remove NVML and DXGI memory accounting from Local AI qualification
• use CUDA driver compatibility and allocator-visible capacity for both discrete RTX and UMA devices

Closes #1191.

Related work: Dallin's investigation in #1237 and Pedro's follow-up in #1239. Rather than adding or excluding DXGI shared memory based on adapter type, CUDA itself supplies the capacity that its allocator can use. This should handle ordinary discrete RTX GPUs and unified-memory NVIDIA systems with one source of truth.

Validation

• LocalInferenceQualificationTests: 14 passed
• LocalAiGpuVerificationTests and SetupPipelineTests: 33 passed
• ARM64 WinUI build: succeeded (1 existing obsolete Icon warning)
• RTX Spark N1X CUDA probe: 48,719,466,496 bytes total; 48,509,751,296 bytes free

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed August 28, 2026, 11:34 AM ET / 15:34 UTC.

ClawSweeper review

What this changes

The PR replaces NVML/DXGI-based Local AI GPU qualification with direct CUDA-driver probing and migrates affected persisted GPU selectors.

Regression provenance

Possible regression — probable (reviewed change; failure trace). No predecessor PR is attributed.

Merge readiness

⛔ Blocked until stronger real behavior proof is added - 9 items remain

Keep this PR open for correction. It can qualify WDDM systems for models that later fail allocation, and its partial GPU-ID migration makes existing verified Local AI installs terminally unreusable.

Priority: P1
Reviewed head: effc11a2a8f63c49ea1142d831a140040b3e93b4

Review scores

Measure	Result	What it means
Overall readiness	🧂 unranked krab (1/6)	Two source-confirmed P1 regressions and insufficient real runtime evidence prevent merge readiness.
Proof confidence	🦪 silver shellfish (2/6)	Needs stronger real behavior proof before merge: The body shows a CUDA probe result but no current-head successful full model allocation or inference; add redacted llama-server runtime output from the affected WDDM hardware, then update the PR body for a fresh review.
Patch quality	🧂 unranked krab (1/6)	2 actionable review findings remain.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs stronger real behavior proof before merge: The body shows a CUDA probe result but no current-head successful full model allocation or inference; add redacted llama-server runtime output from the affected WDDM hardware, then update the PR body for a fresh review.
Evidence reviewed	7 items	Introduced capacity source: The new probe assigns CUDA's reported total and free memory directly to the values used as Local AI GPU capacity. Eligibility consumes that capacity: Qualification now uses GPU-visible memory alone for model selection and eligibility, so the CUDA total directly controls downloads and full-offload attempts. Concrete WDDM counterexample: The linked hardware investigation records CUDA reporting 46.3 GiB while llama-server failed allocation at 15.81 GiB on Windows ARM64, disproving CUDA total as a sufficient allocation-capacity oracle on that path.
Findings	2 actionable findings	[P1] Keep WDDM eligibility capacity conservative [P1] Migrate existing NVML UUID GPU selectors
Security	None	None.

How this fits together

Local AI setup probes NVIDIA hardware, selects a model, persists the chosen GPU in an installation manifest, then launches llama-server with that GPU selector. The reported capacity controls whether setup downloads and attempts to load a full-offload model.

flowchart LR
A[NVIDIA driver] --> B[GPU hardware probe]
B --> C[Model eligibility]
C --> D[Local AI install manifest]
D --> E[llama-server launch]
E --> F[GPU allocation and inference]

Loading

Before merge

• Add real behavior proof - Needs stronger real behavior proof before merge: The body shows a CUDA probe result but no current-head successful full model allocation or inference; add redacted llama-server runtime output from the affected WDDM hardware, then update the PR body for a fresh review.
• Keep WDDM eligibility capacity conservative (P1) - cuMemGetInfo total becomes the sole eligibility capacity through the qualification policy. The linked Windows ARM64 evidence reports 46.3 GiB from CUDA but a llama-server allocation failure at 15.81 GiB, so this can select and download models that cannot fully offload. Retain a proven dedicated-memory limit on WDDM or establish successful allocation capacity before selection.
• Migrate existing NVML UUID GPU selectors (P1) - Existing manifests persist the NVML UUID, but this matcher only accepts the new PCI ID or a cuda:-prefixed PCI ID. An otherwise verified existing install therefore ends in the terminal uninstall instruction instead of being reused. Preserve or migrate the UUID-backed selector and cover that upgrade path.
• Resolve merge risk (P1) - On WDDM unified-memory hardware, the CUDA total can exceed memory that llama-server can allocate, causing a multi-gigabyte download followed by Local AI setup failure.
• Resolve merge risk (P1) - Existing NVML UUID manifests become terminally unreusable because the new PCI selector has no UUID compatibility path.
• Resolve merge risk (P1) - The PR body has no current-head successful full model allocation or inference proof on the hardware class the change intends to support.
• Improve patch quality - Restore a WDDM-safe capacity guard and add a regression test modeled on the documented over-reporting case.
• Improve patch quality - Preserve verified UUID-backed manifests and test reuse without requiring uninstall.
• Improve patch quality - Add redacted current-head full model allocation and inference output for the affected hardware class.

Findings

• [P1] Keep WDDM eligibility capacity conservative — src/OpenClaw.Shared/Inference/CudaHostHardwareProbe.cs:58-71
• [P1] Migrate existing NVML UUID GPU selectors — src/OpenClaw.SetupEngine/LocalAiInstallReconciler.cs:181-184

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Change surface	14 files affected	The rewrite changes probe, qualification, manifest reuse, setup UI, and regression tests.
Production versus test delta	production +217/-603, tests +84/-113	The broad replacement of GPU-memory sources needs stronger upgrade and hardware validation than the current unit coverage provides.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #1191
Summary: This PR is a candidate fix for the false-positive Local AI qualification tracked by the canonical issue, but its CUDA-only implementation does not yet safely solve that behavior.

Members:

• canonical: Do not count DXGI shared system memory as CUDA capacity on discrete GPUs #1191 - The PR explicitly targets the issue's shared-memory qualification problem.
• partial_overlap: fix(inference,setup): shared-memory capacity semantics and destructive rollback (PR #1237 follow-ups) #1239 - Both changes address shared-memory qualification, while that PR also covers rollback behavior outside this PR's scope.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Restore conservative WDDM qualification (recommended)
   Use a capacity source proven to predict successful llama-server allocation on WDDM, preserve existing UUID-backed installs, and add focused regression coverage before merge.
2. Pause the CUDA-only rewrite
   Do not merge this approach until current-head allocation evidence establishes that the reported capacity is safe across the supported Windows GPU configurations.

Technical review

Best possible solution:

Keep a conservative dedicated-memory eligibility baseline on WDDM, prove any UMA exception with successful full model allocation, and preserve verified UUID-backed manifests through an explicit compatibility path.

Do we have a high-confidence way to reproduce the issue?

Yes. On a Windows WDDM system where CUDA reports a larger total than llama-server can allocate, run the selected full-offload setup recipe; the linked hardware report records this outcome, although it was not executed in this review environment.

Is this the best way to solve the issue?

No. The related WDDM allocation trace shows CUDA total is not a safe standalone eligibility signal, so the fix needs a conservative allocation policy and an explicit legacy-selector compatibility path.

Full review comments:

• [P1] Keep WDDM eligibility capacity conservative — src/OpenClaw.Shared/Inference/CudaHostHardwareProbe.cs:58-71
cuMemGetInfo total becomes the sole eligibility capacity through the qualification policy. The linked Windows ARM64 evidence reports 46.3 GiB from CUDA but a llama-server allocation failure at 15.81 GiB, so this can select and download models that cannot fully offload. Retain a proven dedicated-memory limit on WDDM or establish successful allocation capacity before selection.
  Confidence: 0.98
• [P1] Migrate existing NVML UUID GPU selectors — src/OpenClaw.SetupEngine/LocalAiInstallReconciler.cs:181-184
  Existing manifests persist the NVML UUID, but this matcher only accepts the new PCI ID or a cuda:-prefixed PCI ID. An otherwise verified existing install therefore ends in the terminal uninstall instruction instead of being reused. Preserve or migrate the UUID-backed selector and cover that upgrade path.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.98

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 56a169f5b71e.

Labels

Label justifications:

• P1: The introduced eligibility path can make Local AI select and download a model that fails during allocation for real users.
• merge-risk: 🚨 compatibility: The changed GPU identifier format rejects verified installations persisted by the existing NVML-based implementation.
• merge-risk: 🚨 availability: CUDA-reported capacity can overstate successful WDDM allocation capacity and make Local AI setup fail.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The body shows a CUDA probe result but no current-head successful full model allocation or inference; add redacted llama-server runtime output from the affected WDDM hardware, then update the PR body for a fresh review.

Evidence

What I checked:

• Introduced capacity source: The new probe assigns CUDA's reported total and free memory directly to the values used as Local AI GPU capacity. (src/OpenClaw.Shared/Inference/CudaHostHardwareProbe.cs:58, effc11a2a8f6)
• Eligibility consumes that capacity: Qualification now uses GPU-visible memory alone for model selection and eligibility, so the CUDA total directly controls downloads and full-offload attempts. (src/OpenClaw.Shared/Inference/Catalog/LocalInferenceSelector.cs:141, effc11a2a8f6)
• Concrete WDDM counterexample: The linked hardware investigation records CUDA reporting 46.3 GiB while llama-server failed allocation at 15.81 GiB on Windows ARM64, disproving CUDA total as a sufficient allocation-capacity oracle on that path.
• Existing selector format: The default-branch NVML probe persisted the NVML UUID as the stable GPU ID, whereas the new probe uses PCI bus IDs. (src/OpenClaw.Shared/Inference/NvmlHostHardwareProbe.cs:103, 56a169f5b71e)
• Incomplete upgrade compatibility: The introduced matcher accepts equality or a cuda-prefixed PCI selector only, so an existing NVML UUID cannot match the new PCI selector. (src/OpenClaw.SetupEngine/LocalAiInstallReconciler.cs:181, effc11a2a8f6)
• User-visible failure path: A selector mismatch becomes a terminal setup result instructing the user to uninstall before retrying, rather than reusing the verified installation. (src/OpenClaw.SetupEngine/LocalAiSetupSteps.cs:278, effc11a2a8f6)

Likely related people:

• Joel Fernandes: Git history attributes the managed Local AI feature and this CUDA qualification rewrite to Joel. (role: original Local AI implementation author and current GPU-probe contributor; confidence: high; commits: ccd64bbb7d68, effc11a2a8f6; files: src/OpenClaw.Shared/Inference/NvmlHostHardwareProbe.cs, src/OpenClaw.Shared/Inference/CudaHostHardwareProbe.cs, src/OpenClaw.SetupEngine/LocalAiInstallReconciler.cs)
• Pedro Larroy: Pedro authored the recent llama.cpp runtime update in the affected Local AI history. (role: recent Local AI runtime contributor; confidence: medium; commits: f2a96a596c01; files: src/OpenClaw.Shared/Inference/Catalog/LocalInferenceSelector.cs)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (3 earlier review cycles)

• reviewed 2026-08-28T03:41:51.892Z sha c6765a7 :: needs real behavior proof before merge. :: [P1] Keep WDDM allocation capacity conservative
• reviewed 2026-08-28T07:33:01.892Z sha aff93c1 :: needs real behavior proof before merge. :: [P1] Keep WDDM eligibility capacity conservative
• reviewed 2026-08-28T14:40:08.771Z sha f84912d :: needs real behavior proof before merge. :: [P1] Keep WDDM eligibility capacity conservative | [P1] Preserve existing Local AI GPU identifiers on upgrade

[larroy]

Approach looks good.

We should add a test if possible for the case where Cuda detection fails or can't be initialized.

[larroy]

we should surface an error to the user here if cuda is not found or can't be initialized

[larroy]

Suggest to refactor this in a more functional idiom:

private static IReadOnlyList<GpuInfo> CaptureCudaGpus()
{
    if (!OperatingSystem.IsWindows() || CuInit(0) != CudaSuccess ||
        CuDeviceGetCount(out int count) != CudaSuccess)
    {
        return [];
    }

    int? cudaMajorVersion =
        CuDriverGetVersion(out int driverVersion) == CudaSuccess && driverVersion > 0
            ? driverVersion / 1000
            : null;

    return Enumerable.Range(0, count)
        .Select(ordinal => TryCaptureGpu(ordinal, cudaMajorVersion))
        .Where(gpu => gpu is not null)
        .Select(gpu => gpu!)
        .ToList();
}

private static GpuInfo? TryCaptureGpu(int ordinal, int? cudaMajorVersion)
{
    if (CuDeviceGet(out int device, ordinal) != CudaSuccess)
        return null;

    string? name = ReadDeviceName(device);
    string? pciBusId = ReadPciBusId(device);
    if (string.IsNullOrWhiteSpace(name) || string.IsNullOrWhiteSpace(pciBusId))
        return null;

    return WithCudaContext(device, () =>
    {
        if (CuMemGetInfo(out nuint freeBytes, out nuint totalBytes) != CudaSuccess ||
            totalBytes == 0 || totalBytes > long.MaxValue || freeBytes > totalBytes)
        {
            return null;
        }

        return new GpuInfo(
            GpuVendor.Nvidia,
            name,
            GpuVisibleMemoryBytes: (long)totalBytes,
            FreeGpuVisibleMemoryBytes: (long)freeBytes,
            CudaMajorVersion: cudaMajorVersion,
            StableId: ToCudaVisibleDevicesSelector(pciBusId));
    });
}

private static GpuInfo? WithCudaContext(int device, Func<GpuInfo?> action)
{
    if (CuCtxCreate(out IntPtr context, 0, device) != CudaSuccess)
        return null;

    try
    {
        return action();
    }
    finally
    {
        _ = CuCtxDestroy(context);
    }
}