Skip to content

glm: fix CUDA SSD-streaming, and refuse instead of wedging the host - #701

Closed
JonasColmsjo wants to merge 1 commit into
antirez:mainfrom
JonasColmsjo:glm-cuda-streaming-fix
Closed

glm: fix CUDA SSD-streaming, and refuse instead of wedging the host#701
JonasColmsjo wants to merge 1 commit into
antirez:mainfrom
JonasColmsjo:glm-cuda-streaming-fix

Conversation

@JonasColmsjo

Copy link
Copy Markdown

Two bugs on a single-GPU CUDA host, plus the diagnostics needed to find them.

GLM 5.2 produced token soup with --ssd-streaming, and loading it without
--ssd-streaming took the machine down entirely — no error, no log line, just an
unresponsive host. Both are fixed here. Tested on a DGX Spark (GB10, 121 GB
unified) with GLM-5.2-UD-Q2_K_RoutedQ2K.gguf (262 GB).

1. Routed MoE never used the streaming expert compaction on CUDA

ds4_gpu_glm_stream_expert_cache_begin_selected_load_tensor exists and works, but
all five call sites are ROCm, Metal or CPU paths, and neither
ds4_gpu_glm_routed_moe_batch_tensor nor ..._one_tensor reads
g_stream_selected_cache.

So the batch dispatch asked cuda_resolve_weight_ptr for all n_total_expert
experts as one contiguous span
. Streaming can never satisfy that — the cache holds
individual experts — and on single-GPU the resolver answers a miss without an
error (the multi-GPU path logs selective-cache miss). The kernel then multiplied
against memory the model was never written into.

The fix wires it to the same compaction the DeepSeek MXFP4 path already uses at
ds4_cuda.cu:23727: fetch only the experts this batch routes to into a contiguous
buffer, remap the selected ids to index it, and substitute both. The kernels are
untouched
— they already index base + id * stride, which is exactly what
compaction produces. If compaction is unavailable the layer is refused rather than
reading unpopulated memory.

Also stops hardcoding 256 where n_total_expert was already a parameter (it was
explicitly discarded via (void)n_total_expert).

2. The memory guard could not see host memory on Linux

glm_graph_host_memory_bytes() was implemented for macOS only and returned 0
everywhere else, so the guard fell back to a GPU working-set estimate. On an
integrated GPU that is the wrong constraint: device allocations come out of host
RAM, so a model that does not fit does not fail an allocation — it wedges the
machine. This cost four power cycles before I understood it.

Now reads the real total via sysconf(_SC_PHYS_PAGES) and, when refusing, prints
the shortfall and the way out.

3. Diagnostics

None of the above was visible. glm_graph_indexed_prefill_trace_enabled() and
_trace_all() were hardcoded return false, which made every
glm_graph_indexed_prefill_tracef() call site dead code; they now read
DS4_GLM_INDEXED_PREFILL_TRACE / _TRACE_ALL, cached in a static since they are
consulted at every stage boundary of every layer. The two silent entry guards now
name the precondition they rejected instead of returning a bare false out of a
twelve-condition if. Post-router steps and routed-MoE dispatch failures report
themselves, the latter with the expert tensor types — which is what identifies a
quant-support problem. DS4_CUDA_VERIFY_WEIGHT_PTR reports any weight not
resolving to device memory, and an unbounded weight cache on an integrated GPU now
says so.

Verification

  • streaming: "What is the capital of France?"Paris; "reverse a string"
    correct Python
  • compaction engages: GLM batch MoE using compacted experts (compact=166 of 256)
    for a 312-slot batch
  • without --ssd-streaming: refuses in seconds with memory untouched —
    need 249.56 GiB but only 89.63 GiB is available ... retry with --ssd-streaming
  • DeepSeek V4 Flash unaffected, both resident and streaming (regression-checked)

Trace and verify flags are off by default, so output is unchanged unless the env
vars are set.

Supersedes #683, which contained the diagnostics alone.

GLM 5.2 produced token soup on a single-GPU CUDA host with --ssd-streaming, and
loading it without --ssd-streaming took the machine down entirely. Both are
fixed here, along with the diagnostics that were needed to find them.

1. Routed MoE never used the streaming expert compaction on CUDA.

ds4_gpu_glm_stream_expert_cache_begin_selected_load_tensor exists and works,
but all five call sites are ROCm, Metal or CPU paths, and neither
ds4_gpu_glm_routed_moe_batch_tensor nor ..._one_tensor reads
g_stream_selected_cache. So the batch dispatch asked cuda_resolve_weight_ptr
for all n_total_expert experts as one contiguous span. Streaming can never
satisfy that, and on single-GPU the resolver answers a miss without an error,
so the kernel multiplied against memory the model was never written into.

Wire it to the same compaction the DeepSeek MXFP4 path already uses: fetch only
the experts a batch routes to into a contiguous buffer, remap the selected ids
to index it, substitute both. The kernels are untouched -- they already index
base + id * stride, which is what compaction produces. Refuse the layer if
compaction is unavailable rather than reading unpopulated memory. Also stop
hardcoding 256 where n_total_expert was already a parameter.

2. The memory guard could not see host memory on Linux.

glm_graph_host_memory_bytes() was macOS-only and returned 0 elsewhere, so the
guard fell back to a GPU working-set estimate. On an integrated GPU that is the
wrong constraint: device allocations come from host RAM, so an oversized model
does not fail an allocation, it wedges the host. Read the real total via
sysconf(_SC_PHYS_PAGES), and print the shortfall and the remedy when refusing.

3. Diagnostics, because none of this was visible.

The indexed-prefill trace was hardcoded to return false, making every
glm_graph_indexed_prefill_tracef call site dead code; it now reads
DS4_GLM_INDEXED_PREFILL_TRACE / _TRACE_ALL, cached since they are consulted per
stage per layer. The two silent entry guards name the precondition they
rejected. Post-router steps and routed-MoE dispatch failures report themselves,
the latter with expert tensor types. DS4_CUDA_VERIFY_WEIGHT_PTR reports any
weight not resolving to device memory, and an unbounded weight cache on an
integrated GPU now says so.

Verified on a DGX Spark (GB10, 121 GB unified) with
GLM-5.2-UD-Q2_K_RoutedQ2K.gguf (262 GB):
  - streaming: "capital of France" -> Paris; "reverse a string" -> correct Python
  - compaction engages: "compact=166 of 256" for a 312-slot batch
  - without --ssd-streaming: refuses in seconds, memory untouched, and says
    "need 249.56 GiB but only 89.63 GiB is available ... retry with --ssd-streaming"
  - DeepSeek V4 Flash unaffected, resident and streaming

Supersedes antirez#683, which contained the diagnostics alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JonasColmsjo

Copy link
Copy Markdown
Author

Superseded by #717, which covers this plus IQ2_XXS routed-expert support and the host-memory guard, as a single CUDA change. Closing to keep the queue accurate — #717 is the one to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant