glm: fix CUDA SSD-streaming, and refuse instead of wedging the host - #701
Closed
JonasColmsjo wants to merge 1 commit into
Closed
glm: fix CUDA SSD-streaming, and refuse instead of wedging the host#701JonasColmsjo wants to merge 1 commit into
JonasColmsjo wants to merge 1 commit into
Conversation
GLM 5.2 produced token soup on a single-GPU CUDA host with --ssd-streaming, and
loading it without --ssd-streaming took the machine down entirely. Both are
fixed here, along with the diagnostics that were needed to find them.
1. Routed MoE never used the streaming expert compaction on CUDA.
ds4_gpu_glm_stream_expert_cache_begin_selected_load_tensor exists and works,
but all five call sites are ROCm, Metal or CPU paths, and neither
ds4_gpu_glm_routed_moe_batch_tensor nor ..._one_tensor reads
g_stream_selected_cache. So the batch dispatch asked cuda_resolve_weight_ptr
for all n_total_expert experts as one contiguous span. Streaming can never
satisfy that, and on single-GPU the resolver answers a miss without an error,
so the kernel multiplied against memory the model was never written into.
Wire it to the same compaction the DeepSeek MXFP4 path already uses: fetch only
the experts a batch routes to into a contiguous buffer, remap the selected ids
to index it, substitute both. The kernels are untouched -- they already index
base + id * stride, which is what compaction produces. Refuse the layer if
compaction is unavailable rather than reading unpopulated memory. Also stop
hardcoding 256 where n_total_expert was already a parameter.
2. The memory guard could not see host memory on Linux.
glm_graph_host_memory_bytes() was macOS-only and returned 0 elsewhere, so the
guard fell back to a GPU working-set estimate. On an integrated GPU that is the
wrong constraint: device allocations come from host RAM, so an oversized model
does not fail an allocation, it wedges the host. Read the real total via
sysconf(_SC_PHYS_PAGES), and print the shortfall and the remedy when refusing.
3. Diagnostics, because none of this was visible.
The indexed-prefill trace was hardcoded to return false, making every
glm_graph_indexed_prefill_tracef call site dead code; it now reads
DS4_GLM_INDEXED_PREFILL_TRACE / _TRACE_ALL, cached since they are consulted per
stage per layer. The two silent entry guards name the precondition they
rejected. Post-router steps and routed-MoE dispatch failures report themselves,
the latter with expert tensor types. DS4_CUDA_VERIFY_WEIGHT_PTR reports any
weight not resolving to device memory, and an unbounded weight cache on an
integrated GPU now says so.
Verified on a DGX Spark (GB10, 121 GB unified) with
GLM-5.2-UD-Q2_K_RoutedQ2K.gguf (262 GB):
- streaming: "capital of France" -> Paris; "reverse a string" -> correct Python
- compaction engages: "compact=166 of 256" for a 312-slot batch
- without --ssd-streaming: refuses in seconds, memory untouched, and says
"need 249.56 GiB but only 89.63 GiB is available ... retry with --ssd-streaming"
- DeepSeek V4 Flash unaffected, resident and streaming
Supersedes antirez#683, which contained the diagnostics alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bugs on a single-GPU CUDA host, plus the diagnostics needed to find them.
GLM 5.2 produced token soup with
--ssd-streaming, and loading it without--ssd-streamingtook the machine down entirely — no error, no log line, just anunresponsive host. Both are fixed here. Tested on a DGX Spark (GB10, 121 GB
unified) with
GLM-5.2-UD-Q2_K_RoutedQ2K.gguf(262 GB).1. Routed MoE never used the streaming expert compaction on CUDA
ds4_gpu_glm_stream_expert_cache_begin_selected_load_tensorexists and works, butall five call sites are ROCm, Metal or CPU paths, and neither
ds4_gpu_glm_routed_moe_batch_tensornor..._one_tensorreadsg_stream_selected_cache.So the batch dispatch asked
cuda_resolve_weight_ptrfor alln_total_expertexperts as one contiguous span. Streaming can never satisfy that — the cache holds
individual experts — and on single-GPU the resolver answers a miss without an
error (the multi-GPU path logs
selective-cache miss). The kernel then multipliedagainst memory the model was never written into.
The fix wires it to the same compaction the DeepSeek MXFP4 path already uses at
ds4_cuda.cu:23727: fetch only the experts this batch routes to into a contiguousbuffer, remap the selected ids to index it, and substitute both. The kernels are
untouched — they already index
base + id * stride, which is exactly whatcompaction produces. If compaction is unavailable the layer is refused rather than
reading unpopulated memory.
Also stops hardcoding
256wheren_total_expertwas already a parameter (it wasexplicitly discarded via
(void)n_total_expert).2. The memory guard could not see host memory on Linux
glm_graph_host_memory_bytes()was implemented for macOS only and returned0everywhere else, so the guard fell back to a GPU working-set estimate. On an
integrated GPU that is the wrong constraint: device allocations come out of host
RAM, so a model that does not fit does not fail an allocation — it wedges the
machine. This cost four power cycles before I understood it.
Now reads the real total via
sysconf(_SC_PHYS_PAGES)and, when refusing, printsthe shortfall and the way out.
3. Diagnostics
None of the above was visible.
glm_graph_indexed_prefill_trace_enabled()and_trace_all()were hardcodedreturn false, which made everyglm_graph_indexed_prefill_tracef()call site dead code; they now readDS4_GLM_INDEXED_PREFILL_TRACE/_TRACE_ALL, cached in a static since they areconsulted at every stage boundary of every layer. The two silent entry guards now
name the precondition they rejected instead of returning a bare
falseout of atwelve-condition
if. Post-router steps and routed-MoE dispatch failures reportthemselves, the latter with the expert tensor types — which is what identifies a
quant-support problem.
DS4_CUDA_VERIFY_WEIGHT_PTRreports any weight notresolving to device memory, and an unbounded weight cache on an integrated GPU now
says so.
Verification
Paris; "reverse a string" →correct Python
GLM batch MoE using compacted experts (compact=166 of 256)for a 312-slot batch
--ssd-streaming: refuses in seconds with memory untouched —need 249.56 GiB but only 89.63 GiB is available ... retry with --ssd-streamingTrace and verify flags are off by default, so output is unchanged unless the env
vars are set.
Supersedes #683, which contained the diagnostics alone.