glm: make prefill failures diagnosable - #683
Closed
JonasColmsjo wants to merge 1 commit into
Closed
Conversation
A GLM prefill failure surfaces only as "GLM prefill failed" with no layer,
no stage and no cause. Diagnosing one currently means bisecting
glm_graph_forward_indexed_tokens and its callees by hand.
- Enable the indexed-prefill trace. glm_graph_indexed_prefill_trace_enabled()
and _trace_all() were hardcoded to return false, so every
glm_graph_indexed_prefill_tracef() call site was dead code. They now read
DS4_GLM_INDEXED_PREFILL_TRACE / _TRACE_ALL, cached in a static because they
are consulted at every stage boundary of every layer.
- Name the rejected precondition in the two entry guards
(glm_graph_forward_indexed_tokens, the routed-MoE encoder) instead of
returning a bare false out of a twelve-condition if.
- Report which of the four post-router steps failed. No stage boundary
separates them, so they were previously indistinguishable.
- Report a routed-MoE dispatch failure together with the expert tensor types.
Concretely: a GLM 5.2 IQ2_XXS GGUF on CUDA (DGX Spark, GB10) now says
GLM routed_moe_batch_dispatch failed layer=3 pos0=0 n_tokens=943 \
expert types gate/up/down=16/16/16
which points straight at ds4_gpu_glm_routed_moe_batch_tensor accepting only
Q2_K (type 10) experts, so the first routed layer is rejected while the
leading dense Q8_0 layers succeed. Before this the same failure was
indistinguishable from an out-of-memory condition, and I first spent hours
ruling out memory, context size and the SSD-streaming expert cache.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 5, 2026
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When GLM prefill fails, the only output is:
No layer, no stage, no cause.
glm_graph_forward_indexed_tokensis ~1300 linesand reaches that message through a long
ok-chain, so the failure could beanywhere in it or its callees.
I hit this running GLM 5.2 on a DGX Spark (GB10) and spent hours ruling out
things it wasn't: memory (identical failure at 10.95 GiB and 84.82 GiB planned,
with 118 GB free), context size (100000 down to 8192), prompt length, and the
SSD-streaming expert cache (20 GB through 60 GB, with and without explicit
preload). No CUDA error is surfaced either —
cuda_ok()prints one, but thispath never reaches it, so the failure is indistinguishable from an OOM.
Change
Four things, all diagnostic — no behaviour change on the success path.
Enable the indexed-prefill trace.
glm_graph_indexed_prefill_trace_enabled()and
glm_graph_indexed_prefill_trace_all()were hardcodedreturn false, whichmade every
glm_graph_indexed_prefill_tracef()call site dead code. They nowread
DS4_GLM_INDEXED_PREFILL_TRACE/_TRACE_ALL. Both cache the lookup in astatic, since they are consulted at every stage boundary of every layer.
Name the rejected precondition in the two entry guards
(
glm_graph_forward_indexed_tokensand the routed-MoE encoder). Each was asingle
ifof a dozen conditions returning a barefalse.Report which post-router step failed. No stage boundary separates
profile_router_selection_batch,capture_prefill_seed_router_selectedandseed_streaming_expert_cache_from_full_layer, so a failure in any of themlooked the same.
Report routed-MoE dispatch failures with the expert tensor types, which is
what actually identifies a quant-support problem.
Result
The same run now reports:
That points straight at the real cause:
ds4_gpu_glm_routed_moe_batch_tensoraccepts only Q2_K (type 10) experts, so a GGUF with IQ2_XXS (type 16) routed
experts is rejected at the first routed layer, while the leading dense Q8_0
layers succeed. Its own
"unsupported types"message never prints because thedispatch rejects host-side before reaching it.
Tested with
GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf.Verification
make cuda-spark(GB10 / sm_121), no new warnings. Trace off by default, sooutput is unchanged unless the env var is set.