Skip to content

glm: make prefill failures diagnosable - #683

Closed
JonasColmsjo wants to merge 1 commit into
antirez:mainfrom
JonasColmsjo:glm-prefill-diagnostics
Closed

glm: make prefill failures diagnosable#683
JonasColmsjo wants to merge 1 commit into
antirez:mainfrom
JonasColmsjo:glm-prefill-diagnostics

Conversation

@JonasColmsjo

Copy link
Copy Markdown

Problem

When GLM prefill fails, the only output is:

ds4-agent: cuda GLM prefill failed

No layer, no stage, no cause. glm_graph_forward_indexed_tokens is ~1300 lines
and reaches that message through a long ok-chain, so the failure could be
anywhere in it or its callees.

I hit this running GLM 5.2 on a DGX Spark (GB10) and spent hours ruling out
things it wasn't: memory (identical failure at 10.95 GiB and 84.82 GiB planned,
with 118 GB free), context size (100000 down to 8192), prompt length, and the
SSD-streaming expert cache (20 GB through 60 GB, with and without explicit
preload). No CUDA error is surfaced either — cuda_ok() prints one, but this
path never reaches it, so the failure is indistinguishable from an OOM.

Change

Four things, all diagnostic — no behaviour change on the success path.

Enable the indexed-prefill trace. glm_graph_indexed_prefill_trace_enabled()
and glm_graph_indexed_prefill_trace_all() were hardcoded return false, which
made every glm_graph_indexed_prefill_tracef() call site dead code. They now
read DS4_GLM_INDEXED_PREFILL_TRACE / _TRACE_ALL. Both cache the lookup in a
static, since they are consulted at every stage boundary of every layer.

Name the rejected precondition in the two entry guards
(glm_graph_forward_indexed_tokens and the routed-MoE encoder). Each was a
single if of a dozen conditions returning a bare false.

Report which post-router step failed. No stage boundary separates
profile_router_selection_batch, capture_prefill_seed_router_selected and
seed_streaming_expert_cache_from_full_layer, so a failure in any of them
looked the same.

Report routed-MoE dispatch failures with the expert tensor types, which is
what actually identifies a quant-support problem.

Result

The same run now reports:

ds4: GLM stage reached glm_ffn.router layer=3 pos0=0 n_tokens=943
ds4: GLM routed_moe_batch_dispatch failed layer=3 pos0=0 n_tokens=943 \
     expert types gate/up/down=16/16/16

That points straight at the real cause: ds4_gpu_glm_routed_moe_batch_tensor
accepts only Q2_K (type 10) experts, so a GGUF with IQ2_XXS (type 16) routed
experts is rejected at the first routed layer, while the leading dense Q8_0
layers succeed. Its own "unsupported types" message never prints because the
dispatch rejects host-side before reaching it.

Tested with GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf.

Verification

make cuda-spark (GB10 / sm_121), no new warnings. Trace off by default, so
output is unchanged unless the env var is set.

A GLM prefill failure surfaces only as "GLM prefill failed" with no layer,
no stage and no cause. Diagnosing one currently means bisecting
glm_graph_forward_indexed_tokens and its callees by hand.

- Enable the indexed-prefill trace. glm_graph_indexed_prefill_trace_enabled()
  and _trace_all() were hardcoded to return false, so every
  glm_graph_indexed_prefill_tracef() call site was dead code. They now read
  DS4_GLM_INDEXED_PREFILL_TRACE / _TRACE_ALL, cached in a static because they
  are consulted at every stage boundary of every layer.

- Name the rejected precondition in the two entry guards
  (glm_graph_forward_indexed_tokens, the routed-MoE encoder) instead of
  returning a bare false out of a twelve-condition if.

- Report which of the four post-router steps failed. No stage boundary
  separates them, so they were previously indistinguishable.

- Report a routed-MoE dispatch failure together with the expert tensor types.

Concretely: a GLM 5.2 IQ2_XXS GGUF on CUDA (DGX Spark, GB10) now says

  GLM routed_moe_batch_dispatch failed layer=3 pos0=0 n_tokens=943 \
      expert types gate/up/down=16/16/16

which points straight at ds4_gpu_glm_routed_moe_batch_tensor accepting only
Q2_K (type 10) experts, so the first routed layer is rejected while the
leading dense Q8_0 layers succeed. Before this the same failure was
indistinguishable from an out-of-memory condition, and I first spent hours
ruling out memory, context size and the SSD-streaming expert cache.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JonasColmsjo

Copy link
Copy Markdown
Author

Superseded by #717, which covers this plus IQ2_XXS routed-expert support and the host-memory guard, as a single CUDA change. Closing to keep the queue accurate — #717 is the one to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant