Skip to content

[Feature] Add GLM-5.3-Flash F4: mHC four-stream residual - #2107

Open
jayhenry wants to merge 1 commit into
feat/glm53flash-f3-kdafrom
feat/glm53flash-f4-mhc
Open

jayhenry wants to merge 1 commit into
feat/glm53flash-f3-kdafrom
feat/glm53flash-f4-mhc

Conversation

@jayhenry

@jayhenry jayhenry commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Stack (bottom to top):

  1. [Feature] Add GLM-5.3-Flash F0: 25B cropped reference checkpoint builder #2105 feat/glm53flash-materialize-full-f0 → main
  2. [Feature] Add GLM-5.3-Flash F3: Kimi Delta Attention (KDA) #2106 feat/glm53flash-f3-kda → feat/glm53flash-materialize-full-f0
  3. [Feature] Add GLM-5.3-Flash F4: mHC four-stream residual #2107 feat/glm53flash-f4-mhc → feat/glm53flash-f3-kda ← you are here
  4. [Feature] Add GLM-5.3-Flash F5: NoPE DSA + KPool indexer + clamped SwiGLU #2108 feat/glm53flash-f5-nope-dsa → feat/glm53flash-f4-mhc
  5. [Feature] Add GLM-5.3-Flash F1: VL data preprocessing pipeline #2109 feat/glm53flash-f1-vl-data → feat/glm53flash-f5-nope-dsa
  6. [Feature] Add GLM-5.3-Flash F2: vision tower + projector (eager) #2110 feat/glm53flash-f2-vision-tower → feat/glm53flash-f1-vl-data
  7. [Feature] Add GLM-5.3-Flash F6 core: text model + MTP + compose model #2111 feat/glm53flash-f6-text-moe → feat/glm53flash-f2-vision-tower

Base is #2106's branch (layer 2). Review only this PR's own diff.


Summary

Stack layer 3/7 of GLM-5.3-Flash support (base: layer 2, F3 KDA).

Adds xtuner/v1/module/decoder_layer/mhc.py (MHCConfig / hc_split_sinkhorn / hc_pre / hc_post / unshard_hc_params) and xtuner/v1/ops/hc_post.py (fused Triton hc_post_fused), per doc/xtuner_glm5p3flash_design.md F4. The core math is ported from xtuner's dsv4 branch (DeepSeek-V4's Hyper-Connections) into the model-agnostic public location the design doc calls for, dropping V4's XTUNER_V4_HF_PARITY global toggle and the unported 721-line TileKernels backend (left as a documented NotImplementedError gap rather than a blind, unvalidated port).

Adds xtuner/v1/model/moe/glm53/decoder_layer.py: Glm53DenseDecoderLayer/Glm53MoEDecoderLayer, which only override the pre/post-forward seams so the existing EP/dispatcher/domino-micro-batch pipeline stays untouched — the mHC residual rides through as an opaque _MHCResidual payload in place of a plain Tensor. mhc_cfg=None degrades both layers to ordinary residual math unchanged, for reuse by the MTP layer later in the stack.

Also fixes a real bug surfaced while testing: KDA's dt_bias/A_log were left torch.empty-uninitialized; now initialized to match HF's Glm5NextTextForgetGate._init_weights policy.

Test Plan

26 tests across tests/model/test_glm53_mhc.py (Sinkhorn math + exact HF parity in fp32, bf16 parity within ULP tolerance), tests/ops/test_hc_post.py (fused kernel vs eager reference, forward/backward/compile), and tests/model/test_glm53_decoder_layer.py (both decoder layers, mhc_cfg=None passthrough equivalence, gradient flow).

Adds xtuner/v1/module/decoder_layer/mhc.py (MHCConfig / hc_split_sinkhorn /
hc_pre / hc_post / unshard_hc_params) and xtuner/v1/ops/hc_post.py (fused
Triton hc_post_fused), following doc/xtuner_glm5p3flash_design.md F4. The
core math is ported from xtuner's dsv4 branch (DeepSeek-V4's Hyper-
Connections, commit 01c31a8, not merged into this branch) into the
model-agnostic public location the design doc calls for, dropping V4's
XTUNER_V4_HF_PARITY global toggle (the default bf16-fast path already
degrades to HF-exact math under fp32 inputs, which the tests use as the
parity anchor) and the unported 721-line TileKernels backend (left as a
documented NotImplementedError gap rather than a blind port with no
hardware to validate it against).

Adds xtuner/v1/model/moe/glm53/decoder_layer.py: Glm53DenseDecoderLayer
(overrides DenseDecoderLayer._forward) and Glm53MoEDecoderLayer, which only
overrides MoEDecoderLayer's _pre_moe_forward/_post_moe_forward seams so the
~400-line EP/dispatcher/domino-micro-batch pipeline stays untouched; the
mHC residual rides through those methods as an opaque _MHCResidual payload
in place of the base class's plain Tensor. mhc_cfg=None degrades both
layers to the base class's ordinary residual math unchanged, for reuse by
the MTP layer in F6.

Also fixes a real bug surfaced while testing: KDA's dt_bias/A_log were left
torch.empty-uninitialized in xtuner/v1/module/attention/kda.py; now
initialized to match HF's Glm5NextTextForgetGate _init_weights policy
(A_log zeroed when a safe gate lower bound is set, dt_bias log-uniform).

Verified with 26 tests across tests/model/test_glm53_mhc.py (Sinkhorn
math + exact HF parity in fp32, bf16 parity within ULP tolerance),
tests/ops/test_hc_post.py (fused kernel vs eager reference, forward/
backward/compile), and tests/model/test_glm53_decoder_layer.py (both
decoder layers, mhc_cfg=None passthrough equivalence, gradient flow).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant