Skip to content

perf: optimize GLM-5.2 serving - #4827

Draft
CUHKSZzxy wants to merge 13 commits into
InternLM:mainfrom
CUHKSZzxy:perf/glm-5-2-optimization
Draft

perf: optimize GLM-5.2 serving#4827
CUHKSZzxy wants to merge 13 commits into
InternLM:mainfrom
CUHKSZzxy:perf/glm-5-2-optimization

Conversation

@CUHKSZzxy

@CUHKSZzxy CUHKSZzxy commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Optimize GLM-5.2 FP8 serving across attention, MoE, projections, weight loading, and the output head:

  • Use DeepGEMM for DSA index scoring.
  • Extend compact blocked-FP8 MoE to low route density and fuse noaux tensor-core routing.
  • Fuse Q/KV-A projections and shard LM-head vocabulary across TP ranks.
  • Refresh dense FlashMLA metadata correctly during CUDA-graph replay.
  • Load merged blocked-FP8 scales into TP-local destination sections.

Dependency

Depends on #4805; its five commits currently form the branch prefix. GLM-5.2 support PR #4737 has merged.

Performance

GLM-5.2-FP8 TP8 on 8 H200 GPUs, FP8 KV cache, MTP5, concurrency 16, and five-turn SWE-Smith:

  • Two fresh-server runs: 803.60 and 808.51 tok/s; mean 806.06 tok/s.
  • +12.86% versus the 714.22 tok/s DeepGEMM-index baseline.
  • Both runs completed 80/80 requests successfully.

These are cumulative branch measurements and predate the TP-sharded LM head.

Validation

  • Pre-LM-head full GSM8K: 1,267/1,319 (96.06%), exactly matching the reference, with no request or extraction failures.
  • Targeted FlashMLA, cache, DeepGEMM, FP8 MoE, routing, projection, and NSA tests passed after rebase.
  • Full TP8 model loading, CUDA-graph warmup, and repeated SWE runs completed successfully.

Assistance

Assisted with Codex + GPT-5.6-Sol xHigh, reviewed manually

@CUHKSZzxy
CUHKSZzxy force-pushed the perf/glm-5-2-optimization branch from b194e7c to b76c003 Compare August 5, 2026 06:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants