Skip to content

Fix local/unfused Megatron attention and LoRA on Ascend - #9864

Draft
hazelduan wants to merge 3 commits into
modelscope:mainfrom
hazelduan:gkd_dxq
Draft

Fix local/unfused Megatron attention and LoRA on Ascend#9864
hazelduan wants to merge 3 commits into
modelscope:mainfrom
hazelduan:gkd_dxq

Conversation

@hazelduan

@hazelduan hazelduan commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • compare Megatron attention backends through their enum names on Ascend
  • keep use_flash_attn disabled and reject padding-free mode for the local and unfused backends
  • recognize MCore ColumnParallelLinear and RowParallelLinear when resolving all-linear LoRA targets
  • preserve the existing flash/Transformer Engine paths

Root cause

attention_backend is normalized to an enum, but the NPU path compared it with the string "local". The comparison therefore enabled MindSpeed flash attention and generated an NPU flash-attention mask even for local attention. Local MCore attention also does not support padding-free packed sequences.

The same legacy condition treated every backend except local as flash attention. On NPU this silently routed the explicit unfused backend back through MindSpeed FlashAttention. unfused now disables the flash flag and generated flash-attention mask while retaining its distinct backend value.

After selecting a true local MCore layer spec, LoRA target discovery found no modules because it only recognized Transformer Engine linear classes. The local parallel linear classes are now included without changing TE targeting.

Validation

  • pre-commit hooks and Python byte-compilation passed for all changed files
  • regression coverage covers local padding-free handling and NPU mask selection
  • paired mcore-bridge tests cover spec selection, NPU unfused core selection, local norm-weight mapping, and local LoRA dispatch
  • A3 validation used SSH user dxq, MindSpeed, Megatron Core 0.16.0, BF16, Qwen3-0.6B, a fixed 500-row dataset, padding_free=false, and identical seed/configuration for local and flash runs
  • every successful run completed 100 optimizer steps, kept loss/grad metrics finite, and saved checkpoint-100
Path Backend Steps Mean loss Last-10 mean loss Final cumulative s/it End-to-end wall time
SFT + LoRA local 100 1.31381 1.04523 0.10189 45 s
SFT + LoRA flash/TE 100 1.30809 1.04592 0.07201 42 s
SFT + LoRA unfused 100 1.30841 1.04521 0.10207 54 s
GKD + LoRA, independent frozen teacher local 100 0.16752 0.15426 0.10990 45 s
GKD + LoRA, independent frozen teacher flash/TE 100 0.16662 0.15193 0.08711 43 s
GKD + LoRA, independent frozen teacher unfused 100 0.16602 0.15295 0.10008 45 s

All six valid runs completed 100 optimizer steps. SFT unfused/flash per-step loss correlation was 0.99992 with mean absolute delta 0.00546. GKD unfused/flash total-loss correlation was 0.99912 with mean absolute delta 0.00197; JSD-loss correlation was 0.99419. The GKD runs executed separate student and frozen-teacher forwards on every step and logged total loss, JSD loss, SFT loss, and grad norm.

The A3 image required test-only import shims for unused optional acl/FLA paths; these are not repository changes. A separate Qwen3-30B-A3B teacher attempt reached local MoE construction but exposed an existing unsupported MindSpeedGmmExperts grouped-weight loading layout. The controlled dense student/teacher comparison above keeps the test focused on the attention and LoRA changes in this PR.

Paired PR: modelscope/mcore-bridge#159

@hazelduan hazelduan changed the title Fix local attention handling on Ascend Fix local Megatron attention and LoRA handling on Ascend Aug 6, 2026
@hazelduan hazelduan changed the title Fix local Megatron attention and LoRA handling on Ascend Fix local/unfused Megatron attention and LoRA on Ascend Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant