Skip to content

fix_moe_learable-score111#7903

Open
chang-wenbin wants to merge 3 commits into
PaddlePaddle:developfrom
chang-wenbin:fix_moe_learable-score111
Open

fix_moe_learable-score111#7903
chang-wenbin wants to merge 3 commits into
PaddlePaddle:developfrom
chang-wenbin:fix_moe_learable-score111

Conversation

@chang-wenbin
Copy link
Copy Markdown
Collaborator

Motivation

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick,PR标题需遵循格式,在最开始加上[Cherry-Pick]标签,以及最后面加上原PR ID,例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

Modifications

Usage or Command

Accuracy Tests

Checklist

  • Add at least a tag in the PR title.
    • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
    • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@paddle-bot
Copy link
Copy Markdown

paddle-bot Bot commented May 24, 2026

Thanks for your contribution!

PaddlePaddle-bot

This comment was marked as outdated.

@PaddlePaddle-bot
Copy link
Copy Markdown

PaddlePaddle-bot commented May 24, 2026

🤖 Paddle-CI-Agent | ci_status_monitor | 2026-05-25 14:53:42

CI报告基于以下代码生成(30分钟更新一次):


1 任务总览

Required 任务存在 2 个失败,均为 XPU case 日志上传阶段的 GitHub Actions Artifact 存储配额问题;PR 主测试任务 Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage 已通过。建议等待/清理 Artifact 配额后 rerun 失败任务。

总执行(rerun次数) 总任务 ✅ 通过 ❌ 失败 ⏳ 运行中 ⏸️ 等待中 跳过
55(13) 42 36 6 0 0 0

2 任务状态汇总

2.1 Required任务 : 8/10 通过

必选任务阻塞合并,失败需优先处理。

状态 任务 耗时 根因 修复建议 日志 重跑
xpu_4cards_case_test / run_xpu_4cards_cases 27m12s 环境问题:Artifact 存储配额耗尽 清理/等待配额恢复后 rerun Job 🔄×1
xpu_8cards_case_test / run_xpu_8cards_cases 9m42s 环境问题:Artifact 存储配额耗尽 清理/等待配额恢复后 rerun Job 🔄×1
其余 8 个必选任务通过 - - - - -

2.2 可选任务 — 28/32 通过

可选任务不阻塞合并,失败仅供参考。

状态 任务 耗时 日志 重跑
Run iluvatar Tests / run_iluvatar_cases 2m5s Job -
Check PR Template 19s Job -
CI_HPU 1h4m Job -
Trigger Jenkins for PR 14m52s Job -
其余 28 个可选任务通过 - - -

3 失败详情(仅 required)

xpu_4cards_case_test / run_xpu_4cards_cases — 基础设施(置信度: 高)

xpu_4cards_case_test / run_xpu_4cards_cases

  • 状态: ❌ 失败
  • 错误类型: 基础设施
  • 置信度: 高
  • 根因摘要: Artifact 存储配额耗尽
  • 分析器: 通用分析(fallback)

根因详情:
日志显示 Run CI unittest 步骤成功完成,4 卡 cases 14 passed,失败发生在后续 Upload case logs 步骤。actions/upload-artifact@v6 创建 Artifact 时提示组织/仓库 Artifact storage quota 已耗尽,属于 CI 平台配额问题,与本 PR 修改的 MoE 代码无直接关联。

关键日志:

======================= 14 passed in 1484.02s (0:24:44) ========================
============================4卡cases测试通过!============================
##[error]Failed to CreateArtifact: Artifact storage quota has been hit. Unable to upload any new artifacts.
Usage is recalculated every 6-12 hours.

修复建议:

  1. 清理 GitHub Actions Artifact 存储或等待配额 6-12 小时重新计算后,rerun 该任务。
  2. 若短期频繁发生,可临时减少 case_logs 上传量或调整 Artifact 保留策略。

修复建议摘要: 清理/等待配额恢复后 rerun

关联变更: 未发现与 fastdeploy/model_executor/layers/moe/fused_moe_cutlass_backend.py 改动直接相关
链接: 查看日志

xpu_8cards_case_test / run_xpu_8cards_cases — 基础设施(置信度: 高)

xpu_8cards_case_test / run_xpu_8cards_cases

  • 状态: ❌ 失败
  • 错误类型: 基础设施
  • 置信度: 高
  • 根因摘要: Artifact 存储配额耗尽
  • 分析器: 通用分析(fallback)

根因详情:
日志显示 Run CI unittest 步骤成功完成,8 卡 cases 4 passed,失败发生在后续 Upload case logs 步骤。actions/upload-artifact@v6 创建 Artifact 时提示 Artifact storage quota 已耗尽,属于 CI 平台配额问题,与本 PR 修改的 MoE 代码无直接关联。

关键日志:

======================== 4 passed in 395.85s (0:06:35) =========================
============================8卡cases测试通过!============================
##[error]Failed to CreateArtifact: Artifact storage quota has been hit. Unable to upload any new artifacts.
Usage is recalculated every 6-12 hours.

修复建议:

  1. 清理 GitHub Actions Artifact 存储或等待配额 6-12 小时重新计算后,rerun 该任务。
  2. 若短期频繁发生,可临时减少 case_logs 上传量或调整 Artifact 保留策略。

修复建议摘要: 清理/等待配额恢复后 rerun

关联变更: 未发现与 fastdeploy/model_executor/layers/moe/fused_moe_cutlass_backend.py 改动直接相关
链接: 查看日志

Copy link
Copy Markdown

@PaddlePaddle-bot PaddlePaddle-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Paddle-CI-Agent | pr_review | 2026-05-25 13:21:22

📋 Review 摘要

PR 概述:修复 noaux_tc 分支中可学习路由缩放因子(routed_scaling_factor_learnable)因被 moe_expert_dispatch 返回值覆盖而实际未生效的 bug
变更范围fastdeploy/model_executor/layers/moe/fused_moe_cutlass_backend.py
影响面 Tag[OP] [BugFix]

问题

级别 文件 概述
🟡 建议 fused_moe_cutlass_backend.py 其他 MoE backend(deepgemm / triton)中 routed_scaling_factor_learnable 缩放时序是否存在同类问题,需确认
📝 PR 规范 标题缺少官方 Tag,PR body 各 section 均为空

📝 PR 规范检查

标题 fix_moe_learable-score111 缺少官方 Tag,且 PR body 各段落均未填写。

标题建议(可直接复制):

  • [BugFix] Fix routed_scaling_factor_learnable applied to stale topk_weights in apply_tp noaux_tc branch

PR 描述建议(可直接复制,必须复刻 checklist §D2 模板的完整结构):

## Motivation
修复 `fused_moe_cutlass_backend.py``apply_tp``noaux_tc` 路径下,可学习缩放因子(`routed_scaling_factor_learnable`)实际未生效的 bug。原实现将缩放应用于 `get_moe_scores` 的返回值,但其后 `moe_expert_dispatch` 会重新输出新的 `topk_weights``topk_idx`,导致之前的缩放被覆盖丢失。

## Modifications
- `fastdeploy/model_executor/layers/moe/fused_moe_cutlass_backend.py`:将 `apply_tp``noaux_tc` 分支中可学习缩放因子的应用位置,从 `get_moe_scores` 调用后移至 `moe_expert_dispatch` 调用后,确保缩放作用于最终实际使用的 `topk_weights`## Usage or Command
N/A

## Accuracy Tests
N/A

## Checklist

- [x] Add at least a tag in the PR title.
  - Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
  - You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.

总体评价

本 PR 修复思路正确,将可学习缩放因子的应用时机从 get_moe_scores 之后调整到 moe_expert_dispatch 之后,逻辑上解决了缩放未实际生效的问题。建议同步确认 deepgemm / triton 等其他 MoE backend 的相同路径是否存在同类缩放时序问题。

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 25, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@e85f0e2). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             develop    #7903   +/-   ##
==========================================
  Coverage           ?   63.69%           
==========================================
  Files              ?      463           
  Lines              ?    64729           
  Branches           ?     9932           
==========================================
  Hits               ?    41226           
  Misses             ?    20714           
  Partials           ?     2789           
Flag Coverage Δ
GPU 72.79% <100.00%> (?)
XPU 7.10% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants