Skip to content

[feat][evaluation] support async CustomRPC evaluators - #607

Draft
HearyShen wants to merge 15 commits into
mainfrom
codex/trae-work-async-evaluator
Draft

[feat][evaluation] support async CustomRPC evaluators#607
HearyShen wants to merge 15 commits into
mainfrom
codex/trae-work-async-evaluator

Conversation

@HearyShen

@HearyShen HearyShen commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

背景

Trae Work 三方 CustomRPC 评估器耗时较长,需要通过 AsyncInvokeEvaluator 快速接单,由 Provider 异步上报最终结果,并恢复实验调度。

改动

  • IDL/OpenAPI 为 CustomRPC evaluator 增加异步配置和 AsyncInvokeEvaluator SPI。
  • 统一 AsyncRunEvaluator 的 Record -> Context -> Provider 顺序。
  • terminal CAS 区分 Applied / Duplicate / Conflict。
  • refs 持久化后 arm ResumeReady,callback 恢复实验调度。
  • evaluator callback 恢复不重跑 Target,兼容 Agent evaluator 共用异步链路。
  • FaaS HTTP 使用生成的标准 SPI request,不再维护私有 string i64 wire。
  • 异步 callback 自动补算耗时时仅接受正值,避免 MySQL timestamp 秒精度舍入产生负耗时。
  • 保持同步 CustomRPC、old protocol 与普通 evaluator 兼容。

2026-08-12 最终 PPE 验收

  • OSS head:8741a167970c52527e84038fa7faa8d91cba286e,GitHub Checks 全绿。
  • Commercial head:2bb98ef6461cec3eacf86c31de6b69f299d0a45a
  • Evaluation:stone.cozeloop.evaluation/default,SCM 1.0.0.4415,8C16G,HL/LF/LQ 各 1/1;工单 2087543570998784000 成功。外场 lane-default-fornax_commercial 仍为 1.0.0.4322
  • Direct async:RPC 7590117564433612802、FaaS 7590117563496652802、Agent 7590117563496653058 均从 Processing 收敛到 Success,耗时均非负。
  • 核心调度:无 Target、同步 Target、异步 Target 均为 5/5 evaluator Success;Bench Agent 业务失败时其余 4 个 evaluator 仍全部 Success。
  • Retry:无 Target 0 次 Target RPC/INSERT;同步 Target 每个 run 恰好 1 次 Target RPC/INSERT;callback 恢复不二次执行 Target。
  • 协议矩阵:RPC/FaaS immediate、duplicate、conflict、failed、no-report 全覆盖;历史 RPC/FaaS 和同步 RPC/FaaS 正常。
  • 容量:RPC 20/20、FaaS 200/200 Item/Turn/Record Success,0 遗漏。
  • SDK flat error:3 次 callback 均收到 601200702 并携带 PPE Header,Provider 最终失败,0 次假成功。
  • 自动断言:52/52 PASS;全部最终证据文件敏感模式扫描 0 命中。
  • Commercial 全仓 UT:前两次分别命中两个无关 Observability 存量并发/运行时波动;第三次同 commit 原位重跑 13580 tests 全绿,Diff Line Coverage 86.78%。

关联

当前保持 Draft/Open,等待人工 Code Review;不合并、不创建生产发布。

@HearyShen HearyShen changed the title feat(evaluation): support async CustomRPC evaluators [feat][evaluation] support async CustomRPC evaluators Aug 6, 2026
@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.99363% with 22 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...odules/evaluation/domain/service/evaluator_impl.go 87.58% 11 Missing and 7 partials ⚠️
...tion/infra/repo/evaluator/evaluator_record_impl.go 94.64% 2 Missing and 1 partial ⚠️
...aluation/domain/service/expt_run_item_turn_impl.go 95.00% 0 Missing and 1 partial ⚠️

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
+ Coverage   78.16%   78.23%   +0.07%     
==========================================
  Files         693      693              
  Lines       84808    85009     +201     
==========================================
+ Hits        66288    66507     +219     
+ Misses      14577    14551      -26     
- Partials     3943     3951       +8     
Flag Coverage Δ
unittests 78.23% <92.99%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...ation/application/convertor/evaluator/evaluator.go 92.17% <100.00%> (+0.10%) ⬆️
...luation/application/convertor/evaluator/openapi.go 96.55% <100.00%> (+0.02%) ⬆️
...modules/evaluation/application/eval_openapi_app.go 81.73% <100.00%> (+0.14%) ⬆️
...nd/modules/evaluation/application/evaluator_app.go 90.39% <100.00%> (-0.05%) ⬇️
...kend/modules/evaluation/domain/entity/evaluator.go 100.00% <100.00%> (ø)
...tion/domain/entity/evaluator_version_custom_rpc.go 100.00% <100.00%> (ø)
...ckend/modules/evaluation/domain/entity/expt_run.go 98.64% <ø> (ø)
backend/modules/evaluation/domain/entity/param.go 87.71% <ø> (+1.75%) ⬆️
...es/evaluation/domain/service/expt_run_item_impl.go 76.40% <100.00%> (+1.55%) ⬆️
...aluation/domain/service/expt_run_item_turn_impl.go 83.79% <95.00%> (+3.64%) ⬆️
... and 2 more

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 39a1e85...f48a6bd. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HearyShen
HearyShen force-pushed the codex/trae-work-async-evaluator branch from eaf6510 to 3759f6a Compare August 11, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant