Skip to content

feat(bench): add safe benchmark refine attempts#489

Merged
drewstone merged 1 commit into
mainfrom
feat/bench-safe-refine-attempts
Jul 7, 2026
Merged

feat(bench): add safe benchmark refine attempts#489
drewstone merged 1 commit into
mainfrom
feat/bench-safe-refine-attempts

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Summary

  • add LOOP_ATTEMPTS to run-benchmarks so each task can run answer -> judge -> safe feedback -> retry loops
  • reuse the shared runRefineLoop and stop early on pass
  • redact answer/gold/reference fields before feedback reaches retry prompts
  • make cli-bridge cells first-class by passing BRIDGE_URL/BRIDGE_BEARER and preserving full bridge model ids

Verification

  • pnpm exec tsx --test src/run-benchmarks.test.mts src/benchmarks/rag-benchmarks.test.mts: 8/8 pass
  • pnpm exec tsc --noEmit: pass
  • pnpm exec tsx --test $(find src -name '*.test.mts' -print | sort): 51/51 pass
  • git diff --check: pass
  • mergeability: git merge-tree --write-tree origin/main HEAD exit 0
  • live router RAGBench official slice: one-shot 0/5 mean 0.168; LOOP_ATTEMPTS=2 0/5 mean 0.218, 0 infra errors
  • live bridge fixture smoke: BACKEND=bridge CELLS=opencode/deepseek/deepseek-v4-pro LOOP_ATTEMPTS=2 ran 5 tasks; 3 pass, 1 empty-output error, CRAG fake-company fail

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 7f09bba8

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-07T23:37:14Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Value Audit — audit-incomplete

Verdict audit-incomplete
Concerns 0 (none)
Heuristic 0.0s
Duplication 0.1s
Interrogation 8.1s (2 bridge agents)
Total 8.2s

💰 Value — error

value agent produced no parseable value-audit JSON.

  • Model: opencode/deepseek/deepseek-v4-pro
  • Bridge attempts: 3
  • Bridge error: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content; opencode/zai-coding-plan/glm-5.2: bridge stream ended without value-audit content; opencode/deepseek/deepseek-v4-pro: bridge stream ended without value-audit content

🎯 Usefulness — error

usefulness agent produced no parseable value-audit JSON.

  • Model: opencode/deepseek/deepseek-v4-pro
  • Bridge attempts: 3
  • Bridge error: opencode/zai-coding-plan/glm-5.2: bridge stream ended without value-audit content; opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content; opencode/deepseek/deepseek-v4-pro: bridge stream ended without value-audit content

No PR concerns were produced because the value/usefulness agent pass did not complete. Treat this audit as incomplete, not as approval.


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260707T233919Z

@drewstone drewstone merged commit f8af639 into main Jul 7, 2026
1 check passed
@drewstone drewstone deleted the feat/bench-safe-refine-attempts branch July 7, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants