Skip to content

feat(bench): add router-aware metered OpenCode TB agent#488

Merged
drewstone merged 3 commits into
mainfrom
feat/tb-opencode-router-agent-clean
Jul 7, 2026
Merged

feat(bench): add router-aware metered OpenCode TB agent#488
drewstone merged 3 commits into
mainfrom
feat/tb-opencode-router-agent-clean

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Summary

  • adds a Terminal-Bench OpenCode agent that routes arbitrary provider-qualified model ids through the Tangle router
  • records real token usage from OpenCode JSON events instead of returning zero tokens
  • fixes subclass install-template lookup and persists OpenCode event/stderr files into the run log directory

Verification

  • python3 -m py_compile bench/tb_agents/opencode_router_agent.py
  • git diff --check
  • git merge-tree --write-tree origin/main HEAD
  • pnpm run typecheck
  • pnpm test (127 files, 1,254 passed, 2 skipped)
  • pnpm run lint
  • pnpm run build
  • pnpm run verify:package (rerun after build; initial parallel run raced with build cleaning dist)

drewstone and others added 3 commits July 7, 2026 17:28
Custom Terminal-Bench agent that (a) subclasses the published OpenCodeAgent so
the in-container npm install and the task verifier stay the stock open-source
path; (b) injects an OPENCODE_CONFIG opencode.json redefining the model's
provider as an @ai-sdk/openai-compatible provider pointed at the Tangle router,
and forwards OPENAI_API_KEY + OPENAI_BASE_URL into the container, so
-m <provider>/<model> routes through the router for any provider (fixes the
stock agent's zai-coding-plan whitelist ValueError and missing OPENAI_BASE_URL
forwarding); (c) runs opencode --format json, captures the event stream, sums
per-message token usage, and returns a real AgentResult instead of zeros.

Publishes the graded container id to TB_CONTAINER_ID_FILE for a downstream
worker executor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 1644175a

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-07T23:29:59Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Value Audit — audit-incomplete

Verdict audit-incomplete
Concerns 0 (none)
Heuristic 0.0s
Duplication 0.0s
Interrogation 12.8s (2 bridge agents)
Total 12.8s

💰 Value — error

value agent produced no parseable value-audit JSON.

  • Model: opencode/deepseek/deepseek-v4-pro
  • Bridge attempts: 3
  • Bridge error: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content; opencode/zai-coding-plan/glm-5.2: bridge stream ended without value-audit content; opencode/deepseek/deepseek-v4-pro: bridge stream ended without value-audit content

🎯 Usefulness — error

usefulness agent produced no parseable value-audit JSON.

  • Model: opencode/deepseek/deepseek-v4-pro
  • Bridge attempts: 3
  • Bridge error: opencode/zai-coding-plan/glm-5.2: bridge stream ended without value-audit content; opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content; opencode/deepseek/deepseek-v4-pro: bridge stream ended without value-audit content

No PR concerns were produced because the value/usefulness agent pass did not complete. Treat this audit as incomplete, not as approval.


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260707T233156Z

@drewstone drewstone merged commit 2ed4e02 into main Jul 7, 2026
1 check passed
@drewstone drewstone deleted the feat/tb-opencode-router-agent-clean branch July 8, 2026 00:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants