Skip to content

[fix] [v1]: apply agentic-judge task weights when an isolated solver fails - #2223

Open
gtnv wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
gtnv:fix/task-weight
Open

[fix] [v1]: apply agentic-judge task weights when an isolated solver fails#2223
gtnv wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
gtnv:fix/task-weight

Conversation

@gtnv

@gtnv gtnv commented Aug 2, 2026

Copy link
Copy Markdown

closes: #2197

Problem:

In AgenticJudgeEnv(share_runtime=False), a failed solver trace returns before a judge trace is created. finalize() then returns early because no judge exists, so ScoreConfig.task_weight is never applied. With the default task_weight = 0, partial rewards on a failed solver keep their original weights even though the environment is configured to suppress them.

this pr puts task-weight scaling in the no-judge branch before returning. judge-present paths keep existing reward oder.

Note

Fix AgenticJudgeEnv.finalize to apply task weights when the solver runs without a judge

When an agentic-judge episode has no judge trace (i.e. the solver ran in isolation), finalize previously returned without applying the configured score.task_weight. It now iterates over the solver's existing rewards and multiplies each non-None reward.weight by score.task_weight before returning. The change is in env.py. Risk: if both the judge and solver traces are absent, finalize now raises a KeyError.

Macroscope summarized 8074f21.


Note

Low Risk
Small, localized change to reward weighting in an edge path; mirrors existing finalize logic and does not touch auth or infrastructure.

Overview
Fixes reward composition when the judge never runs. In IsolatedAgenticJudgeEnv, a failed solver exits before the judge is invoked, so finalize() only sees the solver trace and used to return immediately.

It now scales the solver trace’s task reward weights by config.score.task_weight in that no-judge path—the same logic already used when a judge verdict exists. With the default task_weight of 0, partial task rewards on failed isolated runs are suppressed as intended instead of keeping their original weights.

Reviewed by Cursor Bugbot for commit 8074f21. Bugbot is set up for automated code reviews on this repo. Configure here.

Scale failed solver task rewards when no judge runs.
@gtnv
gtnv marked this pull request as ready for review August 2, 2026 10:37

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 08d30ae. Configure here.

Comment thread verifiers/v1/envs/agentic_judge/env.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v1: apply agentic-judge task weights when an isolated solver fails

1 participant