Add interception example environments - #2178
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d41b94349e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review 1 blocking correctness issue found. Multiple unresolved review comments identify substantive bugs in these new example environments, including unguarded JSON parsing that could crash rollouts and incorrect configuration that breaks scoring logic. These issues should be addressed before merging. You can customize Macroscope's approvability policy. Learn more. |
295cec1 to
7aae47f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7aae47f960
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
7aae47f to
b1f77a4
Compare
b1f77a4 to
3113425
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 311342505e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
3113425 to
4b92fe4
Compare
8dad0d9 to
b12342b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b12342b8be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b12342b to
ecb3b4b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 38ce15ec82
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: baaf3f33c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6939c63202
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d623180c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1496a4d8f6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51e83618cc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8d62aad59
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2176426ac4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def load(self) -> list[WebSearchInterceptionTask]: | ||
| prompt = ( | ||
| "Use your native web search to find the official OpenAI Responses API " | ||
| "documentation. Cite the source and include the search query." |
There was a problem hiding this comment.
Wrong blocked docs domain
Medium Severity
BLOCKED_WORD is platform.openai.com, but the task asks for the official Responses API docs, which now live on developers.openai.com. Termination with reward is the only scoring path, so runs that correctly search and cite the current docs never match and stay at reward 0.
Reviewed by Cursor Bugbot for commit 30cfbc5. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5c5d47976c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if ( | ||
| call | ||
| and call.name == "bash" | ||
| and json.loads(call.arguments).get("command") == GIT_COMMAND |
There was a problem hiding this comment.
When the model emits a bash tool call whose arguments are invalid JSON or a valid non-object, the Bash harness turns that into an error tool result so the model can retry, but this request hook reparses call.arguments without the defensive handling used above in block_bash. In that scenario json.loads(...).get(...) raises inside @on_request, converting a recoverable tool-argument mistake in the new bash_interception_v1 eval into a TaskError instead of forwarding the tool error result.
Useful? React with 👍 / 👎.
| message = request[-1] | ||
| if not isinstance(message, vf.ToolMessage): | ||
| return None | ||
| call = next( |
There was a problem hiding this comment.
🟠 High bash_interception_v1/taskset.py:35
rewrite_git_result calls json.loads(call.arguments).get("command") without the try/except and isinstance(..., dict) guard used in block_bash. When a model emits malformed JSON arguments, json.loads raises ValueError; when it emits valid non-object JSON like [], .get() raises AttributeError. A tool harness can return an error ToolMessage for such malformed arguments, so this interceptor crashes the rollout instead of passing the error result through unchanged. Consider wrapping the parse in the same try/except and checking isinstance(arguments, dict) before calling .get().
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @environments/bash_interception_v1/bash_interception_v1/taskset.py around line 35:
`rewrite_git_result` calls `json.loads(call.arguments).get("command")` without the `try`/`except` and `isinstance(..., dict)` guard used in `block_bash`. When a model emits malformed JSON arguments, `json.loads` raises `ValueError`; when it emits valid non-object JSON like `[]`, `.get()` raises `AttributeError`. A tool harness can return an error `ToolMessage` for such malformed arguments, so this interceptor crashes the rollout instead of passing the error result through unchanged. Consider wrapping the parse in the same `try`/`except` and checking `isinstance(arguments, dict)` before calling `.get()`.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 48b142a. Configure here.
| == GIT_COMMAND | ||
| ), | ||
| None, | ||
| ) |
There was a problem hiding this comment.
Unguarded tool argument parsing
Medium Severity
block_git parses tool arguments with bare json.loads and then calls .get without checking the value is a dict. Invalid JSON or a non-object payload raises into @on_response and fails the rollout as a TaskError, instead of skipping the call. The sibling block_bash handler already guards both cases.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 48b142a. Configure here.
|
|
||
|
|
||
| class InterceptionTaskConfig(vf.TaskConfig): | ||
| judge: vf.JudgeConfig = vf.JudgeConfig() |
There was a problem hiding this comment.
Judge ignores OpenAI-only credentials
Medium Severity
InterceptionTaskConfig.judge always constructs a default JudgeConfig, which targets Prime. The smoke runner can evaluate with only OPENAI_API_KEY, but judge_guard still calls that Prime-default judge, so the JUDGE_BLOCK example fails auth in OpenAI-only setups even when the main model client is configured correctly.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 48b142a. Configure here.
2 participants
Overview
Adds four small environments that demonstrate
@vf.on_requestand@vf.on_response.What this PR adds
interception-v1: deterministic replacement, judge-backed replacement, termination, and metric-only observation.grayscale-interception-v1: returns a new user message with inline images converted to grayscale.bash-interception-v1: replaces one Bash response outright and blocks another proposed command before execution by returning a syntheticvf.ToolMessage.web-search-interception-v1: uses the Codex harness, inspects native web-search results, and terminates when a blocked word appears.The replacement examples return typed
vf.Messageobjects. The metric-only example callstrace.record_metric()and returnsNone, leaving the response unchanged. The judge example also receivesTraceso its model usage is recorded.Stack
@vf.on_requestand@vf.on_responsehooks.vf.Terminatefinal for rollout and environment scoring.Note
Low Risk
New example-only packages and dependency wiring; no changes to core verifiers runtime or production paths.
Overview
Adds four new v1 example packages that demonstrate
@vf.on_requestand@vf.on_response, and wires them into the rootexamplesdependency group anduveditable sources.interception-v1is a single taskset with four prompts (one marker each): priority-ordered terminate onTERMINATE, deterministic assistant replacement onDETERMINISTIC_BLOCK, async judge replacement onJUDGE_BLOCK(withTracefor judge usage), and metric-only observation onMETRIC_ONLYviatrace.record_metricwith no message change. Rewards treat interceptions or the metric key as success.grayscale-interception-v1uses@vf.on_requestto rewrite inlinedata:image/parts in the last user message to grayscale (Pillow) before the model sees them.bash-interception-v1uses@vf.on_responsein two ways: replace a specific bash tool call with an assistant message before harness execution, and for a second task return a syntheticToolMessagefor a blocked git command; rewards check sentinel files were not created on the runtime.web-search-interception-v1re-exportsCodexHarnessand terminates when a response includes a nativeweb_searchprovider event and serialized content contains a blocked domain string.Lockfile entries register the new editable packages.
Reviewed by Cursor Bugbot for commit 48b142a. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add interception example environments for bash, grayscale, web search, and general patterns
environments/:interception_v1,bash_interception_v1,grayscale_interception_v1, andweb_search_interception_v1, each demonstrating a distinct interception pattern.interception_v1covers four handler types: deterministic block, LLM-judge-based block, terminate-on-keyword, and metric-only recording without message alteration.bash_interception_v1intercepts bash/exec tool calls matching specific commands and returns a blocked message instead of executing them, rewarding runs where the sentinel file was never created.grayscale_interception_v1rewrites inlinedata:imageURLs in user messages to grayscale before the model sees them, using PIL.web_search_interception_v1terminates runs early when a web search response contains a blocked domain (platform.openai.com).examplesdependencies in the root pyproject.toml.Macroscope summarized 48b142a.