Add NeMo Gym V1 integration by xeophon · Pull Request #2032 · PrimeIntellect-ai/verifiers

xeophon · 2026-07-15T16:50:48Z

Overview

Adds a NeMo Gym resources-server integration for Verifiers V1. Existing Verifiers harnesses drive the rollout while NeMo Gym remains responsible for resource-server state, native tools, and scoring.

Details

Loads NeMo Gym JSONL requests into typed V1 tasks and preserves the source row for session setup and verification.
Carries per-rollout cookies and signed MCP session headers through V1 state while exposing upstream tool schemas to MCP-capable harnesses.
Converts completed single-branch traces into NeMo Responses objects and records /verify rewards and numeric metrics.
Adds a Python 3.12+ nemo-gym optional dependency and reuses resource-server classes from the published package, with V1 owning startup and cleanup.
Uses uv dependency metadata to install only the resource-server import slice instead of the full NeMo agent, model, and evaluation stack.
Adds environments/nemo_gym_weather_v1 as a thin, self-starting example package; external resource-server URLs remain supported for custom deployments.

Fixes RES-1090

Note

Medium Risk
New eval/scoring path depends on external HTTP/MCP servers and subprocess lifecycle; trace→response conversion must match Gym’s verifier contract, but changes are isolated behind the new taskset and optional extra.

Overview
Adds NeMo Gym V1 integration so verifiers harnesses can run Gym JSONL tasks against resources servers while Gym keeps session state, tools, and /verify scoring.

NeMoGymTaskset / NeMoGymTask load JSONL rows with responses_create_params, call POST /seed_session per rollout (cookies + optional MCP metadata), expose Gym tools through _NeMoGymToolset (MCP HTTP or direct HTTP), and score by converting a single-branch trace to a NeMo Responses object and posting to /verify.

NeMoGymEnv optionally spawns a packaged resource server via verifiers.v1.tasksets.nemo_gym.server when no resources_url is set; subclasses set resource_server to an import path.

Adds mcp_session in verifiers/v1/mcp/launch.py for short-lived MCP HTTP client sessions. Adds optional verifiers[nemo-gym] (nemo-gym==0.4.0, Python ≥3.12) with uv dependency-metadata to trim the install graph, plus environments/nemo_gym_weather_v1 and docs/v1/nemo_gym.md.

^{Reviewed by Cursor Bugbot for commit 4ee6a93. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add NeMo Gym V1 taskset integration with managed resource server lifecycle

Adds NeMoGymTaskset and NeMoGymEnv in verifiers/v1/tasksets/nemo_gym/taskset.py that load tasks from a JSONL dataset, seed per-rollout server sessions, and score completions by calling a NeMo Gym resources server verifier.
NeMoGymEnv manages a subprocess-based resources server: on start it spawns the server, discovers its port via nemo_gym.port, and tears it down on stop. External server URLs are also supported.
Tool bridging in _NeMoGymToolset supports both MCP HTTP and direct HTTP endpoints, exposing gym tools to agents with consistent MCP-style results.
Adds verifiers/v1/mcp/launch.py mcp_session async context manager for streamable MCP HTTP client sessions with configurable timeout.
Includes a nemo-gym-weather-v1 example environment and documentation in docs/v1/nemo_gym.md.
Risk: nemo-gym==0.4.0 is pinned as an optional dependency (nemo-gym extra) and requires Python >=3.12; the managed server subprocess writes a port file and times out if the server fails to start.

^{Macroscope summarized 4ee6a93.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e42f163ddc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d9bc2c714

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af98d46f17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f18134dd1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6cb4d40. Configure here.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ee6a934da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-30T09:29:23Z

+    for item in output:
+        name = str(item.get("name", ""))
+        bare_name = name.removeprefix("_")
+        if item.get("type") == "function_call" and bare_name in known_names:
+            item["name"] = bare_name


Normalize Codex MCP tool names before verification

When this taskset is used with the Codex harness, its empty MCP server ID (TOOL_PREFIX = None) is exposed to the model under the mcp___ namespace by verifiers/v1/harnesses/codex/harness.py, so recorded function calls retain an mcp__...-qualified name. Removing only one leading underscore leaves that qualification intact, and /verify receives a name that is not among the original Gym tool names even though the tool executed successfully, causing tool-dependent Codex rollouts to be rejected or mis-scored. Normalize the Codex-qualified form back to an entry in known_names.

Useful? React with 👍 / 👎.

xeophon · 2026-07-31T19:27:35Z

Superseded by #2212 after migrating the exact head commit to an upstream-owned branch. Outstanding review context and reviewer requests were carried forward.

xeophon marked this pull request as draft July 15, 2026 16:51

cursor Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread verifiers/v1/tasksets/nemo_gym/server.py Outdated

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py Outdated

xeophon force-pushed the agent/nemo-gym-v1 branch 2 times, most recently from de9f35a to 9cea5a6 Compare July 16, 2026 15:57

xeophon marked this pull request as ready for review July 22, 2026 13:52

cursor Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py Outdated

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread pyproject.toml

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py

cursor Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py

xeophon force-pushed the agent/nemo-gym-v1 branch from 9d9bc2c to af98d46 Compare July 23, 2026 12:25

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py Outdated

xeophon force-pushed the agent/nemo-gym-v1 branch from af98d46 to f18134d Compare July 24, 2026 06:25

chatgpt-codex-connector Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread verifiers/v1/tasksets/nemo_gym/server.py

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py

xeophon requested a review from anravich13-cloud July 26, 2026 16:54

xeophon force-pushed the agent/nemo-gym-v1 branch from 9656d11 to 6cb4d40 Compare July 27, 2026 13:26

cursor Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread verifiers/v1/tasksets/nemo_gym/taskset.py Outdated

xeophon added 14 commits July 30, 2026 11:18

Add NeMo Gym V1 integration

7dca84d

Address NeMo Gym review feedback

afe2bcd

Simplify NeMo Gym V1 adapter

eaef2f2

Further simplify NeMo Gym adapter

db41bf1

Adapt NeMo Gym taskset to current v1 APIs

6cd5ff2

Reuse NeMo Gym resource servers directly

a5f7813

Add NeMo Gym optional dependency

576b8a0

Trim NeMo Gym extra dependencies

e6cd966

Reuse shared MCP sessions for NeMo Gym

fb847d4

Simplify NeMo Gym adapter internals

f0b4959

Move NeMo Gym weather to example environment

0543dc1

Adapt NeMo response serialization to current main

cac9c32

Fix NeMo Gym config discovery

db608f8

Normalize NeMo Gym tool outputs

327e18c

xeophon added 3 commits July 30, 2026 11:19

Adapt NeMo Gym integration to Ruff 0.16

20cabf4

Eliminate NeMo Gym port selection race

9a3d33b

Refresh NeMo Gym lock markers

4ee6a93

xeophon force-pushed the agent/nemo-gym-v1 branch from cc45698 to 4ee6a93 Compare July 30, 2026 09:23

chatgpt-codex-connector Bot reviewed Jul 30, 2026

View reviewed changes

xeophon closed this Jul 31, 2026

Uh oh!

Conversation

xeophon commented Jul 15, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Details

Add NeMo Gym V1 taskset integration with managed resource server lifecycle

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

xeophon commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

xeophon commented Jul 15, 2026 •

edited by macroscopeapp Bot

Loading