fix: user-sim withholds the prompt for the wire, keeps it for scoring

[hallerite]

Follow-up to #2208, which removed mask_prompt/wire_data and had user-sim hand the assistant a prompt-nulled copy of the task instead. The shape is right — nothing threads through Interaction/Rollout anymore — but nulling the prompt outright took judge-verified user-sim with it.

The bug: a plugged judge under user-sim has nothing to grade against

Rollout.close() scores the task it was handed, which is the prompt-nulled copy. judge_question with the default (empty) question_field falls back to prompt_text — "" on that row — and both built-ins default to question_field = "". So a rubric or reference judge on a user-sim taskset judged against an empty question, with no error.

"Keep scoring on non-prompt fields" was the stated contract, but it isn't satisfiable in practice:

• both built-ins call judge_question() unconditionally while assembling their prompt fields (judges/rubric.py:190, judges/reference.py:83), so even a rubric whose template omits {question} needs one;
• no taskset in the tree carries a duplicate question column to point at;
• TaskData is closed to extras (only WireTaskData sets extra="allow"), so user-sim can't add one for an arbitrary taskset either.

And the scenario is exactly what a judge must grade a conversation against, so answer-shaped fields don't substitute. This worked before #2208 (mask_prompt kept the real row for scoring).

The fix

Task.without_prompt() moves the prompt to data.withheld_prompt instead of dropping it, and judge_question's default follows it off the wire: prompt_text, then the withheld text, then a misconfiguration error. A stock taskset now works under user-sim with a plugged judge and no per-taskset configuration.

• prompt_text still reports the wire, which is empty for a withheld row. The rendering is shared with the new withheld_prompt_text rather than made to fall back, so nothing that reads the wire (gepa/adapter.py, envs/agentic_judge) changes meaning.
• Nothing outside judge_question reads the new field. It never enters a request; it rides trace.task.data and bearer-keyed GET /task exactly as gold answer fields already do. It also puts the scenario back on the assistant's trace, which otherwise recorded prompt: null and no record of the goal.
• without_prompt() is idempotent — withholding twice keeps the scenario rather than overwriting it with None.

Also: name the derivation

user-sim built its copy by reconstructing the class:

assistant_task = type(task)(task.data.model_copy(update={"prompt": None}), task.config)

That re-invokes __init__ positionally and drops any non-data instance state — precisely why with_system_prompt copies instead, as gepa/adapter.py notes: "copies rather than reconstructs, so subclass state survives". No Task subclass currently overrides __init__ or carries such state, so it was latent fragility rather than a live bug, but it's the same operation with the worse spelling. without_prompt() now sits alongside with_system_prompt, both routed through one copy.copy + data.model_copy helper, and user-sim reads agents.assistant.interaction(task.without_prompt()).

Leftovers from the mask_prompt removal go too (agent.py's "(or masked) task", test_e2e.py's "masked chat"), and the four places that spelled the maneuver out in prose — interaction()'s docstring, the _turn error message, docs/v1/agent.md, skills/create-environments/SKILL.md — now name the method and the field.

Checks

ruff check --fix . clean, touched files format-clean, ty check verifiers clean. tests/v1/{test_judges,test_scoring,test_taskset,test_envs,test_configs}.py passed against the first commit; the suite is CI's from here.

Verified live end to end, not just by unit test. A throwaway toy taskset — the row's prompt is the user's scenario, scoring is a plugged reference judge left on its default question_field, no duplicate question column — run under --env.id user-sim against deepseek/deepseek-v4-flash:

• assistant's wire prompt: null; withheld_prompt: the scenario; reference reward: 1.0 (judge verdict "yes")
• task metrics confirm all three properties at once: wire_prompt_empty=1.0, scenario_on_row=1.0, scenario_not_leaked=1.0 (the scenario never appears in the transcript)
• spying on Judge.complete shows the scenario verbatim in the judge prompt's Task: block; stripping withheld_prompt from the same row (the pre-fix state) raises the no-question error

Also checked by hand: the move preserves class, config identity, non-prompt fields and subclass instance state, survives a WireTaskData JSON round-trip, handles Messages-form prompts, and is idempotent. No new test files — test_env_id_user_sim_with_tools now asserts the scenario stayed off the wire and on the row, which is the e2e that exercises the whole path.

One usability note found while testing, not addressed here: ReferenceJudge defaults to view="last_reply", which under user-sim shows the judge only the assistant's closing message — a fully successful booking conversation scored 0.0 until the judge was given view="full_trace". Worth a doc line or a different default for conversational envs.

The markdownlint-cli2 / ruff-check / ruff-format / ty pre-commit hooks can't run on this machine (local uv predates uv.lock's revision, so their uv run --locked always fails; markdownlint needs node >= 22). They were skipped at commit time and run by hand as above.

🤖 Generated with Claude Code

Note

Withhold scenario prompt from assistant wire while retaining it for scoring in user-sim env

• Adds Task.without_prompt() in task.py to produce a shallow-copied task with data.prompt=None and the original prompt stored in data.withheld_prompt.
• Updates UserSimEnv.run in env.py to pass task.without_prompt() to the assistant interaction instead of a manually constructed prompt=None copy.
• Updates judge_question in judge.py to fall back to task.withheld_prompt_text when prompt_text is empty, and raises ValueError when neither is available.
• Behavioral Change: graders and reward functions now access the scenario via withheld_prompt_text rather than finding it absent; the e2e test asserts withheld_prompt_text is non-empty.

^{Macroscope summarized 08b5b7d.}

[macroscopeapp]

Approvability

Verdict: Needs human review

Introduces a new without_prompt() method and withheld_prompt field that changes how the judge evaluates tasks in user-simulation scenarios. This is a new feature with runtime behavior changes to the scoring system, warranting human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[mikasenghaas]

from the prompt docstring

"""Initial user prompt; unset if the user opens the conversation."""

doesn't this mean that a judge seeing a null prompt is acc the right behavior? given that the agent turn is seed with the first user message which is on the trace