Conversation
…date registrations Replace the supports_cooperative_stop and system_prompt_semantics ClassVars with one frozen HarnessContract per agent class. AgentRegistry.register now rejects an agent without a contract, and a config class that is not a forbid-extra BaseAgentConfig whose type Literal names the kind. The base environment info records the contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ructor Agent.__init__ is now concrete and takes cost_log_tags; every in-tree agent forwards it through super().__init__. The orchestrator forwards the tags on every LiteLLM route, so the supports_cost_log_tags capability gate is deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and Antigravity Pi maps allowed_tools / disallowed_tools / permission_mode plan to --tools, --no-tools and --exclude-tools. OpenCode writes explicit permission rules and an instructions file for system_prompt into OPENCODE_CONFIG_CONTENT and always passes --auto. Antigravity builds SDK tool-call policies. Every map is the inverse of the adapter's telemetry map. The Codex enabled_tools / disabled_tools forward was a no-op (they are MCP-server keys) and is deleted with its task. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…on-time harness contract check An experiment's defaults.agent may carry by_type.<kind>, applied below the task and selected by the final agent kind, so Claude-only defaults move out of the shared baseline. A gated agent field set on a harness whose contract marks it unsupported now raises HarnessContractError, which plan, run and export treat as a hard config error. The sdk_options guard is derived from the registry, a -D system_prompt_file is inlined after layer 5, and every unenforced-field warning is deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…map per harness HarnessContract gains permission_modes, the permission_mode values a harness honors with their Claude Code meaning. ToolNameMap maps every name in CANONICAL_TOOL_NAMES to the harness's native tools (an empty tuple for a tool the harness lacks), is validated at registration, and replaces the silent .get(name, ()) reads. Pi and OpenCode smoke tasks move to bypassPermissions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mes a harness cannot honor The resolution-time contract check now also rejects a permission_mode value outside the harness's permission_modes and a tool-list name outside CANONICAL_TOOL_NAMES, with a did-you-mean hint. Claude Code keeps its native mcp__ names and permission rule syntax. Orchestrator setup runs the check for library and in-container runs, but not for a re-grade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oder_eval.spi surface CE068 flags a concrete agent config class or an AgentKind member (other than UNKNOWN) named in orchestration/, streaming/ or timing.py. coder_eval.spi re-exports the stable plugin surface with SPI_VERSION = 1, and EXTENDING.md documents the harness contract a plugin agent must declare. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…the harness conformance test make parity-table renders the HarnessContract and ToolNameMap of every in-tree agent into docs/agents/HARNESS_PARITY.md, and CE069 fails on drift. The conformance test derives every rejection from the contracts and requires one offline probe per enforced cell and declared permission mode, including a check that a system prompt never rides the user turn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Docker staging and the Harbor export write only the agent fields a layer set, so a reloaded non-Claude task no longer claims the permission_mode default and fails the contract check in the container (round-trip tests added). - An empty allowed_tools / disallowed_tools restricts nothing and is not a set field. - OpenCode keeps a host rule for external_directory / doom_loop under an allowlist. - coder_eval.spi exports CommandTelemetry, TokenUsage, TranscriptMessage, ResultSummary and ModelPricing, which a plugin needs to emit events and prices. - Correct stale notes and docs: the Codex start() warning, the simulator's allowed_tools=[] claim, plan meaning, resolution vs load, and Pi's ls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eardown and the early-stop ceiling - Pi and OpenCode stop at the (N+1)th turn_start / step_start before it is counted or emitted, so max_turns=N records N turns (it recorded N+1). - Antigravity retries a receive_steps() RuntimeError only when no step was pulled yet; a later error is a real failure and is no longer re-pulled. - A failed Antigravity harness teardown is logged instead of swallowed silently. - The early-stop ceiling fails closed at zero armed weight instead of dividing by zero, matching armed_criteria_passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| subagent tool. | ||
| """ | ||
|
|
||
| TOOL_NAME_ALIASES: Final[dict[str, str]] = {"Task": "Agent"} |
… task caps Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Plan 1 of 3 of the harness target architecture: every agent declares what it honors of the shared agent config, and a task that asks for something the chosen harness cannot do is rejected at resolution, so
coder-eval planfails before any run.HarnessContractper agent class. Each agent declares which ofsystem_prompt,plugins,permission_mode,allowed_toolsanddisallowed_toolsit honors, thepermission_modevalues it honors, and whether it supports cooperative stop. Registration fails without a contract. The ClassVarssupports_cooperative_stop,supports_cost_log_tagsandsystem_prompt_semanticsare gone.CANONICAL_TOOL_NAMES(14 names, checked against the pinned Claude Code CLI) and a per-harnessToolNameMapthat must map every name. Names are checked at resolution, with a did-you-mean hint.--tools/--no-tools/--exclude-tools. OpenCode:permissionrules and aninstructionsfile for the system prompt. Antigravity: SDK tool-call policies.permission_mode: planmeans read-only on all three. The Codex tool forward did nothing (enabled_toolsis an MCP-server key) and is deleted.defaults.agent.by_type.<kind>applies only when the final agent kind matches, below the task. Claude-only defaults inexperiments/default.yamlmoved there.validate_harness_contractraisesHarnessContractError, aTaskResolutionError.plan,runandexportstop on it. Every "set but NOT enforced" warning is deleted.coder_eval.spiwithSPI_VERSION = 1.cost_log_tagsis a base-constructor kwarg onAgent.orchestration/,streaming/andtiming.py.make parity-tablegenerates the contract and tool tables indocs/agents/HARNESS_PARITY.md, and CE069 fails on drift.tests/test_harness_conformance.pyrequires one offline probe per enforced cell and per declared permission mode, and checks that a system prompt never rides the user turn.max_turns. Antigravity re-pulled the step stream on anyRuntimeError, and swallowed a failed teardown silently. The early-stop ceiling could divide by zero.Breaking changes: YAML that loads today is now rejected
permission_mode,allowed_toolsordisallowed_tools.permission_mode: acceptEditsordefaulton Pi, OpenCode or Antigravity (they honorplanandbypassPermissions). A task-levelacceptEditsrun with--type piis rejected too; put it underby_type.claude-code.mcp__…names and rules such asBash(git status:*).plugins,system_prompt,permission_modeor a tool list on atype: nonetask.experiments/default.yamlnow appliesmodel,permission_modeandallowed_toolsto Claude Code only, so--type pino longer inherits the Claude model id.allowed_tools: []restricts nothing on every harness, as on Claude Code.Before merge
coder_eval_uipath): its agent fails registration until it declares acontractand acceptscost_log_tags. Land that change, and movepermission_modeunderby_type.claude-codein the RPA pipeline's shared experiment YAML, together with or before this PR.allowed_tools: [Read]: Pi and OpenCode (Haiku 4.5 via OpenRouter) and Antigravity (gemini-3.5-flash; it runs Gemini only) each wrote no file and scored 0, while an unrestricted control run on the same model passed on all three. An extra Antigravity run with onlydisallowed_tools: [Bash]scored 1.0 with noBashcall, so its policies are selective.Open follow-ups (simulator safeguard, the
acceptEditsowner question, deferred lint ideas) are tracked inc/harness-target-architecture-v2.mdand.claude/harness-candidates.md.Test plan
make verify: 5989 passed, 2 skipped, 93% coverage; ruff, pyright, custom lint (CE000+) and docs budget cleancoder-eval planover every task: exit 0;plan -eover every shipped experiment: exit 0 except the intendedwith-pluginrejection on the agentless taskmkdocs build --strict; new anchors verified in the built HTML🤖 Generated with Claude Code