Skip to content

Add an Atomic SDK harness for the hosted Planner - #210

Open
lavaman131 wants to merge 6 commits into
mainfrom
atomic-harness
Open

lavaman131 wants to merge 6 commits into
mainfrom
atomic-harness

Conversation

@lavaman131

@lavaman131 lavaman131 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds HARNESS=atomic, a HarnessV1 adapter (apps/server/src/harness/atomic/adapter.ts) that embeds @bastani/atomic 0.9.23 in the Chopin server process through its headless SDK: createAgentSession, ModelRuntime, DefaultResourceLoader, and in-memory session and settings managers. It does not spawn the atomic CLI, use RPC mode, or alias Pi. The copilot-sdk and pi harnesses are unchanged.

  • 07c5852 adds the adapter, its unit tests (atomic/adapter.test.ts), the shared contract suite (harness/atomic.contract.test.ts), harnesses.ts/config.ts wiring and tests, and docs (docs/hosted-agent.md, docs/self-hosting.md, docs/architecture.md, README.md).
  • 878147a keeps the research notes that map the SDK to HarnessV1 and list the caveats.
  • 4c685ef documents that auto also honors the legacy PI_CODING_AGENT_DIR. Atomic checks ATOMIC_CODING_AGENT_DIR first, then PI_CODING_AGENT_DIR, and a set variable disables the ~/.pi fallback.

This branch was rebased onto main (99204aa). The only conflict was bun.lock. I regenerated it from main's lockfile, and it matches the branch's original lockfile except for main's impeccable entries.

Security boundary

  • Only Chopin's host tools are exposed. Every shipped Atomic package is disabled (workflows, subagents, MCP, web access, Intercom), and Atomic's coding tools are not registered.
  • The resource loader discovers no context files, skills, prompt templates, themes, or extensions. The working and configuration directory is an empty private temporary directory.
  • A per-turn hook replaces the whole system prompt, so Atomic's coding-agent preamble never reaches the model.
  • The active tool set is re-checked before every model request, and any unexpected tool fails the turn. The session is also checked before the first request for extensions, context files, skills, templates, or prompt drift.
  • An unknown MODEL fails the turn before any model request. Atomic would otherwise silently pick another model.
  • Structured output uses a terminating result tool, enabled only when the turn requests a JSON response format and blocked otherwise. It does not use Atomic's createStructuredOutputTool, which makes a second model call.

Auth modes

HARNESS=atomic requires an explicit HARNESS_AUTH:

  • auto copies the host's Atomic login into memory ($ATOMIC_CODING_AGENT_DIR, then $PI_CODING_AGENT_DIR, otherwise ~/.atomic/agent over ~/.pi/agent), plus env keys and ambient cloud credentials. Startup refuses it unless SERVER_HOST is loopback-only.
  • ai-gateway reads AI_GATEWAY_API_KEY and accepts only vercel-ai-gateway models. It is the only mode allowed on a public bind.

No credentials, models.json, or models-store.json are written to disk.

Documented caveats

  • Atomic's SDK defaults suit a local coding agent. The adapter overrides them and fails closed on the ones it can observe, but a future release could add a default the suite does not check. Run the contract suite before bumping @bastani/atomic.
  • Sessions live only in memory and cannot resume across restarts. Compaction, turn suspension, and skills are unsupported. Atomic's automatic retries stay on.
  • Under auto, an in-memory OAuth refresh can rotate the host CLI's refresh token. !command API-key entries in auth.json are executed.
  • There is no per-session credit limit, so a worker's maxAiCredits does not apply.
  • @bastani/atomic adds more than 250 MB of dependencies. The rebuilt chopin-atomic image is 2.1 GB.
  • Atomic uses typebox 1.3.27 while Chopin pins 1.3.7. Host tool schemas cross as plain JSON Schema, so the two versions never meet.

Verification on this machine

These checks ran on a new checkout after the rebase. Earlier results from the original machine are in the 07c5852 commit trailers.

The Playwright suite does not exercise HARNESS=atomic; its agent servers use AGENT=off or the fake harness. I tested the Atomic harness separately in a real browser against a real model.

Browser test of the Atomic harness with a live model

I started a server on loopback with AGENT=on HARNESS=atomic HARNESS_AUTH=auto MODEL=github-copilot/gpt-6-luna, using the E2E preloads (e2e/github.ts and e2e/harness/preload.ts). Those preloads fake only GitHub's REST and OAuth endpoints and the GitHub MCP URL. Model traffic went to GitHub Copilot through the host's Atomic login. With agent-browser I signed in through the server's real OAuth callback, added octo-org/score in the UI, created a document, typed a sentence, and sent two @chopin turns.

  1. Read turn. The Planner called read_plan and list_pull_requests, then answered with the document's sentence and the fixture PR title "Restring the harp section". The fake MCP log shows list_pull_requests was called for octo-org/score with a bearer token, readonly=true, and toolsets=pull_requests.

A read turn with read_plan and list_pull_requests

  1. Edit turn. The Planner called read_plan and then edit_plan twice to append a three-item list and keep the existing sentence. The document and the transcript were unchanged after a page reload.

An edit turn that survived a reload

The server's [agent] diagnostic listed exactly Chopin's 16 host tools plus chopin_submit_atomic_result. It showed no bash, filesystem, fetch, or Intercom tool. Nothing in ~/.atomic/agent was modified during the run.

Assistant-verification: docker build passed: docker build --no-cache -t chopin-atomic . completed; both bun install --frozen-lockfile stages fetched from registry.npmjs.org (build 995 packages, production 908 packages)
Assistant-verification: docker runtime import passed: docker run --rm chopin-atomic bun -e "await import('./apps/server/src/harness/harnesses.ts')" and a direct import of atomic/adapter.ts both loaded @bastani/atomic in the production image
Assistant-verification: bun run fix passed: no changes beyond the doc edit; one existing oxlint warning in the untouched github-tools.test.ts
Assistant-verification: bun test passed: 1688 pass, 2 PostgreSQL skips, 0 fail across 190 files; apps/server/src/harness alone 127 pass, covering atomic, copilot-sdk and pi
Assistant-verification: bun run types passed: all workspaces and E2E
Assistant-verification: bun run ci passed: dprint, oxlint (0 errors), tokens, type scale, Impeccable design check
Assistant-verification: bun run e2e passed: 236 Chromium tests after confirming ports 8788, 8789, 8791, 8792 and 8797 were free
Assistant-verification: agent-browser E2E passed: live github-copilot/gpt-6-luna turns through HARNESS=atomic HARNESS_AUTH=auto ran read_plan, list_pull_requests over the fake GitHub MCP (bearer, readonly, pull_requests toolset) and edit_plan; the edit and transcript persisted across reload; ~/.atomic/agent was not modified
Assistant-verification: atomic source check passed: @bastani/atomic 0.9.23 dist/config.js reads ATOMIC_CODING_AGENT_DIR, then PI_CODING_AGENT_DIR

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-duration: 12m converged, estimated none given
User-preference: Build the Atomic harness on Atomic's headless SDK, which separates runtime from hosts
Co-authored-by: Alex Lavaee lavaman131@github.com

@coolify-githubnext-app

coolify-githubnext-app Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The preview deployment for chopin is ready. 🟢

Open app | Open Build Logs | Open Application Logs

Last updated at: 2026-10-01 16:21:12 CET

policy.hostNames = hostNames;
policy.structured = false;
if (!live) {
directory ??= await mkdtemp(join(tmpdir(), "chopin-atomic-planner-"));

@MaggieAppleton MaggieAppleton Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closing a session while its first turn is still setting up can leave a temporary folder behind. I reproduced this by starting doPromptTurn() and immediately calling doDestroy(): cleanup finishes before the folder exists, then setup creates it and stops because the session is closed. The folder never gets deleted.

Could we delete the folder in the if (leak || closed) branch too? Putting that deletion in a finally block would ensure it happens even if session.dispose() fails. A test that closes the session during setup and checks the folder is gone would cover this.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, reproduced and fixed in 12815c8. When the session closes during setup, the working directory is now deleted on that path, in a finally so a failed dispose() still removes it. The new lifecycle test closes the session while its first turn is still setting up and checks that no chopin-atomic-planner-* directory is left. It fails without the fix and passes with it.

Assistant-workflow: inline
Assistant-verification: fail-before/pass-after passed: the lifecycle test left a directory behind without the fix
Co-authored-by: Alex Lavaee lavaman131@github.com

HARNESS=atomic embeds Atomic 0.9.23 in the server process through its
headless SDK (createAgentSession, ModelRuntime, DefaultResourceLoader,
and in-memory session and settings managers). It does not spawn the
atomic CLI, use RPC mode, or alias Pi's runtime.

Each harness session owns one Atomic AgentSession in a private temporary
directory. Every shipped Atomic package is off, the tool allowlist holds
only the host tools and a result tool, and the loader discovers no
extensions, skills, prompt templates, or context files. A per-turn hook
replaces the whole system prompt, so the model never sees Atomic's
coding-agent preamble. The adapter fails the turn before any model
request if the live session reports an extra extension, tool, context
file, skill, prompt template, or system prompt. It checks the tools
offered to the model again before every model request.

Host tools round-trip through submitToolResult and settle on abort.
Structured output uses a terminating result tool enabled only from the
turn's JSON response format, which captures the calling model's
arguments without a second inference. An unrecognized provider/model
fails the turn instead of letting Atomic pick another model, and MODEL
is now required for HARNESS=atomic as it is for Pi.

HARNESS_AUTH is required and is either auto or ai-gateway. auto copies
the host's Atomic login into memory and is refused off loopback;
ai-gateway accepts only vercel-ai-gateway models. Credentials are never
written to disk. The Pi auth checks now share the same table with
unchanged messages.

Docs cover selection, trust, credentials, and Atomic's caveats.

Assistant-workflow: ralph (run 0cfa5220-f488-43ef-8bb5-2f54cf5141f2)
Assistant-model: Claude Opus 5.5
Assistant-duration: 30m converged, estimated 65m
Assistant-verification: bun test passed: 1683 pass, 2 PostgreSQL skips, 0 fail, including the atomic contract suite and the unchanged copilot-sdk and pi suites
Assistant-verification: bun run types passed: all workspaces and E2E
Assistant-verification: bun run ci passed: dprint, oxlint with no errors, tokens
Assistant-verification: bun run fix passed: only formatting changes, inspected
Assistant-verification: mutation checks passed: turning builtins back on, enabling skill discovery, or offering the result tool on plain turns each fail the atomic suite
Assistant-verification: agent-browser E2E passed: a real Chopin server with HARNESS=atomic and a local stub model ran an @chopin turn that called read_plan and streamed the reply into Chat; the model was offered exactly PLANNER_TOOL_NAMES and Chopin's Planner prompt
Assistant-verification: qlty smells passed: no blocking findings; complexity warnings match the Copilot adapter's closure shape
User-preference: Build the Atomic harness on Atomic's headless SDK, which separates runtime from hosts
Co-authored-by: Alex Lavaee <lavaman131@github.com>
The ralph run's research stage mapped Atomic's SDK onto HarnessV1:
event translation, host tool round trips, the tool boundary, auth
modes, and caveats such as no cross-process resume and oversized tool
result truncation. Keep it with the branch so review can continue on
another machine.

The run was stopped during its first review round. Both reviewers had
no blocking findings; reviewer B noted one doc nit: self-hosting.md
names only ATOMIC_CODING_AGENT_DIR, but Atomic also honors legacy
PI_CODING_AGENT_DIR for the auto-mode auth.json path. The Docker build
could not be verified on this machine because its Docker Desktop network
fails TLS to registry.npmjs.org (npm.pkg.github.com and api.github.com
still work), so bun install inside the image cannot fetch the new
lockfile entries.

Assistant-workflow: ralph (run 0cfa5220-f488-43ef-8bb5-2f54cf5141f2)
Assistant-model: Claude Opus 5.5
Assistant-duration: 72m abandoned, estimated 90m
Assistant-verification: docker build unavailable: container TLS to registry.npmjs.org fails with UNKNOWN_CERTIFICATE_VERIFICATION_ERROR on this Docker Desktop host
User-preference: Stop the workflow and hand off when the local environment blocks verification; continue on another machine
Co-authored-by: Alex Lavaee <lavaman131@github.com>
Atomic reads PI_CODING_AGENT_DIR when ATOMIC_CODING_AGENT_DIR is unset,
so the auto-mode auth.json lookup lists both variables in precedence order.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-verification: source check passed: @bastani/atomic 0.9.23 dist/config.js getEnvValue prefers ATOMIC_CODING_AGENT_DIR, then PI_CODING_AGENT_DIR, and a set variable disables the ~/.pi fallback
Assistant-verification: dprint ci passed: bun run ci
Co-authored-by: Alex Lavaee <lavaman131@github.com>
The release keeps the same five builtin packages, so BUILTINS_OFF still
covers every one and the Planner session stays isolated to Chopin's host
tools.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-verification: bun test passed: 1688 pass, 2 PostgreSQL skips, 0 fail, including the atomic contract and isolation suites
Assistant-verification: bun run types passed: all workspaces and E2E
Assistant-verification: bun run ci passed: dprint, oxlint, tokens, type scale, Impeccable baseline
Assistant-verification: source check passed: @bastani/atomic 0.9.24-alpha.1 dist/builtin lists intercom, mcp, subagents, web-access, workflows, matching BUILTINS_OFF
Co-authored-by: Alex Lavaee <lavaman131@github.com>
Closing an Atomic session while its first turn was still setting up ran
cleanup before the working directory existed; setup then created it,
saw the session closed, and stopped without deleting it. Delete the
directory on that closed path, in a finally so a failed dispose still
removes it.

Reported by Maggie Appleton in review.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-verification: fail-before/pass-after passed: the new lifecycle test left a chopin-atomic-planner directory behind without the fix and passes with it
Assistant-verification: bun test passed: 1824 pass, 2 PostgreSQL skips, 0 fail
Assistant-verification: bun run types passed: all workspaces
Assistant-verification: bun run ci passed: dprint, oxlint, tokens, design contract, design record, Impeccable (no new findings)
Co-authored-by: Alex Lavaee <lavaman131@github.com>
0.9.25 is the stable release that rolls up the 0.9.25 prereleases,
including SDK workflow run control and the native MCP client. It ships
the same five builtin packages, so BUILTINS_OFF still turns off each one
for the isolated Planner.

Assistant-model: Claude Opus 5.5
Assistant-workflow: inline
Assistant-verification: bun test passed: 1824 pass, 2 PostgreSQL skips, 0 fail
Assistant-verification: bun run types passed: all workspaces
Assistant-verification: bun run ci passed: dprint, oxlint, tokens, design contract, design record, Impeccable (no new findings)
Assistant-verification: source check passed: @bastani/atomic 0.9.25 dist/builtin lists intercom, mcp, subagents, web-access, workflows
Co-authored-by: Alex Lavaee <lavaman131@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants