diff --git a/skills/agent-runtime-adoption/SKILL.md b/skills/agent-runtime-adoption/SKILL.md index 7bc3d23e..0e8b5aeb 100644 --- a/skills/agent-runtime-adoption/SKILL.md +++ b/skills/agent-runtime-adoption/SKILL.md @@ -54,10 +54,12 @@ Topology is the **one recursive agent tree**: each round an agent decides to ref construction; the body is harness-re-verified, so an authored strategy can't fabricate a win. Use when the right shape is task-dependent (scout-then-fanout, refine-then-branch, decompose). -- **`createCoordinationTools`** — the agent-driving-agent loop: a driver agent - spawns / steers / awaits child agents (and sub-drivers) through MCP verbs over a - live `Scope`, recursively. Use when a driver should reason about and orchestrate - its workers in natural language. +- **`createCoordinationTools`** (from `@tangle-network/agent-runtime/mcp`) — the + agent-driving-agent loop: a driver agent spawns / steers / awaits child agents + (and sub-drivers) through MCP verbs over a live `Scope`, recursively. Use when a + driver should reason about and orchestrate its workers in natural language. From + `/loops` the equivalent surfaces are `serveCoordinationMcp` (the verbs as an HTTP + MCP over a live `Scope`) and the offline `driverAgent`. Topology is **orthogonal to harness** — a strategy decides the shape; the executor decides which harness (claude-code / codex / opencode / pi / router) runs each @@ -100,6 +102,12 @@ agent-driving-agent loop), expose `createCoordinationTools` over a live `Scope` - `runLoop` validates `ctx.sandboxClient.create` exists or throws `ValidationError`. Never stub a `null` client. +- Build that client with `resolveSandboxClient({ backend })` (from + `@tangle-network/agent-runtime/loops`) — the one call that selects the sandbox / + bridge (cli-bridge) / router transport `runLoop` drives; do not hand-construct it. + Its sibling `resolveAgentBackend` is a DIFFERENT resolver — it resolves the CHAT + leg (`runChatThroughRuntime` / `runAgentTaskStream`) and returns an + `AgentExecutionBackend`, not a feeder for `resolveSandboxClient`. - The kernel emits `loop.started / iteration.dispatch / iteration.ended / decision / ended` via `ctx.traceEmitter`. Wire it to the same OTLP sink as the chat path so loop telemetry is queryable. @@ -108,17 +116,19 @@ agent-driving-agent loop), expose `createCoordinationTools` over a live `Scope` - Dynamic driver: set the kernel's `runLoop` `maxIterations >=` the driver's so the driver's cap governs and the loop closes on a clean `'done'`. -## Campaign bridge — `loopDispatch` +## Campaign bridge — `loopCampaignDispatch` / `loopDispatch` To run `runLoop` as an agent-eval campaign cell, do NOT hand-build the ExecCtx + forward trace + report usage every time (the third is silent — forgetting it -yields a `{0,0}` cell `assertRealBackend` reads as a stub). Use the one bridge, -`loopDispatch` (the old `loopCampaignDispatch` name was consolidated away; verify -in `src/runtime/index.ts`): +yields a `{0,0}` cell `assertRealBackend` reads as a stub). Use the bridge. Both +are exported from `src/runtime/index.ts` and are distinct sibling adapters, NOT a +rename: `loopCampaignDispatch` returns a `DispatchFn` for plain `runCampaign` / +`runEvalCampaign`; `loopDispatch` returns a `ProfileDispatchFn` and is the +`runProfileMatrix` variant (it adds the profile axis). ```ts -import { loopDispatch } from '@tangle-network/agent-runtime/loops' -const dispatch = loopDispatch({ +import { loopCampaignDispatch } from '@tangle-network/agent-runtime/loops' +const dispatch = loopCampaignDispatch({ sandboxClient, toLoopOptions: (scenario, profile) => ({ driver, agentRun, output, validator, task: toTask(scenario) }), // toArtifact? — defaults to result.winner?.output @@ -126,19 +136,59 @@ const dispatch = loopDispatch({ // pass `dispatch` to runCampaign / runEvalCampaign; usage + trace are auto-forwarded ``` -`loopDispatch` doubles as the `runProfileMatrix` variant (the `profile` arg is an axis). +For the common shape — a fixed set of `cases` + a `prompt` builder + a `score` +fn, swept across profiles — prefer the declarative facade +`defineLeaderboard({ cases, prompt, score })` (from +`@tangle-network/agent-runtime/loops`). It composes +`expandProfileAxes × loopDispatch × naiveDriver` into one call, exposes +`.run(argv?)` (CLI-flag parsing + matrix) and `.toBenchmarkAdapter()`, and yields +a ranked leaderboard. Reach for raw `loopDispatch` only when a cell needs a custom +driver/validator. -## Identity-gated optimization — agent-eval's `selfImprove` +## Identity-gated optimization — agent-runtime's `improve()` (facade over agent-eval's `selfImprove`) -The optimization entry point is **`selfImprove`** (`@tangle-network/agent-eval/contract`), -NOT agent-runtime — agent-runtime contributes the code-surface `improvementDriver` -(`/improvement`, the git-worktree path) you pass to it as `driver` to optimize CODE -instead of a string. `selfImprove` optimizes any text/config surface (system / -planner / judge rubric) and is **identity-gated by construction**: it runs evals, -proposes candidates (default driver `gepaDriver`), and a held-out gate ships a winner -only if it beats the baseline. `result.winner.surface` is the **baseline unless -`result.gateDecision === 'ship'`** — so registering a surface for optimization can -never regress it; it only improves when held-out data earns it. +**Start with `improve()`** — the one pluggable RSI verb, exported at the +`@tangle-network/agent-runtime` package ROOT (its own header: "the ONE public, +surface-pluggable RSI verb. A thin facade over agent-eval's `selfImprove`"). Real +signature is 3-arg, NOT a single options object: + +```ts +improve( + profile: AgentProfile, + findings: unknown[], + opts: ImproveOptions, +): Promise +``` + +`opts`: `surface?: 'prompt'|'skills'|'tools'|'mcp'|'hooks'|'code'` (default +`'prompt'`), `scenarios`, `judge`, `agent`, `gate?: 'holdout'|'none'` (default +`'holdout'`; `'none'` forces `generations = 0`), plus `budget?` / `llm?` / +`generator?` / `code?` / `skills?` / `runDir?`. It picks the default proposer for +the surface (`gepaProposer` for `'prompt'`, `skillOptProposer` for `'skills'`; +`'code'`/`'tools'`/`'mcp'`/`'hooks'` throw `ConfigError` unless you pass +`opts.generator` or `opts.code`), extracts the baseline from the profile, runs +`selfImprove` with the held-out gate, and on a ship verdict writes the winner back +into the profile field. Returns `ImproveResult { profile, shipped, lift, +gateDecision, raw }` — deploy with `if (out.shipped) deploy(out.profile)`: + +```ts +import { improve } from '@tangle-network/agent-runtime' +const out = await improve(profile, findings, { + surface: 'prompt', scenarios, judge, agent, gate: 'holdout', llm, +}) +if (out.shipped) deploy(out.profile) +``` + +**Drop to `selfImprove`** (`@tangle-network/agent-eval/contract`) only when you +need finer control — a custom proposer/gate, or the code-surface git-worktree path +via agent-runtime's `improvementDriver` (`/improvement`), which you pass to it as +`proposer` to optimize CODE instead of a string. `selfImprove` optimizes any +text/config surface (system / planner / judge rubric) and is **identity-gated by +construction**: it runs evals, proposes candidates (default proposer +`gepaProposer`), and a held-out gate ships a winner only if it beats the baseline. +`result.winner.surface` is the **baseline unless `result.gateDecision === 'ship'`** +— so registering a surface for optimization can never regress it; it only improves +when held-out data earns it. ```ts import { selfImprove } from '@tangle-network/agent-eval/contract' @@ -148,8 +198,8 @@ const result = await selfImprove({ scenarios, judge, budget: { holdoutScenarios, generations: 3, populationSize: 2 }, - llm: { baseUrl, apiKey, model: REFLECTION_MODEL }, // drives the default gepaDriver - // driver? — pass agent-runtime's improvementDriver to optimize CODE (worktree) instead of a string + llm: { baseUrl, apiKey, model: REFLECTION_MODEL }, // drives the default gepaProposer + // proposer? — pass agent-runtime's improvementDriver to optimize CODE (worktree) instead of a string // gate? — defaults to a held-out gate; pass defaultProductionGate for red-team hardening }) // use result.winner.surface unconditionally: it's the baseline until a candidate genuinely wins @@ -157,7 +207,7 @@ const result = await selfImprove({ ### selfImprove gotchas — read before wiring -- **`gepaDriver` mutates TEXT only**, and its only structural guard is `##` H2 +- **`gepaProposer` mutates TEXT only**, and its only structural guard is `##` H2 headings (`preserveSections`) + `maxSentenceEdits`. Make load-bearing sections of your prompt real `##` headings, and treat the output schema as fixed code — GEPA optimizes the prose, never the envelope/contract. diff --git a/skills/build-with-agent-runtime/SKILL.md b/skills/build-with-agent-runtime/SKILL.md index 9474ab32..ab139294 100644 --- a/skills/build-with-agent-runtime/SKILL.md +++ b/skills/build-with-agent-runtime/SKILL.md @@ -92,8 +92,8 @@ to its native default (`HARNESS_NATIVE_MODEL`) — never silently dropped. | **Spawn N coding agents on isolated git worktrees, keep the one whose patch passes checks** | `worktreeFanout` + `createWorktreeCliExecutor` + `gateOnDeliverable(DeliverableSpec)` over a raw `WorktreePatchArtifact`, winner via `selectValidWinner` — `/loops` — NOT a hand-rolled spawn-loop / "coder" role | canonical-api §3.1 / §5 | | **Sandbox coding rollout** (fresh box/round, or persistent+resume) | `runLoop(options)` / `openSandboxRun(client, opts, deliverable)` — `/loops` | canonical-api §3.1 | | **Optimize a CODE surface** in a gated loop | `improvementDriver({ worktree, generator })` — root `.` | canonical-api §3.4 | -| **Optimize a PROMPT/config surface** (one call) | `selfImprove({ agent, scenarios, judge, baselineSurface })` — `agent-eval/contract` | canonical-api §3.4 | -| **Gate: ship/hold a candidate** (campaign ctx) | `defaultProductionGate` / `heldOutGate` / `composeGate` — `agent-eval/contract` | canonical-api §3.4 | +| **Optimize a PROMPT/config surface** (one call) — START HERE | `improve(profile, findings, { surface, gate })` — root `.` (the one pluggable RSI verb; picks the default proposer from `surface` — `gepaProposer` for prompt, `skillOptProposer` for skills — and wraps `selfImprove`; drop to `selfImprove({ agent, scenarios, judge, baselineSurface })` from `agent-eval/contract` only for the lower-level loop) | canonical-api §3.4 | +| **Gate: ship/hold a candidate** (campaign ctx) | `defaultProductionGate` / `heldOutGate` / `composeGate` — `agent-eval/contract`; `neutralizationGate` (footprint-matched PLACEBO gate — proves a held-out lift is CONTENT, not added prompt/mount footprint) — `agent-eval/campaign` | canonical-api §3.4 | | **Gate: ship/hold from a `BenchmarkReport`** (per-task cells) | `promotionGate({ report, incumbent, candidate })` — `/loops` | canonical-api §3.4 | | **Run the full multi-generation flywheel + certify** | `runStrategyEvolution(config)` — `/loops` | canonical-api §3.4 | | **Observe a run** (cost/time waterfall, OTLP) | `createWaterfallCollector()` — `/loops`; `createOtelExporter` attached via `composeRuntimeHooks(...)` — root `.` | canonical-api §2 | @@ -110,7 +110,7 @@ holds the load-bearing invariant the parallel breaks: `loopUntil` + `runPersonified` (threads executor seams; equal-k; selector≠judge firewall; journal/replay — a parallel runner silently fails to wire the seams). - "skill optimizer" / "topology mutator" that opens branches + applies patches - **≈** `improvementDriver` (code surface) or `selfImprove`/`gepaDriver` (prompt + **≈** `improvementDriver` (code surface) or `selfImprove`/`gepaProposer` (prompt surface) — both gated on a frozen holdout. - "profile-seam" / agent-config wrapper carrying model+prompt+tools+role **≈** `AgentProfile` (it IS that bundle) + `definePersona` (the run record); diff --git a/skills/loop-writer/SKILL.md b/skills/loop-writer/SKILL.md index 254d3770..bbfdf335 100644 --- a/skills/loop-writer/SKILL.md +++ b/skills/loop-writer/SKILL.md @@ -30,7 +30,8 @@ The driver owns strategy. | Review from several lenses | `panel` | | Simulated user/product eval | `defineConversation` + `runConversation` | | Dynamic topology / drivers of drivers | `Scope` or sandbox driver + `createCoordinationTools` | -| Mutate a shared repo | git branch/clone loop with typed merge outcomes | +| Run N coding workers on isolated worktrees, gate each, pick best patch | `worktreeFanout` | +| Mutate a shared repo | git branch/clone loop with typed merge outcomes (`gitWorkspace` seam) | If a fixed combinator solves it, do not use a dynamic driver. @@ -110,9 +111,9 @@ const result = await createSupervisor().run(driver, task, supervis ``` When the driver lives in a sandbox, expose the same verbs through -`createCoordinationTools`: `spawn_worker`, `await_event`, `observe_worker`, -`steer_worker`, `list_questions`, `answer_question`, `ask_parent`, `stop`, and -optional analyst tools. +`createCoordinationTools`: `spawn_agent`, `await_event`, `observe_agent`, +`steer_agent`, `list_questions`, `answer_question`, `ask_parent`, `stop`, and +optional analyst tools (`list_analysts`, `run_analyst`). ## Role Boundaries @@ -133,7 +134,7 @@ with unresolved `blocks-run` questions. Steer sparingly: only when an analyst finds a concrete mistake, a loop is duplicating work, a parent/Pi answers a blocker, or a verifier reveals a specific fix a running worker can still use. Delivery is through `Scope.send` or -`steer_worker`; failed delivery means spawn a fresh corrected attempt. +`steer_agent`; failed delivery means spawn a fresh corrected attempt. ## Workspace Loops diff --git a/skills/supervise/SKILL.md b/skills/supervise/SKILL.md index 35239342..c40df78a 100644 --- a/skills/supervise/SKILL.md +++ b/skills/supervise/SKILL.md @@ -10,7 +10,7 @@ You are a supervisor. You do NOT do the work yourself — you design and drive s ## Loop 1. **Decompose** the task into the smallest set of sub-tasks a single focused worker can each deliver. -2. **Author** a worker per sub-task by calling `spawn_worker` with a complete `profile`: +2. **Author** a worker per sub-task by calling `spawn_agent` with a complete `profile`: - `name` — a short id. - `skills` — the skill files the worker should carry (by name), OR `systemPrompt` — rich, specific instructions for this sub-task. - `model` — the model best suited to this sub-task (optional). @@ -21,4 +21,4 @@ You are a supervisor. You do NOT do the work yourself — you design and drive s ## Authoring sub-supervisors -If a sub-task is itself too large for one worker, author it as a **sub-supervisor**: give its profile a `skills` list that includes `supervise`. It will decompose and drive its own workers one level deeper. This is not a special call — it is the same `spawn_worker`, just a profile that carries this skill. +If a sub-task is itself too large for one worker, author it as a **sub-supervisor**: give its profile a `skills` list that includes `supervise`. It will decompose and drive its own workers one level deeper. This is not a special call — it is the same `spawn_agent`, just a profile that carries this skill.