Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 74 additions & 24 deletions skills/agent-runtime-adoption/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,12 @@ Topology is the **one recursive agent tree**: each round an agent decides to ref
construction; the body is harness-re-verified, so an authored strategy can't
fabricate a win. Use when the right shape is task-dependent (scout-then-fanout,
refine-then-branch, decompose).
- **`createCoordinationTools`** — the agent-driving-agent loop: a driver agent
spawns / steers / awaits child agents (and sub-drivers) through MCP verbs over a
live `Scope`, recursively. Use when a driver should reason about and orchestrate
its workers in natural language.
- **`createCoordinationTools`** (from `@tangle-network/agent-runtime/mcp`) — the
agent-driving-agent loop: a driver agent spawns / steers / awaits child agents
(and sub-drivers) through MCP verbs over a live `Scope`, recursively. Use when a
driver should reason about and orchestrate its workers in natural language. From
`/loops` the equivalent surfaces are `serveCoordinationMcp` (the verbs as an HTTP
MCP over a live `Scope`) and the offline `driverAgent`.

Topology is **orthogonal to harness** — a strategy decides the shape; the executor
decides which harness (claude-code / codex / opencode / pi / router) runs each
Expand Down Expand Up @@ -100,6 +102,12 @@ agent-driving-agent loop), expose `createCoordinationTools` over a live `Scope`

- `runLoop` validates `ctx.sandboxClient.create` exists or throws
`ValidationError`. Never stub a `null` client.
- Build that client with `resolveSandboxClient({ backend })` (from
`@tangle-network/agent-runtime/loops`) — the one call that selects the sandbox /
bridge (cli-bridge) / router transport `runLoop` drives; do not hand-construct it.
Its sibling `resolveAgentBackend` is a DIFFERENT resolver — it resolves the CHAT
leg (`runChatThroughRuntime` / `runAgentTaskStream`) and returns an
`AgentExecutionBackend`, not a feeder for `resolveSandboxClient`.
- The kernel emits `loop.started / iteration.dispatch / iteration.ended /
decision / ended` via `ctx.traceEmitter`. Wire it to the same OTLP sink as the
chat path so loop telemetry is queryable.
Expand All @@ -108,37 +116,79 @@ agent-driving-agent loop), expose `createCoordinationTools` over a live `Scope`
- Dynamic driver: set the kernel's `runLoop` `maxIterations >=` the driver's so
the driver's cap governs and the loop closes on a clean `'done'`.

## Campaign bridge — `loopDispatch`
## Campaign bridge — `loopCampaignDispatch` / `loopDispatch`

To run `runLoop` as an agent-eval campaign cell, do NOT hand-build the ExecCtx +
forward trace + report usage every time (the third is silent — forgetting it
yields a `{0,0}` cell `assertRealBackend` reads as a stub). Use the one bridge,
`loopDispatch` (the old `loopCampaignDispatch` name was consolidated away; verify
in `src/runtime/index.ts`):
yields a `{0,0}` cell `assertRealBackend` reads as a stub). Use the bridge. Both
are exported from `src/runtime/index.ts` and are distinct sibling adapters, NOT a
rename: `loopCampaignDispatch` returns a `DispatchFn` for plain `runCampaign` /
`runEvalCampaign`; `loopDispatch` returns a `ProfileDispatchFn` and is the
`runProfileMatrix` variant (it adds the profile axis).

```ts
import { loopDispatch } from '@tangle-network/agent-runtime/loops'
const dispatch = loopDispatch({
import { loopCampaignDispatch } from '@tangle-network/agent-runtime/loops'
const dispatch = loopCampaignDispatch({
sandboxClient,
toLoopOptions: (scenario, profile) => ({ driver, agentRun, output, validator, task: toTask(scenario) }),
// toArtifact? — defaults to result.winner?.output
})
// pass `dispatch` to runCampaign / runEvalCampaign; usage + trace are auto-forwarded
```

`loopDispatch` doubles as the `runProfileMatrix` variant (the `profile` arg is an axis).
For the common shape — a fixed set of `cases` + a `prompt` builder + a `score`
fn, swept across profiles — prefer the declarative facade
`defineLeaderboard({ cases, prompt, score })` (from
`@tangle-network/agent-runtime/loops`). It composes
`expandProfileAxes × loopDispatch × naiveDriver` into one call, exposes
`.run(argv?)` (CLI-flag parsing + matrix) and `.toBenchmarkAdapter()`, and yields
a ranked leaderboard. Reach for raw `loopDispatch` only when a cell needs a custom
driver/validator.

## Identity-gated optimization — agent-eval's `selfImprove`
## Identity-gated optimization — agent-runtime's `improve()` (facade over agent-eval's `selfImprove`)

The optimization entry point is **`selfImprove`** (`@tangle-network/agent-eval/contract`),
NOT agent-runtime — agent-runtime contributes the code-surface `improvementDriver`
(`/improvement`, the git-worktree path) you pass to it as `driver` to optimize CODE
instead of a string. `selfImprove` optimizes any text/config surface (system /
planner / judge rubric) and is **identity-gated by construction**: it runs evals,
proposes candidates (default driver `gepaDriver`), and a held-out gate ships a winner
only if it beats the baseline. `result.winner.surface` is the **baseline unless
`result.gateDecision === 'ship'`** — so registering a surface for optimization can
never regress it; it only improves when held-out data earns it.
**Start with `improve()`** — the one pluggable RSI verb, exported at the
`@tangle-network/agent-runtime` package ROOT (its own header: "the ONE public,
surface-pluggable RSI verb. A thin facade over agent-eval's `selfImprove`"). Real
signature is 3-arg, NOT a single options object:

```ts
improve<TScenario, TArtifact>(
profile: AgentProfile,
findings: unknown[],
opts: ImproveOptions,
): Promise<ImproveResult>
```

`opts`: `surface?: 'prompt'|'skills'|'tools'|'mcp'|'hooks'|'code'` (default
`'prompt'`), `scenarios`, `judge`, `agent`, `gate?: 'holdout'|'none'` (default
`'holdout'`; `'none'` forces `generations = 0`), plus `budget?` / `llm?` /
`generator?` / `code?` / `skills?` / `runDir?`. It picks the default proposer for
the surface (`gepaProposer` for `'prompt'`, `skillOptProposer` for `'skills'`;
`'code'`/`'tools'`/`'mcp'`/`'hooks'` throw `ConfigError` unless you pass
`opts.generator` or `opts.code`), extracts the baseline from the profile, runs
`selfImprove` with the held-out gate, and on a ship verdict writes the winner back
into the profile field. Returns `ImproveResult { profile, shipped, lift,
gateDecision, raw }` — deploy with `if (out.shipped) deploy(out.profile)`:

```ts
import { improve } from '@tangle-network/agent-runtime'
const out = await improve(profile, findings, {
surface: 'prompt', scenarios, judge, agent, gate: 'holdout', llm,
})
if (out.shipped) deploy(out.profile)
```

**Drop to `selfImprove`** (`@tangle-network/agent-eval/contract`) only when you
need finer control — a custom proposer/gate, or the code-surface git-worktree path
via agent-runtime's `improvementDriver` (`/improvement`), which you pass to it as
`proposer` to optimize CODE instead of a string. `selfImprove` optimizes any
text/config surface (system / planner / judge rubric) and is **identity-gated by
construction**: it runs evals, proposes candidates (default proposer
`gepaProposer`), and a held-out gate ships a winner only if it beats the baseline.
`result.winner.surface` is the **baseline unless `result.gateDecision === 'ship'`**
— so registering a surface for optimization can never regress it; it only improves
when held-out data earns it.

```ts
import { selfImprove } from '@tangle-network/agent-eval/contract'
Expand All @@ -148,16 +198,16 @@ const result = await selfImprove({
scenarios,
judge,
budget: { holdoutScenarios, generations: 3, populationSize: 2 },
llm: { baseUrl, apiKey, model: REFLECTION_MODEL }, // drives the default gepaDriver
// driver? — pass agent-runtime's improvementDriver to optimize CODE (worktree) instead of a string
llm: { baseUrl, apiKey, model: REFLECTION_MODEL }, // drives the default gepaProposer
// proposer? — pass agent-runtime's improvementDriver to optimize CODE (worktree) instead of a string
// gate? — defaults to a held-out gate; pass defaultProductionGate for red-team hardening
})
// use result.winner.surface unconditionally: it's the baseline until a candidate genuinely wins
```

### selfImprove gotchas — read before wiring

- **`gepaDriver` mutates TEXT only**, and its only structural guard is `##` H2
- **`gepaProposer` mutates TEXT only**, and its only structural guard is `##` H2
headings (`preserveSections`) + `maxSentenceEdits`. Make load-bearing sections
of your prompt real `##` headings, and treat the output schema as fixed code —
GEPA optimizes the prose, never the envelope/contract.
Expand Down
6 changes: 3 additions & 3 deletions skills/build-with-agent-runtime/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ to its native default (`HARNESS_NATIVE_MODEL`) — never silently dropped.
| **Spawn N coding agents on isolated git worktrees, keep the one whose patch passes checks** | `worktreeFanout` + `createWorktreeCliExecutor` + `gateOnDeliverable(DeliverableSpec)` over a raw `WorktreePatchArtifact`, winner via `selectValidWinner` — `/loops` — NOT a hand-rolled spawn-loop / "coder" role | canonical-api §3.1 / §5 |
| **Sandbox coding rollout** (fresh box/round, or persistent+resume) | `runLoop(options)` / `openSandboxRun(client, opts, deliverable)` — `/loops` | canonical-api §3.1 |
| **Optimize a CODE surface** in a gated loop | `improvementDriver({ worktree, generator })` — root `.` | canonical-api §3.4 |
| **Optimize a PROMPT/config surface** (one call) | `selfImprove({ agent, scenarios, judge, baselineSurface })` `agent-eval/contract` | canonical-api §3.4 |
| **Gate: ship/hold a candidate** (campaign ctx) | `defaultProductionGate` / `heldOutGate` / `composeGate` — `agent-eval/contract` | canonical-api §3.4 |
| **Optimize a PROMPT/config surface** (one call) — START HERE | `improve(profile, findings, { surface, gate })` — root `.` (the one pluggable RSI verb; picks the default proposer from `surface` — `gepaProposer` for prompt, `skillOptProposer` for skills — and wraps `selfImprove`; drop to `selfImprove({ agent, scenarios, judge, baselineSurface })` from `agent-eval/contract` only for the lower-level loop) | canonical-api §3.4 |
| **Gate: ship/hold a candidate** (campaign ctx) | `defaultProductionGate` / `heldOutGate` / `composeGate` — `agent-eval/contract`; `neutralizationGate` (footprint-matched PLACEBO gate — proves a held-out lift is CONTENT, not added prompt/mount footprint) — `agent-eval/campaign` | canonical-api §3.4 |
| **Gate: ship/hold from a `BenchmarkReport`** (per-task cells) | `promotionGate({ report, incumbent, candidate })` — `/loops` | canonical-api §3.4 |
| **Run the full multi-generation flywheel + certify** | `runStrategyEvolution(config)` — `/loops` | canonical-api §3.4 |
| **Observe a run** (cost/time waterfall, OTLP) | `createWaterfallCollector()` — `/loops`; `createOtelExporter` attached via `composeRuntimeHooks(...)` — root `.` | canonical-api §2 |
Expand All @@ -110,7 +110,7 @@ holds the load-bearing invariant the parallel breaks:
`loopUntil` + `runPersonified` (threads executor seams; equal-k; selector≠judge
firewall; journal/replay — a parallel runner silently fails to wire the seams).
- "skill optimizer" / "topology mutator" that opens branches + applies patches
**≈** `improvementDriver` (code surface) or `selfImprove`/`gepaDriver` (prompt
**≈** `improvementDriver` (code surface) or `selfImprove`/`gepaProposer` (prompt
surface) — both gated on a frozen holdout.
- "profile-seam" / agent-config wrapper carrying model+prompt+tools+role **≈**
`AgentProfile` (it IS that bundle) + `definePersona` (the run record);
Expand Down
11 changes: 6 additions & 5 deletions skills/loop-writer/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ The driver owns strategy.
| Review from several lenses | `panel` |
| Simulated user/product eval | `defineConversation` + `runConversation` |
| Dynamic topology / drivers of drivers | `Scope` or sandbox driver + `createCoordinationTools` |
| Mutate a shared repo | git branch/clone loop with typed merge outcomes |
| Run N coding workers on isolated worktrees, gate each, pick best patch | `worktreeFanout` |
| Mutate a shared repo | git branch/clone loop with typed merge outcomes (`gitWorkspace` seam) |

If a fixed combinator solves it, do not use a dynamic driver.

Expand Down Expand Up @@ -110,9 +111,9 @@ const result = await createSupervisor<Task, Output>().run(driver, task, supervis
```

When the driver lives in a sandbox, expose the same verbs through
`createCoordinationTools`: `spawn_worker`, `await_event`, `observe_worker`,
`steer_worker`, `list_questions`, `answer_question`, `ask_parent`, `stop`, and
optional analyst tools.
`createCoordinationTools`: `spawn_agent`, `await_event`, `observe_agent`,
`steer_agent`, `list_questions`, `answer_question`, `ask_parent`, `stop`, and
optional analyst tools (`list_analysts`, `run_analyst`).

## Role Boundaries

Expand All @@ -133,7 +134,7 @@ with unresolved `blocks-run` questions.
Steer sparingly: only when an analyst finds a concrete mistake, a loop is
duplicating work, a parent/Pi answers a blocker, or a verifier reveals a specific
fix a running worker can still use. Delivery is through `Scope.send` or
`steer_worker`; failed delivery means spawn a fresh corrected attempt.
`steer_agent`; failed delivery means spawn a fresh corrected attempt.

## Workspace Loops

Expand Down
4 changes: 2 additions & 2 deletions skills/supervise/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ You are a supervisor. You do NOT do the work yourself — you design and drive s
## Loop

1. **Decompose** the task into the smallest set of sub-tasks a single focused worker can each deliver.
2. **Author** a worker per sub-task by calling `spawn_worker` with a complete `profile`:
2. **Author** a worker per sub-task by calling `spawn_agent` with a complete `profile`:
- `name` — a short id.
- `skills` — the skill files the worker should carry (by name), OR `systemPrompt` — rich, specific instructions for this sub-task.
- `model` — the model best suited to this sub-task (optional).
Expand All @@ -21,4 +21,4 @@ You are a supervisor. You do NOT do the work yourself — you design and drive s

## Authoring sub-supervisors

If a sub-task is itself too large for one worker, author it as a **sub-supervisor**: give its profile a `skills` list that includes `supervise`. It will decompose and drive its own workers one level deeper. This is not a special call — it is the same `spawn_worker`, just a profile that carries this skill.
If a sub-task is itself too large for one worker, author it as a **sub-supervisor**: give its profile a `skills` list that includes `supervise`. It will decompose and drive its own workers one level deeper. This is not a special call — it is the same `spawn_agent`, just a profile that carries this skill.
Loading