Skip to content
Open
225 changes: 214 additions & 11 deletions ARCHITECTURE.md

Large diffs are not rendered by default.

36 changes: 36 additions & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ To disable the dreamer entirely, set `dreamer.disable: true`. To disable a singl
| `refresh-primers` | `0 3 * * *` | Re-investigate stale primers against current code and refresh their answers. |
| `evaluate-smart-notes` | `0 3 * * *` | Surface smart notes whose `ctx_note` conditions have come true. |
| `review-user-memories` | `0 3 * * *` | Promote recurring behavioral observations into the `<user-profile>` block (privacy-sensitive). |
| `distill-skill-memory` | `""` (off) | **Opt-in** — add a schedule to enable. Merge near-duplicate skill notes, prune stale low-hit notes, promote recurring gotchas to `pinned=1`, enforce per-skill note caps. Requires the `skill_memory` table (migration v50) — auto-created on upgrade. |

### Retrospective privacy

Expand Down Expand Up @@ -589,6 +590,41 @@ Tier boundaries are hardcoded to keep behavior predictable and prevent cache-bus

**When to enable.** Enable alongside `ctx_reduce_enabled: false` if you find historian/heuristics insufficient for your workload — typically sessions with very long pasted content or verbose agent explanations that the automatic pipeline doesn't reach. Leaves `ctx_reduce_enabled: true` sessions untouched.

---

## Skill-Memory (per-skill frontmatter)

Skill-memory is the "motor memory" for skills — per-skill, cross-session recall of gotchas, discoveries, fixes, and workflow steps. The plugin transparently augments opencode's built-in `skill` tool: when a skill declares `skill-memory: { enabled: true }` in its YAML frontmatter, accumulated notes for that skill surface in a `<skill-memory>` block appended to the skill tool's RESULT on every load. Agents write back via `ctx_skill_note`; explicit recall (without re-loading) is `ctx_skill_recall`.

Unlike every other setting in this file, **skill-memory is configured per-skill in each `SKILL.md`'s frontmatter, not in `magic-context.jsonc`**. Absent or malformed block = inert. A bad config in one skill cannot break other skills.

```yaml
---
name: test-driven-development
description: ...
skill-memory:
enabled: true # required: true to activate
max_tokens: 1500 # default 1500 — token budget for unpinned notes
max_pinned_tokens: 4000 # default 4000 — separate cap for pinned notes
dedup_threshold: 0.92 # default 0.92 — P2 cosine near-dedup threshold
---
```

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | `boolean` | (required `true`) | Master switch per skill. When absent or `false`, the transparent after-hook skips this skill entirely and `ctx_skill_recall` returns "skill-memory is not enabled for '<skill>'". |
| `max_tokens` | `number` | `1500` | Hard cap on tokens for unpinned notes in the injected block. Greedy fill by composite score (P1: recency × hit_count). |
| `max_pinned_tokens` | `number` | `4000` | Separate cap for pinned notes. Pinned notes are always included first; on cap overflow, least-used pinned notes are truncated in ascending `hit_count` order with an "N pinned notes omitted" marker. |
| `dedup_threshold` | `number` | `0.92` | P2 cosine near-dedup threshold. P1 ships without embeddings, so this is reserved for the P2 rollout. Tune per-skill in the `0.85`–`0.95` range. |

**Cache safety.** The injected block lands in the tool RESULT = conversation tail, never the cached m[0]/m[1] prefix. This is the same pattern as Channel-1 (`maybeInjectChannel1Nudge`) and is why skill-memory cannot regress the prompt-cache hit rate.

**Write-back (`ctx_skill_note`).** The injected block's footer prompts: *"After using this skill, call `ctx_skill_note` — record only gotchas, novel discoveries, or error→fix; skip routine successes."* The `kind` parameter is a hard gate: `kind: "general"` is rejected at the tool level — general observations belong in `ctx_memory` with an appropriate category.

**Dreamer integration.** Add `"distill-skill-memory"` to your `dreamer.tasks` list to opt in to overnight maintenance (merges near-duplicates, prunes stale zero-hit notes older than 30 days, promotes recurring gotchas to pinned). It is **not** a default task — the feature is opt-in like `maintain-docs`.

**P1 vs P2.** P1 (shipped) is flat recall (recency × hit_count, no embeddings). P2 (planned) adds intent-aware ranking via the project's existing embedding provider. The per-skill `dedup_threshold` field is reserved for P2 cosine near-dedup and has no effect on P1.

## Commands

| Command | Description |
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ Because it runs during idle time, the dreamer pairs well with local models, even

- **`ctx_expand`**: pull a compressed history range back to the original `U:`/`A:` transcript when the agent needs the exact details.
- **`ctx_note`**: a scratchpad for deferred intentions. Notes resurface at natural boundaries (after commits, after historian runs, when todos finish). **Smart notes** carry an open-ended condition the dreamer watches for.
- **Skill-memory (motor memory for skills)**: a per-skill `<skill-memory>` block is appended to the skill tool's RESULT on every load, surfacing accumulated gotchas, discoveries, fixes, and workflow steps the agent has recorded for that skill. Per-skill opt-in via the skill's `SKILL.md` frontmatter (`skill-memory: { enabled: true }`); write back with **`ctx_skill_note`**, recall without re-loading with **`ctx_skill_recall`**. The block lands in the tool result, not the cached prompt prefix, so it never thrashes the cache.

Recall works **across sessions** (a new session inherits everything) and **across harnesses** (write a memory in OpenCode, retrieve it in Pi).

Expand All @@ -198,6 +199,8 @@ Recall works **across sessions** (a new session inherits everything) and **acros
| `ctx_search` | Recall | Search memories, conversation history, and git commits |
| `ctx_expand` | Recall | Decompress a history range back to the transcript |
| `ctx_note` | Recall | Deferred intentions and dreamer-evaluated smart notes |
| `ctx_skill_note` | Recall | Write back a per-skill note (gotcha/discovery/fix/workflow) for future loads |
| `ctx_skill_recall` | Recall | Explicitly recall skill-memory notes without re-loading the skill |

---

Expand Down
20 changes: 14 additions & 6 deletions STRUCTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,14 @@

**`src/features/`:**
- Purpose: Group reusable subsystem logic by feature.
- Contains: Magic-context services (storage, scheduler, tagger, search, message-index, overflow detection, compaction markers), dreamer runtime, sidekick support, memory system, user-memory pipeline, git-commit indexer, tool-definition token measurement, schema migrations, built-in commands.
- Key subdirs: `src/features/magic-context/dreamer/`, `src/features/magic-context/memory/`, `src/features/magic-context/sidekick/`, `src/features/magic-context/user-memory/`, `src/features/magic-context/git-commits/`, `src/features/builtin-commands/`
- Key files: `src/features/magic-context/storage-db.ts`, `src/features/magic-context/storage.ts` (barrel), `src/features/magic-context/migrations.ts`, `src/features/magic-context/message-index.ts`, `src/features/magic-context/search.ts`, `src/features/magic-context/overflow-detection.ts`, `src/features/magic-context/dreamer/runner.ts`, `src/features/magic-context/memory/storage-memory.ts`, `src/features/magic-context/user-memory/storage-user-memory.ts`, `src/features/builtin-commands/commands.ts`
- Contains: Magic-context services (storage, scheduler, tagger, search, message-index, overflow detection, compaction markers), dreamer runtime, sidekick support, memory system, user-memory pipeline, key-files pinning, git-commit indexer, tool-definition token measurement, schema migrations, built-in commands, **skill-memory (per-skill motor memory)**.
- Key subdirs: `src/features/magic-context/dreamer/`, `src/features/magic-context/memory/`, `src/features/magic-context/sidekick/`, `src/features/magic-context/user-memory/`, `src/features/magic-context/key-files/`, `src/features/magic-context/git-commits/`, `src/features/magic-context/skill-memory/`, `src/features/builtin-commands/`
- Key files: `src/features/magic-context/storage-db.ts`, `src/features/magic-context/storage.ts` (barrel), `src/features/magic-context/migrations.ts`, `src/features/magic-context/message-index.ts`, `src/features/magic-context/search.ts`, `src/features/magic-context/overflow-detection.ts`, `src/features/magic-context/dreamer/runner.ts`, `src/features/magic-context/memory/storage-memory.ts`, `src/features/magic-context/skill-memory/{frontmatter,provenance,storage,recall}.ts`, `src/features/magic-context/user-memory/storage-user-memory.ts`, `src/features/builtin-commands/commands.ts`

**`src/tools/`:**
- Purpose: Define the agent-facing tool surface.
- Contains: One directory per tool with constants, types, implementation, and tests. Five tools: `ctx-reduce`, `ctx-expand`, `ctx-note`, `ctx-memory`, `ctx-search`.
- Key files: `src/tools/ctx-reduce/tools.ts`, `src/tools/ctx-expand/tools.ts`, `src/tools/ctx-note/tools.ts`, `src/tools/ctx-memory/tools.ts`, `src/tools/ctx-search/tools.ts`
- Contains: One directory per tool with constants, types, implementation, and tests. Seven tools: `ctx-reduce`, `ctx-expand`, `ctx-note`, `ctx-memory`, `ctx-search`, `ctx-skill-note`, `ctx-skill-recall`. The two `ctx_skill_*` tools share the `recallSkillMemoryBlock` core with the transparent after-hook path.
- Key files: `src/tools/ctx-reduce/tools.ts`, `src/tools/ctx-expand/tools.ts`, `src/tools/ctx-note/tools.ts`, `src/tools/ctx-memory/tools.ts`, `src/tools/ctx-search/tools.ts`, `src/tools/ctx-skill-note/tools.ts`, `src/tools/ctx-skill-recall/tools.ts`

**`src/shared/`:**
- Purpose: Keep cross-feature utilities small and dependency-light.
Expand Down Expand Up @@ -101,16 +101,24 @@
- `src/hooks/magic-context/strip-content.ts`: Strip and replay reasoning, inline thinking, structural noise, dropped placeholders, merged-assistant reasoning, processed images, and system-injected messages.
- `src/hooks/magic-context/caveman.ts`: Experimental age-tier text compression for primary sessions with `ctx_reduce_enabled=false`.
- `src/hooks/magic-context/todo-view.ts`: Build the deterministic synthetic todowrite tool part and compute its hash-based `call_id`.
- `src/hooks/magic-context/skill-tool-definition.ts`: `injectSkillIntentParam` — adds an optional `intent` parameter to the `skill` tool's schema via the `tool.definition` hook (Effect-Schema strips it before the skill runs; the before-hook captures it pre-validation).
- `src/hooks/magic-context/inject-compartments.ts`: m[0]/m[1] history layout — `renderM0`/`renderM1`/`materializeM0`/`mustMaterialize` (mirrored in Pi's `inject-compartments-pi.ts`).
- `src/hooks/magic-context/decay-curve.ts`: Council-validated deterministic tier-decay math (half-life, log-cost tier boundaries, budget pressure).
- `src/hooks/magic-context/decay-render.ts`: Shared OpenCode+Pi compartment renderer built on the decay curve (replaces the removed LLM compressor).
- `src/hooks/magic-context/compartment-runner-incremental.ts`: v2 historian publish path — bounded reference blocks, tiered/scored compartments, faithful per-chunk facts, discard-last, events + `p1_embedding` on publish.
- `src/hooks/magic-context/reference-retrieval.ts` (+ `reference-seeds.generated.ts`): 4 rotating seed compartments + last-6 recency references for the historian prompt.
- `src/hooks/magic-context/historian-prompt.generated.ts`: Generated v8.7.3 historian system prompt (source: `.alfonso/.../historian-prompt-v8.7.3.md`; re-exported via `compartment-prompt.ts`).
- `src/hooks/magic-context/hook-handlers.ts` (skill-memory branches): `createToolExecuteBeforeHook` (stashes per-callID `intent` in a bounded closure map), `maybeInjectSkillMemory` (appends the `<skill-memory>` block to `output.output` BEFORE the Channel-1 nudge), and the `createToolExecuteAfterHook` branch that parses the `Base directory` line + reads `SKILL.md` from disk to populate the session-scoped `SkillLoadRegistry`.
- `src/features/magic-context/memory/memory-migration.ts`: `/ctx-session-upgrade` 9-cat→5-cat memory re-eval (active-only, permanent-safe, epoch-bumping).
- `src/features/magic-context/skill-memory/frontmatter.ts`: Minimal YAML frontmatter parser for the per-skill `skill-memory:` block — returns `null` (inert) when absent or malformed; a bad config in one skill cannot break other skills.
- `src/features/magic-context/skill-memory/provenance.ts`: `parseSkillProvenance` (cross-platform `fileURLToPath` parser for the `Base directory for this skill: file:///...` line), `deriveSkillTier` / `deriveSkillSource` (path-based classification), and the session-scoped `SkillLoadRegistry` (`Map<sessionId:skillId, {resolvedPath, tier, skillSource, frontmatterConfig, loadedAt}>` — NOT persisted, cleaned in `onSessionDeleted`).
- `src/features/magic-context/skill-memory/storage.ts`: `skill_memory` table CRUD — `insertSkillMemoryNote` (UNIQUE-violation on duplicate returns null so callers can `bumpHitCount`), `getSkillMemoryNotes` (window-function flat ranking for P1), `findExistingNote`, `bumpHitCount`. The `UNIQUE(skill_id, tier, project_identity, normalized_hash)` constraint plus the `idx_skill_memory_lookup` and `idx_skill_memory_fts_prep` indexes live in migration v37.
- `src/features/magic-context/skill-memory/recall.ts`: `recallSkillMemoryBlock` — the shared recall+format core (used by BOTH the transparent after-hook AND `ctx_skill_recall`). `flatRecall` does P1's recency × hit_count greedy-fill; `buildSkillMemoryBlock` formats the `<skill-memory>` XML with the `ctx_skill_note` write-back footer. Lives in the feature layer to avoid a tools→hooks layering violation.
- `src/tools/ctx-skill-note/tools.ts`: `ctx_skill_note` — write-back tool. Hard gate rejects `kind: "general"` (general observations belong in `ctx_memory`). Exact-dedup on `normalized_hash` (reuses `computeNormalizedHash` from `memory/normalize-hash.ts`) bumps `hit_count`. Resolves `(skill_id, tier, project_identity, resolved_path)` from the session-scoped `SkillLoadRegistry`.
- `src/tools/ctx-skill-recall/tools.ts`: `ctx_skill_recall` — explicit recall companion. Registry-first resolution (exact, free, no disk I/O when the skill was loaded this session) with a cold-start disk fallback walking opencode's real `discoverSkills()` order (project dirs first — they shadow global). Returns distinct messages for SKILL.md-not-found vs disabled vs cold-start-no-notes.
- `src/features/magic-context/storage-db.ts`: Create durable storage; run versioned migrations; resolve runtime SQLite backend.
- `src/features/magic-context/storage-meta-persisted.ts`: Read and write per-session persisted scalars and JSON blobs.
- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v44 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep).
- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v50 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep). v50 adds the `skill_memory` table with `(skill_id, tier, project_identity, normalized_hash)` UNIQUE plus `idx_skill_memory_lookup` and `idx_skill_memory_fts_prep` indexes.
- `src/features/magic-context/message-index.ts`: FTS-backed raw-message index for `ctx_search`.
- `src/features/magic-context/search.ts`: Unified retrieval over memories, raw messages, and git commits.

Expand Down
54 changes: 54 additions & 0 deletions assets/magic-context.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,10 @@
"refresh-primers": {
"schedule": "0 3 * * *",
"timeout_minutes": 20
},
"distill-skill-memory": {
"schedule": "",
"timeout_minutes": 20
}
},
"type": "object",
Expand Down Expand Up @@ -959,6 +963,56 @@
"minimum": 5
}
}
},
"distill-skill-memory": {
"default": {
"schedule": "",
"timeout_minutes": 20
},
"type": "object",
"properties": {
"schedule": {
"default": "",
"type": "string",
"description": "5-field cron schedule (e.g. \"0 3 * * *\"), or \"\" to disable this task."
},
"model": {
"description": "Per-task model override (inherits dreamer.model)",
"type": "string"
},
"fallback_models": {
"description": "Per-task fallback chain (inherits dreamer.fallback_models)",
"anyOf": [
{
"type": "string"
},
{
"type": "array",
"items": {
"type": "string"
}
}
]
},
"thinking_level": {
"description": "Pi only: per-task thinking level",
"type": "string",
"enum": [
"off",
"minimal",
"low",
"medium",
"high",
"xhigh"
]
},
"timeout_minutes": {
"default": 20,
"description": "Minutes allowed for this task before it is aborted",
"type": "number",
"minimum": 5
}
}
}
},
"description": "Per-task scheduling + model config. Each task has its own cron schedule and may override the dreamer-level model."
Expand Down
4 changes: 2 additions & 2 deletions packages/cli/src/lib/dreamer-setup.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,11 @@ describe("runDreamerSetup", () => {
const prompts = new MockPrompts({
confirms: [false],
autos: ["x/y"],
selects: Array(11).fill("cron:0 3 * * *"),
selects: Array(12).fill("cron:0 3 * * *"),
});
const result = await runDreamerSetup(prompts, ["x/y"]);
expect(result.tasks).toBeDefined();
expect(Object.keys(result.tasks ?? {}).length).toBe(11);
expect(Object.keys(result.tasks ?? {}).length).toBe(12);
expect(result.tasks?.verify.schedule).toBe("0 3 * * *");
expect(result.tasks?.curate.schedule).toBe("0 3 * * *");
expect(result.tasks?.["classify-memories"].schedule).toBe("0 3 * * *");
Expand Down
2 changes: 2 additions & 0 deletions packages/cli/src/lib/dreamer-setup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ const TASK_DESCRIPTIONS: Record<DreamTaskName, string> = {
"review-user-memories": "Promote recurring behaviors into your user profile",
"promote-primers": "Promote recurring project questions into Primers",
"refresh-primers": "Refresh answers for active project Primers",
"distill-skill-memory": "Opt-in: distills per-skill memory (merge/prune/promote)",
};

/** v1-behavior-preserving default schedules (must match the Zod schema defaults). */
Expand All @@ -50,6 +51,7 @@ const DEFAULT_TASK_SCHEDULES: Record<DreamTaskName, string> = {
"review-user-memories": "0 3 * * *",
"promote-primers": "0 3 * * *",
"refresh-primers": "0 3 * * *",
"distill-skill-memory": "",
};

const PRESET_CUSTOM = "__custom__";
Expand Down
4 changes: 3 additions & 1 deletion packages/dashboard/src-tauri/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ pub fn resolve_project_config_path(project_path: &str) -> PathBuf {
/// frontend DreamerTasksField list). The dashboard renders this fixed set so
/// every project shows the same tasks regardless of its (possibly stale) per-
/// project scheduler snapshot in task_schedule_state.
pub const CANONICAL_DREAM_TASKS: [&str; 11] = [
pub const CANONICAL_DREAM_TASKS: [&str; 12] = [
"map-memories",
"verify",
"verify-broad",
Expand All @@ -40,6 +40,7 @@ pub const CANONICAL_DREAM_TASKS: [&str; 11] = [
"review-user-memories",
"promote-primers",
"refresh-primers",
"distill-skill-memory",
];

/// Default cron per task (mirrors DEFAULT_TASK_SCHEDULES in the plugin schema and
Expand All @@ -58,6 +59,7 @@ pub fn default_task_schedule(task: &str) -> &'static str {
"review-user-memories" => "0 3 * * *",
"promote-primers" => "0 3 * * *",
"refresh-primers" => "0 3 * * *",
"distill-skill-memory" => "",
_ => "",
}
}
Expand Down
Loading
Loading