feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session memory: a cache-safe <skill-memory> block is appended to each skill tool RESULT with intent-scoped recall (embeddings + FTS) and historian auto-extraction into global notes. Ships the opt-in distill-skill-memory dreamer task, ctx_skill_note/ctx_skill_recall tools, UI/CLI stats, and DB migrations to v52 with a self-heal path.

• New Features

  • Transparent per-skill recall: skill tool advertises optional intent; before-hook stashes it and after-hook injects recall; ctx_skill_note writes and ctx_skill_recall reads; Status/TUI/ctx-status show per-project totals and pinned counts.
  • Ranking + embeddings: cosine blend over intent_embedding + delta_embedding with FTS5 fallback; programmatic re-embed backfills NULL/stale vectors before distill-skill-memory.
  • Historian auto-extraction: TC: skill(<name>) markers → <skill_observations>; both runners promote via promoteSkillObservations as global '*' notes (source_type='historian').
  • Schema/migrations/self-heal: skill_memory table + FTS vtable/columns; LATEST_SUPPORTED_VERSION=52; initializeDatabase re-creates missing objects and rebuilds skill_memory_fts when needed.

• Bug Fixes

  • Cross-partition recall: union a skill’s partition with global '*' so project-local skills surface historian-written notes; writes/dedup remain exact-partition.
  • Disabled-path safety and routing: guard ctx_skill_note when the plugin is disabled; correctly classify singular ~/.config/opencode/skill/** as global; normalize Windows paths.
  • Frontmatter parsing: accept inline skill-memory: { enabled: true }; anchor to start-of-file; tolerate leading BOM/whitespace; strip inline # from unquoted scalars.
  • Historian marker + intent stash: emit TC: skill(<name>) verbatim (')' preserved); only CR/LF/tab are unsafe; key stashed intents by sessionID:callID and prune on session delete.
  • Provenance parsing: anchor the "Base directory" regex to line-start and take the last match to avoid content-shadowing; ensures correct skill identity resolution.
  • Observability + health: log recall errors; rebuild skill_memory_fts when the vtable exists but is empty; escape apostrophes in SQL.
  • Token budgeting + tier derivation: count per-note XML framing in budgets and clamp pinned budget; derive tier via dirname(resolvedPath) for robustness.

^{Written for commit ef5cba1. Summary will update on new commits.}

Greptile Summary

This PR adds skill-memory — a per-skill cross-session recall system that appends a <skill-memory> block to the skill tool result. It is fully opt-in (frontmatter skill-memory: { enabled: true }), cache-safe by design (appended to the tool result, not the system prompt), and ships in four coherent phases: transparent recall, embedding-ranked recall, historian extraction foundation, and historian auto-extraction with both runners.

• Core storage and recall (skill-memory/storage.ts, recall.ts): the union-partition predicate correctly surfaces historian global '*' notes alongside project-specific notes; budgetFill properly clamps the pinned sub-budget to maxTokens; the previously bare catch {} in recallSkillMemoryBlock now logs errors before swallowing them.
• DB migrations v50\u2013v52 (migrations.ts, storage-db.ts): idempotent via IF NOT EXISTS / columnExists guards; v52 collision-merge runs in a transaction with correct pre-aggregated sums; initializeDatabase self-heal includes the FTS rebuild guard fixed in the previous review round.
• Hook wiring (hook-handlers.ts, hook.ts, index.ts): intent stash now keyed sessionId:callId and pruned by prefix on session delete; single hoisted dynamic import of the provenance module shared by both try blocks; fail-loud guard in index.ts catches a missing skillLoadRegistry at startup rather than at the first ctx_skill_note call.

Confidence Score: 5/5

Safe to merge. All three previously-reported blocking issues are correctly addressed and regression-tested. The new code is defensive throughout: recall errors are logged and swallowed, historian promotion is best-effort per-observation, and the fail-loud guard at plugin init catches wiring regressions before any agent can invoke the tools.

All three previously-reported blocking issues (intent-map session scoping, FTS rebuild in the self-heal path, project-local skill recall orphaning) are correctly addressed. The migrations are idempotent and correctly versioned, hook wiring is properly cleaned up in onSessionDeleted, and no functional regressions were found. The only open items are cosmetic and do not affect correctness.

No files require special attention. ctx-skill-recall/tools.ts has a cold-start limitation for custom skill paths but this affects only users with non-standard cfg.skills.paths configuration and degrades gracefully to a clear error message.

Important Files Changed

Filename	Overview
packages/plugin/src/features/magic-context/skill-memory/storage.ts	New file: all read/write helpers for skill_memory table. Union-partition recall predicate correctly surfaces historian '*' notes alongside per-project notes. getDedupCandidates intentionally uses exact-partition. No bugs found.
packages/plugin/src/features/magic-context/skill-memory/recall.ts	New file: 4-rung recall cascade (cosine to FTS5 to flat recency x hit, pinned always first). recallSkillMemoryBlock now logs errors before returning empty string. budgetFill ordering guarantee prevents pinnedTokens counter from being prematurely incremented.
packages/plugin/src/features/magic-context/skill-memory/promote.ts	New file: historian-extracted notes promoted as global '*' tier with hash dedup and best-effort per-observation error handling. originProject stores caller-provided string as-is -- inconsistent between runners but column is metadata-only.
packages/plugin/src/features/magic-context/migrations.ts	Adds migrations v50-v52. Idempotent via IF NOT EXISTS and columnExists guards. v52 collision-merge runs inside a transaction with correct pre-aggregated sums. LATEST_SUPPORTED_VERSION bumped to 52 in lockstep.
packages/plugin/src/features/magic-context/storage-db.ts	Self-heal path in initializeDatabase now includes the FTS rebuild guard (empty fts + non-empty table triggers rebuild). All new ensureColumn calls for skill_memory added. Guarded rebuild fires once on the gap case, not on every boot.
packages/plugin/src/hooks/magic-context/hook-handlers.ts	Intent stash now keyed sessionId:callId; pruneIntentsForSession prefix-prunes on session delete. Dynamic import hoisted above both try blocks. Non-fatal wrappers prevent skill-memory failures from surfacing in tool results.
packages/plugin/src/hooks/magic-context/hook.ts	Wires up intentByCallId map, skillLoadRegistry, sessionDirectoryBySession, and both before/after hooks. Session cleanup prunes both structures by prefix. skillLoadRegistry exposed on return value for use by createCtxSkillNoteTool.
packages/plugin/src/features/magic-context/skill-memory/frontmatter.ts	New file: hand-rolled YAML frontmatter parser for skill-memory block. Handles BOM, inline flow-mapping, inline comments, multi-line block form. Anchored to file start to avoid mid-doc false positives. All failure paths return null.
packages/plugin/src/features/magic-context/skill-memory/provenance.ts	New file: extracts skill tier/source/path from opencode's Base directory output line. Uses last-match anchor to defeat content-shadowing. Correctly classifies all global dirs and handles Windows path normalization.
packages/plugin/src/tools/ctx-skill-note/tools.ts	New tool: hash-dedup then semantic-dedup (delta only, 200 candidates, model-matched), then insert. Guards against kind=general with actionable error. Correctly uses toolContext.directory for project identity resolution.
packages/plugin/src/tools/ctx-skill-recall/tools.ts	New tool: registry-first resolution with disk-fallback for cold-start. Cold-start search covers standard skill directories but does NOT include custom paths from cfg.skills?.paths, yielding a misleading error for non-standard configurations.
packages/plugin/src/hooks/magic-context/compartment-parser.ts	Added ParsedSkillObservation and skillObservations parsing. SKILL_OBS_ITEM_REGEX correctly captures skillId
packages/pi-plugin/src/pi-historian-runner.ts	Adds promoteSkillObservations call behind the same promotionActive + !discardedLast gate used for facts/primers. Passes raw projectPath as originProject while OpenCode runner passes a hash -- inconsistent provenance but metadata-only.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Model
    participant hook as tool.execute hooks
    participant stash as intentByCallId map
    participant registry as skillLoadRegistry
    participant db as SQLite (skill_memory)
    participant historian as Historian runner

    Note over Model,historian: Transparent recall path
    Model->>hook: "tool.execute.before {tool:skill, intent}"
    hook->>stash: stashIntent(sessionId:callId, intent)
    Model->>hook: "tool.execute.after {tool:skill, output}"
    hook->>hook: parseSkillProvenance + parseFrontmatterConfig
    hook->>registry: set(sessionId:skillId, entry)
    hook->>stash: getAndDeleteIntent
    hook->>db: recallSkillMemoryBlock
    db-->>hook: skill-memory block
    hook->>Model: append block to output

    Note over Model,historian: Agent writes a note
    Model->>hook: "ctx_skill_note {skill, kind, delta}"
    hook->>registry: getSkillLoad
    hook->>db: hash dedup + semantic dedup
    hook->>db: insertSkillMemoryNote

    Note over Model,historian: Historian extraction
    historian->>historian: parse skill_observations
    historian->>db: "promoteSkillObservations to global '*'"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Model
    participant hook as tool.execute hooks
    participant stash as intentByCallId map
    participant registry as skillLoadRegistry
    participant db as SQLite (skill_memory)
    participant historian as Historian runner

    Note over Model,historian: Transparent recall path
    Model->>hook: "tool.execute.before {tool:skill, intent}"
    hook->>stash: stashIntent(sessionId:callId, intent)
    Model->>hook: "tool.execute.after {tool:skill, output}"
    hook->>hook: parseSkillProvenance + parseFrontmatterConfig
    hook->>registry: set(sessionId:skillId, entry)
    hook->>stash: getAndDeleteIntent
    hook->>db: recallSkillMemoryBlock
    db-->>hook: skill-memory block
    hook->>Model: append block to output

    Note over Model,historian: Agent writes a note
    Model->>hook: "ctx_skill_note {skill, kind, delta}"
    hook->>registry: getSkillLoad
    hook->>db: hash dedup + semantic dedup
    hook->>db: insertSkillMemoryNote

    Note over Model,historian: Historian extraction
    historian->>historian: parse skill_observations
    historian->>db: "promoteSkillObservations to global '*'"

Loading

Comments Outside Diff (1)

1. packages/plugin/src/features/magic-context/skill-memory/recall.ts, line 481-484 (link)

   Bare catch {} makes recall errors invisible

   Any exception thrown inside recallSkillMemoryBlock — a SQLite error, embedding provider crash, or a coding mistake — is silently swallowed and returns an empty string. From the caller's perspective this is indistinguishable from "no notes exist" or "skill-memory disabled". At minimum a sessionLog or console.warn call here would let a developer diagnose why the recall block is absent without attaching a debugger.

_{Reviews (5): Last reviewed commit: "fix(skill-memory): address third review ..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/plugin/src/features/magic-context/dreamer/task-executor.ts, line 419:

<comment>Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</comment>

<file context>
@@ -412,7 +413,18 @@ export function createDreamTaskExecutor(deps: DreamTaskExecutorDeps): TaskExecut
+            // Skill-memory P2: re-embed stale/NULL skill-note vectors BEFORE the
+            // distill agentic pass so the prompt sees a fully-embedded corpus and
+            // P1-era notes promote from the FTS rung to the cosine rung.
+            if (config.task === "distill-skill-memory") {
+                try {
+                    await reembedStaleSkillNotes(db, projectIdentity);
</file context>

[iceteaSA]

Holding this one as working-as-intended. The catch is not silent — it logs [dreamer] distill-skill-memory reembed pre-step failed: ${e} (task-executor.ts, the line directly inside the catch). The suppression is deliberate: re-embedding is a best-effort optimization pre-step, not a prerequisite — if it fails, the affected notes simply stay on the FTS recall rung instead of the cosine rung, and the distill report still runs correctly. Letting a reembed failure fail the whole task would block a report that doesn't depend on it. The failure is observable via the log; happy to escalate to a metric/warning if you'd prefer louder signal.

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>