diff --git a/CLAUDE.md b/CLAUDE.md index c669dcd2..b61dbf16 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,7 +26,7 @@ Read the relevant docs before starting work on a subsystem. - **Tech stack**: Bun, TypeScript, PostgreSQL 18 (pgvector/halfvec, pg_textsearch BM25, ltree, citext, JSONB), **postgres.js** driver. One database, one pool. - **Schemas** (three, one database): `auth` (better-auth-shaped: `users`, `sessions`, `accounts`, `device_authorization`), `core` (control plane: `principal`, `space`, `principal_space`, `group_member`, `tree_access`, `api_key`), and per-space `me_` (data plane: the single `memory` table). `auth.users.id == core.principal.id` for user principals. -- **Memory table** (per space): `content`, `meta` (JSONB), `tree` (ltree), `temporal` (tstzrange), `embedding` (halfvec(1536)). +- **Memory table** (per space): `content`, `name` (text — optional filename-like leaf slug, unique within `(tree, name)` via a partial unique index where name is not null), `meta` (JSONB), `tree` (ltree), `temporal` (tstzrange), `embedding` (halfvec(1536)). Addressed by immutable `id` (`memory.get`/`delete`) or by `folder/name` path (`memory.getByPath`/`deleteByPath`, split at the final `/`); `deleteTree` removes a subtree. Wire/display paths are canonical leading-slash (`/share/auth`, `~/notes`, root `/`); the leading slash is optional on input. ltree storage and the access-grant shorthand below (`owner@home.`) stay dotted because that is the literal ltree representation. - **Search**: hybrid BM25 + semantic via Reciprocal Rank Fusion, computed in SQL functions. - **Access**: no RLS. `core.build_tree_access(principalId, spaceId)` produces a `_tree_access` jsonb (rows of `tree_path` + `access`) passed into the space SQL functions (`search_memory`, `get_memory`, …). Three additive levels: **1 = read, 2 = write, 3 = owner**; `owner@root` (the empty ltree path) owns the whole space, and an owner grant at any path delegates access-management within that subtree. Two axes: **structural** authority (`principal_space.admin` — roster mutations, groups, invitations) vs **data** authority (owner@path); an admin may also grant data and can self-grant `owner@root`. The auth gate is a non-empty `build_tree_access` (every member holds ≥1 grant). - **Tree conventions**: two reserved roots — per-member `home.` (`~` is input sugar for it; a joining **user** is granted `owner@home.`, and a joining **agent** `owner@home..` — nested under its owner's home so the owner's home grant covers it and the `agent_tree_access` clamp keeps it effective) and the shared `share`. A space **creator** gets `admin` + `owner@home` + `owner@share`, **not** `owner@root` — so it sees `share` and its own `~` but not other members' homes (as an admin it can self-grant `owner@root`). `memory.create`/`batchCreate` **require** an explicit `tree` (callers choose `share` vs `~` deliberately); only the file importers (`me import memories`, the `me_memory_import` MCP tool) default a tree-less record to `share` (`SHARE_NAMESPACE`, canonically defined in `@memory.build/protocol` and re-exported by `@memory.build/database`). @@ -210,7 +210,7 @@ so drop it too.) - **CLI credentials**: split across `~/.config/me/` — **`config.yaml`** (non-secret: default server + per-server **active space** / the X-Me-Space) and **`credentials.yaml`** (0600, secret session-token *fallback* only). The **session token** lives in the OS keychain when available (macOS `security`, Linux `secret-tool` via libsecret; `ME_NO_KEYCHAIN=1` forces off), else in `credentials.yaml` (empty/absent on keychain hosts); a pre-split `credentials.yaml` is migrated on first read. `me logout` clears the session secret but keeps the non-secret config (so re-login resumes). **Api keys are never persisted** — an agent key only ever comes from `ME_API_KEY` (humans authenticate with sessions; `apiKey.create` prints the key once for the operator to place where the agent runs). Env: `ME_SERVER` / `ME_API_KEY` / `ME_SPACE` / `ME_SESSION_TOKEN` / `ME_NO_KEYCHAIN`. - **Header constants** (`CLIENT_VERSION_HEADER`, `SPACE_HEADER`) live in `@memory.build/protocol/headers`. - **MCP compatibility**: all tool parameters are required (nullable for optional). Uses `z.record(z.string(), z.any())` for meta instead of `z.record(z.unknown())` (which crashes the MCP SDK). -- **batchCreate conflict semantics**: a duplicate explicit id is skipped, or — with `replaceIfMetaDiffers: ""` — replaced in place when the stored row's value for that key differs (the session importers pass `importer_version` so version bumps re-render server-side). Result is `{ids, updatedIds}` (inserted / replaced); ids in neither were skipped. Single `memory.create` on a duplicate id errors with CONFLICT. +- **create / batchCreate conflict semantics**: the idempotency key is a named row's `(tree, name)` slot (name wins over id), else the explicit id. `onConflict` governs a clash on that key: `error` (default) raises CONFLICT, `replace` overwrites in place when content/meta/temporal differ (a no-op when identical; the id-path also compares tree/name since it can move/rename), `ignore` skips. create/batchCreate report a per-row `{id, status}` (`status` = `inserted` | `updated` | `skipped`); batchCreate returns `{results: [...]}`, one entry per input in input order (so a skip is visible and ids map back to inputs). The session/git importers pass `onConflict: 'replace'` and stamp `meta.importer_version` (deterministic meta, no per-run timestamp), so an unchanged re-import is a no-op while a version bump makes meta differ and re-renders. The file importers (`me import memories`, `me_memory_import`, `me pack install`) pass `onConflict: 'ignore'` so a re-import/re-install is a no-op. (There is no `replaceIfMetaDiffers` — content-aware `replace` subsumed it.) ## Database driver: postgres.js diff --git a/docs/access-control.md b/docs/access-control.md index 3426e1d0..6d90ad56 100644 --- a/docs/access-control.md +++ b/docs/access-control.md @@ -42,25 +42,25 @@ A grant attaches an access **level** to a principal at a **tree path**. Levels a | 2 | **write** | Read + create, update, move, and delete memories. | | 3 | **owner** | Write + manage access (grant/revoke) within the subtree. | -Grants are **hierarchical**: a grant at `share.work` also covers `share.work.projects`, `share.work.projects.api`, and so on. An `owner` grant at a path delegates access-management for that whole subtree; `owner@root` (the empty path) owns the entire space. +Grants are **hierarchical**: a grant at `/share/work` also covers `/share/work/projects`, `/share/work/projects/api`, and so on. An `owner` grant at a path delegates access-management for that whole subtree; ownership at the root `/` (the empty path) owns the entire space. ```bash # Grant read access to a subtree -me access grant alice@example.com share.work r +me access grant alice@example.com /share/work r # Grant write access -me access grant bob@example.com share.work.backend w +me access grant bob@example.com /share/work/backend w # Grant ownership of a subtree (lets the grantee manage access below it) -me access grant team-leads share.work o +me access grant team-leads /share/work o # List grants in the active space (optionally scope to one principal or path) me access list me access list alice@example.com -me access list --path share.work +me access list --path /share/work # Remove a grant -me access rm-grant bob@example.com share.work.backend +me access rm-grant bob@example.com /share/work/backend ``` The level argument accepts `r` (read), `w` (write), or `o` (owner). @@ -69,15 +69,15 @@ The level argument accepts `r` (read), `w` (write), or `o` (owner). Every space has two conventional roots: -- **`share`** — the shared root. Memories everyone in the space should see go here. This is where the file importers default a tree-less record, and where `me memory create` / `me_memory_create` callers usually place memories. -- **`home.`** — a per-member private root. The input shortcut **`~`** expands to your own home, so `~.notes` means `home..notes` and displays back as `~.notes`. An **agent**'s home nests under its owner's — `home..` — so its owner can see what the agent stores under `~` (an agent's access is capped at its owner's regardless). +- **`/share`** — the shared root. Memories everyone in the space should see go here. This is where the file importers default a tree-less record, and where `me memory create` / `me_memory_create` callers usually place memories. +- **`/home/`** — a per-member private root. The input shortcut **`~`** expands to your own home, so `~/notes` means `/home//notes` and displays back as `~/notes`. An **agent**'s home nests under its owner's — `/home//` — so its owner can see what the agent stores under `~` (an agent's access is capped at its owner's regardless). -`.` is the canonical path separator (`/` is also accepted on input and normalized). Labels must match `[A-Za-z0-9_-]`. +`/` is the canonical path separator (the leading slash is optional on input). Labels must match `[A-Za-z0-9_-]`. ### Default grants - A space **creator** gets `admin` + `owner@home` + `owner@share` — **not** `owner@root`. So the creator sees `share` and their own `~`, but not other members' homes. Because they're an admin, they can self-grant `owner@root` if they need the whole space. -- A **user** who joins a space is granted `owner@home` (their own private root). An **agent** who joins is likewise granted owner over its home — nested under its owner's (`home..`) — so it's usable immediately and the grant isn't clamped away. An admin then grants whatever shared access is appropriate (often via `me space invite --share`). +- A **user** who joins a space is granted `owner@home` (their own private root). An **agent** who joins is likewise granted owner over its home — nested under its owner's (`/home//`) — so it's usable immediately and the grant isn't clamped away. An admin then grants whatever shared access is appropriate (often via `me space invite --share`). ## How it's enforced @@ -106,11 +106,11 @@ me group add backend alice@example.com me group add backend bob@example.com # Grant the group write access to a subtree (members inherit it) -me access grant backend share.work.backend w +me access grant backend /share/work/backend w # Add one of your agents to the space and give it write access to share me agent add ci-bot -me access grant ci-bot share w +me access grant ci-bot /share w ``` See [`me access`](cli/me-access.md), [`me space`](cli/me-space.md), [`me group`](cli/me-group.md), and [`me agent`](cli/me-agent.md) for full command references. diff --git a/docs/cli/agent-session-imports.md b/docs/cli/agent-session-imports.md index bb466461..0de79db0 100644 --- a/docs/cli/agent-session-imports.md +++ b/docs/cli/agent-session-imports.md @@ -6,7 +6,7 @@ Shared reference for the agent-session import subcommands: - `me import codex` ([`me codex import`](me-codex.md#me-codex-import) is its alias) - `me import opencode` ([`me opencode import`](me-opencode.md#me-opencode-import) is its alias) -Each source-native message becomes one memory. Re-running the same command only inserts newly-seen messages (deterministic UUIDs make re-imports idempotent). +Each source-native message becomes one memory, named `msg_` under a per-session tree node. Re-running the same command only inserts newly-seen messages — the `(tree, name)` slot makes re-imports idempotent. ## Shared options @@ -18,7 +18,7 @@ All three subcommands accept the same flags (with one extra flag on the Claude i | `--project ` | Only import sessions whose cwd equals or is below this path. | | `--since ` | Only import sessions started at or after this ISO 8601 timestamp. | | `--until ` | Only import sessions started at or before this ISO 8601 timestamp. | -| `--tree-root ` | Tree root under which `.` nodes are placed. Default: `share.projects`. Accepts ltree labels (`[A-Za-z0-9_-]`) separated by `.` or `/`, with an optional leading `~` for your home (e.g. `~.projects`). | +| `--tree-root ` | Tree root under which `/` nodes are placed. Default: `/share/projects`. Accepts ltree labels (`[A-Za-z0-9_-]`) separated by `/`, with an optional leading `~` for your home (e.g. `~/projects`). | | `--sessions-node-name ` | Per-project node name for imported agent sessions. Default: `agent_sessions`. Must match `[a-z0-9_]+`. | | `--full-transcript` | Also store reasoning, tool calls, and tool results as their own message memories (default: user + assistant text only). | | `--include-temp-cwd` | Include sessions whose cwd is a system temp directory (`/tmp`, `/private/var/folders/...`). Off by default. | @@ -34,19 +34,19 @@ All three subcommands accept the same flags (with one extra flag on the Claude i ## Tree layout -Each imported message is stored under: +Each session is its own tree node, and each message is a named leaf under it: ``` -.. +////msg_ ``` -For example, a Claude message from a session run in `/Users/me/dev/memory-engine` ends up under `share.projects.memory_engine.agent_sessions` by default. Every message from every session in a project shares that same tree node; individual sessions are distinguished by `meta.source_session_id`. +For example, a Claude message from a session run in `/Users/me/dev/memory-engine` ends up at `/share/projects/memory_engine/agent_sessions//msg_` by default. Each session is browsable as a folder, and an individual message is addressable by its path (`me get /share/projects/memory_engine/agent_sessions//msg_`). The session id is normalized to an ltree label for the node; the raw id is also kept in `meta.source_session_id`. Project slugs come from the git repo root directory name when the cwd is inside a repo, or from `basename(cwd)` otherwise. Slug collisions (two different cwds that normalize to the same label) are resolved automatically by appending a 4-char hash suffix -- the first cwd seen gets the plain slug, subsequent ones get `slug_`. The full cwd is always preserved in `meta.source_cwd`. ## Idempotency -Each imported message gets a deterministic UUIDv7 derived from `(tool, session_id, message_id, timestamp)`. Re-imports reconcile **server-side**: every planned message is submitted through the engine's conditional upsert, which inserts new ids, rewrites in place any row whose stored `meta.importer_version` differs from the current importer's (so a version bump re-renders previously-imported messages in the same batched pass), and skips rows that are already current. There is no per-session lookup and no session-size limit — a session with tens of thousands of imported messages reconciles exactly like a small one. +Idempotency is keyed on `(tree, name)` — the per-session node plus the `msg_` leaf. (The id is a timestamp-prefixed UUIDv7 with a random tail, so messages still sort chronologically by id; the same message gets a fresh id each run, but the `(tree, name)` slot keeps it on the existing row.) Re-imports reconcile **server-side**: every planned message is submitted with `onConflict: 'replace'`, which inserts new slots and rewrites an existing one only when content/meta/temporal differ. Since `meta.importer_version` is part of meta, an importer-version bump makes meta differ and re-renders previously-imported messages in the same batched pass, while an unchanged re-import is a no-op. There is no per-session lookup and no session-size limit — a session with tens of thousands of imported messages reconciles exactly like a small one. Source files are append-only for all three tools, so re-importing an in-progress session simply inserts its newly-appended messages on the next run. The live-capture hook additionally narrows each submission to the messages after the newest already-imported one (a single `limit 1` search) — purely a bandwidth optimization; correctness never depends on it. @@ -85,7 +85,6 @@ Each imported memory carries: | `source_tool_name` | Tool name for `tool_call` / `tool_result` messages. | | `source_file` | Absolute path of the session file on disk. | | `content_mode` | `"default"` or `"full_transcript"`. | -| `imported_at` | ISO 8601 timestamp of this import run. | | `importer_version` | Version tag of the importer schema. | Temporal is a point-in-time at the message's timestamp. diff --git a/docs/cli/me-access.md b/docs/cli/me-access.md index e8794cc8..c066aa69 100644 --- a/docs/cli/me-access.md +++ b/docs/cli/me-access.md @@ -2,7 +2,7 @@ Manage tree-access grants in the active space. -A grant attaches an access **level** to a principal (user, agent, or group) at a **tree path**. Levels are additive and hierarchical — a grant at `share.work` also covers everything below it: +A grant attaches an access **level** to a principal (user, agent, or group) at a **tree path**. Levels are additive and hierarchical — a grant at `/share/work` also covers everything below it: | Level | Flag | Capabilities | |-------|------|--------------| @@ -37,8 +37,8 @@ me access grant | `level` | yes | Access level: `r` (read), `w` (write), or `o` (owner). | ```bash -me access grant alice@example.com share.work r -me access grant backend share.work.api w +me access grant alice@example.com /share/work r +me access grant backend /share/work/api w me access grant lead@example.com "" o # owner@root — whole space ``` diff --git a/docs/cli/me-agent.md b/docs/cli/me-agent.md index f79ff2ef..752305a5 100644 --- a/docs/cli/me-agent.md +++ b/docs/cli/me-agent.md @@ -72,7 +72,7 @@ me agent delete ## me agent add -Add one of your agents to the active space's roster. It joins with owner over its own home — nested under yours (`home..`), so you can see what it stores under `~`. Grant it shared access (e.g. on `share`) with [`me access`](me-access.md). +Add one of your agents to the active space's roster. It joins with owner over its own home — nested under yours (`/home//`), so you can see what it stores under `~`. Grant it shared access (e.g. on `share`) with [`me access`](me-access.md). ``` me agent add diff --git a/docs/cli/me-claude.md b/docs/cli/me-claude.md index 6a64e6aa..5b3667ae 100644 --- a/docs/cli/me-claude.md +++ b/docs/cli/me-claude.md @@ -69,7 +69,7 @@ Setup is a list of independent steps, grouped by source: Claude Code sessions an | Claude Code sessions | Install the Claude Code plugin (ongoing capture) | `--skip-plugin-install` | Runs the same install as [`me claude install`](#me-claude-install) (full plugin, `user` scope, login-session auth) — its hooks capture each new session as you work, plus slash commands and MCP tools. Hidden when the `claude` binary isn't on PATH; when `claude plugin list` already shows the plugin, the picker offers it unchecked as "Reinstall … (already installed)" (non-interactive runs report it as a ✓ line and skip it). | | Git history | Import existing commit history (one-time backfill) | `--skip-git-import` | Imports the repo's full commit history — the same import as [`me import git`](me-import.md#me-import-git). Skipped automatically when the current directory is not inside a git repo. | | Git history | Install a git post-commit hook (ongoing capture) | `--skip-git-hook` | Installs the managed hook from [`me import git-hook`](me-import.md#me-import-git-hook) so each new commit triggers a background incremental import. Hidden outside a git repo or when a `core.hooksPath` manager owns the hook path; when the hook is already installed, the picker offers it unchecked as "Reinstall … (already installed)" (non-interactive runs report it as a ✓ line and skip it). | -| Project config | Add a memory pointer to CLAUDE.md | `--skip-claude-md` | Upserts a managed block into the project's CLAUDE.md naming the project tree (`share.projects.`), its `agent_sessions` and `git_history` nodes, and how to search them. Idempotent — re-runs replace the block in place. When the block is already present and up to date, the picker offers it unchecked as "Rewrite … (already present)" (non-interactive runs report it as a ✓ line and skip it); a stale block (e.g. the active space changed) keeps the step pre-checked so the re-run refreshes it. | +| Project config | Add a memory pointer to CLAUDE.md | `--skip-claude-md` | Upserts a managed block into the project's CLAUDE.md naming the project tree (`/share/projects/`), its `agent_sessions` and `git_history` nodes, and how to search them. Idempotent — re-runs replace the block in place. When the block is already present and up to date, the picker offers it unchecked as "Rewrite … (already present)" (non-interactive runs report it as a ✓ line and skip it); a stale block (e.g. the active space changed) keeps the step pre-checked so the re-run refreshes it. | Re-running `init` is safe: both imports are incremental/idempotent and the CLAUDE.md block is replaced, not duplicated. After the steps run, `init` closes with a recap of what is now covered — historical data imported, hooks keeping it updated going forward. diff --git a/docs/cli/me-import.md b/docs/cli/me-import.md index f92d60bc..aa185fed 100644 --- a/docs/cli/me-import.md +++ b/docs/cli/me-import.md @@ -62,19 +62,19 @@ me import git [repo] [options] | `--full` | Walk the full history (skip the incremental high-water lookup). | | `--no-merges` | Drop all merge commits. | | `--no-file-list` | Omit the changed-file list from commit memories. | -| `--tree-root ` | Tree root under which `.git_history` is placed. Default: `share.projects`. | +| `--tree-root ` | Tree root under which `/git_history` is placed. Default: `/share/projects`. | | `--dry-run` | Parse and report what would be imported without writing. | | `-v, --verbose` | Per-commit progress output. | ### Tree layout -Commits are stored under: +Each commit is a named leaf (the commit ``) under the project's `git_history` node: ``` -..git_history +//git_history/ ``` -The project slug is derived exactly as for [agent session imports](agent-session-imports.md#tree-layout) (git remote repo name, else repo root directory name), so a project's commit history sits next to its `agent_sessions` node — e.g. `share.projects.memory_engine.git_history`. +The project slug is derived exactly as for [agent session imports](agent-session-imports.md#tree-layout) (git remote repo name, else repo root directory name), so a project's commit history sits next to its `agent_sessions` node — e.g. a commit lands at `/share/projects/memory_engine/git_history/` and is addressable by that path. ### Content shape @@ -84,9 +84,9 @@ Merge commits with no message body (`Merge branch 'x'` boilerplate) are skipped ### Idempotency and incremental re-runs -Each commit gets a deterministic UUIDv7 keyed by `(tree, sha)` with the commit date as its timestamp half. Re-imports are server-side no-ops: an already-imported commit is skipped, never duplicated. +Idempotency is keyed on `(tree, sha)` — each commit is named by its sha. The id is a timestamp-prefixed UUIDv7 (commit date in the prefix, random tail), so commits sort by date on the id. Re-imports are server-side no-ops: an already-imported commit is skipped, never duplicated. -Re-runs are also incremental: the newest already-imported commit is looked up server-side, and when it is an ancestor of the target rev only `..` is walked. After a force-push (or when importing a different branch) the walk falls back to the full log — still safe, because the deterministic ids dedupe the overlap. Explicit bounds (`--since`, `--until`, `--max-count`, `--full`) always walk exactly what they say. +Re-runs are also incremental: the newest already-imported commit is looked up server-side, and when it is an ancestor of the target rev only `..` is walked. After a force-push (or when importing a different branch) the walk falls back to the full log — still safe, because the `(tree, sha)` key dedupes the overlap. Explicit bounds (`--since`, `--until`, `--max-count`, `--full`) always walk exactly what they say. ### Metadata @@ -100,7 +100,6 @@ Re-runs are also incremental: the newest already-imported commit is looked up se | `author_date` / `commit_date` | ISO 8601 author and committer dates. | | `files_changed` / `insertions` / `deletions` | Change stats (binary files excluded from line counts). | | `is_merge` | `true` on merge commits (absent otherwise). | -| `imported_at` | ISO 8601 timestamp of this import run. | | `importer_version` | Version tag of the importer schema. | Temporal is a point-in-time at the commit date. diff --git a/docs/cli/me-mcp.md b/docs/cli/me-mcp.md index 87531891..47853c0e 100644 --- a/docs/cli/me-mcp.md +++ b/docs/cli/me-mcp.md @@ -53,4 +53,4 @@ claude plugin marketplace add timescale/memory-engine claude plugin install memory-engine@memory-engine [--scope user|project|local] ``` -Then start Claude Code, run `/plugin`, select `memory-engine`, and configure the options (all optional except `server`): leave `api_key` blank to use your `me login` session, leave `space` blank to use your active space, and `tree_root` defaults to `share.projects`. +Then start Claude Code, run `/plugin`, select `memory-engine`, and configure the options (all optional except `server`): leave `api_key` blank to use your `me login` session, leave `space` blank to use your active space, and `tree_root` defaults to `/share/projects`. diff --git a/docs/cli/me-memory.md b/docs/cli/me-memory.md index b9b6d06e..359b5199 100644 --- a/docs/cli/me-memory.md +++ b/docs/cli/me-memory.md @@ -2,15 +2,16 @@ Manage memories. -Memories are the core data type in Memory Engine. Each memory has content, optional metadata, an optional tree path for hierarchical organization, and an optional temporal range. +Memories are the core data type in Memory Engine. Each memory has content, an optional tree path for hierarchical organization, an optional filename-like `name` (unique within the tree), optional metadata, and an optional temporal range. ## Commands - [me memory create](#me-memory-create) -- create a memory -- [me memory get](#me-memory-get) -- get a memory by ID +- [me memory get](#me-memory-get) -- get a memory by ID or path - [me memory search](#me-memory-search) -- search memories - [me memory update](#me-memory-update) -- update a memory -- [me memory delete](#me-memory-delete) -- delete a memory or tree +- [me memory delete](#me-memory-delete) -- delete a single memory +- [me memory deltree](#me-memory-deltree) -- delete a subtree - [me memory edit](#me-memory-edit) -- open a memory in your editor - [me memory count](#me-memory-count) -- count memories matching a tree filter - [me memory tree](#me-memory-tree) -- show tree structure @@ -36,30 +37,38 @@ me memory create [content] [options] | Option | Description | |--------|-------------| | `--content ` | Memory content (alternative to positional argument). | -| `--tree ` | **Required.** Tree path where the memory is stored (e.g., `share.work.projects`). Use `share` for memories the rest of the space should see, or `~` (your private home, e.g. `~.notes`) for memories that must stay private to you. | +| `--tree ` | **Required.** Tree path where the memory is stored (e.g., `/share/work/projects`). Use `/share` for memories the rest of the space should see, or `~` (your private home, e.g. `~/notes`) for memories that must stay private to you. | +| `--name ` | Optional filename-like leaf name, unique within the tree (e.g. `jwt-rotation`). Lets you later address the memory by path (`/share/auth/jwt-rotation`) and re-create idempotently. | | `--meta ` | Metadata as a JSON string. | | `--temporal ` | Temporal range as `start[,end]` (ISO 8601). | +| `--replace` | On a conflict (a `--name` already taken in that tree), replace the existing memory in place when content/meta/temporal differ -- a no-op when identical. | +| `--ignore` | On a conflict, skip silently and leave the existing memory untouched. | -Content can come from the positional argument, the `--content` flag, or piped via stdin. A `--tree` path is required. +Content can come from the positional argument, the `--content` flag, or piped via stdin. A `--tree` path is required. Without `--replace`/`--ignore`, creating a second memory with a `--name` already used in that tree errors with `CONFLICT`. --- ## me memory get -Get a memory by ID. In a TTY, renders the content as ANSI-formatted markdown with dimmed YAML frontmatter. When piped or redirected, outputs raw Markdown with YAML frontmatter (suitable for `> file.md`). +Get a memory by ID or by its `tree/name` path. In a TTY, renders the content as ANSI-formatted markdown with dimmed YAML frontmatter. When piped or redirected, outputs raw Markdown with YAML frontmatter (suitable for `> file.md`). ``` -me memory get [options] +me memory get [options] ``` | Argument | Required | Description | |----------|----------|-------------| -| `id` | yes | Memory ID (UUIDv7). | +| `id-or-path` | yes | A memory ID (UUIDv7), or a named memory's `tree/name` path (e.g. `/share/auth/jwt-rotation`, `~/notes/todo`). A UUID is fetched by id; anything else is resolved by path (split at the final `/`). | | Option | Description | |--------|-------------| | `--raw` | Output raw Markdown with YAML frontmatter (no ANSI), even in a TTY. | +```bash +me memory get 0194a000-0001-7000-8000-000000000001 # by id +me memory get /share/auth/jwt-rotation # by path +``` + --- ## me memory search @@ -103,7 +112,7 @@ me memory search "how does authentication work" me memory search --fulltext "pgvector ltree" # Hybrid with tree filter -me memory search --semantic "embedding performance" --fulltext "nomic" --tree "me.design.*" +me memory search --semantic "embedding performance" --fulltext "nomic" --tree "/me/design/*" # Browse by metadata me memory search --meta '{"type": "decision"}' --limit 20 @@ -127,31 +136,52 @@ me memory update [options] |--------|-------------| | `--content ` | New content (use `-` for stdin). | | `--tree ` | New tree path. | +| `--name ` | Set or rename the memory's name. Pass an empty string (`--name ""`) to clear it. | | `--meta ` | New metadata as JSON (replaces existing). | | `--temporal ` | New temporal range as `start[,end]`. | -At least one update option is required. Metadata is fully replaced, not merged. +At least one update option is required. Metadata is fully replaced, not merged. Update is id-addressed; you can pass a `tree/name` path as the `` argument and the CLI resolves it to an id first. --- ## me memory delete -Delete a memory by ID, or all memories under a tree path. +Delete a **single** memory, by ID or by its `tree/name` path. To delete a whole subtree, use [`me memory deltree`](#me-memory-deltree). + +``` +me memory delete +``` + +| Argument | Required | Description | +|----------|----------|-------------| +| `id-or-path` | yes | A memory ID (UUIDv7), or a memory's `tree/name` path (e.g. `/share/auth/jwt-rotation`). | + +A UUIDv7 deletes that one memory by id; anything else is a `tree/name` path (split at the final `/`) that deletes at most that one named memory. It never deletes a subtree — a path that names no existing memory reports "not found" rather than removing everything beneath it. + +Alias: `me memory rm`. + +--- + +## me memory deltree + +Delete **every** memory at or under a tree path (a subtree). ``` -me memory delete [options] +me memory deltree [options] ``` | Argument | Required | Description | |----------|----------|-------------| -| `id-or-tree` | yes | Memory ID (UUIDv7) or tree path. | +| `tree` | yes | A tree path; all memories at or under it are deleted (e.g. `/share/old-project`). | | Option | Description | |--------|-------------| -| `--dry-run` | Preview what would be deleted (tree mode only). | -| `-y, --yes` | Skip the confirmation prompt (tree mode only). | +| `--dry-run` | Preview the count without deleting anything. | +| `-y, --yes` | Skip the confirmation prompt. | + +Always previews the count first, so `--dry-run` can never delete. Without `--yes`, an interactive run shows the count and asks to confirm before deleting. -If the argument is a UUIDv7, deletes a single memory. If it is a tree path, deletes all memories under that path after showing a count and confirming. +Alias: `me memory rmtree`. --- @@ -292,9 +322,9 @@ Supports Markdown (with YAML frontmatter), YAML, JSON, and NDJSON. Format is aut ### Skipped memories -Memories with an explicit `id` that already exists in the space are silently skipped server-side (a conflict skip in `create_memory`) rather than failing the whole batch. The command surfaces these as `skipped` so re-imports of unchanged data and id collisions with unrelated memories are observable. Memories without an `id` get a server-generated UUIDv7 and never collide. +Import submits with `onConflict: 'ignore'`, so a record whose idempotency key already exists -- a named record's `(tree, name)` slot (name takes precedence), else its explicit `id` -- is silently skipped rather than failing the whole batch. The command surfaces these as `skipped` so re-imports of unchanged data and id collisions with unrelated memories are observable. Records without an `id` and without a `name` get a server-generated UUIDv7 and never collide. -JSON output adds `skipped` (count) and `skippedIds` (array of conflicting ids). Text output appends `(K skipped — id already exists)` to the summary, or prints `Imported 0 memories (N already exist, no changes)` when everything was a re-import. Run with `--verbose` to see each skipped id inline. +JSON output adds `skipped` (count) and `skippedIds` (array of the skipped rows' stored ids). Text output appends `(K skipped — already exist)` to the summary, or prints `Imported 0 memories (N already exist, no changes)` when everything was a re-import. Run with `--verbose` to see each skipped id inline. The server reports a per-row status, so a named, id-less record skipped on its `(tree, name)` slot is counted too. Skipped memories do not contribute to the exit code; only parse and server errors do. @@ -304,7 +334,7 @@ Skipped memories do not contribute to the exit code; only parse and server error Large imports are sliced into multiple `batchCreate` requests under the hood to fit under the server's request-body limit. Each chunk is sent sequentially. If a chunk fails (network error, server error), siblings are not affected -- the successful chunks still land. The failed chunk's items are reported as `failed`, and the chunk-level error message appears in the `errors` array (sourced as `chunk N (K items)`). -This means partial failures are now possible: `imported > 0` and `failed > 0` can both be true in the same run. Re-running the import with the same input will pick up where the previous run left off (already-inserted ids are skipped via `ON CONFLICT DO NOTHING`, missing ids are inserted). +This means partial failures are now possible: `imported > 0` and `failed > 0` can both be true in the same run. Re-running the import with the same input will pick up where the previous run left off (already-present rows are skipped via `onConflict: 'ignore'`, missing ones are inserted). --- @@ -330,4 +360,4 @@ me memory export [file] [options] | `--temporal-overlaps ` | Memory must overlap this range. | | `--temporal-within ` | Memory must be within this range. | -For `md` format with a directory output, each memory is written as an individual `.md` file with YAML frontmatter. Exported content is compatible with `me memory import`. See [File Formats](../formats.md) for full schema documentation. +For `md` format with a directory output, the directory mirrors the tree: each memory is written to `//.md`. A named memory uses its name as the filename; an unnamed one falls back to `{id}.md`. Frontmatter includes `name` when set. Exported content is compatible with `me memory import`. See [File Formats](../formats.md) for full schema documentation. diff --git a/docs/cli/me-pack.md b/docs/cli/me-pack.md index 3b6fe655..c10a307b 100644 --- a/docs/cli/me-pack.md +++ b/docs/cli/me-pack.md @@ -53,7 +53,7 @@ The install process: 4. Deletes stale memories from previous versions (with confirmation). 5. Creates all memories from the pack with `pack.*` tree prefixes and pack metadata. -Inserts use server-side `ON CONFLICT DO NOTHING`, so existing rows with the same id are left untouched. The command classifies and reports any skips: +Inserts submit with `onConflict: 'ignore'`, so existing rows with the same id are left untouched. The command classifies and reports any skips: - **Already present** -- id is already tagged with this pack name and version. A benign no-op (e.g. re-running install on an unchanged pack). - **Conflict** -- id is held by something else (a different pack, a different version, or a non-pack memory). Surfaced as a warning and listed by id so a real collision isn't silently masked. Exit code remains `0`. @@ -84,7 +84,7 @@ JSON mode (`--format json`) returns: | `version` | Pack version. | | `installed` | Memories actually inserted on this run. | | `staleRemoved` | Previous-version memories deleted before insert. | -| `skipped` | Total memories skipped by `ON CONFLICT DO NOTHING`. | +| `skipped` | Total memories skipped via `onConflict: 'ignore'`. | | `skippedIdempotent` | Skipped because already present at this version. | | `skippedConflict` | Skipped because the id is held by something not from this pack/version. | | `skippedConflictIds` | Array of conflicting ids (only present when `skippedConflict > 0`). | diff --git a/docs/concepts.md b/docs/concepts.md index 078e2a0d..cc3446e5 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -5,10 +5,13 @@ A memory is a single piece of knowledge. Every memory has: - **content** (required) -- the text of the memory. Be specific and self-contained. -- **tree** -- a hierarchical dot-path for organizing and browsing (e.g., `work.projects.api`). +- **tree** -- a hierarchical path for organizing and browsing (e.g., `/share/auth` or `/work/projects/api`). +- **name** (optional) -- a human-chosen, filename-like slug, unique within its tree (e.g., `jwt-rotation`). Lets you address the memory as a path like `/share/auth/jwt-rotation` instead of by UUID, and serves as the upsert key for re-runs. Mutable. Matches `^[A-Za-z0-9][A-Za-z0-9._-]*$`, ≤128 chars -- dots are allowed, slashes are not. Distinct from, and in addition to, the memory's immutable UUID. - **meta** -- key-value metadata for filtering (e.g., `{"type": "decision", "confidence": "high"}`). - **temporal** -- a time association, either a point-in-time or a date range. +Every memory also has an immutable **id** (a UUIDv7) -- the stable identity that survives renames and moves. The server mints it; callers may supply one only to preserve identity across import/export. + Each space stores its memories in a single PostgreSQL table (the `me_` schema). There are no separate tables for different "types" of memory -- the type is a convention in `meta`, not a schema distinction. This keeps queries simple and the data model flexible. ### Best practices @@ -41,18 +44,20 @@ Each space stores its memories in a single PostgreSQL table (the `me_` sch ## Tree Paths -Tree paths organize memories into a browsable hierarchy using dot-separated labels: +Tree paths organize memories into a browsable hierarchy of labels: ``` -work -work.projects -work.projects.api -work.projects.api.auth -personal.reading -personal.reading.books +/work +/work/projects +/work/projects/api +/work/projects/api/auth +/personal/reading +/personal/reading/books ``` -Tree paths use PostgreSQL's `ltree` extension. Labels match `[A-Za-z0-9_-]` (letters, digits, underscores, and hyphens); use `.` as the separator (`/` is also accepted on input and normalized). +Tree paths use PostgreSQL's `ltree` extension under the hood. Each label matches `[A-Za-z0-9_-]` (letters, digits, underscores, and hyphens). The **canonical form uses `/` with a leading slash**: the root is `/`, an absolute path is `/share/auth`, and your home is `~/notes`. This is what the API and CLI display, and what you should write (the leading slash is optional when you type a path). + +> The tree-filter patterns below use lquery / ltxtquery operators (`*`, `{}`, `|`, `!`, `&`) layered on top of these paths. Keep paths **2-4 levels deep**. Deeper nesting rarely helps findability. @@ -60,8 +65,8 @@ Keep paths **2-4 levels deep**. Deeper nesting rarely helps findability. Every space has two conventional roots: -- **`share`** -- the shared root. Memories the rest of the space should see go here (`share.work.projects`, etc.). The file importers default a tree-less record to `share`. -- **`home.`** -- your private per-member root. The input shortcut **`~`** expands to your own home, so `~.notes` is stored as `home..notes` and displays back as `~.notes`. An **agent**'s home nests under its owner's (`home..`), so the agent's `~` is visible to its owner. +- **`/share`** -- the shared root. Memories the rest of the space should see go here (`/share/work/projects`, etc.). The file importers default a tree-less record to `share`. +- **`/home/`** -- your private per-member root. The input shortcut **`~`** expands to your own home, so `~/notes` resolves to `/home//notes` and displays back as `~/notes`. An **agent**'s home nests under its owner's (`/home//`), so the agent's `~` is visible to its owner. `me memory create` (and the `me_memory_create` MCP tool) **require** an explicit tree -- choose `share` for shared memories or `~` for private ones. See [Access Control](access-control.md) for how grants attach to these paths. @@ -69,24 +74,24 @@ Every space has two conventional roots: When filtering by tree (in search, export, or browse), the system auto-detects which syntax you're using: -**Exact match (ltree)** -- plain dot-separated path. Matches that node and all descendants. +**Exact match** -- a plain path. Matches that node and all descendants. | Pattern | Matches | |---------|---------| -| `work.projects` | `work.projects`, `work.projects.api`, `work.projects.api.auth`, etc. | +| `/work/projects` | `/work/projects`, `/work/projects/api`, `/work/projects/api/auth`, etc. | **Pattern matching (lquery)** -- triggered when the pattern contains `*`, `!`, `{`, `}`, `|`, `@`, or `%`. Uses wildcards and quantifiers. | Pattern | Meaning | |---------|---------| -| `work.projects.*` | All descendants of `work.projects` (any depth) | -| `work.*{1}` | Direct children of `work` only (exactly 1 level) | -| `work.*{2,4}` | Descendants 2-4 levels below `work` | -| `work.*{0,}` | `work` itself plus all descendants (equivalent to ltree `work`) | -| `*.api.*` | Any path containing the label `api` at any position | -| `*.!draft.*` | Any path that does NOT contain the label `draft` | -| `work|personal.*` | Paths starting with `work` or `personal`, then anything | -| `me.!archived.*{0,}` | Everything under `me` except the `me.archived` subtree | +| `/work/projects/*` | All descendants of `/work/projects` (any depth) | +| `/work/*{1}` | Direct children of `/work` only (exactly 1 level) | +| `/work/*{2,4}` | Descendants 2-4 levels below `/work` | +| `/work/*{0,}` | `/work` itself plus all descendants (equivalent to `/work`) | +| `*/api/*` | Any path containing the label `api` at any position | +| `*/!draft/*` | Any path that does NOT contain the label `draft` | +| `/work\|personal/*` | Paths starting with `work` or `personal`, then anything | +| `/me/!archived/*{0,}` | Everything under `/me` except the `/me/archived` subtree | **Label search (ltxtquery)** -- triggered when the pattern contains `&`. Boolean search over path labels. @@ -101,12 +106,33 @@ When filtering by tree (in search, export, or browse), the system auto-detects w Below the two reserved roots, tree paths are user-defined. There is no mandated hierarchy. Common patterns: ``` -share.work.projects. # shared per-project knowledge -share.design. # shared design decisions -pack. # installed memory packs (their own root) -~.notes. # private notes +/share/work/projects/ # shared per-project knowledge +/share/design/ # shared design decisions +/pack/ # installed memory packs (their own root) +~/notes/ # private notes ``` +## Addressing & Conflicts + +A memory can be addressed two ways: + +- **By id** -- the immutable UUID (`memory.get`, `memory.delete`; `me get `). Stable across renames and moves. +- **By path** -- a named memory's `tree/name`, split at the final `/` (`memory.getByPath`, `memory.deleteByPath`; `me get /share/auth/jwt-rotation`). The last segment is the name; the rest is the tree. A name may contain dots (`config.yaml`) but never a slash. + +The CLI's `me get` / `me delete` auto-detect: a UUID is treated as an id, anything else as a `tree/name` path -- and `me delete` only ever removes that single memory. `me update` is id-addressed (it resolves a path to an id first). Deleting a whole subtree is `me deltree ` / `memory.deleteTree`. + +### Conflict handling + +Create and batch-create take an `onConflict` policy, applied against the memory's **idempotency key** -- a named memory's `(tree, name)` slot (the name takes precedence over any explicit id), or the explicit id for an unnamed one: + +- **`error`** (default) -- a clash raises `CONFLICT`. +- **`replace`** -- overwrite in place, but only when something actually differs (content, meta, or temporal); an identical re-submit is a no-op. The id is preserved, and the embedding is recomputed only when content changes. +- **`ignore`** -- skip the conflicting row, leaving the existing one untouched. + +`onConflict` governs a clash on that idempotency key only. A *named* memory whose explicit id happens to collide with a **different** existing row still raises a primary-key violation regardless of `ignore`/`replace` -- so `ignore` means "ignore an idempotency-key conflict", not "ignore any conflict". (Importers mint random ids, so this doesn't arise in practice.) + +This makes re-runs idempotent. The transcript and git importers submit with `replace` and stamp `meta.importer_version`, so an unchanged re-import does nothing while a parser-version bump re-renders. The file importers (`me import memories`, the `me_memory_import` tool, `me pack install`) submit with `ignore`, so re-importing or re-installing is a no-op. (There is no separate "upsert" flag -- content-aware `replace` covers it.) + ## Metadata Metadata is a JSON object attached to each memory. Use it for structured attributes that you want to filter on: diff --git a/docs/formats.md b/docs/formats.md index 0f29ad5f..c0e8af16 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -4,14 +4,15 @@ Import and export use the same memory structure across all formats. This page is ## Memory fields -Every memory has one required field (`content`) and four optional fields: +Every memory has one required field (`content`) and several optional fields: | Field | Type | Required | Description | |-------|------|----------|-------------| -| `id` | `string` | no | UUIDv7. Enables idempotent imports -- re-importing the same ID won't create a duplicate. Must match `^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`. | +| `id` | `string` | no | UUIDv7. Preserves identity across import/export and makes re-import idempotent -- a record whose id (or `(tree, name)` slot) already exists is skipped. Must match `^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`. | | `content` | `string` | **yes** | The memory text. Must be non-empty. | +| `name` | `string` | no | Optional filename-like leaf slug, unique within its tree (e.g. `jwt-rotation`). Matches `^[A-Za-z0-9][A-Za-z0-9._-]*$`, ≤128 chars -- dots allowed, no slashes. Lets the memory be addressed as `tree/name`. | | `meta` | `object` | no | Arbitrary key-value metadata. Any valid JSON object. | -| `tree` | `string` | no | Hierarchical path using dot-separated labels (e.g. `share.work.projects.api`). Labels match `[A-Za-z0-9_-]`; `/` is also accepted as a separator and a leading `~` expands to your private home. When omitted, the file importers (`me import memories`, `me_memory_import`) default the record to the shared root `share`. | +| `tree` | `string` | no | Hierarchical path, `/`-separated with a leading slash (e.g. `/share/work/projects/api`); a leading `~` expands to your private home. The leading slash is optional on input. Each label matches `[A-Za-z0-9_-]`. When omitted, the file importers (`me import memories`, `me_memory_import`) default the record to the shared root `/share`. | | `temporal` | varies | no | Time range for the memory. Accepted shapes depend on format -- see below. | ### Temporal input shapes @@ -47,13 +48,14 @@ A JSON array of memory objects. This is the default export format. { "id": "0194a000-0001-7000-8000-000000000001", "content": "Project started with three engineers", - "tree": "work.projects.api", + "tree": "/work/projects/api", + "name": "kickoff", "meta": { "source": "import", "author": "jane" }, "temporal": { "start": "2024-01-15T00:00:00Z" } }, { "content": "Switched to PostgreSQL for the queue", - "tree": "work.projects.api", + "tree": "/work/projects/api", "meta": { "type": "decision" } } ] @@ -64,7 +66,7 @@ A single object (not wrapped in an array) is also accepted: ```json { "content": "Single memory import", - "tree": "notes" + "tree": "/notes" } ``` @@ -77,9 +79,9 @@ A single object (not wrapped in an array) is also accepted: Newline-delimited JSON -- one JSON object per line. Useful for streaming or large datasets. ``` -{"content": "First memory", "tree": "notes"} -{"content": "Second memory", "tree": "notes", "meta": {"priority": "high"}} -{"content": "Third memory", "tree": "notes"} +{"content": "First memory", "tree": "/notes"} +{"content": "Second memory", "tree": "/notes", "name": "second", "meta": {"priority": "high"}} +{"content": "Third memory", "tree": "/notes"} ``` NDJSON is auto-detected when the content contains multiple lines that each start with `{`. It is parsed using the JSON parser internally. @@ -97,7 +99,8 @@ A YAML array of memory objects. ```yaml - id: "0194a000-0001-7000-8000-000000000001" content: Project started with three engineers - tree: work.projects.api + tree: /work/projects/api + name: kickoff meta: source: import author: jane @@ -106,7 +109,7 @@ A YAML array of memory objects. end: "2024-12-31T23:59:59Z" - content: Switched to PostgreSQL for the queue - tree: work.projects.api + tree: /work/projects/api meta: type: decision ``` @@ -115,7 +118,7 @@ A single object (not wrapped in an array) is also accepted: ```yaml content: Single memory import -tree: notes +tree: /notes ``` **File extensions**: `.yaml`, `.yml` @@ -129,7 +132,8 @@ A Markdown file with optional YAML frontmatter. The frontmatter carries the meta ```markdown --- id: 0194a000-0001-7000-8000-000000000001 -tree: work.projects.api +tree: /work/projects/api +name: queue-backend meta: source: import type: decision @@ -155,10 +159,10 @@ No metadata, tree, or temporal information. ### Markdown export -When exporting to Markdown, each memory is written as an individual `{id}.md` file in a directory. Frontmatter includes `created_at` (CLI only) in addition to the standard fields. The `created_at` field is informational and is ignored on re-import. +When exporting to Markdown, the directory mirrors the tree: each memory is written to `//.md`. A named memory uses its name as the filename (`.../share/auth/jwt-rotation.md`); an unnamed one falls back to its `{id}.md`. Frontmatter includes `name` (when set) and `created_at` (CLI only) in addition to the standard fields. The `created_at` field is informational and is ignored on re-import. - **CLI**: requires a directory path when exporting multiple memories. Single-memory export to stdout is allowed. -- **MCP**: when `path` is provided, creates or uses it as a directory of `.md` files. When `path` is null (inline), only single-memory export is allowed -- multiple memories will return an error asking for a directory path. +- **MCP**: when `path` is provided, creates or uses it as a directory tree of `.md` files. When `path` is null (inline), only single-memory export is allowed -- multiple memories will return an error asking for a directory path. --- @@ -205,8 +209,8 @@ When importing from a file path, the file is read server-side and the 1 MB reque ## Round-trip compatibility -Exported files can be re-imported directly. The export output uses the same field names and structure as the import schema. +Exported files can be re-imported directly. The export output uses the same field names and structure as the import schema; `tree` is written in the canonical `/`-prefixed form, which re-imports cleanly (input is lenient). -The `id` field is preserved in exports, so re-importing an export is idempotent -- existing memories with the same ID are not duplicated. +The `id` and `name` fields are preserved in exports, so re-importing is idempotent: the file importers submit with `onConflict: 'ignore'`, and a record whose idempotency key -- a named record's `(tree, name)` slot (name takes precedence), else its `id` -- already exists is skipped rather than duplicated. Fields that appear in exports but are not part of the import schema (like `created_at` in Markdown frontmatter) are silently ignored on re-import. diff --git a/docs/getting-started.md b/docs/getting-started.md index 4f6ee62f..0f2a07d2 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -38,11 +38,12 @@ me version ```bash me memory create "PostgreSQL 18 supports native UUIDv7 generation." \ - --tree share.notes.postgres \ + --tree share/notes/postgres \ + --name uuidv7 \ --meta '{"topic": "database"}' ``` -A `--tree` is required. Put memories the rest of your space should see under `share.*`, and personal ones under `~.*` (your private home). See [Core Concepts](concepts.md#reserved-roots). +A `--tree` is required. Put memories the rest of your space should see under `share/*`, and personal ones under `~/*` (your private home). The optional `--name` gives the memory a filename-like slug (unique within its tree) so you can later address it by path -- `me get share/notes/postgres/uuidv7`. See [Core Concepts](concepts.md#reserved-roots). ## Search @@ -90,7 +91,7 @@ me claude install # full plugin me claude install --mcp-only # or just the MCP server ``` -This drives Claude Code's native plugin flow for you (`claude plugin marketplace add` + `claude plugin install`), passing your resolved server/space/api_key through `--config`. Afterwards, restart Claude Code (or run `/plugin`) to load the hooks and slash commands; you can re-run `/plugin` → `memory-engine` → Configure to adjust options. All are optional except `server`: leave `api_key` blank to use your `me login` session, leave `space` blank to use your active space, and `tree_root` defaults to `share.projects`. +This drives Claude Code's native plugin flow for you (`claude plugin marketplace add` + `claude plugin install`), passing your resolved server/space/api_key through `--config`. Afterwards, restart Claude Code (or run `/plugin`) to load the hooks and slash commands; you can re-run `/plugin` → `memory-engine` → Configure to adjust options. All are optional except `server`: leave `api_key` blank to use your `me login` session, leave `space` blank to use your active space, and `tree_root` defaults to `/share/projects`. After installation, your AI agent has access to memory tools -- create, search, get, update, delete, and more. diff --git a/docs/mcp-integration.md b/docs/mcp-integration.md index d875d05f..b2357366 100644 --- a/docs/mcp-integration.md +++ b/docs/mcp-integration.md @@ -153,8 +153,10 @@ Once connected, the agent has access to: | `me_memory_create` | Store a new memory | | `me_memory_search` | Search by meaning, keywords, or filters | | `me_memory_get` | Retrieve a memory by ID | +| `me_memory_get_by_path` | Retrieve a named memory by its `tree/name` path | | `me_memory_update` | Modify an existing memory | -| `me_memory_delete` | Delete a memory | +| `me_memory_delete` | Delete a memory by ID | +| `me_memory_delete_by_path` | Delete a named memory by its `tree/name` path | | `me_memory_delete_tree` | Bulk delete by tree prefix | | `me_memory_count` | Count memories matching a tree filter | | `me_memory_copy` | Copy memories between tree paths | @@ -185,9 +187,9 @@ This project uses Memory Engine for persistent knowledge. ## Memory Map -- `share.design.*` -- architecture decisions and design docs -- `share.research.*` -- research findings and comparisons -- `share.bugs.*` -- known issues and workarounds +- `/share/design/*` -- architecture decisions and design docs +- `/share/research/*` -- research findings and comparisons +- `/share/bugs/*` -- known issues and workarounds ## How to Search @@ -208,7 +210,7 @@ me_memory_search({semantic: "how does authentication work"}) me_memory_search({fulltext: "OAuth JWT"}) # Browse a section -me_memory_search({tree: "share.design.*"}) +me_memory_search({tree: "/share/design/*"}) ``` ## Troubleshooting diff --git a/docs/mcp/me_memory_create.md b/docs/mcp/me_memory_create.md index dab3e87f..cfbc56cc 100644 --- a/docs/mcp/me_memory_create.md +++ b/docs/mcp/me_memory_create.md @@ -6,11 +6,13 @@ Store a new memory. | Name | Type | Required | Description | |------|------|----------|-------------| -| `id` | `string \| null` | no | UUIDv7 for idempotent creates. Omit or pass `null` to auto-generate. | +| `id` | `string \| null` | no | UUIDv7 to preserve identity (import/export). Omit or pass `null` to auto-generate. | | `content` | `string` | yes | The content of the memory. Must be non-empty. | +| `name` | `string \| null` | no | Optional filename-like leaf slug, unique within the tree (e.g. `jwt-rotation`). Matches `^[A-Za-z0-9][A-Za-z0-9._-]*$`, ≤128 chars -- dots allowed, no slashes. Lets the memory be addressed as `/share/auth/jwt-rotation`. Omit or pass `null` for an unnamed memory. | | `meta` | `object \| null` | no | Key-value metadata pairs. Omit or pass `null` to skip. | -| `tree` | `string` | yes | Hierarchical path where the memory is stored, using dot-separated labels (e.g., `share.work.projects`). Choose deliberately: most memories should go under `share` so the rest of the space can see them; use `~` (your private home, e.g. `~.notes`) only for memories that must stay private to you. | +| `tree` | `string` | yes | Hierarchical path where the memory is stored (e.g., `/share/work/projects`). The canonical form is `/`-separated with a leading slash (the leading slash is optional on input). Choose deliberately: most memories should go under `/share` so the rest of the space can see them; use `~` (your private home, e.g. `~/notes`) only for memories that must stay private to you. | | `temporal` | `object \| null` | no | Time range for the memory. Omit or pass `null` to skip. | +| `on_conflict` | `string \| null` | no | What to do when the idempotency key (a named memory's `(tree, name)` slot, which takes precedence over any `id`; else the explicit `id`) already exists: `"error"` (default -- raise CONFLICT), `"replace"` (overwrite in place when content/meta/temporal differ; a no-op when identical), or `"ignore"` (skip and return the existing memory). | ### temporal @@ -28,7 +30,8 @@ The full memory object as created: "id": "0194a000-0001-7000-8000-000000000001", "content": "PostgreSQL 18 supports native UUID v7 generation.", "meta": { "topic": "database" }, - "tree": "notes.postgres", + "tree": "/notes/postgres", + "name": "uuidv7", "temporal": null, "hasEmbedding": false, "createdAt": "2025-04-15T12:00:00Z", @@ -42,7 +45,8 @@ The full memory object as created: | `id` | `string` | UUIDv7 identifier. | | `content` | `string` | The memory content. | | `meta` | `object` | Metadata key-value pairs (empty `{}` if none). | -| `tree` | `string` | Tree path (empty string if root). | +| `tree` | `string` | Tree path (canonical `/`-form; `/` if root). | +| `name` | `string \| null` | The leaf name, or `null` if unnamed. | | `temporal` | `object \| null` | Time range with `start` and `end`, or `null`. | | `hasEmbedding` | `boolean` | Whether a vector embedding has been computed yet. | | `createdAt` | `string` | ISO 8601 creation timestamp. | @@ -55,7 +59,8 @@ The full memory object as created: { "content": "Use ltree for hierarchical path queries in PostgreSQL.", "meta": { "source": "docs", "confidence": "high" }, - "tree": "research.postgres", + "tree": "/research/postgres", + "name": "ltree-paths", "temporal": { "start": "2025-04-15T00:00:00Z" } @@ -65,7 +70,7 @@ The full memory object as created: ## Notes - **One idea per memory.** Three decisions = three memories. Search first to avoid duplicates. -- Tree labels must be lowercase alphanumeric with underscores only -- no spaces, hyphens, or uppercase (e.g., `work.my_project`, not `work.my-project`). -- When `id` is provided, the call is idempotent -- creating the same ID twice returns the existing memory. +- Tree labels match `[A-Za-z0-9_-]` (letters, digits, `_`, `-`) and are `/`-separated. A memory's `name` is a separate leaf that additionally allows dots. +- By default a conflict on the idempotency key (a named memory's `(tree, name)` slot, which takes precedence over any `id`; else the explicit `id`) raises `CONFLICT`. Pass `on_conflict: "ignore"` to make the call idempotent (returns the existing memory) or `"replace"` to overwrite in place when something differs. This governs the idempotency-key conflict only — a named memory whose `id` collides with a *different* existing row still raises regardless of `on_conflict`. - `meta` is fully replaced, not merged. Store the complete metadata object each time. Values support any JSON type (strings, numbers, arrays, nested objects). - Embeddings are computed asynchronously after creation. `hasEmbedding` will be `false` initially. Fulltext search works immediately; semantic search is available after ~10-30 seconds. diff --git a/docs/mcp/me_memory_delete.md b/docs/mcp/me_memory_delete.md index 7473c375..0d6d90ea 100644 --- a/docs/mcp/me_memory_delete.md +++ b/docs/mcp/me_memory_delete.md @@ -2,7 +2,7 @@ Permanently remove a memory by ID. -This is irreversible. Consider archiving (via a meta update) or moving (via `me_memory_mv`) instead. +This is irreversible. Consider archiving (via a meta update) or moving (via `me_memory_mv`) instead. To delete a named memory by its `tree/name` path use [me_memory_delete_by_path](me_memory_delete_by_path.md); to remove a whole subtree use [me_memory_delete_tree](me_memory_delete_tree.md). ## Parameters diff --git a/docs/mcp/me_memory_delete_by_path.md b/docs/mcp/me_memory_delete_by_path.md new file mode 100644 index 00000000..94da0ca4 --- /dev/null +++ b/docs/mcp/me_memory_delete_by_path.md @@ -0,0 +1,24 @@ +# me_memory_delete_by_path + +Permanently remove a single named memory by its `tree/name` path +(e.g. `/share/auth/jwt-rotation`). + +Deletes only that one named memory. Use `me_memory_delete_tree` to remove a +whole subtree, or `me_memory_delete` to delete by UUID. + +## Parameters + +| Name | Type | Required | Description | +|------|------|----------|-------------| +| `path` | `string` | yes | The `tree/name` path, e.g. `/share/auth/jwt-rotation`. | + +## Returns + +```json +{ "deleted": true } +``` + +## Notes + +- Irreversible. +- Returns NOT_FOUND if no named memory matches the path. diff --git a/docs/mcp/me_memory_delete_tree.md b/docs/mcp/me_memory_delete_tree.md index 6c2c056c..b07ac43f 100644 --- a/docs/mcp/me_memory_delete_tree.md +++ b/docs/mcp/me_memory_delete_tree.md @@ -29,7 +29,7 @@ Use `dry_run: true` to preview how many memories would be deleted without actual ```json { - "tree": "pack.datasync", + "tree": "/pack/datasync", "dry_run": true } ``` @@ -38,14 +38,14 @@ Use `dry_run: true` to preview how many memories would be deleted without actual ```json { - "tree": "pack.datasync", + "tree": "/pack/datasync", "dry_run": false } ``` ## Notes -- This deletes memories at the exact path **and** all descendants. `tree: "work"` deletes `work`, `work.projects`, `work.projects.me`, etc. +- This deletes memories at the exact path **and** all descendants. `tree: "/work"` deletes `/work`, `/work/projects`, `/work/projects/me`, etc. - The deletion is **atomic** -- all memories are deleted together or none are. - Always preview with `dry_run: true` first to avoid surprises. - This operation is irreversible. Consider using [me_memory_mv](me_memory_mv.md) to archive instead of deleting. diff --git a/docs/mcp/me_memory_export.md b/docs/mcp/me_memory_export.md index cacc91f4..e8de3624 100644 --- a/docs/mcp/me_memory_export.md +++ b/docs/mcp/me_memory_export.md @@ -69,7 +69,7 @@ For `md` format with a directory path: ```json { - "tree": "me.design.*", + "tree": "/me/design/*", "format": "yaml", "path": "/Users/me/memories/design-export.yaml" } @@ -79,13 +79,13 @@ For `md` format with a directory path: ```json { - "tree": "me.design.*", + "tree": "/me/design/*", "format": "md", "path": "/Users/me/memories/design-export" } ``` -Each memory is written as `{id}.md` with YAML frontmatter. The directory is created if it does not exist. +The directory mirrors the tree: each memory is written to `/.md` with YAML frontmatter (including `name` when set). A named memory uses its name as the filename; an unnamed one falls back to `{id}.md`. The directory is created if it does not exist. ### Export inline for inspection @@ -101,7 +101,7 @@ Each memory is written as `{id}.md` with YAML frontmatter. The directory is crea - **Prefer `path` for large exports** to avoid returning large payloads through the conversation. Omit `path` only for small result sets or when you need to inspect the content. - The exported content is directly compatible with [me_memory_import](me_memory_import.md). Exported files and directories can be re-imported directly. -- **Markdown format**: use a directory path for multi-memory export. Each memory is written as `{id}.md`. Inline Markdown export (omitting `path`) is only supported for single-memory results. +- **Markdown format**: use a directory path for multi-memory export. The directory mirrors the tree -- each memory is written to `/.md`. Inline Markdown export (omitting `path`) is only supported for single-memory results. - Results are sorted in ascending order by creation time. -- The `tree` filter supports exact match, wildcards, negation, and label search. See [Tree filter syntax](../concepts.md#tree-filter-syntax) for the full reference. Use `me.!archived.*{0,}` to export everything under `me` except archived content. +- The `tree` filter supports exact match, wildcards, negation, and label search. See [Tree filter syntax](../concepts.md#tree-filter-syntax) for the full reference. Use `/me/!archived/*{0,}` to export everything under `/me` except archived content. - See [File Formats](../formats.md) for full schema documentation and format details. diff --git a/docs/mcp/me_memory_get.md b/docs/mcp/me_memory_get.md index 9fbdf3f3..96b24242 100644 --- a/docs/mcp/me_memory_get.md +++ b/docs/mcp/me_memory_get.md @@ -2,7 +2,7 @@ Retrieve a single memory by its ID. -Returns the full memory including content, tree, meta, temporal, and embedding status. Use after search to get full details, or before update to see current state. +Returns the full memory including content, tree, name, meta, temporal, and embedding status. Use after search to get full details, or before update to see current state. To fetch a named memory by its `tree/name` path instead, use [me_memory_get_by_path](me_memory_get_by_path.md). ## Parameters @@ -19,7 +19,8 @@ The full memory object: "id": "0194a000-0001-7000-8000-000000000001", "content": "PostgreSQL 18 supports native UUID v7 generation.", "meta": { "topic": "database" }, - "tree": "notes.postgres", + "tree": "/notes/postgres", + "name": "uuidv7", "temporal": null, "hasEmbedding": true, "createdAt": "2025-04-15T12:00:00Z", @@ -33,7 +34,8 @@ The full memory object: | `id` | `string` | UUIDv7 identifier. | | `content` | `string` | The memory content. | | `meta` | `object` | Metadata key-value pairs (empty `{}` if none). | -| `tree` | `string` | Tree path (empty string if root). | +| `tree` | `string` | Tree path (canonical `/`-form; `/` if root). | +| `name` | `string \| null` | The leaf name, or `null` if unnamed. | | `temporal` | `object \| null` | Time range with `start` and `end`, or `null`. | | `hasEmbedding` | `boolean` | Whether a vector embedding has been computed. | | `createdAt` | `string` | ISO 8601 creation timestamp. | diff --git a/docs/mcp/me_memory_get_by_path.md b/docs/mcp/me_memory_get_by_path.md new file mode 100644 index 00000000..5ca1e32e --- /dev/null +++ b/docs/mcp/me_memory_get_by_path.md @@ -0,0 +1,33 @@ +# me_memory_get_by_path + +Retrieve a single named memory by its `tree/name` path. + +The last path segment is the name; the rest is the tree. For example, +`/share/auth/jwt-rotation` is the memory named `jwt-rotation` under the tree +`/share/auth`, and `~/notes/todo` resolves under your home. Returns an error +(NOT_FOUND) if no such named memory exists. + +Use `me_memory_get` when you already have the UUID. + +## Parameters + +| Name | Type | Required | Description | +|------|------|----------|-------------| +| `path` | `string` | yes | The `tree/name` path, e.g. `/share/auth/jwt-rotation`. | + +## Returns + +The full memory object — same shape as `me_memory_get`, including its `name`. + +## Example + +```json +{ + "path": "/share/auth/jwt-rotation" +} +``` + +## Notes + +- The split is on the final `/`: a name may contain dots (`config.yaml`) but never a slash. +- Returns NOT_FOUND if no named memory matches, or the caller lacks read access. diff --git a/docs/mcp/me_memory_import.md b/docs/mcp/me_memory_import.md index a9bef343..3ff59429 100644 --- a/docs/mcp/me_memory_import.md +++ b/docs/mcp/me_memory_import.md @@ -18,7 +18,9 @@ One of `path` or `content` must be provided. JSON (array or single object), NDJSON, YAML (array or single object), and Markdown (YAML frontmatter + body, one memory per file). -Each memory object supports fields: `id`, `content` (required), `meta`, `tree`, `temporal`. Unlike `me_memory_create` (which requires an explicit `tree`), a record with no `tree` is imported into the shared root `share`. +Each memory object supports fields: `id`, `content` (required), `name`, `meta`, `tree`, `temporal`. Unlike `me_memory_create` (which requires an explicit `tree`), a record with no `tree` is imported into the shared root `share`. + +Import submits with `onConflict: 'ignore'`, so a record whose idempotency key -- its `id`, or its `(tree, name)` slot -- already exists is skipped rather than erroring. Re-importing the same data is a no-op. See [File Formats](../formats.md) for full schema documentation, examples, and format detection rules. @@ -49,7 +51,7 @@ See [File Formats](../formats.md) for full schema documentation, examples, and f | `skippedIds` | `string[]` | The explicit ids that were skipped because they already existed. Always present (may be empty). Inspect any of these with `me_memory_get` to see what's there. | | `errors` | `Array<{ chunkIndex, itemCount, ids, error }>` | One entry per failed chunk. Always present (may be empty). | -The tool is idempotent for memories with explicit ids: re-calling with the same arguments leaves the space in the same state, with all previously-imported ids appearing in `skippedIds` instead of `ids`. Memories submitted without an explicit `id` get a server-generated UUIDv7 and never collide. +The tool is idempotent: re-calling with the same arguments leaves the space in the same state, with previously-imported explicit ids appearing in `skippedIds` instead of `ids`. A record with neither an `id` nor a `name` gets a server-generated UUIDv7 and never collides; a named record is keyed on its `(tree, name)` slot (such skips aren't listed by id in `skippedIds`). ### Chunking and partial failures diff --git a/docs/mcp/me_memory_mv.md b/docs/mcp/me_memory_mv.md index 9e7e1239..0e4c3cc3 100644 --- a/docs/mcp/me_memory_mv.md +++ b/docs/mcp/me_memory_mv.md @@ -30,23 +30,23 @@ Works like `mv` in a filesystem -- all memories under the source prefix get thei ```json { - "source": "work.old_project", - "destination": "work.new_project", + "source": "/work/old_project", + "destination": "/work/new_project", "dry_run": false } ``` This moves: -- `work.old_project` -> `work.new_project` -- `work.old_project.api` -> `work.new_project.api` -- `work.old_project.api.auth` -> `work.new_project.api.auth` +- `/work/old_project` -> `/work/new_project` +- `/work/old_project/api` -> `/work/new_project/api` +- `/work/old_project/api/auth` -> `/work/new_project/api/auth` ### Preview a move ```json { - "source": "scratch", - "destination": "archive.scratch", + "source": "/scratch", + "destination": "/archive/scratch", "dry_run": true } ``` diff --git a/docs/mcp/me_memory_search.md b/docs/mcp/me_memory_search.md index a73f242e..0374726b 100644 --- a/docs/mcp/me_memory_search.md +++ b/docs/mcp/me_memory_search.md @@ -24,11 +24,11 @@ Supports three search modes: **semantic** (meaning-based), **fulltext** (keyword The system auto-detects the syntax from the pattern. Quick reference: -- Bare path (`work.projects`) -- matches that node and all descendants. -- Wildcard (`work.projects.*`) -- all descendants at any depth. -- Depth-limited (`work.*{2}`) -- descendants up to 2 levels deep. -- Negation (`*.!draft.*`) -- paths that do NOT contain `draft`. -- Pattern (`*.api.*`) -- any path containing `api`. +- Bare path (`/work/projects`) -- matches that node and all descendants. +- Wildcard (`/work/projects/*`) -- all descendants at any depth. +- Depth-limited (`/work/*{2}`) -- descendants up to 2 levels deep. +- Negation (`*/!draft/*`) -- paths that do NOT contain `draft`. +- Pattern (`*/api/*`) -- any path containing `api`. - Label search (`api & v2`) -- boolean search over path labels. See [Tree filter syntax](../concepts.md#tree-filter-syntax) for the full reference with examples. @@ -57,7 +57,7 @@ See [Tree filter syntax](../concepts.md#tree-filter-syntax) for the full referen "id": "0194a000-0001-7000-8000-000000000001", "content": "Use ltree for hierarchical path queries.", "meta": { "source": "docs" }, - "tree": "research.postgres", + "tree": "/research/postgres", "temporal": null, "hasEmbedding": true, "createdAt": "2025-04-15T12:00:00Z", @@ -104,7 +104,7 @@ See [Tree filter syntax](../concepts.md#tree-filter-syntax) for the full referen { "semantic": "embedding performance", "fulltext": "nomic ollama", - "tree": "me.design.*", + "tree": "/me/design/*", "limit": 5 } ``` @@ -114,7 +114,7 @@ See [Tree filter syntax](../concepts.md#tree-filter-syntax) for the full referen ```json { "meta": { "type": "decision" }, - "tree": "me.strategy.*", + "tree": "/me/strategy/*", "limit": 20, "order_by": "desc" } diff --git a/docs/mcp/me_memory_tree.md b/docs/mcp/me_memory_tree.md index 454ff869..1744a233 100644 --- a/docs/mcp/me_memory_tree.md +++ b/docs/mcp/me_memory_tree.md @@ -8,7 +8,7 @@ Shows how memories are organized and how many exist at each level. Use this to u | Name | Type | Required | Description | |------|------|----------|-------------| -| `tree` | `string \| null` | no | Root path to display from (e.g., `work.projects`). Omit or pass `null` for the full tree. | +| `tree` | `string \| null` | no | Root path to display from (e.g., `/work/projects`). Omit or pass `null` for the full tree. | | `levels` | `integer \| null` | no | Maximum depth to display. Omit or pass `null` for unlimited. | ## Returns @@ -16,11 +16,11 @@ Shows how memories are organized and how many exist at each level. Use this to u ```json { "nodes": [ - { "path": "me", "count": 45 }, - { "path": "me.design", "count": 30 }, - { "path": "me.design.auth", "count": 8 }, - { "path": "me.strategy", "count": 15 }, - { "path": "pack", "count": 120 } + { "path": "/me", "count": 45 }, + { "path": "/me/design", "count": 30 }, + { "path": "/me/design/auth", "count": 8 }, + { "path": "/me/strategy", "count": 15 }, + { "path": "/pack", "count": 120 } ] } ``` @@ -45,7 +45,7 @@ Shows how memories are organized and how many exist at each level. Use this to u ```json { - "tree": "me.design" + "tree": "/me/design" } ``` diff --git a/docs/mcp/me_memory_update.md b/docs/mcp/me_memory_update.md index e7815240..525af91f 100644 --- a/docs/mcp/me_memory_update.md +++ b/docs/mcp/me_memory_update.md @@ -10,6 +10,7 @@ Provide the ID and any fields to change. Omitted fields remain unchanged. |------|------|----------|-------------| | `id` | `string` | yes | The UUID of the memory to update. | | `content` | `string \| null` | no | New content. Omit or pass `null` to keep existing. | +| `name` | `string \| null` | no | Set or rename the leaf name. Pass an empty string (`""`) to clear it; omit or pass `null` to keep existing. Same slug rules as `me_memory_create`. | | `meta` | `object \| null` | no | New metadata. Omit or pass `null` to keep existing. | | `tree` | `string \| null` | no | New tree path. Omit or pass `null` to keep existing. | | `temporal` | `object \| null` | no | New time range. Omit or pass `null` to keep existing. | @@ -30,7 +31,8 @@ The full updated memory object: "id": "0194a000-0001-7000-8000-000000000001", "content": "Updated content here.", "meta": { "topic": "database", "reviewed": true }, - "tree": "notes.postgres", + "tree": "/notes/postgres", + "name": "uuidv7", "temporal": null, "hasEmbedding": true, "createdAt": "2025-04-15T12:00:00Z", diff --git a/docs/memory-packs.md b/docs/memory-packs.md index e7528288..2d99ac08 100644 --- a/docs/memory-packs.md +++ b/docs/memory-packs.md @@ -40,7 +40,7 @@ A pack is a YAML array of memory objects with a header comment: # ID prefix: 019b0300 - id: "019b0300-0001-7000-8000-000000000001" - tree: "pack.typescript.naming" + tree: "/pack/typescript/naming" meta: pack: name: "my-pack" @@ -51,7 +51,7 @@ A pack is a YAML array of memory objects with a header comment: PascalCase for types and classes. - id: "019b0300-0002-7000-8000-000000000002" - tree: "pack.typescript.error_handling" + tree: "/pack/typescript/error_handling" meta: pack: name: "my-pack" diff --git a/docs/typescript-client.md b/docs/typescript-client.md index 2044f783..6d226a93 100644 --- a/docs/typescript-client.md +++ b/docs/typescript-client.md @@ -28,10 +28,10 @@ const me = createMemoryClient({ space: "abc123def456", // the X-Me-Space slug }); -// Create a memory (tree is required — choose share.* or ~.* deliberately) +// Create a memory (tree is required — choose /share/* or ~/* deliberately) await me.memory.create({ content: "TypeScript was released in 2012", - tree: "share.knowledge.programming", + tree: "/share/knowledge/programming", }); // Search @@ -63,56 +63,69 @@ me.setSpace("otherslug1234"); ### create -`tree` is required. Use `share.*` for memories the rest of the space should see, or `~.*` for your private home. +`tree` is required. Use `/share/*` for memories the rest of the space should see, or `~/*` for your private home. An optional `name` (a filename-like slug, unique within the tree) lets you address the memory by path. `onConflict` governs a clash on the idempotency key (the `id` if given, else the `(tree, name)` slot): `"error"` (default), `"replace"` (content-aware), or `"ignore"`. ```typescript const memory = await me.memory.create({ content: "The fact to remember", - tree: "share.work.projects.acme", // required + tree: "/share/work/projects/acme", // required (leading slash optional on input) + name: "kickoff", // optional, unique within the tree meta: { source: "meeting-notes" }, // optional JSON metadata temporal: { // optional time range start: "2025-01-01T00:00:00Z", end: "2025-01-31T23:59:59Z", }, + onConflict: "replace", // optional; default "error" }); -// memory.id, memory.content, memory.tree, memory.meta, ... +// memory.id, memory.content, memory.tree, memory.name, memory.meta, ... ``` ### batchCreate -Create up to 1,000 memories in a single call. Each memory requires a `tree`. +Create up to 1,000 memories in a single call. Each memory requires a `tree`. A batch-level `onConflict` applies to every row (importers pass `"replace"` or `"ignore"`). ```typescript -const { ids } = await me.memory.batchCreate({ +const { results } = await me.memory.batchCreate({ memories: [ - { content: "First memory", tree: "share.notes" }, - { content: "Second memory", tree: "share.notes" }, + { content: "First memory", tree: "/share/notes" }, + { content: "Second memory", tree: "/share/notes", name: "second" }, ], + onConflict: "ignore", // optional; default "error" }); +// `results` has one { id, status } per input, in order — status is +// "inserted" | "updated" | "skipped". Filter by status for counts/ids: +const insertedIds = results + .filter((r) => r.status === "inserted") + .map((r) => r.id); ``` -### get +### get / getByPath ```typescript const memory = await me.memory.get({ id: "019..." }); +// Or address a named memory by its tree/name path: +const byPath = await me.memory.getByPath({ path: "/share/auth/jwt-rotation" }); ``` ### update -Only provided fields are changed. Pass `null` to clear optional fields. +Only provided fields are changed. Pass `null` to clear optional fields (e.g. `name: null` clears the name). Update is id-addressed. ```typescript const updated = await me.memory.update({ id: "019...", content: "Updated content", + name: "jwt-rotation", // set/rename; null clears meta: { reviewed: true }, }); ``` -### delete +### delete / deleteByPath ```typescript const { deleted } = await me.memory.delete({ id: "019..." }); +// Or delete a named memory by its tree/name path: +await me.memory.deleteByPath({ path: "/share/auth/jwt-rotation" }); ``` ### deleteTree @@ -120,8 +133,8 @@ const { deleted } = await me.memory.delete({ id: "019..." }); Delete all memories under a tree prefix. ```typescript -const { count } = await me.memory.deleteTree({ tree: "share.old.project", dryRun: true }); -const { count: deleted } = await me.memory.deleteTree({ tree: "share.old.project" }); +const { count } = await me.memory.deleteTree({ tree: "/share/old/project", dryRun: true }); +const { count: deleted } = await me.memory.deleteTree({ tree: "/share/old/project" }); ``` ### move @@ -130,8 +143,8 @@ Move memories from one tree prefix to another, preserving subtree structure. ```typescript const { count } = await me.memory.move({ - source: "share.drafts.api", - destination: "share.published.api", + source: "/share/drafts/api", + destination: "/share/published/api", }); ``` @@ -141,9 +154,9 @@ View the hierarchical tree structure with counts at each node. ```typescript const { nodes } = await me.memory.tree(); -// [{ path: "share", count: 5 }, { path: "share.work", count: 3 }, ...] +// [{ path: "/share", count: 5 }, { path: "/share/work", count: 3 }, ...] -const { nodes } = await me.memory.tree({ tree: "share.work", levels: 2 }); +const { nodes } = await me.memory.tree({ tree: "/share/work", levels: 2 }); ``` ## Search @@ -158,7 +171,7 @@ const { results } = await me.memory.search({ // Filters (all optional, combined with AND) grep: "regex.*pattern", // POSIX regex on content - tree: "share.work.projects.*", // ltree/lquery filter + tree: "/share/work/projects/*", // ltree/lquery filter meta: { source: "meeting-notes" }, // JSONB containment temporal: { // time-based filter contains: "2025-06-15T00:00:00Z", // point-in-time @@ -211,8 +224,8 @@ const { groups } = await me.group.listForMember({ memberId: "019..." }); Levels are `1` (read), `2` (write), `3` (owner). ```typescript -await me.grant.set({ principalId: "019...", treePath: "share.work", access: 2 }); -await me.grant.remove({ principalId: "019...", treePath: "share.work" }); +await me.grant.set({ principalId: "019...", treePath: "/share/work", access: 2 }); +await me.grant.remove({ principalId: "019...", treePath: "/share/work" }); const { grants } = await me.grant.list(); // optionally { principalId } / { treePath } ``` diff --git a/e2e/cli.e2e.test.ts b/e2e/cli.e2e.test.ts index 1c3b7f4e..c3dda024 100644 --- a/e2e/cli.e2e.test.ts +++ b/e2e/cli.e2e.test.ts @@ -209,6 +209,12 @@ describe.skipIf( return { stdout, stderr, code }; } + // The canonical (leading-slash) display form of a dotted ltree path — what + // the API/CLI return. Used to assert returned `tree` values against the + // dotted paths the tests build for input / ltree queries. + const toSlashPath = (dotted: string): string => + `/${dotted.replace(/\./g, "/")}`; + // Count memories under a tree in this run's space schema. async function countUnder(treePrefix: string): Promise { const [row] = await sql.unsafe( @@ -338,7 +344,7 @@ describe.skipIf( expect(await countUnder(`${base}.keep`)).toBe(1); const fetched = await meJson<{ tree: string }>(["memory", "get", first.id]); - expect(fetched.tree).toBe(`${src}.one`); + expect(fetched.tree).toBe(toSlashPath(`${src}.one`)); }); test("2d. memory move previews and relocates a subtree", async () => { @@ -381,7 +387,7 @@ describe.skipIf( expect(await countUnder(`${base}.keep`)).toBe(1); const fetched = await meJson<{ tree: string }>(["memory", "get", first.id]); - expect(fetched.tree).toBe(`${dst}.one`); + expect(fetched.tree).toBe(toSlashPath(`${dst}.one`)); }); test("2e. export alias writes matching memories as JSON", async () => { @@ -402,8 +408,8 @@ describe.skipIf( "export probe two", ]); expect(exported.map((m) => m.tree).sort()).toEqual([ - `${branch}.one`, - `${branch}.two`, + toSlashPath(`${branch}.one`), + toSlashPath(`${branch}.two`), ]); }); @@ -464,7 +470,7 @@ describe.skipIf( ]); expect(updated.content).toBe("edited content"); - const del = await me(["memory", "delete", created.id, "--yes"]); + const del = await me(["memory", "delete", created.id]); expect(del.code).toBe(0); // Getting it now fails with a non-zero exit. @@ -472,6 +478,149 @@ describe.skipIf( expect(get.code).not.toBe(0); }); + test("6b. name: create --name, get by path, conflict modes, rename, delete --name", async () => { + const created = await meJson<{ id: string; name: string | null }>([ + "create", + "rotation runbook", + "--tree", + "share/auth", + "--name", + "jwt-rotation", + ]); + expect(created.name).toBe("jwt-rotation"); + + // get by the folder/name path resolves to the same memory. + const got = await meJson<{ id: string; name: string | null }>([ + "get", + "share/auth/jwt-rotation", + ]); + expect(got.id).toBe(created.id); + expect(got.name).toBe("jwt-rotation"); + + // a bare name conflict errors; --replace overwrites in place (same id). + const dup = await me([ + "create", + "v2", + "--tree", + "share/auth", + "--name", + "jwt-rotation", + ]); + expect(dup.code).not.toBe(0); + const replaced = await meJson<{ id: string; content: string }>([ + "create", + "v2", + "--tree", + "share/auth", + "--name", + "jwt-rotation", + "--replace", + ]); + expect(replaced.id).toBe(created.id); + expect(replaced.content).toBe("v2"); + + // rename via update addressed by path. + const renamed = await meJson<{ name: string | null }>([ + "update", + "share/auth/jwt-rotation", + "--name", + "rotation", + ]); + expect(renamed.name).toBe("rotation"); + + // delete the named memory by its path (no flag needed — a non-UUID arg is + // always a tree/name path that deletes at most one memory). + const del = await meJson<{ deleted: boolean }>([ + "delete", + "share/auth/rotation", + ]); + expect(del.deleted).toBe(true); + }); + + test("6c. update --name '' clears the name", async () => { + const created = await meJson<{ id: string }>([ + "create", + "clearable", + "--tree", + "share", + "--name", + "tmp", + ]); + const cleared = await meJson<{ name: string | null }>([ + "update", + created.id, + "--name", + "", + ]); + expect(cleared.name).toBeNull(); + }); + + test("6d. deltree: --dry-run previews without deleting; --yes deletes the subtree", async () => { + await meJson(["create", "a", "--tree", "share/deltree_demo"]); + await meJson(["create", "b", "--tree", "share/deltree_demo/sub"]); + + // delete only ever targets a single named memory — with no memory + // named 'deltree_demo' at share/, it errors and never touches the subtree + // beneath it, and the error points at deltree. + const single = await me(["delete", "share/deltree_demo"]); + expect(single.code).not.toBe(0); + expect(`${single.stdout}${single.stderr}`).toContain( + "deltree share/deltree_demo", + ); + expect( + (await meJson<{ count: number }>(["count", "share/deltree_demo"])).count, + ).toBe(2); + + // deltree --dry-run reports the count but deletes nothing. + const dry = await meJson<{ dryRun: boolean; count: number }>([ + "deltree", + "share/deltree_demo", + "--dry-run", + ]); + expect(dry.dryRun).toBe(true); + expect(dry.count).toBe(2); + expect( + (await meJson<{ count: number }>(["count", "share/deltree_demo"])).count, + ).toBe(2); + + // deltree --yes deletes the whole subtree. + const del = await meJson<{ count: number }>([ + "deltree", + "share/deltree_demo", + "--yes", + ]); + expect(del.count).toBe(2); + expect( + (await meJson<{ count: number }>(["count", "share/deltree_demo"])).count, + ).toBe(0); + }); + + test("6e. name validation: create --name '' rejected; bad path fails validation not NOT_FOUND", async () => { + // create --name "" must fail fast (empty is never a valid name) rather than + // silently creating an unnamed memory. + const emptyName = await me([ + "create", + "x", + "--tree", + "share", + "--name", + "", + ]); + expect(emptyName.code).not.toBe(0); + // and nothing was created at that tree under an empty name + expect( + `${emptyName.stdout}${emptyName.stderr}`.toLowerCase(), + ).not.toContain("created memory"); + + // A path with a trailing slash (empty leaf) is a validation error, not a + // NOT_FOUND — the leaf must be a valid memory name. + const badPath = await me(["get", "share/auth/"]); + expect(badPath.code).not.toBe(0); + expect(`${badPath.stdout}${badPath.stderr}`.toLowerCase()).not.toContain( + "not found", + ); + }); + // ------------------------------------------------------------------------- // Extended scenarios // ------------------------------------------------------------------------- @@ -1166,6 +1315,51 @@ describe.skipIf( expect(await countUnder("share.importgroup")).toBe(2); }); + test("9d. `me import memories` is idempotent — re-import skips, never errors", async () => { + // Named record (no id): the (tree, name) slot is the idempotency key. + const named = JSON.stringify({ + content: "idempotent named probe", + tree: "share/idem", + name: "probe", + }); + const first = await meStdin(["import", "memories", "-", "--json"], named); + expect(first.code, first.stderr).toBe(0); + expect(JSON.parse(first.stdout).imported).toBe(1); + + // Re-import identical content: skipped server-side, exit 0 (the + // raise-by-default introduced in the SQL conflict model would otherwise + // error here), and no duplicate row materializes. + const again = await meStdin(["import", "memories", "-", "--json"], named); + expect(again.code, again.stderr).toBe(0); + expect(JSON.parse(again.stdout).imported).toBe(0); + expect(await countUnder("share.idem")).toBe(1); + + // Explicit-id record: the id is the idempotency key, and a skipped id is + // reported back in `skippedIds`. + const id = Bun.randomUUIDv7(); + const withId = JSON.stringify({ + content: "idempotent id probe", + tree: "share/idem", + id, + }); + const idFirst = await meStdin( + ["import", "memories", "-", "--json"], + withId, + ); + expect(idFirst.code, idFirst.stderr).toBe(0); + expect(JSON.parse(idFirst.stdout).ids).toContain(id); + + const idAgain = await meStdin( + ["import", "memories", "-", "--json"], + withId, + ); + expect(idAgain.code, idAgain.stderr).toBe(0); + const r = JSON.parse(idAgain.stdout); + expect(r.imported).toBe(0); + expect(r.skippedIds).toContain(id); + expect(await countUnder("share.idem")).toBe(2); + }); + test("10. failure modes: bad space and missing auth exit non-zero", async () => { const badSpace = await me(["search", "--fulltext", "fox"], { ME_SPACE: "doesnotexist1", diff --git a/packages/cli/chunk.test.ts b/packages/cli/chunk.test.ts index f916944b..2b69bad4 100644 --- a/packages/cli/chunk.test.ts +++ b/packages/cli/chunk.test.ts @@ -2,7 +2,10 @@ * Tests for the byte-aware chunker in `chunk.ts`. */ import { describe, expect, test } from "bun:test"; -import type { MemoryCreateParams } from "@memory.build/protocol/memory"; +import type { + MemoryCreateParams, + MemoryWriteResult, +} from "@memory.build/protocol/memory"; import { approxMemoryBytes, type BatchCreateClient, @@ -113,62 +116,62 @@ describe("batchCreateChunked", () => { const stubClient = ( handler: ( memories: MemoryCreateParams[], - replaceIfMetaDiffers?: string, - ) => Promise<{ ids: string[]; updatedIds?: string[] }>, + onConflict?: "error" | "replace" | "ignore", + ) => Promise<{ results: MemoryWriteResult[] }>, ): BatchCreateClient => ({ memory: { - batchCreate: async ({ memories, replaceIfMetaDiffers }) => { - const res = await handler(memories, replaceIfMetaDiffers); - // Old servers omit updatedIds; the helper must tolerate that, so the - // stub passes whatever the handler chose to return. - return res as { ids: string[]; updatedIds: string[] }; - }, + batchCreate: ({ memories, onConflict }) => handler(memories, onConflict), }, }); + /** Build inserted results for a chunk, keyed on each memory's id. */ + const inserted = (memories: MemoryCreateParams[]): MemoryWriteResult[] => + memories.map((m) => ({ id: m.id ?? "auto", status: "inserted" as const })); + test("single chunk, all succeed", async () => { const calls: number[] = []; const client = stubClient(async (memories) => { calls.push(memories.length); - return { ids: memories.map((m) => m.id ?? "auto") }; + return { results: inserted(memories) }; }); const result = await batchCreateChunked(client, [mem("a"), mem("b")]); - expect(result.insertedIds).toEqual(["a", "b"]); - expect(result.failedIds).toEqual([]); + expect(result.results).toEqual([ + { id: "a", status: "inserted" }, + { id: "b", status: "inserted" }, + ]); expect(result.errors).toEqual([]); expect(calls).toEqual([2]); // single batchCreate call }); - test("two chunks succeed, insertedIds accumulate across chunks", async () => { - // Force two chunks via a tight byte budget by using big content. We - // can't override the 768 KiB default through the public API, so use - // many small memories and rely on the count cap... actually easier: - // use one big enough that two would overflow. + test("two chunks succeed, results accumulate across chunks in order", async () => { + // Force two chunks via big content (the 768 KiB default isn't overridable + // through the public API). We assert results accumulate, not boundaries. const big = mem("big", 700_000); const small = mem("small", 10); const client = stubClient(async (memories) => ({ - ids: memories.map((m) => m.id ?? "auto"), + results: inserted(memories), })); const result = await batchCreateChunked(client, [big, small]); - // Both items land; we don't assert chunk boundaries here, only that - // ids are accumulated correctly across however many chunks fired. - expect(result.insertedIds.sort()).toEqual(["big", "small"]); - expect(result.failedIds).toEqual([]); + expect(result.results.map((r) => r.id).sort()).toEqual(["big", "small"]); + expect(result.results.every((r) => r.status === "inserted")).toBe(true); expect(result.errors).toEqual([]); }); - test("second chunk fails: insertedIds from first only, failedIds from second", async () => { + test("second chunk fails: first inserted, second is an 'error' row", async () => { const big1 = mem("a", 700_000); const big2 = mem("b", 700_000); let call = 0; const client = stubClient(async (memories) => { call++; if (call === 2) throw new Error("server boom"); - return { ids: memories.map((m) => m.id ?? "auto") }; + return { results: inserted(memories) }; }); const result = await batchCreateChunked(client, [big1, big2]); - expect(result.insertedIds).toEqual(["a"]); - expect(result.failedIds).toEqual(["b"]); + // One row per input, in order: a inserted, b's chunk failed → 'error'. + expect(result.results).toEqual([ + { id: "a", status: "inserted" }, + { id: "b", status: "error" }, + ]); expect(result.errors).toHaveLength(1); expect(result.errors[0]).toMatchObject({ chunkIndex: 1, @@ -178,58 +181,78 @@ describe("batchCreateChunked", () => { }); }); - test("all chunks fail: insertedIds empty, failedIds covers all explicit ids", async () => { + test("all chunks fail: every input is an 'error' row", async () => { const big1 = mem("a", 700_000); const big2 = mem("b", 700_000); const client = stubClient(async () => { throw new Error("network down"); }); const result = await batchCreateChunked(client, [big1, big2]); - expect(result.insertedIds).toEqual([]); - expect(result.failedIds.sort()).toEqual(["a", "b"]); + expect(result.results).toEqual([ + { id: "a", status: "error" }, + { id: "b", status: "error" }, + ]); expect(result.errors).toHaveLength(2); expect(result.errors[0]?.chunkIndex).toBe(0); expect(result.errors[1]?.chunkIndex).toBe(1); }); - test("server returns shorter ids than requested (simulating ON CONFLICT)", async () => { - // Caller submits 3 memories, server inserts 2 (one was a duplicate id, - // skipped by the conditional upsert). The helper should faithfully - // report the 2 inserted; classifying the missing one as "skipped" is - // the caller's job. + test("a failed input with no explicit id gets a null id 'error' row", async () => { + const noId: MemoryCreateParams = { content: "x", tree: "t" }; + const client = stubClient(async () => { + throw new Error("boom"); + }); + const result = await batchCreateChunked(client, [noId]); + expect(result.results).toEqual([{ id: null, status: "error" }]); + expect(result.errors[0]?.ids).toEqual([]); // no explicit id to report + }); + + test("carries per-row status (inserted/skipped) through unchanged", async () => { + // Caller submits 3 memories; the server skips one (its idempotency key + // already existed). The helper faithfully reports each row's status in + // submission order — classifying the skip is the caller's job. const client = stubClient(async (memories) => ({ - ids: memories.map((m) => m.id ?? "auto").filter((id) => id !== "dup"), // server "drops" the dup id - updatedIds: [], + results: memories.map((m) => ({ + id: m.id ?? "auto", + status: m.id === "dup" ? ("skipped" as const) : ("inserted" as const), + })), })); const result = await batchCreateChunked(client, [ mem("a"), mem("dup"), mem("b"), ]); - expect(result.insertedIds).toEqual(["a", "b"]); - expect(result.updatedIds).toEqual([]); - expect(result.failedIds).toEqual([]); // no chunk failed - expect(result.errors).toEqual([]); + expect(result.results).toEqual([ + { id: "a", status: "inserted" }, + { id: "dup", status: "skipped" }, + { id: "b", status: "inserted" }, + ]); + expect(result.errors).toEqual([]); // no chunk failed }); - test("passes replaceIfMetaDiffers through and accumulates updatedIds", async () => { - // Two chunks (big payloads); the server reports the first id of each + test("passes onConflict through every chunk and accumulates updated rows", async () => { + // Two chunks (big payloads); the server reports the first row of each // chunk as updated and the rest as inserted. - const seenKeys: Array = []; - const client = stubClient(async (memories, replaceIfMetaDiffers) => { - seenKeys.push(replaceIfMetaDiffers); - const ids = memories.map((m) => m.id ?? "auto"); - return { ids: ids.slice(1), updatedIds: ids.slice(0, 1) }; + const seen: Array = []; + const client = stubClient(async (memories, onConflict) => { + seen.push(onConflict); + return { + results: memories.map((m, i) => ({ + id: m.id ?? "auto", + status: i === 0 ? ("updated" as const) : ("inserted" as const), + })), + }; }); const result = await batchCreateChunked( client, [mem("a", 700_000), mem("b", 10), mem("c", 700_000), mem("d", 10)], - { replaceIfMetaDiffers: "importer_version" }, + { onConflict: "replace" }, ); - expect(seenKeys.length).toBeGreaterThan(1); // multiple chunks - expect(new Set(seenKeys)).toEqual(new Set(["importer_version"])); - expect(result.updatedIds.length).toBe(seenKeys.length); - expect([...result.insertedIds, ...result.updatedIds].sort()).toEqual([ + expect(seen.length).toBeGreaterThan(1); // multiple chunks + expect(new Set(seen)).toEqual(new Set(["replace"])); + const updated = result.results.filter((r) => r.status === "updated"); + expect(updated.length).toBe(seen.length); // one updated per chunk + expect(result.results.map((r) => r.id).sort()).toEqual([ "a", "b", "c", @@ -237,26 +260,28 @@ describe("batchCreateChunked", () => { ]); }); - test("tolerates a pre-upsert server omitting updatedIds", async () => { - const client = stubClient(async (memories) => ({ - ids: memories.map((m) => m.id ?? "auto"), - // no updatedIds field at all - })); - const result = await batchCreateChunked(client, [mem("a")]); - expect(result.insertedIds).toEqual(["a"]); - expect(result.updatedIds).toEqual([]); + test("leaves onConflict unset when no option is given", async () => { + let seen: string | undefined = "sentinel"; + const client: BatchCreateClient = { + memory: { + batchCreate: async ({ memories, onConflict }) => { + seen = onConflict; + return { results: inserted(memories) }; + }, + }, + }; + await batchCreateChunked(client, [mem("a")]); + expect(seen).toBeUndefined(); }); test("empty input never calls the server", async () => { let calls = 0; const client = stubClient(async () => { calls++; - return { ids: [], updatedIds: [] }; + return { results: [] }; }); const result = await batchCreateChunked(client, []); - expect(result.insertedIds).toEqual([]); - expect(result.updatedIds).toEqual([]); - expect(result.failedIds).toEqual([]); + expect(result.results).toEqual([]); expect(result.errors).toEqual([]); expect(calls).toBe(0); }); diff --git a/packages/cli/chunk.ts b/packages/cli/chunk.ts index f65069cb..33a55814 100644 --- a/packages/cli/chunk.ts +++ b/packages/cli/chunk.ts @@ -14,7 +14,10 @@ * that callers should reach for unless they need a custom budget. */ -import type { MemoryCreateParams } from "@memory.build/protocol/memory"; +import type { + MemoryCreateParams, + MemoryWriteResult, +} from "@memory.build/protocol/memory"; /** * Hard cap on memories per `memory.batchCreate` call. Matches the protocol @@ -115,39 +118,52 @@ export interface BatchCreateClient { memory: { batchCreate: (params: { memories: MemoryCreateParams[]; - replaceIfMetaDiffers?: string; - }) => Promise<{ ids: string[]; updatedIds: string[] }>; + onConflict?: "error" | "replace" | "ignore"; + }) => Promise<{ results: MemoryWriteResult[] }>; }; } /** Options applied to every chunk of a `batchCreateChunked` run. */ export interface BatchCreateChunkedOptions { /** - * Meta key for the server's conditional replace: a memory whose explicit - * id already exists is rewritten in place when the stored row's value for - * this key differs (importers pass "importer_version" so version bumps - * re-render existing rows), else skipped. Unset: duplicates are skipped. + * Conflict policy for every chunk's idempotency key (each row's id when + * given, else its (tree, name) slot). The server defaults to "error" + * (raise); file importers pass "ignore" so a re-import is a no-op rather + * than failing, and "replace" overwrites in place when content/meta/temporal + * differ — deterministic-id importers pass "replace" and stamp + * meta.importer_version, so a version bump makes meta differ and re-renders. */ - replaceIfMetaDiffers?: string; + onConflict?: "error" | "replace" | "ignore"; +} + +/** + * One submitted memory's outcome from a chunked run. A superset of the wire + * `MemoryWriteResult`: successful chunks yield the server's `{ id, status }` + * (status 'inserted' | 'updated' | 'skipped', `id` always present); a failed + * chunk yields `status: 'error'` for each of its rows, with `id` the row's + * explicit id when it had one (echoed back) or `null` when it was submitted + * without an id. The failure *message* lives once per chunk in `errors[]`. + */ +export interface ChunkWriteResult { + id: string | null; + status: MemoryWriteResult["status"] | "error"; } /** Result of a chunked `batchCreate` run. */ export interface BatchCreateChunkedResult { - /** Ids the server confirmed inserted (across all successful chunks). */ - insertedIds: string[]; - /** Existing rows rewritten in place via `replaceIfMetaDiffers`. */ - updatedIds: string[]; /** - * Explicit ids submitted in chunks that errored, flattened across all - * failed chunks for callers that just need a set of "ids to exclude - * from skip classification." For per-chunk error attribution use - * `errors[].ids` instead. - * - * These were never processed by the server, so they are neither - * inserted nor skipped. + * One row per submitted memory, in submission order — so `results[i]` is the + * outcome of the i-th input (the same contract as the wire `batchCreate`, + * extended with an 'error' status for inputs whose chunk failed). Filter by + * status for inserted/updated/skipped/error counts and ids. + */ + results: ChunkWriteResult[]; + /** + * One entry per failed chunk, carrying the shared error message. Every row in + * a failed chunk also appears in `results` as an 'error' row; this view groups + * those failures with their message (and reports the full item count, which + * includes rows submitted without an explicit id). */ - failedIds: string[]; - /** One entry per failed chunk. */ errors: Array<{ /** 0-based index of the failed chunk in submission order. */ chunkIndex: number; @@ -163,24 +179,18 @@ export interface BatchCreateChunkedResult { * Run `client.memory.batchCreate` over `memories`, automatically slicing * the input into chunks that fit under the server's request-body limit. * - * Chunks are sent sequentially. A failed chunk is recorded once in - * `errors` and its explicit ids are added to `failedIds`; it does not - * abort siblings. Successful chunks contribute to `insertedIds` and - * `updatedIds`. - * - * A submitted explicit id in neither array (and not in a failed chunk) was - * skipped server-side — it already exists, at a matching meta-key value - * when `replaceIfMetaDiffers` is set. Use `computeSkippedIds` (or, for - * packs, `classifySkips` with `failedIds`) to classify the missing ids. + * Chunks are sent sequentially and a failed chunk does not abort its siblings. + * Every input gets one `results` row in submission order: successful chunks + * contribute the server's `{ id, status }`, and a failed chunk contributes an + * 'error' row per input (its explicit id, else `null`). The failure message is + * recorded once per chunk in `errors`. */ export async function batchCreateChunked( client: BatchCreateClient, memories: MemoryCreateParams[], options: BatchCreateChunkedOptions = {}, ): Promise { - const insertedIds: string[] = []; - const updatedIds: string[] = []; - const failedIds: string[] = []; + const results: ChunkWriteResult[] = []; const errors: BatchCreateChunkedResult["errors"] = []; let chunkIndex = 0; @@ -188,19 +198,21 @@ export async function batchCreateChunked( try { const res = await client.memory.batchCreate({ memories: chunk, - ...(options.replaceIfMetaDiffers !== undefined - ? { replaceIfMetaDiffers: options.replaceIfMetaDiffers } + ...(options.onConflict !== undefined + ? { onConflict: options.onConflict } : {}), }); - insertedIds.push(...res.ids); - // A pre-upsert server doesn't return updatedIds; treat as none updated. - updatedIds.push(...(res.updatedIds ?? [])); + results.push(...res.results); } catch (error) { const msg = error instanceof Error ? error.message : String(error); + // Every row in the failed chunk gets an 'error' result (its explicit id, + // else null), preserving the one-row-per-input contract. + for (const p of chunk) { + results.push({ id: p.id ?? null, status: "error" }); + } const ids = chunk .map((p) => p.id) .filter((x): x is string => typeof x === "string"); - failedIds.push(...ids); errors.push({ chunkIndex, itemCount: chunk.length, @@ -211,5 +223,5 @@ export async function batchCreateChunked( chunkIndex++; } - return { insertedIds, updatedIds, failedIds, errors }; + return { results, errors }; } diff --git a/packages/cli/commands/import-git.ts b/packages/cli/commands/import-git.ts index b5495d81..c382514d 100644 --- a/packages/cli/commands/import-git.ts +++ b/packages/cli/commands/import-git.ts @@ -2,15 +2,16 @@ * `me import git` — import a repo's commit history as memories. * * One memory per commit (message + capped changed-file list) under - * `..git_history`, with the commit date as the - * memory's temporal and a deterministic id keyed by `(tree, sha)` — so - * re-runs are idempotent (existing commits become server-side skips). + * `..git_history`, named with the commit `` and + * with the commit date as the memory's temporal. Idempotency is keyed on + * `(tree, sha)`, so re-runs become server-side skips; the id is a + * timestamp-prefixed UUIDv7 (random tail) so commits sort by date on the id. * * Re-runs are also incremental: the newest already-imported commit is looked * up server-side (one search) and, when it is an ancestor of the target rev, * only `..` is walked. Any doubt (force-push, other branch, - * explicit bounds) falls back to the full walk, which deterministic ids make - * safe. `--full` forces the full walk. + * explicit bounds) falls back to the full walk, which the (tree, sha) key + * makes safe. `--full` forces the full walk. */ import { resolve } from "node:path"; import * as clack from "@clack/prompts"; @@ -29,7 +30,7 @@ import { import { createProgressReporter, DEFAULT_TREE_ROOT, - dedupByMemoryId, + dedupBy, } from "../importers/index.ts"; import { SlugRegistry } from "../importers/slug.ts"; import { getOutputFormat, output } from "../output.ts"; @@ -40,7 +41,6 @@ import { requireSpace, } from "../util.ts"; import { VALID_TREE_ROOT_RE } from "./import.ts"; -import { computeSkippedIds } from "./memory-import.ts"; /** Parsed options for one git import run. */ export interface GitImportOptions { @@ -204,7 +204,6 @@ export async function runGitImport( fmt === "text" ? createProgressReporter(process.stderr) : undefined; progress?.start(); - const importedAt = new Date().toISOString(); const planned: Array<{ memoryId: string; payload: MemoryCreateParams }> = []; let commitsWalked = 0; let skippedMerges = 0; @@ -231,7 +230,6 @@ export async function runGitImport( projectSlug: slug, gitRemote, fileList: opts.fileList !== false, - importedAt, }); if ("error" in built) { failed++; @@ -250,23 +248,29 @@ export async function runGitImport( handleError(error, fmt); } - const { unique } = dedupByMemoryId(planned); + // Dedup on the commit sha (the (tree, name) key), not the random id. + const { unique } = dedupBy(planned, (p) => p.payload.name ?? p.memoryId); let inserted = 0; let skipped = 0; if (opts.dryRun) { inserted = unique.length; } else if (unique.length > 0) { - const submitted = unique.map((p) => p.memoryId); + // Re-import is idempotent via content-aware replace: an unchanged commit is + // a no-op (status 'skipped'); a version bump changes meta and re-renders in + // place ('updated'). Without a directive a re-submitted commit would be a + // hard (tree, name) conflict. const result = await batchCreateChunked( engine, unique.map((p) => p.payload), + { onConflict: "replace" }, ); - inserted = result.insertedIds.length; - const failedSet = new Set(result.failedIds); - skipped = computeSkippedIds(submitted, result.insertedIds).filter( - (id) => !failedSet.has(id), + // A commit "imported" if it was inserted or re-rendered; skipped = + // unchanged. 'error' rows are tallied via errors[] (failed) below. + inserted = result.results.filter( + (r) => r.status === "inserted" || r.status === "updated", ).length; + skipped = result.results.filter((r) => r.status === "skipped").length; for (const e of result.errors) { failed += e.itemCount; errors.push({ sha: `chunk ${e.chunkIndex}`, error: e.error }); diff --git a/packages/cli/commands/memory-count.test.ts b/packages/cli/commands/memory-count.test.ts index 3c3fe4a5..c8c12943 100644 --- a/packages/cli/commands/memory-count.test.ts +++ b/packages/cli/commands/memory-count.test.ts @@ -1,5 +1,9 @@ import { describe, expect, test } from "bun:test"; -import { formatMemoryCount, parseMaxCount } from "./memory.ts"; +import { + formatMemoryCount, + parseMaxCount, + uniqueExportFilename, +} from "./memory.ts"; describe("parseMaxCount", () => { test("returns undefined when omitted", () => { @@ -34,3 +38,43 @@ describe("formatMemoryCount", () => { expect(formatMemoryCount(2, 3)).toBe("2 memories"); }); }); + +describe("uniqueExportFilename", () => { + test("appends .md once and leaves a non-colliding name clean", () => { + const used = new Map>(); + expect(uniqueExportFilename("/d", "foo", "id1", used)).toBe("foo.md"); + expect(uniqueExportFilename("/d", "bar.md", "id2", used)).toBe("bar.md"); + }); + + test("disambiguates `foo` vs `foo.md` (same on-disk name) by id", () => { + const used = new Map>(); + expect(uniqueExportFilename("/d", "foo", "id1", used)).toBe("foo.md"); + // `foo.md` would overwrite the first file → gets the id inserted. + expect(uniqueExportFilename("/d", "foo.md", "id2", used)).toBe( + "foo.id2.md", + ); + }); + + test("treats names colliding only by case as a clash (portable to case-insensitive FS)", () => { + const used = new Map>(); + expect(uniqueExportFilename("/d", "Foo", "id1", used)).toBe("Foo.md"); + expect(uniqueExportFilename("/d", "foo", "id2", used)).toBe("foo.id2.md"); + }); + + test("scopes collisions per directory", () => { + const used = new Map>(); + expect(uniqueExportFilename("/a", "foo", "id1", used)).toBe("foo.md"); + // Same name in a different directory is fine — no disambiguation. + expect(uniqueExportFilename("/b", "foo", "id2", used)).toBe("foo.md"); + }); + + test("throws if even the id-disambiguated name is already taken", () => { + const used = new Map>(); + uniqueExportFilename("/d", "foo", "id1", used); // foo.md + uniqueExportFilename("/d", "foo.X", "id3", used); // foo.X.md (named like an id) + // `foo.md` (id X) → foo.md taken → foo.X.md also taken → error, not a guess. + expect(() => uniqueExportFilename("/d", "foo.md", "X", used)).toThrow( + /already taken/, + ); + }); +}); diff --git a/packages/cli/commands/memory-edit.test.ts b/packages/cli/commands/memory-edit.test.ts new file mode 100644 index 00000000..13e5486d --- /dev/null +++ b/packages/cli/commands/memory-edit.test.ts @@ -0,0 +1,63 @@ +/** + * Tests for the interactive editor's pure helpers — specifically that `name` + * round-trips so a rename/clear in `me memory edit` isn't silently dropped. + */ +import { describe, expect, test } from "bun:test"; +import { parseMarkdown } from "../parsers/markdown.ts"; +import { formatForEdit, hasChanges } from "./memory-edit.ts"; + +describe("formatForEdit", () => { + test("emits the name in frontmatter when set, and it round-trips through the parser", () => { + const text = formatForEdit({ + id: "0194a000-0001-7000-8000-000000000001", + content: "body", + name: "jwt-rotation", + tree: "share.auth", + }); + expect(text).toContain("name: jwt-rotation"); + const parsed = parseMarkdown(text)[0]; + expect(parsed?.name).toBe("jwt-rotation"); + }); + + test("omits the name line for an unnamed memory", () => { + const text = formatForEdit({ + id: "0194a000-0001-7000-8000-000000000001", + content: "body", + tree: "share.auth", + }); + expect(text).not.toContain("name:"); + }); +}); + +describe("hasChanges (name)", () => { + const base = { content: "body", tree: "share.auth", name: "jwt-rotation" }; + + test("detects a rename", () => { + expect(hasChanges(base, { content: "body", name: "jwt-rotate" })).toBe( + true, + ); + }); + + test("detects clearing the name (line removed)", () => { + expect(hasChanges(base, { content: "body" })).toBe(true); + }); + + test("detects adding a name where there was none", () => { + expect( + hasChanges( + { content: "body", tree: "share.auth" }, + { content: "body", tree: "share.auth", name: "jwt-rotation" }, + ), + ).toBe(true); + }); + + test("no change when the name is untouched", () => { + expect( + hasChanges(base, { + content: "body", + tree: "share.auth", + name: "jwt-rotation", + }), + ).toBe(false); + }); +}); diff --git a/packages/cli/commands/memory-edit.ts b/packages/cli/commands/memory-edit.ts index ab8046e5..87a3f7f3 100644 --- a/packages/cli/commands/memory-edit.ts +++ b/packages/cli/commands/memory-edit.ts @@ -16,6 +16,7 @@ import { parseMarkdown } from "../parsers/markdown.ts"; interface ParsedMemory { id?: string; content: string; + name?: string; meta?: Record; tree?: string; temporal?: { start: string; end?: string }; @@ -40,8 +41,10 @@ function openInEditor(filePath: string): boolean { /** * Format a memory as Markdown with YAML frontmatter for editing. + * + * Exported for unit testing. */ -function formatForEdit(memory: Record): string { +export function formatForEdit(memory: Record): string { const frontmatter: Record = { id: memory.id }; if (memory.createdAt) frontmatter.created_at = memory.createdAt; if ( @@ -51,6 +54,7 @@ function formatForEdit(memory: Record): string { ) { frontmatter.meta = memory.meta; } + if (memory.name) frontmatter.name = memory.name; if (memory.tree) frontmatter.tree = memory.tree; if (memory.temporal) frontmatter.temporal = memory.temporal; @@ -83,8 +87,10 @@ function stripErrorComments(content: string): string { /** * Check if a memory has changed. + * + * Exported for unit testing. */ -function hasChanges( +export function hasChanges( original: Record, parsed: ParsedMemory, ): boolean { @@ -98,6 +104,10 @@ function hasChanges( const parsedTree = parsed.tree || null; if (origTree !== parsedTree) return true; + const origName = (original.name as string) || null; + const parsedName = parsed.name || null; + if (origName !== parsedName) return true; + const origTemporal = original.temporal as { start: string; end?: string | null; @@ -171,6 +181,11 @@ export async function editMemory( if (parsed.tree !== undefined) updateParams.tree = parsed.tree; if (parsed.temporal !== undefined) updateParams.temporal = parsed.temporal; + // name: a value sets/renames; removing the line from a previously-named + // memory clears it (null). Only sent when it actually changed. + const origName = ((original as { name?: string }).name as string) || null; + const parsedName = parsed.name || null; + if (parsedName !== origName) updateParams.name = parsedName; try { await engine.memory.update( diff --git a/packages/cli/commands/memory-import.test.ts b/packages/cli/commands/memory-import.test.ts deleted file mode 100644 index b317ab10..00000000 --- a/packages/cli/commands/memory-import.test.ts +++ /dev/null @@ -1,44 +0,0 @@ -/** - * Tests for `me import memories` (alias `me memory import`) helpers. - * - * The skip-detection helper exists because `engine.memory.batchCreate` - * silently drops conflicting ids (post-#64). Memory import — unlike pack - * install — has no metadata to classify skips against, so this is just - * a set difference between explicit-id requests and inserted ids. - */ -import { describe, expect, test } from "bun:test"; -import { computeSkippedIds } from "./memory-import.ts"; - -describe("computeSkippedIds", () => { - test("returns empty when every explicit id was inserted", () => { - expect(computeSkippedIds(["a", "b", "c"], ["a", "b", "c"])).toEqual([]); - }); - - test("returns ids that are absent from inserted", () => { - expect(computeSkippedIds(["a", "b", "c"], ["b"])).toEqual(["a", "c"]); - }); - - test("ignores extra inserted ids that weren't in the request", () => { - // Auto-generated ids land in `insertedIds` but were never in - // `explicitIds`, so they don't affect the skip count. - expect(computeSkippedIds(["a"], ["a", "auto-1", "auto-2"])).toEqual([]); - }); - - test("handles mixed explicit + auto-generated requests", () => { - // Caller submitted 2 explicit-id memories and 3 auto-id memories. - // 1 explicit id collided; the other 4 inserts succeeded. - const explicit = ["a", "b"]; - const inserted = ["b", "auto-1", "auto-2", "auto-3"]; - expect(computeSkippedIds(explicit, inserted)).toEqual(["a"]); - }); - - test("handles empty input", () => { - expect(computeSkippedIds([], [])).toEqual([]); - expect(computeSkippedIds([], ["auto-1"])).toEqual([]); - expect(computeSkippedIds(["a"], [])).toEqual(["a"]); - }); - - test("preserves request order in the skipped list", () => { - expect(computeSkippedIds(["c", "a", "b"], [])).toEqual(["c", "a", "b"]); - }); -}); diff --git a/packages/cli/commands/memory-import.ts b/packages/cli/commands/memory-import.ts index 8894ebd8..926c8c67 100644 --- a/packages/cli/commands/memory-import.ts +++ b/packages/cli/commands/memory-import.ts @@ -70,25 +70,6 @@ interface ImportResult { errors: Array<{ source: string; error: string }>; } -/** - * Compute which explicit ids were skipped by the server. - * - * `engine.memory.batchCreate` uses `ON CONFLICT (id) DO NOTHING` server-side - * (post-#64), so the returned `ids` array can be shorter than the request - * when conflicts occur. Memories submitted without an explicit `id` get a - * server-generated UUIDv7 that statistically can't collide, so only - * explicit-id requests can be skipped. - * - * Pure function exported for unit testing. - */ -export function computeSkippedIds( - explicitIds: string[], - insertedIds: string[], -): string[] { - const inserted = new Set(insertedIds); - return explicitIds.filter((id) => !inserted.has(id)); -} - export function createMemoryImportCommand(name = "import"): Command { return new Command(name) .description("import memories from files or stdin") @@ -274,26 +255,29 @@ export function createMemoryImportCommand(name = "import"): Command { // tree is required on the wire; records without one default to `share`. tree: mem.tree ?? SHARE_NAMESPACE, ...(mem.id ? { id: mem.id } : {}), + ...(mem.name ? { name: mem.name } : {}), ...(mem.meta ? { meta: mem.meta } : {}), ...(mem.temporal ? { temporal: mem.temporal } : {}), })); - const explicitIds = createParams - .map((p) => p.id) - .filter((id): id is string => typeof id === "string"); - // Chunked batch create — large imports are sliced under the // server's request-body limit, and a single failed chunk doesn't // take down the rest of the import. - const { - insertedIds, - failedIds, - errors: chunkErrors, - } = await batchCreateChunked(engine, createParams); + const { results: writeResults, errors: chunkErrors } = + await batchCreateChunked(engine, createParams, { + // Re-importing the same file is a no-op: skip rows whose idempotency + // key ((tree, name), else id) already exists rather than erroring. + onConflict: "ignore", + }); - result.imported = insertedIds.length; - result.ids = insertedIds; - result.failed = failedIds.length; + // inserted/skipped rows come from a successful chunk, so their id is + // always present; flatMap drops the null only TS insists on. + result.ids = writeResults.flatMap((r) => + r.status === "inserted" && r.id !== null ? [r.id] : [], + ); + result.imported = result.ids.length; + // Failed = rows whose chunk threw (one 'error' row per input). + result.failed = writeResults.filter((r) => r.status === "error").length; for (const e of chunkErrors) { result.errors.push({ source: `chunk ${e.chunkIndex} (${e.itemCount} items)`, @@ -301,13 +285,12 @@ export function createMemoryImportCommand(name = "import"): Command { }); } - // Skipped = explicit ids requested but neither inserted nor in a - // failed chunk. Failed-chunk ids never reached the server, so they - // are not "skipped due to id collision" — they're a separate class - // already accounted for in `result.failed`. - const failedSet = new Set(failedIds); - skippedIds = computeSkippedIds(explicitIds, insertedIds).filter( - (id) => !failedSet.has(id), + // Skipped = rows the server left as-is because their idempotency key + // already existed (onConflict 'ignore'). The per-row status counts a + // named-but-id-less skip too — a pre-status explicit-id-only tally + // missed those. + skippedIds = writeResults.flatMap((r) => + r.status === "skipped" && r.id !== null ? [r.id] : [], ); // Output results @@ -334,7 +317,7 @@ export function createMemoryImportCommand(name = "import"): Command { ); } for (const id of skippedIds) { - console.log(` ⊝ ${id} (id already exists)`); + console.log(` ⊝ ${id} (already exists)`); } for (const { source, error } of result.errors) { console.log(` ✗ ${source}: ${error}`); @@ -358,7 +341,7 @@ export function createMemoryImportCommand(name = "import"): Command { ); } else if (skipped > 0) { console.log( - `Imported ${result.imported} ${result.imported === 1 ? "memory" : "memories"} (${skipped} skipped — id already exists)`, + `Imported ${result.imported} ${result.imported === 1 ? "memory" : "memories"} (${skipped} skipped — already exist)`, ); } else { console.log( diff --git a/packages/cli/commands/memory.ts b/packages/cli/commands/memory.ts index e960c419..9a5f77aa 100644 --- a/packages/cli/commands/memory.ts +++ b/packages/cli/commands/memory.ts @@ -5,7 +5,8 @@ * - me memory get : Get a memory by ID (ANSI-rendered in TTY, raw markdown when piped) * - me memory search [query]: Hybrid search * - me memory update : Update a memory - * - me memory delete : Delete memory or tree + * - me memory delete : Delete a single memory (by ID or tree/name path) + * - me memory deltree : Delete every memory under a tree path * - me memory edit : Open in $EDITOR * - me memory count : Count memories matching a tree filter * - me memory tree [filter]: Show tree structure @@ -25,6 +26,7 @@ import { getOutputFormat, output, table } from "../output.ts"; import { buildMemoryClient, handleError, + isAppErrorCode, requireMemoryAuth, requireSpace, } from "../util.ts"; @@ -88,6 +90,46 @@ export function formatMemoryCount(count: number, maxCount?: number): string { return `${count} ${noun}`; } +/** + * Resolve a collision-free `.md` export filename for `base` (a memory name, or + * its id when unnamed) within `dir`, recording the choice in `used`. + * + * Memory names are unique in the database, but distinct names can still map to + * the same file on disk: `foo` and `foo.md` both want `foo.md`, and a + * case-insensitive filesystem also conflates `Foo` and `foo`. `used` maps each + * directory to the set of filenames already claimed there (compared + * lowercased). On a clash the memory's unique id is inserted before the `.md` + * extension so nothing is silently overwritten; the common no-collision case is + * unchanged (`.md`). + */ +export function uniqueExportFilename( + dir: string, + base: string, + id: string, + used: Map>, +): string { + let claimed = used.get(dir); + if (!claimed) { + claimed = new Set(); + used.set(dir, claimed); + } + const stem = base.endsWith(".md") ? base.slice(0, -3) : base; + let candidate = `${stem}.md`; + if (claimed.has(candidate.toLowerCase())) { + // Disambiguate with the unique id. This can only itself clash if another + // memory is literally *named* `${stem}.${id}` — astronomically unlikely, + // so surface it as an error rather than silently guessing another name. + candidate = `${stem}.${id}.md`; + if (claimed.has(candidate.toLowerCase())) { + throw new Error( + `Cannot pick a unique export filename in '${dir}': '${candidate}' is already taken (a memory named like another's id?).`, + ); + } + } + claimed.add(candidate.toLowerCase()); + return candidate; +} + /** * Format a memory for Markdown output (frontmatter + content). */ @@ -104,6 +146,7 @@ export function formatMemoryAsMarkdown( frontmatter.meta = memory.meta; } if (memory.tree) frontmatter.tree = memory.tree; + if (memory.name) frontmatter.name = memory.name; if (memory.temporal) frontmatter.temporal = memory.temporal; const yaml = yamlStringify(frontmatter, { lineWidth: 0 }).trimEnd(); @@ -123,8 +166,11 @@ function createMemoryCreateCommand(): Command { "--tree ", "tree path ('share' for shared, '~' for private home)", ) + .option("--name ", "filename-like leaf name, unique within the tree") .option("--meta ", "metadata as JSON") .option("--temporal ", "temporal range (start[,end])") + .option("--replace", "on conflict, replace the existing memory in place") + .option("--ignore", "on conflict, skip and keep the existing memory") .action(async (positionalContent: string | undefined, opts, cmd) => { const globalOpts = cmd.optsWithGlobals(); const creds = resolveCredentials(globalOpts.server); @@ -164,13 +210,32 @@ function createMemoryCreateCommand(): Command { process.exit(1); } + // `--name ""` is an error, not "unnamed": omit --name for an unnamed + // memory. (Clearing an existing name is an update-only op: `update + // --name ""`.) Caught here so the user gets a clear message rather than a + // schema rejection round-trip. + if (opts.name === "") { + if (fmt === "text") { + clack.log.error( + "Empty --name. Omit --name for an unnamed memory, or pass a filename-like slug.", + ); + } else { + output({ error: "Empty --name is not a valid name" }, fmt, () => {}); + } + process.exit(1); + } + const client = buildMemoryClient(creds); try { const params: Record = { content }; params.tree = opts.tree; + // Empty is rejected above; here name is either omitted or a real slug. + if (opts.name) params.name = opts.name; if (opts.meta) params.meta = parseMeta(opts.meta); if (opts.temporal) params.temporal = parseTemporal(opts.temporal); + if (opts.replace) params.onConflict = "replace"; + else if (opts.ignore) params.onConflict = "ignore"; const memory = await client.memory.create( params as Parameters[0], @@ -179,6 +244,7 @@ function createMemoryCreateCommand(): Command { output(memory, fmt, () => { clack.log.success(`Created memory ${memory.id}`); if (memory.tree) console.log(` Tree: ${memory.tree}`); + if (memory.name) console.log(` Name: ${memory.name}`); }); } catch (error) { handleError(error, fmt); @@ -188,10 +254,10 @@ function createMemoryCreateCommand(): Command { function createMemoryGetCommand(): Command { return new Command("get") - .description("get a memory by ID") - .argument("", "memory ID") + .description("get a memory by ID or by its tree/name path") + .argument("", "memory ID (UUIDv7) or tree/name path") .option("--raw", "output raw Markdown with YAML frontmatter (no ANSI)") - .action(async (id: string, opts, cmd) => { + .action(async (ref: string, opts, cmd) => { const globalOpts = cmd.optsWithGlobals(); const creds = resolveCredentials(globalOpts.server); const fmt = getOutputFormat(globalOpts); @@ -201,7 +267,9 @@ function createMemoryGetCommand(): Command { const client = buildMemoryClient(creds); try { - const memory = await client.memory.get({ id }); + const memory = UUIDV7_RE.test(ref) + ? await client.memory.get({ id: ref }) + : await client.memory.getByPath({ path: ref }); // --json / --yaml: structured output if (fmt !== "text") { @@ -222,6 +290,7 @@ function createMemoryGetCommand(): Command { // TTY: ANSI-rendered markdown with dimmed frontmatter const frontmatter: Record = { id: memory.id }; if (memory.tree) frontmatter.tree = memory.tree; + if (memory.name) frontmatter.name = memory.name; if ( memory.meta && typeof memory.meta === "object" && @@ -397,13 +466,17 @@ function createMemorySearchCommand(): Command { function createMemoryUpdateCommand(): Command { return new Command("update") - .description("update a memory") - .argument("", "memory ID") + .description("update a memory (by ID or tree/name path)") + .argument("", "memory ID (UUIDv7) or tree/name path") .option("--content ", "new content (use - for stdin)") - .option("--tree ", "new tree path") + .option("--tree ", "new tree path (moves the memory)") + .option( + "--name ", + "new leaf name (renames; pass an empty string to clear it)", + ) .option("--meta ", "new metadata (replaces existing)") .option("--temporal ", "new temporal range (start[,end])") - .action(async (id: string, opts, cmd) => { + .action(async (ref: string, opts, cmd) => { const globalOpts = cmd.optsWithGlobals(); const creds = resolveCredentials(globalOpts.server); const fmt = getOutputFormat(globalOpts); @@ -416,9 +489,15 @@ function createMemoryUpdateCommand(): Command { content = (await Bun.stdin.text()).trimEnd(); } - if (!content && !opts.tree && !opts.meta && !opts.temporal) { + if ( + !content && + !opts.tree && + opts.name === undefined && + !opts.meta && + !opts.temporal + ) { const msg = - "At least one update field required (--content, --tree, --meta, or --temporal)."; + "At least one update field required (--content, --tree, --name, --meta, or --temporal)."; if (fmt === "text") { clack.log.error(msg); } else { @@ -430,9 +509,18 @@ function createMemoryUpdateCommand(): Command { const client = buildMemoryClient(creds); try { + // update is id-addressed; resolve a tree/name ref to its id first. + const id = UUIDV7_RE.test(ref) + ? ref + : (await client.memory.getByPath({ path: ref })).id; const params: Record = { id }; if (content) params.content = content; if (opts.tree) params.tree = opts.tree; + // --name "" clears the name (empty is never a valid name); a non-empty + // value renames. + if (opts.name !== undefined) { + params.name = opts.name === "" ? null : opts.name; + } if (opts.meta) params.meta = parseMeta(opts.meta); if (opts.temporal) params.temporal = parseTemporal(opts.temporal); @@ -452,11 +540,9 @@ function createMemoryUpdateCommand(): Command { function createMemoryDeleteCommand(): Command { return new Command("delete") .alias("rm") - .description("delete a memory by ID, or all memories under a tree path") - .argument("", "memory ID (UUIDv7) or tree path") - .option("--dry-run", "preview what would be deleted (tree mode)") - .option("-y, --yes", "skip confirmation (tree mode)") - .action(async (idOrTree: string, opts, cmd) => { + .description("delete a single memory by ID or by its tree/name path") + .argument("", "memory ID (UUIDv7) or tree/name path") + .action(async (ref: string, _opts, cmd) => { const globalOpts = cmd.optsWithGlobals(); const creds = resolveCredentials(globalOpts.server); const fmt = getOutputFormat(globalOpts); @@ -466,63 +552,95 @@ function createMemoryDeleteCommand(): Command { const client = buildMemoryClient(creds); try { - if (UUIDV7_RE.test(idOrTree)) { - // Single memory delete - const result = await client.memory.delete({ id: idOrTree }); - output(result, fmt, () => { - if (result.deleted) { - clack.log.success(`Deleted memory ${idOrTree}`); - } else { - clack.log.warn("Memory not found."); + // A UUID deletes by id; anything else is a tree/name path (the segment + // after the final '/') and deletes at most that one named memory. To + // delete a whole subtree, use `deltree`. A ref that matches nothing — + // id or path — raises NOT_FOUND (caught below), so reaching `output` + // means it was deleted. + const result = UUIDV7_RE.test(ref) + ? await client.memory.delete({ id: ref }) + : await client.memory.deleteByPath({ path: ref }); + output(result, fmt, () => { + clack.log.success(`Deleted memory ${ref}`); + }); + } catch (error) { + // A non-UUID ref that matched no single memory but has memories beneath + // it was almost certainly meant as a subtree delete — point at deltree. + if (!UUIDV7_RE.test(ref) && isAppErrorCode(error, "NOT_FOUND")) { + try { + const { count } = await client.memory.countTree({ tree: ref }); + if (count > 0) { + const noun = count === 1 ? "memory" : "memories"; + handleError( + new Error( + `No memory at '${ref}'. ${count} ${noun} exist under that tree — to delete the whole subtree run: me memory deltree ${ref}`, + ), + fmt, + ); + return; } - }); - } else { - // Tree delete — always dry-run first - const preview = await client.memory.deleteTree({ - tree: idOrTree, - dryRun: true, - }); - - if (preview.count === 0) { - output({ count: 0 }, fmt, () => { - clack.log.warn(`No memories found under '${idOrTree}'`); - }); - return; - } - - if (fmt === "text") { - console.log( - ` ${preview.count} ${preview.count === 1 ? "memory" : "memories"} will be deleted under '${idOrTree}'`, - ); + } catch { + // Couldn't probe the subtree — fall through to the original error. } + } + handleError(error, fmt); + } + }); +} - if (opts.dryRun) { - output({ dryRun: true, count: preview.count }, fmt, () => {}); - return; - } +function createMemoryDeltreeCommand(): Command { + return new Command("deltree") + .alias("rmtree") + .description("delete every memory at or under a tree path (a subtree)") + .argument("", "tree path; all memories at or under it are deleted") + .option("--dry-run", "preview what would be deleted without deleting") + .option("-y, --yes", "skip the confirmation prompt") + .action(async (tree: string, opts, cmd) => { + const globalOpts = cmd.optsWithGlobals(); + const creds = resolveCredentials(globalOpts.server); + const fmt = getOutputFormat(globalOpts); + requireMemoryAuth(creds, fmt); + requireSpace(creds, fmt); - // Confirm unless --yes - if (fmt === "text" && !opts.yes) { - const confirmed = await clack.confirm({ - message: `Delete ${preview.count} ${preview.count === 1 ? "memory" : "memories"}?`, - initialValue: false, - }); - if (clack.isCancel(confirmed) || !confirmed) { - clack.cancel("Cancelled."); - process.exit(0); - } - } + const client = buildMemoryClient(creds); - const result = await client.memory.deleteTree({ - tree: idOrTree, - dryRun: false, + try { + // Always preview first so --dry-run can NEVER delete, and so the + // confirmation prompt shows an accurate count. + const preview = await client.memory.deleteTree({ tree, dryRun: true }); + if (preview.count === 0) { + output({ count: 0 }, fmt, () => { + clack.log.warn(`No memories found under '${tree}'`); }); - output(result, fmt, () => { - clack.log.success( - `Deleted ${result.count} ${result.count === 1 ? "memory" : "memories"}`, - ); + return; + } + + const noun = preview.count === 1 ? "memory" : "memories"; + if (fmt === "text") { + console.log( + ` ${preview.count} ${noun} will be deleted under '${tree}'`, + ); + } + if (opts.dryRun) { + output({ dryRun: true, count: preview.count }, fmt, () => {}); + return; + } + if (fmt === "text" && !opts.yes) { + const confirmed = await clack.confirm({ + message: `Delete ${preview.count} ${noun}?`, + initialValue: false, }); + if (clack.isCancel(confirmed) || !confirmed) { + clack.cancel("Cancelled."); + process.exit(0); + } } + const result = await client.memory.deleteTree({ tree, dryRun: false }); + output(result, fmt, () => { + clack.log.success( + `Deleted ${result.count} ${result.count === 1 ? "memory" : "memories"}`, + ); + }); } catch (error) { handleError(error, fmt); } @@ -772,6 +890,7 @@ function toExportable( result.meta = memory.meta; } if (memory.tree) result.tree = memory.tree; + if (memory.name) result.name = memory.name; if (memory.temporal) result.temporal = memory.temporal; return result; } @@ -869,14 +988,38 @@ function createMemoryExportCommand(): Command { ); } } else { - // Write directory + // Write a directory tree mirroring the memory tree: + // //.md + // Named files get a legible filename (`.md` appended unless already + // present); unnamed ones fall back to the uuid. Distinct names can + // still map to one file on disk (`foo` vs `foo.md`, or case-insensitive + // filesystems), so `uniqueExportFilename` disambiguates by id. if (!existsSync(file)) { mkdirSync(file, { recursive: true }); } + const usedByDir = new Map>(); for (const mem of memories) { - const filename = `${mem.id}.md`; - const filepath = join(file, filename); - writeFileSync(filepath, formatMemoryAsMarkdown(mem), "utf-8"); + const treeDir = + typeof mem.tree === "string" + ? mem.tree.replace(/^\//, "") // drop the absolute leading slash + : ""; + const base = + typeof mem.name === "string" && mem.name + ? mem.name + : String(mem.id); + const dir = treeDir ? join(file, treeDir) : file; + mkdirSync(dir, { recursive: true }); + const filename = uniqueExportFilename( + dir, + base, + String(mem.id), + usedByDir, + ); + writeFileSync( + join(dir, filename), + formatMemoryAsMarkdown(mem), + "utf-8", + ); } output({ count: memories.length, directory: file }, fmt, () => { clack.log.success( @@ -1024,6 +1167,7 @@ function memorySubcommands(): Command[] { createMemorySearchCommand(), createMemoryUpdateCommand(), createMemoryDeleteCommand(), + createMemoryDeltreeCommand(), createMemoryEditCommand(), createMemoryCountCommand(), createMemoryTreeCommand(), diff --git a/packages/cli/commands/pack.test.ts b/packages/cli/commands/pack.test.ts index cf92aa9b..7083fcca 100644 --- a/packages/cli/commands/pack.test.ts +++ b/packages/cli/commands/pack.test.ts @@ -2,18 +2,20 @@ * Tests for `me pack` helpers. * * The skip-classification helper exists because `engine.memory.batchCreate` - * silently drops conflicting ids (post-#64) — pack install needs to tell - * benign re-installs (already at this version) from suspicious id collisions - * (some other pack or a non-pack memory holds the id). + * reports a row whose deterministic id already existed as `status: 'skipped'` + * — pack install needs to tell benign re-installs (already at this version) + * from suspicious id collisions (some other pack or a non-pack memory holds the + * id). The caller passes the already-known skipped ids (filtered from the + * per-row write results); a failed-chunk row never reaches `results`, so it + * can't be mis-classified here. */ import { describe, expect, test } from "bun:test"; import { classifySkips } from "./pack.ts"; describe("classifySkips", () => { - test("returns empty buckets when every requested id was inserted", () => { + test("returns empty buckets when nothing was skipped", () => { const result = classifySkips({ - requestedIds: ["a", "b", "c"], - insertedIds: ["a", "b", "c"], + skippedIds: [], existing: [], packName: "foo", packVersion: "1", @@ -24,8 +26,7 @@ describe("classifySkips", () => { test("classifies a skipped id as idempotent when same pack+version is present", () => { const result = classifySkips({ - requestedIds: ["a", "b"], - insertedIds: ["b"], + skippedIds: ["a"], existing: [{ id: "a", meta: { pack: { name: "foo", version: "1" } } }], packName: "foo", packVersion: "1", @@ -38,8 +39,7 @@ describe("classifySkips", () => { // batchCreate skipped "a" but the step-3 search didn't find it tagged // with this pack — so something else (a non-pack memory) holds the id. const result = classifySkips({ - requestedIds: ["a", "b"], - insertedIds: ["b"], + skippedIds: ["a"], existing: [], packName: "foo", packVersion: "1", @@ -50,8 +50,7 @@ describe("classifySkips", () => { test("classifies as conflict when the existing row belongs to a different pack", () => { const result = classifySkips({ - requestedIds: ["a"], - insertedIds: [], + skippedIds: ["a"], existing: [{ id: "a", meta: { pack: { name: "other", version: "1" } } }], packName: "foo", packVersion: "1", @@ -62,8 +61,7 @@ describe("classifySkips", () => { test("classifies as conflict when version differs (caller bug — stale should have been deleted)", () => { const result = classifySkips({ - requestedIds: ["a"], - insertedIds: [], + skippedIds: ["a"], existing: [{ id: "a", meta: { pack: { name: "foo", version: "0" } } }], packName: "foo", packVersion: "1", @@ -72,10 +70,9 @@ describe("classifySkips", () => { expect(result.conflict).toEqual(["a"]); }); - test("separates idempotent from conflict in a mixed batch", () => { + test("separates idempotent from conflict in a mixed skip set", () => { const result = classifySkips({ - requestedIds: ["a", "b", "c", "d"], - insertedIds: ["b"], + skippedIds: ["a", "c", "d"], existing: [ { id: "a", meta: { pack: { name: "foo", version: "1" } } }, // idempotent { id: "c", meta: { pack: { name: "other", version: "1" } } }, // conflict (other pack) @@ -90,8 +87,7 @@ describe("classifySkips", () => { test("treats malformed meta defensively as a conflict", () => { const result = classifySkips({ - requestedIds: ["a", "b", "c"], - insertedIds: [], + skippedIds: ["a", "b", "c"], existing: [ { id: "a", meta: undefined }, { id: "b", meta: { pack: "not-an-object" } }, @@ -104,10 +100,9 @@ describe("classifySkips", () => { expect(result.conflict).toEqual(["a", "b", "c"]); }); - test("preserves request order in the classification arrays", () => { + test("preserves skip order in the classification arrays", () => { const result = classifySkips({ - requestedIds: ["c", "a", "b"], - insertedIds: [], + skippedIds: ["c", "a", "b"], existing: [ { id: "a", meta: { pack: { name: "foo", version: "1" } } }, { id: "b", meta: { pack: { name: "foo", version: "1" } } }, @@ -120,13 +115,12 @@ describe("classifySkips", () => { expect(result.conflict).toEqual([]); }); - test("ignores existing rows whose ids weren't requested", () => { + test("ignores existing rows whose ids weren't skipped", () => { // The step-3 search may return rows that aren't in the new pack // (e.g. memories removed in a version bump, before step-6 deletion - // runs). Those should not count toward classification. + // runs). With nothing skipped, those extra existing rows are ignored. const result = classifySkips({ - requestedIds: ["a"], - insertedIds: ["a"], + skippedIds: [], existing: [ { id: "a", meta: { pack: { name: "foo", version: "1" } } }, { id: "removed", meta: { pack: { name: "foo", version: "1" } } }, @@ -137,38 +131,4 @@ describe("classifySkips", () => { expect(result.idempotent).toEqual([]); expect(result.conflict).toEqual([]); }); - - test("excludes failedIds from classification (failed != skipped)", () => { - // Chunk containing "b" errored — "b" never reached the server, so it - // must not be counted as either idempotent or conflict. Without the - // failedIds parameter it would have been mis-classified as conflict - // (no existing row → looks like a non-pack id collision). - const result = classifySkips({ - requestedIds: ["a", "b", "c"], - insertedIds: ["a"], - failedIds: ["b"], - existing: [{ id: "c", meta: { pack: { name: "foo", version: "1" } } }], - packName: "foo", - packVersion: "1", - }); - expect(result.idempotent).toEqual(["c"]); - expect(result.conflict).toEqual([]); - // "b" is in neither bucket — caller tracks it under `failed`. - }); - - test("handles all four categories in one classification", () => { - const result = classifySkips({ - requestedIds: ["inserted", "idem", "conflict", "failed"], - insertedIds: ["inserted"], - failedIds: ["failed"], - existing: [ - { id: "idem", meta: { pack: { name: "foo", version: "1" } } }, - { id: "conflict", meta: { pack: { name: "other", version: "1" } } }, - ], - packName: "foo", - packVersion: "1", - }); - expect(result.idempotent).toEqual(["idem"]); - expect(result.conflict).toEqual(["conflict"]); - }); }); diff --git a/packages/cli/commands/pack.ts b/packages/cli/commands/pack.ts index ac4a553a..49015ff5 100644 --- a/packages/cli/commands/pack.ts +++ b/packages/cli/commands/pack.ts @@ -233,6 +233,7 @@ function createPackInstallCommand(): Command { const createParams = memories.map((mem) => ({ id: mem.id, content: mem.content, + ...(mem.name ? { name: mem.name } : {}), meta: { ...(mem.meta ?? {}), pack: { name: packName, version: packVersion }, @@ -244,35 +245,38 @@ function createPackInstallCommand(): Command { // Chunked batch create — large packs are sliced under the // server's request-body limit, and a single failed chunk doesn't // take down its siblings (re-running install will self-heal). - const { - insertedIds, - failedIds, - errors: chunkErrors, - } = await batchCreateChunked(client, createParams); + const { results: writeResults, errors: chunkErrors } = + await batchCreateChunked(client, createParams, { + // Packs carry deterministic ids; re-installing skips rows that + // already exist rather than erroring on the raise-by-default server. + onConflict: "ignore", + }); spin?.stop("Done"); - // Post-#64 `batchCreate` returns only ids it actually inserted — - // conflicting ids are silently skipped. Classify the skips so the - // user sees benign re-installs vs real id collisions, excluding - // failed-chunk ids (those never reached the server). - const requestedIds = createParams - .map((p) => p.id) - .filter((x): x is string => typeof x === "string"); + // 'ignore' never updates, so a row is inserted, skipped, or error. + // Classify the skipped ids so the user sees benign re-installs vs real + // id collisions; 'error' rows are a failed chunk (a different bucket), + // so they can't be mis-classified as conflicts. Pack ids are always + // explicit, so inserted/skipped ids are never null. + const installed = writeResults.filter( + (r) => r.status === "inserted", + ).length; + const skippedIds = writeResults.flatMap((r) => + r.status === "skipped" && r.id !== null ? [r.id] : [], + ); const { idempotent, conflict } = classifySkips({ - requestedIds, - insertedIds, - failedIds, + skippedIds, existing: existing.results, packName, packVersion, }); - const installed = insertedIds.length; const skippedIdempotent = idempotent.length; const skippedConflict = conflict.length; const skipped = skippedIdempotent + skippedConflict; - const failed = failedIds.length; + const failed = writeResults.filter((r) => r.status === "error").length; + const failedIds = chunkErrors.flatMap((e) => e.ids); const jsonOut: Record = { pack: packName, @@ -421,49 +425,38 @@ function createPackListCommand(): Command { } // ============================================================================= -// Skip classification (post-#64 batchCreate semantics) +// Skip classification (onConflict: 'ignore' batchCreate semantics) // ============================================================================= /** - * `client.memory.batchCreate` uses `ON CONFLICT (id) DO NOTHING` server-side, - * so the returned `ids` array can be shorter than the request when conflicts - * occur. For pack install, ids that didn't land fall into three buckets: + * Pack install calls `client.memory.batchCreate` with `onConflict: 'ignore'`, + * which reports each row's `status`. The rows that came back `skipped` (their + * deterministic id already existed) fall into two buckets: * * - **idempotent**: the row is already present and tagged with this pack * name + version (a benign re-install of the same version) * - **conflict**: the id is held by something else — a different pack, * a different version, or a non-pack memory the user wrote themselves. * Surfaced as a warning so a real id collision isn't silently masked. - * - **failed (excluded here)**: the id was in a chunk that errored before - * reaching the server. Callers pass these via `failedIds` so they don't - * get mis-classified as conflicts; they're tracked separately under - * the `failed` bucket in the install output. + * + * Failed-chunk ids never reached the server, so they aren't in `skippedIds` + * and are tracked separately under the install output's `failed` bucket. * * Pure function exported for unit testing. */ export function classifySkips(args: { - requestedIds: string[]; - insertedIds: string[]; - /** - * Ids that were submitted but never reached the server because their - * containing chunk errored. Optional — if omitted, treated as empty. - */ - failedIds?: string[]; + skippedIds: string[]; existing: ReadonlyArray<{ id: string; meta?: unknown }>; packName: string; packVersion: string; }): { idempotent: string[]; conflict: string[] } { - const inserted = new Set(args.insertedIds); - const failed = new Set(args.failedIds ?? []); const existingById = new Map( args.existing.map((m) => [m.id, m.meta]), ); const idempotent: string[] = []; const conflict: string[] = []; - for (const id of args.requestedIds) { - if (inserted.has(id) || failed.has(id)) continue; - + for (const id of args.skippedIds) { const meta = existingById.get(id); const packMeta = meta && typeof meta === "object" diff --git a/packages/cli/importers/git.test.ts b/packages/cli/importers/git.test.ts index e2cea991..9cf38f12 100644 --- a/packages/cli/importers/git.test.ts +++ b/packages/cli/importers/git.test.ts @@ -172,19 +172,20 @@ function ctx( projectSlug: "demo", gitRemote: "git@github.com:org/demo.git", fileList: true, - importedAt: "2026-06-10T00:00:00.000Z", ...overrides, }; } describe("buildCommitMemory", () => { - test("builds content, meta, temporal, and a deterministic id", () => { + test("builds content, meta, temporal, name, and a timestamp-prefixed id", () => { const built = buildCommitMemory(commit(), ctx()); if ("error" in built) throw new Error(built.error); expect(built.content).toBe( "fix: a thing\n\ndetails here\n\nFiles:\n src/a.ts (+10 -2)", ); expect(built.tree).toBe("share.projects.demo.git_history"); + // The sha is the leaf name — the (tree, name) idempotency key. + expect(built.name).toBe(SHA_A); // Commit date, normalized to UTC. expect(built.temporal).toEqual({ start: "2026-01-02T01:04:06.000Z" }); expect(built.meta).toEqual({ @@ -199,17 +200,23 @@ describe("buildCommitMemory", () => { files_changed: 1, insertions: 10, deletions: 2, - imported_at: "2026-06-10T00:00:00.000Z", importer_version: "1", }); - // Deterministic: same inputs → same id; different tree → different id. + // The id is a v7 whose 48-bit prefix is the commit time (random tail), so + // commits sort by date on the id; identity/idempotency comes from the sha. + const id = built.id as string; + expect(id).toMatch( + /^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i, + ); + const tsHex = id.replace(/-/g, "").slice(0, 12); + expect(Number.parseInt(tsHex, 16)).toBe(Date.parse(commit().commitDate)); + + // Random tail → two builds of the same commit differ in id, same name. const again = buildCommitMemory(commit(), ctx()); if ("error" in again) throw new Error(again.error); - expect(again.id).toBe(built.id); - const moved = buildCommitMemory(commit(), ctx({ tree: "share.other" })); - if ("error" in moved) throw new Error(moved.error); - expect(moved.id).not.toBe(built.id); + expect(again.id).not.toBe(built.id); + expect(again.name).toBe(SHA_A); }); test("omits remote and merge marker when absent, sets them when present", () => { diff --git a/packages/cli/importers/git.ts b/packages/cli/importers/git.ts index c17ab858..e6a2c316 100644 --- a/packages/cli/importers/git.ts +++ b/packages/cli/importers/git.ts @@ -22,7 +22,7 @@ import { execFile } from "node:child_process"; import { promisify } from "node:util"; import type { MemoryCreateParams } from "@memory.build/protocol/memory"; -import { deterministicUuidV7 } from "./uuid.ts"; +import { uuidv7At } from "./uuid.ts"; const execFileAsync = promisify(execFile); @@ -285,8 +285,6 @@ export interface CommitMemoryContext { gitRemote?: string; /** Render the changed-file list into the content. */ fileList: boolean; - /** `meta.imported_at` for this run. */ - importedAt: string; } /** @@ -303,7 +301,9 @@ export function buildCommitMemory( return { error: `invalid commit date: ${commit.commitDate}` }; } - const id = deterministicUuidV7(`git:${ctx.tree}:${commit.sha}`, commitMs); + // Idempotency is keyed on (tree, name) where name is the commit sha; the id + // is a timestamp-prefixed v7 (random tail) so commits sort by date on the id. + const id = uuidv7At(commitMs); let content = commit.subject; const body = truncateUtf8(commit.body, BODY_BYTES_CAP); @@ -342,7 +342,6 @@ export function buildCommitMemory( files_changed: commit.files.length, insertions, deletions, - imported_at: ctx.importedAt, importer_version: GIT_IMPORTER_VERSION, }; if (ctx.gitRemote) meta.source_git_repo = ctx.gitRemote; @@ -350,6 +349,7 @@ export function buildCommitMemory( return { id, + name: commit.sha, content, meta, tree: ctx.tree, diff --git a/packages/cli/importers/import-transcript.test.ts b/packages/cli/importers/import-transcript.test.ts index 1dcc158d..2bd3dc05 100644 --- a/packages/cli/importers/import-transcript.test.ts +++ b/packages/cli/importers/import-transcript.test.ts @@ -28,15 +28,23 @@ const WRITE: WriteOptions = { verbose: false, }; -/** A mock engine backed by an in-memory id→row store, mimicking the server. */ +/** A mock engine backed by an in-memory store keyed on (tree, name), mimicking + * the server: named rows dedup on (tree, name), keeping the existing row's id. */ function mockEngine() { const store = new Map< - string, - { id: string; meta: Record; content: string } + string, // `${tree} ${name}` + { + id: string; + tree: string; + name: string; + meta: Record; + content: string; + } >(); const client = { memory: { - // Filter by source_session_id, order by id desc (server default), slice to limit. + // Filter by source_session_id, order by id desc (server default for + // filter-only — id encodes the message time), slice to limit. search: async (p: { meta?: Record; limit?: number }) => { const sid = p.meta?.source_session_id; const all = [...store.values()] @@ -45,33 +53,52 @@ function mockEngine() { const limit = p.limit ?? 10; return { results: all.slice(0, limit), total: all.length, limit }; }, - // The server's conditional upsert: insert new ids; replace an existing - // row when its meta value for `replaceIfMetaDiffers` differs; else skip. + // The server's content-aware replace, keyed on (tree, name): insert a new + // (tree, name) with the submitted id; for an existing slot under + // onConflict 'replace', rewrite it (KEEPING the existing row's id) when + // content or meta differ, else skip. importer_version lives in meta, so a + // version bump makes meta differ and re-renders. batchCreate: async (p: { memories: Array<{ id: string; + tree: string; + name: string; meta: Record; content: string; }>; - replaceIfMetaDiffers?: string; + onConflict?: "error" | "replace" | "ignore"; }) => { - const ids: string[] = []; - const updatedIds: string[] = []; + // One {id, status} per input, in input order (mirrors the server). + const results: Array<{ + id: string; + status: "inserted" | "updated" | "skipped"; + }> = []; for (const m of p.memories) { - const existing = store.get(m.id); + const key = `${m.tree} ${m.name}`; + const existing = store.get(key); if (!existing) { - store.set(m.id, { id: m.id, meta: m.meta, content: m.content }); - ids.push(m.id); + store.set(key, { + id: m.id, + tree: m.tree, + name: m.name, + meta: m.meta, + content: m.content, + }); + results.push({ id: m.id, status: "inserted" }); } else if ( - p.replaceIfMetaDiffers !== undefined && - existing.meta[p.replaceIfMetaDiffers] !== - m.meta[p.replaceIfMetaDiffers] + p.onConflict === "replace" && + (existing.content !== m.content || + JSON.stringify(existing.meta) !== JSON.stringify(m.meta)) ) { - store.set(m.id, { id: m.id, meta: m.meta, content: m.content }); - updatedIds.push(m.id); + store.set(key, { ...existing, meta: m.meta, content: m.content }); + // (tree, name) conflict keeps the existing row's id. + results.push({ id: existing.id, status: "updated" }); + } else { + // Existing slot, nothing differs (or not replacing) → no-op. + results.push({ id: existing.id, status: "skipped" }); } } - return { ids, updatedIds }; + return { results }; }, }, } as unknown as MemoryClient; @@ -175,8 +202,8 @@ describe("importTranscriptFile", () => { }); // The hook (importTranscriptFile) and `me import claude` (runImport) must be - // idempotent w.r.t. each other: both derive the same tree + deterministic ids - // from the same parse, so importing a session via one path and then the other + // idempotent w.r.t. each other: both derive the same (tree, name) keys from + // the same parse, so importing a session via one path and then the other // inserts nothing the second time. Guards the shared-derivation assumption. test("hook capture then `me import claude` over the same session is a no-op", async () => { const { client, store } = mockEngine(); diff --git a/packages/cli/importers/index.test.ts b/packages/cli/importers/index.test.ts index c4c4e678..a3a3bccc 100644 --- a/packages/cli/importers/index.test.ts +++ b/packages/cli/importers/index.test.ts @@ -2,56 +2,56 @@ * Tests for importer helpers in `index.ts`. */ import { describe, expect, test } from "bun:test"; -import { dedupByMemoryId } from "./index.ts"; +import { dedupBy } from "./index.ts"; -describe("dedupByMemoryId", () => { - test("returns input unchanged when all ids are unique", () => { +const byKey = (item: { key: string }) => item.key; + +describe("dedupBy", () => { + test("returns input unchanged when all keys are unique", () => { const items = [ - { memoryId: "a", value: 1 }, - { memoryId: "b", value: 2 }, - { memoryId: "c", value: 3 }, + { key: "a", value: 1 }, + { key: "b", value: 2 }, + { key: "c", value: 3 }, ]; - const result = dedupByMemoryId(items); + const result = dedupBy(items, byKey); expect(result.unique).toEqual(items); expect(result.duplicates).toBe(0); }); test("removes duplicates, keeping the first occurrence", () => { - const a1 = { memoryId: "a", value: 1 }; - const a2 = { memoryId: "a", value: 2 }; // duplicate id, different payload - const b = { memoryId: "b", value: 3 }; - const result = dedupByMemoryId([a1, a2, b]); + const a1 = { key: "a", value: 1 }; + const a2 = { key: "a", value: 2 }; // duplicate key, different payload + const b = { key: "b", value: 3 }; + const result = dedupBy([a1, a2, b], byKey); expect(result.unique).toEqual([a1, b]); expect(result.duplicates).toBe(1); }); - test("counts duplicates accurately when an id repeats more than twice", () => { - const result = dedupByMemoryId([ - { memoryId: "a" }, - { memoryId: "a" }, - { memoryId: "a" }, - { memoryId: "b" }, - ]); - expect(result.unique.map((u) => u.memoryId)).toEqual(["a", "b"]); + test("counts duplicates accurately when a key repeats more than twice", () => { + const result = dedupBy( + [{ key: "a" }, { key: "a" }, { key: "a" }, { key: "b" }], + byKey, + ); + expect(result.unique.map((u) => u.key)).toEqual(["a", "b"]); expect(result.duplicates).toBe(2); }); test("handles empty input", () => { - const result = dedupByMemoryId([]); + const result = dedupBy([] as { key: string }[], byKey); expect(result.unique).toEqual([]); expect(result.duplicates).toBe(0); }); - test("preserves insertion order across distinct ids", () => { + test("preserves insertion order across distinct keys", () => { const items = [ - { memoryId: "c" }, - { memoryId: "a" }, - { memoryId: "b" }, - { memoryId: "a" }, // dup - { memoryId: "d" }, + { key: "c" }, + { key: "a" }, + { key: "b" }, + { key: "a" }, // dup + { key: "d" }, ]; - const result = dedupByMemoryId(items); - expect(result.unique.map((u) => u.memoryId)).toEqual(["c", "a", "b", "d"]); + const result = dedupBy(items, byKey); + expect(result.unique.map((u) => u.key)).toEqual(["c", "a", "b", "d"]); expect(result.duplicates).toBe(1); }); }); diff --git a/packages/cli/importers/index.ts b/packages/cli/importers/index.ts index cfc89db6..615e1c13 100644 --- a/packages/cli/importers/index.ts +++ b/packages/cli/importers/index.ts @@ -4,25 +4,28 @@ * Each per-tool importer (claude, codex, opencode) exposes a * `discoverSessions` async generator that yields `ImportedSession` * objects. `runImport` then walks each session's `messages[]` and - * writes one memory per message, using deterministic UUIDv7s keyed - * by `(tool, sessionId, messageId)` so re-imports are idempotent. + * writes one memory per message, named `msg_` under a per-session + * tree node — so `(tree, name)` is the idempotency key and re-imports collapse + * onto the same rows. The id is a timestamp-prefixed UUIDv7 (random tail) so + * memories still sort chronologically by id. * * Reconciliation happens server-side: every planned message is submitted - * through the conditional upsert (`memory.batchCreate` with - * `replaceIfMetaDiffers: "importer_version"`) — new ids insert, rows whose - * stored `importer_version` differs are rewritten in place, and - * already-current rows are skipped, all classified from the batch - * response. No existing-state pre-fetch, so sessions of any size (including - * past the 1000-row search page) reconcile exactly. Per session that is - * ceil(n/chunk) `memory.batchCreate` calls; the live-capture hook adds one - * `memory.search` to narrow the submission to the new suffix. + * through `memory.batchCreate` with `onConflict: "replace"` — new ids insert, + * and an existing row is rewritten in place only when content/meta/temporal + * differ. The deterministic meta carries `importer_version`, so a parser change + * (version bump) makes meta differ and re-renders, while an unchanged re-import + * is a no-op; all outcomes are classified from the batch response. No + * existing-state pre-fetch, so sessions of any size (including past the + * 1000-row search page) reconcile exactly. Per session that is ceil(n/chunk) + * `memory.batchCreate` calls; the live-capture hook adds one `memory.search` + * to narrow the submission to the new suffix. */ import type { MemoryCreateParams } from "@memory.build/protocol/memory"; import { batchCreateChunked } from "../chunk.ts"; import type { MemoryClient } from "../client.ts"; import type { ProgressReporter } from "./progress.ts"; -import { SlugRegistry } from "./slug.ts"; +import { boundedUniqueLabel, normalizeSlug, SlugRegistry } from "./slug.ts"; import { renderMessageContent, synthesizeTitle } from "./transcript.ts"; import type { ConversationMessage, @@ -30,23 +33,61 @@ import type { ImporterOptions, ImporterStats, } from "./types.ts"; -import { deterministicMessageUuidV7 } from "./uuid.ts"; +import { uuidv7At } from "./uuid.ts"; /** * Version tag stored in `meta.importer_version`. Bumping this forces a - * re-render of every previously-imported message on the next run: the - * server's conditional upsert replaces any row whose stored value for - * `IMPORTER_VERSION_KEY` differs from the submitted one, so parser changes - * propagate without manual intervention. + * re-render of every previously-imported message on the next run: the new + * value changes meta, so the server's content-aware `onConflict: "replace"` + * rewrites every previously-imported row, propagating parser changes without + * manual intervention. * * Locked at "1" during pre-release iteration — bump only after the first * real release so early adopters get parser fixes without a manual wipe. */ export const IMPORTER_VERSION = "1"; -/** The meta key the server compares for the conditional replace. */ +/** Meta key carrying the importer version (provenance; a bump re-renders via the meta diff). */ const IMPORTER_VERSION_KEY = "importer_version"; +/** Max length of one ltree label (well under Postgres' per-label limit). */ +const SESSION_LABEL_MAX = 200; +/** Memory-name length cap (DB CHECK), minus the `msg_` prefix below. */ +const MESSAGE_NAME_BODY_MAX = 128 - "msg_".length; + +/** + * The ltree node for one session: `...`. + * The session id is mapped to a valid, collision-free ltree label via + * `boundedUniqueLabel` — `normalizeSlug` alone is lossy (e.g. it merges a UUID's + * dashes), so distinct session ids could otherwise share one node. Each session + * is its own node so its messages are browsable as named leaves under it. + */ +function sessionTree( + options: WriteOptions, + slug: string, + sessionId: string, +): string { + const label = boundedUniqueLabel(sessionId, normalizeSlug, SESSION_LABEL_MAX); + return `${options.treeRoot}.${slug}.${options.sessionsNodeName}.${label}`; +} + +/** + * A message's leaf name within its session node: `msg_`, mapped to + * the name charset and capped at 128 chars. `boundedUniqueLabel` appends a hash + * of the full id when the mapping is lossy or over-length, so distinct ids + * (`a/b`, `a:b`, `a_b`) don't collapse to one slot. `(tree, name)` is the + * idempotency key, so the same message always lands in the same slot across + * re-imports. + */ +function messageName(messageId: string): string { + const body = boundedUniqueLabel( + messageId, + (s) => s.replace(/[^A-Za-z0-9._-]/g, "_"), + MESSAGE_NAME_BODY_MAX, + ); + return `msg_${body}`; +} + /** * Default capture layout, shared by `me import claude` and the Claude Code capture * hook so live + imported sessions land in the same place: @@ -145,7 +186,7 @@ export async function runImport( sessionsProcessed++; const { slug, gitRoot, gitRemote } = await slugs.resolve(session.cwd); - const tree = `${writeOptions.treeRoot}.${slug}.${writeOptions.sessionsNodeName}`; + const tree = sessionTree(writeOptions, slug, session.sessionId); const outcome = await writeSession( engine, @@ -210,7 +251,7 @@ export async function importTranscriptFile( const { slug, gitRoot, gitRemote } = await new SlugRegistry().resolve( session.cwd, ); - const tree = `${options.treeRoot}.${slug}.${options.sessionsNodeName}`; + const tree = sessionTree(options, slug, session.sessionId); const plan = planSession(session, tree, slug, gitRoot, gitRemote, options); const outcome: SessionOutcome = { @@ -289,7 +330,7 @@ interface PlanResult { /** * Render + dedup a session's messages into write payloads (no RPCs). Skips * messages that render empty under the chosen mode, records bad timestamps as - * failures, and collapses events sharing a deterministic id (resume/replay + * failures, and collapses events sharing a (tree, name) (resume/replay * artefacts) so the batch can't trip the unique constraint server-side. */ function planSession( @@ -322,12 +363,6 @@ function planSession( }); continue; } - const memoryId = deterministicMessageUuidV7( - session.tool, - session.sessionId, - message.messageId, - timestampMs, - ); const meta = buildMeta( session, message, @@ -337,14 +372,19 @@ function planSession( options, ); const temporal = { start: new Date(timestampMs).toISOString() }; + const name = messageName(message.messageId); + const id = uuidv7At(timestampMs); planned.push({ message, - memoryId, - payload: { id: memoryId, content, meta, tree, temporal }, + memoryId: id, + payload: { id, name, content, meta, tree, temporal }, }); } - const dedup = dedupByMemoryId(planned); + // Dedup on (tree, name) — the idempotency key — so resume/replay artefacts + // (the same messageId twice in one file) collapse before submit. tree is + // constant within a session, so the name alone distinguishes them. + const dedup = dedupBy(planned, (p) => p.payload.name ?? ""); return { planned: dedup.unique, skipped: skipped + dedup.duplicates, @@ -397,12 +437,11 @@ async function writeSession( } /** - * Submit planned messages through the conditional upsert and fold the - * outcome into `outcome`: new ids insert, rows whose stored - * `importer_version` differs are rewritten in place (the version-bump - * re-render), and already-current rows are skipped — all classified from - * the batch response, independent of how many messages the session already - * has server-side. + * Submit planned messages through `onConflict: "replace"` and fold the outcome + * into `outcome`: new ids insert, rows whose content/meta/temporal differ are + * rewritten in place (a version bump changes meta → the re-render), and + * unchanged rows are skipped — all classified from the batch response, + * independent of how many messages the session already has server-side. * * Chunks are cut by byte budget OR count cap (see batchCreateChunked) so * each request body stays under the server's size limit; a failed chunk @@ -423,25 +462,24 @@ async function submitPlanned( return; } - const { insertedIds, updatedIds, errors } = await batchCreateChunked( + const { results, errors } = await batchCreateChunked( engine, planned.map((p) => p.payload), - { replaceIfMetaDiffers: IMPORTER_VERSION_KEY }, + { onConflict: "replace" }, ); - outcome.inserted += insertedIds.length; - outcome.updated += updatedIds.length; - let failedCount = 0; + // Per-row status: inserted (new), updated (re-rendered), skipped (unchanged — + // a content-aware replace no-op). 'error' rows are tallied via errors[] below. + for (const r of results) { + if (r.status === "inserted") outcome.inserted += 1; + else if (r.status === "updated") outcome.updated += 1; + else if (r.status === "skipped") outcome.skipped += 1; + } for (const e of errors) { - failedCount += e.itemCount; outcome.failed += e.itemCount; for (const id of e.ids) { outcome.errors.push({ messageId: id, error: e.error }); } } - // Whatever the server neither inserted, updated, nor failed already - // exists at the current importer_version. - outcome.skipped += - planned.length - insertedIds.length - updatedIds.length - failedCount; } /** Build the full meta object for one message memory. */ @@ -463,7 +501,8 @@ function buildMeta( source_project_slug: projectSlug, source_file: session.sourceFile, content_mode: options.fullTranscript ? "full_transcript" : "default", - imported_at: new Date().toISOString(), + // No per-run timestamp here: meta must be deterministic so a re-import is a + // content-aware-replace no-op (the row's created_at/updated_at carry timing). [IMPORTER_VERSION_KEY]: IMPORTER_VERSION, }; @@ -510,24 +549,27 @@ function logOutcome( } /** - * Drop items whose `memoryId` has already been seen, preserving order. - * Exported so the dedup behavior can be unit-tested without standing up - * a fake MemoryClient. Used by `writeSession` to absorb sessions whose - * JSONL has duplicate `event.uuid` entries (which would otherwise produce - * two planned memories with the same deterministic UUIDv7). + * Drop items whose `key` has already been seen, preserving order. Callers key + * on the idempotency slot: the transcript planner passes the `(tree, name)` + * key, the git importer the commit sha — so resume/replay artefacts (the same + * record twice in one batch) collapse before submit and don't trip the unique + * constraint server-side. Exported so the dedup behavior can be unit-tested + * without standing up a fake MemoryClient. */ -export function dedupByMemoryId( +export function dedupBy( items: T[], + key: (item: T) => string, ): { unique: T[]; duplicates: number } { const seen = new Set(); const unique: T[] = []; let duplicates = 0; for (const item of items) { - if (seen.has(item.memoryId)) { + const k = key(item); + if (seen.has(k)) { duplicates++; continue; } - seen.add(item.memoryId); + seen.add(k); unique.push(item); } return { unique, duplicates }; @@ -537,4 +579,4 @@ export type { ProgressReporter } from "./progress.ts"; export { createProgressReporter } from "./progress.ts"; export { SlugRegistry } from "./slug.ts"; export { synthesizeTitle } from "./transcript.ts"; -export { deterministicMessageUuidV7 } from "./uuid.ts"; +export { uuidv7At } from "./uuid.ts"; diff --git a/packages/cli/importers/slug.test.ts b/packages/cli/importers/slug.test.ts index 9f9735bd..c8d85c8b 100644 --- a/packages/cli/importers/slug.test.ts +++ b/packages/cli/importers/slug.test.ts @@ -6,7 +6,12 @@ * `undefined` and the fallback to `basename(cwd)` is exercised. */ import { describe, expect, test } from "bun:test"; -import { normalizeSlug, repoNameFromRemote, SlugRegistry } from "./slug.ts"; +import { + boundedUniqueLabel, + normalizeSlug, + repoNameFromRemote, + SlugRegistry, +} from "./slug.ts"; describe("repoNameFromRemote", () => { test("extracts the repo name from https and ssh remotes (sans .git)", () => { @@ -41,6 +46,51 @@ describe("normalizeSlug", () => { }); }); +describe("boundedUniqueLabel", () => { + // The name-charset normalizer used by messageName (dashes/dots/underscores ok). + const nameNorm = (s: string) => s.replace(/[^A-Za-z0-9._-]/g, "_"); + + test("returns a clean, fitting id unchanged (no hash suffix)", () => { + expect(boundedUniqueLabel("343a75a0-8037-4579", nameNorm, 124)).toBe( + "343a75a0-8037-4579", + ); + expect(boundedUniqueLabel("a_b", nameNorm, 124)).toBe("a_b"); + }); + + test("keeps distinct ids distinct when normalization would collapse them", () => { + // a/b, a:b, a_b all normalize to "a_b" — the hash of the original keeps + // them in three different slots. + const a = boundedUniqueLabel("a/b", nameNorm, 124); + const b = boundedUniqueLabel("a:b", nameNorm, 124); + const c = boundedUniqueLabel("a_b", nameNorm, 124); + expect(new Set([a, b, c]).size).toBe(3); + expect(c).toBe("a_b"); // already clean → unchanged + expect(a.startsWith("a_b_")).toBe(true); // lossy → disambiguated + }); + + test("caps length and stays unique when truncating", () => { + const a = boundedUniqueLabel("x".repeat(300), nameNorm, 124); + const b = boundedUniqueLabel(`${"x".repeat(299)}y`, nameNorm, 124); + expect(a.length).toBeLessThanOrEqual(124); + expect(b.length).toBeLessThanOrEqual(124); + expect(a).not.toBe(b); // share the truncated prefix but differ by hash + }); + + test("is deterministic (stable as an idempotency key)", () => { + expect(boundedUniqueLabel("a/b", nameNorm, 124)).toBe( + boundedUniqueLabel("a/b", nameNorm, 124), + ); + }); + + test("disambiguates a lossy ltree label via normalizeSlug", () => { + const dashed = boundedUniqueLabel("sess-1", normalizeSlug, 200); + const under = boundedUniqueLabel("sess_1", normalizeSlug, 200); + expect(under).toBe("sess_1"); // already a clean ltree label + expect(dashed).not.toBe(under); // "sess-1" → sess_1, disambiguated + expect(dashed.startsWith("sess_1_")).toBe(true); + }); +}); + describe("SlugRegistry", () => { test("returns unknown for undefined cwd", async () => { const reg = new SlugRegistry(); diff --git a/packages/cli/importers/slug.ts b/packages/cli/importers/slug.ts index c3567c9d..eefa566d 100644 --- a/packages/cli/importers/slug.ts +++ b/packages/cli/importers/slug.ts @@ -133,8 +133,31 @@ function getGitInfo( /** * Short hex hash (4 chars) of an arbitrary disambiguation string. */ -function shortHash(input: string): string { - return createHash("sha256").update(input).digest("hex").slice(0, 4); +function shortHash(input: string, chars = 4): string { + return createHash("sha256").update(input).digest("hex").slice(0, chars); +} + +/** + * Map an arbitrary id to a deterministic, length-bounded, collision-free label. + * + * `normalize` slugifies the id to the target charset — a lossy step, so distinct + * ids can collapse (`a/b`, `a:b`, and `a_b` all normalize to `a_b`; for an ltree + * label the dashes in a UUID likewise merge to `_`). To keep distinct ids + * distinct, a hash of the FULL original id is appended whenever normalization + * changed the id (lossy) or the result would exceed `maxLen`; an id that + * normalizes to itself and already fits is returned unchanged. Pure and stable, + * so the result is safe as part of a `(tree, name)` idempotency key. + */ +export function boundedUniqueLabel( + id: string, + normalize: (s: string) => string, + maxLen: number, +): string { + const normalized = normalize(id); + if (normalized === id && normalized.length <= maxLen) return normalized; + const suffix = `_${shortHash(id, 8)}`; + const head = normalized.slice(0, Math.max(0, maxLen - suffix.length)); + return `${head}${suffix}`; } /** diff --git a/packages/cli/importers/uuid.test.ts b/packages/cli/importers/uuid.test.ts index 5a3b126f..2e913cf0 100644 --- a/packages/cli/importers/uuid.test.ts +++ b/packages/cli/importers/uuid.test.ts @@ -1,125 +1,43 @@ /** - * Tests for deterministic UUIDv7 derivation. + * Tests for timestamp-prefixed UUIDv7 minting. */ import { describe, expect, test } from "bun:test"; -import { deterministicMessageUuidV7, deterministicUuidV7 } from "./uuid.ts"; +import { uuidv7At } from "./uuid.ts"; const UUIDV7_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; -describe("deterministicMessageUuidV7", () => { +describe("uuidv7At", () => { test("produces a valid UUIDv7", () => { - const id = deterministicMessageUuidV7( - "claude", - "session-123", - "msg-1", - 1_700_000_000_000, - ); - expect(id).toMatch(UUIDV7_RE); - }); - - test("is deterministic for the same inputs", () => { - const a = deterministicMessageUuidV7( - "claude", - "abc", - "m1", - 1_700_000_000_000, - ); - const b = deterministicMessageUuidV7( - "claude", - "abc", - "m1", - 1_700_000_000_000, - ); - expect(a).toBe(b); - }); - - test("changes when tool changes", () => { - const a = deterministicMessageUuidV7( - "claude", - "abc", - "m1", - 1_700_000_000_000, - ); - const b = deterministicMessageUuidV7( - "codex", - "abc", - "m1", - 1_700_000_000_000, - ); - expect(a).not.toBe(b); - }); - - test("changes when sessionId changes", () => { - const a = deterministicMessageUuidV7( - "claude", - "abc", - "m1", - 1_700_000_000_000, - ); - const b = deterministicMessageUuidV7( - "claude", - "xyz", - "m1", - 1_700_000_000_000, - ); - expect(a).not.toBe(b); - }); - - test("changes when messageId changes", () => { - const a = deterministicMessageUuidV7( - "claude", - "abc", - "m1", - 1_700_000_000_000, - ); - const b = deterministicMessageUuidV7( - "claude", - "abc", - "m2", - 1_700_000_000_000, - ); - expect(a).not.toBe(b); + expect(uuidv7At(1_700_000_000_000)).toMatch(UUIDV7_RE); }); test("encodes the timestamp in the leading 48 bits", () => { const ts = 1_700_000_000_000; - const id = deterministicMessageUuidV7("claude", "abc", "m1", ts); - // Strip dashes, take first 12 hex chars = 48 bits = 6 bytes. - const tsHex = id.replace(/-/g, "").slice(0, 12); - const decoded = Number.parseInt(tsHex, 16); - expect(decoded).toBe(ts); + const tsHex = uuidv7At(ts).replace(/-/g, "").slice(0, 12); + expect(Number.parseInt(tsHex, 16)).toBe(ts); }); test("version nibble is 7 and variant bits are 10", () => { - const id = deterministicMessageUuidV7( - "opencode", - "ses_123", - "msg_1", - 1_700_000_000_000, - ); - // Position 14 (after first two dashes) = version nibble. + const id = uuidv7At(1_700_000_000_000); expect(id.charAt(14)).toBe("7"); - // Position 19 = variant high nibble; top 2 bits must be 10 → hex 8/9/a/b. expect(["8", "9", "a", "b"]).toContain(id.charAt(19)); }); - test("equals deterministicUuidV7 over the tool:session:message key", () => { - // The message variant is a thin wrapper; the key format is load-bearing - // for ids already written to engines, so lock it down. + test("is random: two calls at the same timestamp differ but share the prefix", () => { const ts = 1_700_000_000_000; - expect(deterministicMessageUuidV7("claude", "abc", "m1", ts)).toBe( - deterministicUuidV7("claude:abc:m1", ts), + const a = uuidv7At(ts); + const b = uuidv7At(ts); + expect(a).not.toBe(b); + // Same 48-bit timestamp prefix → they sort together by time. + expect(a.replace(/-/g, "").slice(0, 12)).toBe( + b.replace(/-/g, "").slice(0, 12), ); }); -}); -describe("deterministicUuidV7", () => { - test("namespaced keys produce distinct ids at the same timestamp", () => { - const ts = 1_700_000_000_000; - const a = deterministicUuidV7("git:share.projects.x.git_history:abc", ts); - const b = deterministicUuidV7("git:share.projects.y.git_history:abc", ts); - expect(a).toMatch(UUIDV7_RE); - expect(a).not.toBe(b); + test("later timestamps sort after earlier ones (lexicographic by id)", () => { + const earlier = uuidv7At(1_700_000_000_000); + const later = uuidv7At(1_700_000_001_000); + expect(later > earlier).toBe(true); }); }); diff --git a/packages/cli/importers/uuid.ts b/packages/cli/importers/uuid.ts index 2077921e..86c56c2f 100644 --- a/packages/cli/importers/uuid.ts +++ b/packages/cli/importers/uuid.ts @@ -1,34 +1,27 @@ /** - * Deterministic UUIDv7 derivation for idempotent imports. + * UUIDv7 minting for importers. * - * We need stable UUIDs so that re-importing the same record collides - * with the existing row in the database and becomes a no-op. Regular - * UUIDv7 is random, so we derive a deterministic variant: + * Importers key idempotency on `(tree, name)` (a source-coordinate name like + * `msg_` or a commit ``), not the id — so the id no longer + * needs to be derived from the source. It only needs to: * - * - 48 bits: Unix ms timestamp (record timestamp) — keeps chronological sort - * - 4 bits: version = 7 - * - 12 bits: rand_a ← SHA-256(key), bits 0..11 - * - 2 bits: variant = 10 - * - 62 bits: rand_b ← SHA-256(key), bits 12..73 + * - pass the engine's `uuid_extract_version(id) = 7` check, and + * - carry the record's timestamp in the 48-bit prefix so memories sort + * chronologically by id (the import watermark orders newest-first by id). * - * The result passes the `uuid_extract_version(id) = 7` check in the engine's - * memory schema, sorts by record time, and is stable across re-imports of - * the same source data. + * So we mint a v7 with the record timestamp in the prefix and a random tail. + * A re-import mints a *different* id for the same record, but the server dedups + * on `(tree, name)` and keeps the existing row's id, so identity stays stable. */ -import { createHash } from "node:crypto"; -import type { SourceTool } from "./types.ts"; /** - * Compute a deterministic UUIDv7 from an identity `key` and a timestamp. - * - * Same inputs always return the same UUID; different inputs produce - * different UUIDs (cryptographically, with SHA-256). Each importer owns - * its key format (messages: `tool:sessionId:messageId`; git commits: - * `git::`) — keys must be namespaced so importers can't collide. + * Mint a UUIDv7 whose 48-bit timestamp prefix is `timestampMs` and whose low + * bits are random. Two calls with the same timestamp return different ids that + * share a prefix (so they sort together, by time). */ -export function deterministicUuidV7(key: string, timestampMs: number): string { - // 16 bytes = 128 bits. - const bytes = new Uint8Array(16); +export function uuidv7At(timestampMs: number): string { + // 16 random bytes; we overwrite the timestamp prefix and the version/variant. + const bytes = crypto.getRandomValues(new Uint8Array(16)); // Bytes 0..5 (48 bits): timestamp in ms, big-endian. const ts = Math.max(0, Math.floor(timestampMs)); @@ -39,38 +32,14 @@ export function deterministicUuidV7(key: string, timestampMs: number): string { bytes[4] = Math.floor(ts / 2 ** 8) & 0xff; bytes[5] = ts & 0xff; - // SHA-256 over the key gives 32 bytes of deterministic pseudo-random. - // We only need 74 bits (12 + 62) so 10 bytes is plenty. - const digest = createHash("sha256").update(key, "utf8").digest(); - - // Bytes 6..7: version (4 bits = 0x7) + rand_a (12 bits). - const randA = ((digest[0] ?? 0) << 8) | (digest[1] ?? 0); - bytes[6] = 0x70 | ((randA >> 8) & 0x0f); - bytes[7] = randA & 0xff; - - // Byte 8: variant (2 bits = 0b10) + top 6 bits of rand_b. - // Bytes 9..15: remaining 56 bits of rand_b (from digest[3..10]). - bytes[8] = 0x80 | ((digest[2] ?? 0) & 0x3f); - for (let i = 0; i < 7; i++) { - bytes[9 + i] = digest[3 + i] ?? 0; - } + // Byte 6: version (4 bits = 0x7) over the random nibble. + bytes[6] = 0x70 | ((bytes[6] ?? 0) & 0x0f); + // Byte 8: variant (2 bits = 0b10) over the random bits. + bytes[8] = 0x80 | ((bytes[8] ?? 0) & 0x3f); return bytesToUuid(bytes); } -/** - * Compute a deterministic UUIDv7 from `(tool, sessionId, messageId, timestampMs)`. - * The message-import key format; see `deterministicUuidV7`. - */ -export function deterministicMessageUuidV7( - tool: SourceTool, - sessionId: string, - messageId: string, - timestampMs: number, -): string { - return deterministicUuidV7(`${tool}:${sessionId}:${messageId}`, timestampMs); -} - /** Format 16 bytes as a canonical UUID string. */ function bytesToUuid(bytes: Uint8Array): string { const hex: string[] = []; diff --git a/packages/cli/mcp/server.ts b/packages/cli/mcp/server.ts index 9328dcb8..3ca9637c 100644 --- a/packages/cli/mcp/server.ts +++ b/packages/cli/mcp/server.ts @@ -17,7 +17,10 @@ import { CLIENT_VERSION } from "../../../version"; import { batchCreateChunked } from "../chunk.ts"; import type { MemoryClient } from "../client.ts"; import { createMemoryClient } from "../client.ts"; -import { formatMemoryAsMarkdown } from "../commands/memory.ts"; +import { + formatMemoryAsMarkdown, + uniqueExportFilename, +} from "../commands/memory.ts"; import { detectFormatFromExtension, type ImportFormat, @@ -74,7 +77,14 @@ Docs: ${docUrl("me_memory_create")}`, tree: z .string() .describe( - "Hierarchical path where the memory is stored (required — choose deliberately). Most memories should go under `share` (e.g. `share.work.projects`) so the rest of the space can see them. Use `~` — your private home (e.g. `~.notes`) — only for memories that must stay private to you.", + "Hierarchical path where the memory is stored (required — choose deliberately). Most memories should go under `share` (e.g. `/share/work/projects`) so the rest of the space can see them. Use `~` — your private home (e.g. `~/notes`) — only for memories that must stay private to you.", + ), + name: z + .string() + .optional() + .nullable() + .describe( + 'Optional filename-like leaf name, unique within the tree (e.g. "jwt-rotation", "config.yaml"). Lets you address the memory later as `tree/name` and dedupe re-tells.', ), temporal: z .object({ @@ -90,6 +100,13 @@ Docs: ${docUrl("me_memory_create")}`, .optional() .nullable() .describe("Time range for the memory"), + on_conflict: z + .enum(["error", "replace", "ignore"]) + .optional() + .nullable() + .describe( + "On a conflict on the idempotency key (a named memory's tree+name, which takes precedence over id; else the id): 'error' (default) fails, 'replace' overwrites it in place, 'ignore' keeps the existing one.", + ), }, annotations: { title: "Create Memory", @@ -104,12 +121,14 @@ Docs: ${docUrl("me_memory_create")}`, content: args.content, meta: args.meta ?? undefined, tree: args.tree, + name: args.name ?? undefined, temporal: args.temporal ? { start: args.temporal.start, end: args.temporal.end ?? undefined, } : undefined, + onConflict: args.on_conflict ?? undefined, }); return { content: [ @@ -309,6 +328,41 @@ Docs: ${docUrl("me_memory_get")}`, }, ); + // me_memory_get_by_path + server.registerTool( + "me_memory_get_by_path", + { + title: "Get Memory by Path", + description: `Retrieve a single named memory by its tree/name path. + +The last path segment is the name; the rest is the tree — e.g. "/share/auth/jwt-rotation" is the memory named "jwt-rotation" under "/share/auth". NOT_FOUND if no such named memory exists. Use me_memory_get when you have the UUID. + +Docs: ${docUrl("me_memory_get_by_path")}`, + inputSchema: { + path: z + .string() + .min(1) + .describe( + 'tree/name path, e.g. "/share/auth/jwt-rotation" or "~/notes/todo"', + ), + }, + annotations: { + title: "Get Memory by Path", + readOnlyHint: true, + destructiveHint: false, + idempotentHint: true, + }, + }, + async (args) => { + const result = await client.memory.getByPath({ path: args.path }); + return { + content: [ + { type: "text" as const, text: JSON.stringify(result, null, 2) }, + ], + }; + }, + ); + // me_memory_update server.registerTool( "me_memory_update", @@ -336,6 +390,13 @@ Docs: ${docUrl("me_memory_update")}`, .optional() .nullable() .describe("New tree path (omit or null to keep existing)"), + name: z + .string() + .optional() + .nullable() + .describe( + 'New leaf name — renames the memory; pass an empty string "" to clear the name (omit or null to keep existing).', + ), temporal: z .object({ start: z.string().describe("ISO timestamp for start of time range"), @@ -364,6 +425,8 @@ Docs: ${docUrl("me_memory_update")}`, content: args.content ?? undefined, meta: args.meta ?? undefined, tree: args.tree ?? undefined, + // "" clears the name; a non-empty value renames; null/omit keeps it. + name: args.name === "" ? null : (args.name ?? undefined), temporal: args.temporal ? { start: args.temporal.start, @@ -409,6 +472,39 @@ Docs: ${docUrl("me_memory_delete")}`, }, ); + // me_memory_delete_by_path + server.registerTool( + "me_memory_delete_by_path", + { + title: "Delete Memory by Path", + description: `Permanently remove a single named memory by its tree/name path (e.g. "/share/auth/jwt-rotation"). + +Irreversible. Deletes only that one named memory — use me_memory_delete_tree to remove a whole subtree. + +Docs: ${docUrl("me_memory_delete_by_path")}`, + inputSchema: { + path: z + .string() + .min(1) + .describe('tree/name path, e.g. "/share/auth/jwt-rotation"'), + }, + annotations: { + title: "Delete Memory by Path", + readOnlyHint: false, + destructiveHint: true, + idempotentHint: true, + }, + }, + async (args) => { + const result = await client.memory.deleteByPath({ path: args.path }); + return { + content: [ + { type: "text" as const, text: JSON.stringify(result, null, 2) }, + ], + }; + }, + ); + // me_memory_delete_tree server.registerTool( "me_memory_delete_tree", @@ -658,12 +754,12 @@ Docs: ${docUrl("me_memory_import")}`, title: "Import Memories", readOnlyHint: false, destructiveHint: false, - // Server-side `ON CONFLICT (id) DO NOTHING` makes repeat calls with - // the same explicit ids land the engine in the same state. With - // chunking, a partial-failure call can be retried safely: ids - // already inserted are skipped, ids in failed chunks are - // re-attempted, and the final state converges to "all submitted - // ids present" once at least one call gets each chunk through. + // Import passes `onConflict: 'ignore'`, so repeat calls with the same + // ids (or (tree, name) slots) land the engine in the same state. With + // chunking, a partial-failure call can be retried safely: rows already + // present are skipped, ids in failed chunks are re-attempted, and the + // final state converges to "all submitted rows present" once at least + // one call gets each chunk through. idempotentHint: true, }, }, @@ -673,6 +769,7 @@ Docs: ${docUrl("me_memory_import")}`, content: string; tree: string; id?: string; + name?: string; meta?: Record; temporal?: { start: string; end?: string }; }> = []; @@ -717,6 +814,7 @@ Docs: ${docUrl("me_memory_import")}`, // tree is required on the wire; default bare records to `share`. tree: mem.tree ?? SHARE_NAMESPACE, ...(mem.id ? { id: mem.id } : {}), + ...(mem.name ? { name: mem.name } : {}), ...(mem.meta ? { meta: mem.meta } : {}), ...(mem.temporal ? { temporal: mem.temporal } : {}), }); @@ -736,6 +834,7 @@ Docs: ${docUrl("me_memory_import")}`, // tree is required on the wire; default bare records to `share`. tree: mem.tree ?? SHARE_NAMESPACE, ...(mem.id ? { id: mem.id } : {}), + ...(mem.name ? { name: mem.name } : {}), ...(mem.meta ? { meta: mem.meta } : {}), ...(mem.temporal ? { temporal: mem.temporal } : {}), }); @@ -749,6 +848,7 @@ Docs: ${docUrl("me_memory_import")}`, // tree is required on the wire; default bare records to `share`. tree: mem.tree ?? SHARE_NAMESPACE, ...(mem.id ? { id: mem.id } : {}), + ...(mem.name ? { name: mem.name } : {}), ...(mem.meta ? { meta: mem.meta } : {}), ...(mem.temporal ? { temporal: mem.temporal } : {}), }); @@ -757,16 +857,21 @@ Docs: ${docUrl("me_memory_import")}`, throw new Error("Either path or content is required."); } - const explicitIds = allMemories - .map((m) => m.id) - .filter((id): id is string => typeof id === "string"); - // Chunked batch create — large imports are sliced under the // server's request-body limit, and a single failed chunk doesn't // take down the rest of the import. - const { insertedIds, failedIds, errors } = await batchCreateChunked( + const { results: writeResults, errors } = await batchCreateChunked( client, allMemories, + // Re-importing the same content is a no-op: skip rows whose + // idempotency key ((tree, name), else id) already exists. + { onConflict: "ignore" }, + ); + + // inserted/skipped rows come from a successful chunk, so their id is + // always present; flatMap drops the null only TS insists on. + const insertedIds = writeResults.flatMap((r) => + r.status === "inserted" && r.id !== null ? [r.id] : [], ); // Throw only on total failure — the agent should see partial-success @@ -779,15 +884,15 @@ Docs: ${docUrl("me_memory_import")}`, ); } - // Server-side `ON CONFLICT (id) DO NOTHING` may silently drop - // duplicate ids; surface those so the caller can investigate. - // Failed-chunk ids never reached the server, so they're not - // skipped — they're reported separately under `failed`/`errors`. - const insertedSet = new Set(insertedIds); - const failedSet = new Set(failedIds); - const skippedIds = explicitIds.filter( - (id) => !insertedSet.has(id) && !failedSet.has(id), + // With `onConflict: 'ignore'` the server skips rows whose idempotency + // key already exists; surface their stored ids so the caller can + // investigate. Per-row status counts named-but-id-less skips too. 'error' + // rows are a failed chunk that never reached the server — counted under + // `failed`, with messages in `errors`. + const skippedIds = writeResults.flatMap((r) => + r.status === "skipped" && r.id !== null ? [r.id] : [], ); + const failed = writeResults.filter((r) => r.status === "error").length; return { content: [ @@ -797,7 +902,7 @@ Docs: ${docUrl("me_memory_import")}`, { imported: insertedIds.length, skipped: skippedIds.length, - failed: failedIds.length, + failed, ids: insertedIds, skippedIds, errors, @@ -911,6 +1016,7 @@ Docs: ${docUrl("me_memory_export")}`, ? { meta: r.meta } : {}), ...(r.tree ? { tree: r.tree } : {}), + ...(r.name ? { name: r.name } : {}), ...(r.temporal ? { temporal: r.temporal } : {}), })); @@ -924,10 +1030,30 @@ Docs: ${docUrl("me_memory_export")}`, } const stat = statSync(resolved); if (stat.isDirectory()) { + // Mirror the memory tree: //.md. + // Distinct names can map to one file on disk (`foo` vs `foo.md`, or a + // case-insensitive filesystem), so disambiguate by id on a clash. + const usedByDir = new Map>(); for (const mem of memories) { - const filename = `${mem.id}.md`; - const filepath = join(resolved, filename); - writeFileSync(filepath, formatMemoryAsMarkdown(mem), "utf-8"); + const treeDir = + typeof mem.tree === "string" ? mem.tree.replace(/^\//, "") : ""; + const base = + typeof mem.name === "string" && mem.name + ? mem.name + : String(mem.id); + const dir = treeDir ? join(resolved, treeDir) : resolved; + mkdirSync(dir, { recursive: true }); + const fname = uniqueExportFilename( + dir, + base, + String(mem.id), + usedByDir, + ); + writeFileSync( + join(dir, fname), + formatMemoryAsMarkdown(mem), + "utf-8", + ); } return { content: [ diff --git a/packages/cli/parsers/import.test.ts b/packages/cli/parsers/import.test.ts index 1b338123..ceffdf85 100644 --- a/packages/cli/parsers/import.test.ts +++ b/packages/cli/parsers/import.test.ts @@ -2,7 +2,7 @@ * Unit tests for memory import parsers. */ import { describe, expect, test } from "bun:test"; -import { parseMarkdown, parseYaml } from "./index.ts"; +import { parseJson, parseMarkdown, parseYaml } from "./index.ts"; describe("memory import temporal parsing", () => { test("accepts YAML temporal objects emitted by export", () => { @@ -66,3 +66,29 @@ Exported markdown memory ]); }); }); + +describe("memory import name parsing", () => { + test("passes a filename-like name through JSON", () => { + expect( + parseJson('{"content":"x","tree":"share/auth","name":"jwt-rotation"}'), + ).toEqual([{ content: "x", tree: "share/auth", name: "jwt-rotation" }]); + }); + + test("passes a name through YAML and Markdown frontmatter", () => { + expect(parseYaml("content: x\nname: config.yaml\n")).toEqual([ + { content: "x", name: "config.yaml" }, + ]); + expect(parseMarkdown("---\nname: README.md\n---\n\nbody\n")).toEqual([ + { content: "body", name: "README.md" }, + ]); + }); + + test("rejects names with a slash or other invalid characters", () => { + expect(() => parseJson('{"content":"x","name":"a/b"}')).toThrow( + /Invalid name/, + ); + expect(() => parseJson('{"content":"x","name":".hidden"}')).toThrow( + /Invalid name/, + ); + }); +}); diff --git a/packages/cli/parsers/index.ts b/packages/cli/parsers/index.ts index 551ff086..c7b9ba67 100644 --- a/packages/cli/parsers/index.ts +++ b/packages/cli/parsers/index.ts @@ -14,6 +14,8 @@ import { parseYaml } from "./yaml.ts"; export interface ParsedMemory { id?: string; content: string; + /** Optional filename-like leaf slug, unique within its tree. */ + name?: string; meta?: Record; tree?: string; temporal?: { start: string; end?: string }; diff --git a/packages/cli/parsers/validation.ts b/packages/cli/parsers/validation.ts index c225ae50..ebade5e0 100644 --- a/packages/cli/parsers/validation.ts +++ b/packages/cli/parsers/validation.ts @@ -8,6 +8,10 @@ import type { ImportFormat, ParsedMemory } from "./index.ts"; const UUIDV7_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; +// Mirrors `memoryNameSchema` in @memory.build/protocol (kept literal here so the +// parsers stay Zod-free): a filename-like leaf slug, 1–128 chars, no slashes. +const MEMORY_NAME_RE = /^[A-Za-z0-9][A-Za-z0-9._-]*$/; + /** * Validate a memory object from parsed input. */ @@ -44,6 +48,20 @@ export function validateMemoryObject( } } + // Validate name if present + if (record.name !== undefined && record.name !== null) { + if ( + typeof record.name !== "string" || + record.name.length === 0 || + record.name.length > 128 || + !MEMORY_NAME_RE.test(record.name) + ) { + throw new Error( + `Invalid name: must be a filename-like slug (letters, digits, '.', '-', '_'; no leading '.'/'-'), 1–128 chars${inLoc}`, + ); + } + } + // Validate meta if present if (record.meta !== undefined) { if ( @@ -65,6 +83,9 @@ export function validateMemoryObject( return { content: record.content, ...(record.id !== undefined ? { id: record.id as string } : {}), + ...(record.name !== undefined && record.name !== null + ? { name: record.name as string } + : {}), ...(record.meta !== undefined ? { meta: record.meta as Record } : {}), diff --git a/packages/client/memory.ts b/packages/client/memory.ts index c966c2ff..259b0a2e 100644 --- a/packages/client/memory.ts +++ b/packages/client/memory.ts @@ -23,10 +23,12 @@ import type { MemoryCountTreeParams, MemoryCountTreeResult, MemoryCreateParams, + MemoryDeleteByPathParams, MemoryDeleteParams, MemoryDeleteResult, MemoryDeleteTreeParams, MemoryDeleteTreeResult, + MemoryGetByPathParams, MemoryGetParams, MemoryMoveParams, MemoryMoveResult, @@ -102,8 +104,10 @@ export interface MemoryNamespace { params: MemoryBatchCreateParams, ): Promise; get(params: MemoryGetParams): Promise; + getByPath(params: MemoryGetByPathParams): Promise; update(params: MemoryUpdateParams): Promise; delete(params: MemoryDeleteParams): Promise; + deleteByPath(params: MemoryDeleteByPathParams): Promise; search(params: MemorySearchParams): Promise; tree(params?: MemoryTreeParams): Promise; copy(params: MemoryCopyParams): Promise; @@ -196,8 +200,10 @@ export function createMemoryClient( create: (p) => writeRpc("memory.create", p), batchCreate: (p) => writeRpc("memory.batchCreate", p), get: (p) => readRpc("memory.get", p), + getByPath: (p) => readRpc("memory.getByPath", p), update: (p) => writeRpc("memory.update", p), delete: (p) => writeRpc("memory.delete", p), + deleteByPath: (p) => writeRpc("memory.deleteByPath", p), search: (p) => readRpc("memory.search", p), tree: (p) => readRpc("memory.tree", p ?? {}), copy: (p) => writeRpc("memory.copy", p), diff --git a/packages/database/space/migrate/idempotent/001_memory.sql b/packages/database/space/migrate/idempotent/001_memory.sql index 0c9da95b..1e520f5f 100644 --- a/packages/database/space/migrate/idempotent/001_memory.sql +++ b/packages/database/space/migrate/idempotent/001_memory.sql @@ -66,11 +66,11 @@ $func$ language sql immutable strict security invoker ------------------------------------------------------------------------------- -- get memory ------------------------------------------------------------------------------- --- Removing the `default null` from `_id` changes the parameter defaults, which --- create-or-replace cannot do ("cannot remove parameter defaults from existing --- function"). Drop the old definition only when it still carries a default --- (pronargdefaults > 0); when already current — or absent — skip the drop so it --- isn't churned every migration run. The create-or-replace below then recreates it. +-- get_memory gained a `name` return column (a return-type change, which +-- create-or-replace cannot make → 42P13 "cannot change return type"). Drop a +-- prior definition only when it lacks `name` among its columns — this also +-- covers the historical `_id default null` variant — a no-op on fresh schemas +-- and once current. The create-or-replace below then recreates it. do $$ begin if exists ( @@ -79,7 +79,7 @@ do $$ begin join pg_namespace n on n.oid = p.pronamespace where n.nspname = '{{schema}}' and p.proname = 'get_memory' - and p.pronargdefaults > 0 + and not ('name' = any(coalesce(p.proargnames, array[]::text[]))) ) then drop function {{schema}}.get_memory(jsonb, uuid); end if; @@ -94,6 +94,7 @@ returns table , meta jsonb , temporal tstzrange , content text +, name text , created_at timestamptz , updated_at timestamptz , has_embedding bool @@ -105,6 +106,7 @@ as $func$ , m.meta , m.temporal , m.content + , m.name , m.created_at , m.updated_at , m.embedding is not null @@ -115,33 +117,139 @@ $func$ language sql stable strict rows 1 security invoker set search_path to pg_catalog, {{schema}}, public, pg_temp ; +------------------------------------------------------------------------------- +-- resolve memory id +-- +-- Translate a `(tree, name)` reference to the memory's id, gated on read access +-- (level 1) so a non-reader can't probe existence. Returns null when there is +-- no such named memory or the caller can't read it. The RPC layer resolves a +-- `folder/name` address to an id with this, then calls get/patch/delete by id. +------------------------------------------------------------------------------- +create or replace function {{schema}}.resolve_memory_id +( _tree_access jsonb +, _tree ltree +, _name text +) +returns uuid +as $func$ + select m.id + from {{schema}}.memory m + where m.tree = _tree + and m.name = _name + and {{schema}}.has_tree_access(_tree_access, m.tree, 1) +$func$ language sql stable strict security invoker +set search_path to pg_catalog, {{schema}}, public, pg_temp +; + +------------------------------------------------------------------------------- +-- raise_conflict +-- +-- Raises a unique_violation (23505 → CONFLICT at the RPC boundary). Called from +-- the create path's ON CONFLICT ... WHERE so that a conflict on the idempotency +-- key (the explicit id, or the (tree, name) slot) under the default +-- _on_conflict ('error') is a hard error rather than a silent skip (the +-- 'replace'/'ignore' arms short-circuit before reaching here). Returns boolean +-- only so it can sit in a WHERE expression; +-- it never actually returns. +------------------------------------------------------------------------------- +create or replace function {{schema}}.raise_conflict() +returns boolean +as $func$ +begin + raise exception 'memory already exists (id or tree/name conflict)' + using errcode = 'unique_violation'; +end; +$func$ language plpgsql volatile +set search_path to pg_catalog, {{schema}}, public, pg_temp +; + +------------------------------------------------------------------------------- +-- raise_no_write_access +-- +-- Raises insufficient_privilege (→ FORBIDDEN at the RPC boundary). Sits in the +-- create path's ON CONFLICT ... WHERE for the case where an explicit-id row +-- collides with an EXISTING row in a tree the caller can't write: under +-- 'replace' that replace can't be performed, so it's a hard error rather than a +-- silent no-op. Returns boolean only so it can sit in a WHERE expression; it +-- never actually returns. +------------------------------------------------------------------------------- +create or replace function {{schema}}.raise_no_write_access() +returns boolean +as $func$ +begin + raise exception 'insufficient tree access' + using errcode = 'insufficient_privilege'; +end; +$func$ language plpgsql volatile +set search_path to pg_catalog, {{schema}}, public, pg_temp +; + ------------------------------------------------------------------------------- -- batch create memory -- --- The canonical memory insert: one set-based statement for a whole batch +-- The canonical memory insert: one set-based call for a whole batch -- (create_memory below is a one-row wrapper). Parallel arrays, aligned by --- position, carry the rows. Per-row, on a duplicate explicit id the outcome --- depends on _replace_if_meta_differs: --- - null (default): skip — the existing row is left untouched. --- - a meta key name: the existing row is REPLACED (tree/meta/temporal/ --- content) when its meta->>key value differs from the new record's, and --- skipped when it matches. Deterministic-id importers use this to push --- re-renders by bumping a version value in meta (importer_version). --- The replace arm additionally requires write access on the EXISTING --- row's tree; without it the row is silently skipped (not raised, unlike --- patch_memory) so one inaccessible row can't fail a whole batch. +-- position, carry the rows; _names is optional. The idempotency key — name +-- takes precedence over id: +-- - NAMED (name present, with OR without an explicit id) → dedup on +-- (tree, name). A supplied id is used only as the row's identity on INSERT +-- (importers mint a timestamp-prefixed v7 for chronological sort); on a +-- (tree, name) conflict the existing row — and its id — is kept. +-- - UNNAMED with an explicit id → dedup on the id (import/export identity). +-- - UNNAMED, no id → anonymous; always inserts. +-- On a conflict against that key the action is _on_conflict: +-- - 'replace' → replace in place, but only when content/meta/temporal differ +-- (a no-op when identical, so a re-import is idempotent; an +-- importer-version bump re-renders because the version lives in +-- meta, so meta differs — this subsumes the old +-- replaceIfMetaDiffers override) +-- - 'ignore' → skip, leaving the existing row (insert-if-absent) +-- - 'error' (default) → RAISE unique_violation (→ CONFLICT) +-- (An id-keyed replace also requires write access on the EXISTING row's tree, +-- since an id can move the row across trees; a (tree, name) replace stays in +-- the same tree, covered by the up-front check.) The (tree, name) unique index +-- is enforced on every path. `_on_conflict` governs the row's OWN idempotency +-- key; a NAMED row whose explicit id happens to collide with a DIFFERENT row's +-- id still raises a pk unique_violation regardless of 'ignore'/'replace' (the +-- id is taken) — importers mint random-tailed ids, so this never bites them. +-- +-- Returns ONE row per input, in input order: (ord, id, status) where status is +-- 'inserted' | 'updated' | 'skipped' and id is the row's stored id (for a +-- skip/update on a (tree, name) key that is the EXISTING row's id, which may +-- differ from a submitted id). So a caller can map every result back to its +-- input by ord and see exactly what happened. Embedding columns are never set +-- here; the update trigger re-embeds only on content change, so a meta-only +-- replace does not re-embed. -- --- Returns one row (id, inserted) per insert/replace — inserted distinguishes --- a fresh insert (true, xmax = 0) from a replace (false); skipped rows are --- absent. The target-tree access check is all-or-nothing up front (one bad --- row raises before anything is written), and an explicit id repeated WITHIN --- the batch collapses to its first occurrence (a single INSERT cannot touch --- the same row twice); later occurrences are skipped. +-- A duplicate idempotency key WITHIN one batch is rejected up front +-- (invalid_parameter_value): a repeated explicit id, or a repeated (tree, name). +-- The caller can't express two outcomes for one key, and splitting the work +-- into per-key partitions would otherwise miss an id shared across a named and +-- an unnamed row. Target-tree write access is all-or-nothing up front. -- --- Embedding columns are never set here: the update triggers invalidate and --- re-enqueue the embedding only when content actually changed, so a --- meta-only replace does not re-embed. +-- The return type changed from (id, inserted) to (ord, id, status), which +-- create-or-replace cannot make (42P13). The plain drops first remove older +-- arg-signature overloads (pre-name 7-arg, _replace_if_meta_differs 9-arg), so +-- the only batch_create_memory that can remain is the current 8-arg; the +-- guarded do-block then drops THAT only when it still returns the old shape +-- (lacks `status`), so it doesn't churn the live function every boot. No-op on +-- fresh schemas and once current. ------------------------------------------------------------------------------- +drop function if exists {{schema}}.batch_create_memory(jsonb, uuid[], ltree[], text[], jsonb, tstzrange[], text); +drop function if exists {{schema}}.batch_create_memory(jsonb, uuid[], ltree[], text[], jsonb, tstzrange[], text, text[], text); +do $$ begin + if exists + ( + select 1 + from pg_proc p + join pg_namespace n on n.oid = p.pronamespace + where n.nspname = '{{schema}}' + and p.proname = 'batch_create_memory' + and not ('status' = any(coalesce(p.proargnames, array[]::text[]))) + ) then + drop function {{schema}}.batch_create_memory(jsonb, uuid[], ltree[], text[], jsonb, tstzrange[], text[], text); + end if; +end $$; create or replace function {{schema}}.batch_create_memory ( _tree_access jsonb , _ids uuid[] -- null elements get a generated uuidv7 @@ -149,12 +257,13 @@ create or replace function {{schema}}.batch_create_memory , _contents text[] , _metas jsonb -- json ARRAY of meta objects; null elements default to '{}' , _temporals tstzrange[] -, _replace_if_meta_differs text default null +, _names text[] default null -- per-row leaf name; null = unnamed +, _on_conflict text default 'error' -- 'error' | 'replace' | 'ignore' ) -returns table (id uuid, inserted boolean) +returns table (ord bigint, id uuid, status text) -- status: inserted | updated | skipped as $func$ --- The out columns (id, inserted) shadow table columns inside the body; the --- body never reads them as variables, so resolve ambiguity to the columns. +-- The out columns (id, ...) shadow table columns inside the body; the body +-- never reads them as variables, so resolve ambiguity to the columns. #variable_conflict use_column begin -- _metas is one jsonb array (not jsonb[]): drivers pass json values @@ -165,11 +274,40 @@ begin or cardinality(_ids) is distinct from cardinality(_contents) or cardinality(_ids) is distinct from jsonb_array_length(_metas) or cardinality(_ids) is distinct from cardinality(_temporals) + or (_names is not null and cardinality(_ids) is distinct from cardinality(_names)) then raise exception 'batch arrays must have equal lengths' using errcode = 'invalid_parameter_value'; end if; + if _on_conflict is null or _on_conflict not in ('error', 'replace', 'ignore') then + raise exception 'invalid _on_conflict: %', _on_conflict + using errcode = 'invalid_parameter_value'; + end if; + + -- A duplicate idempotency key within the batch is ambiguous (two outcomes for + -- one key) and would otherwise slip past the per-key partitions below. + if exists + ( + select 1 from unnest(_ids) u(id) + where u.id is not null + group by u.id having count(*) > 1 + ) then + raise exception 'duplicate explicit id within batch' + using errcode = 'invalid_parameter_value'; + end if; + if _names is not null and exists + ( + select 1 from unnest(_trees, _names) u(tree, name) + where u.name is not null + group by u.tree, u.name having count(*) > 1 + ) then + raise exception 'duplicate (tree, name) within batch' + using errcode = 'invalid_parameter_value'; + end if; + + -- Check access to write targets before probing existing rows, so callers + -- can't learn whether ids or (tree, name) slots exist outside their grant. if exists ( select 1 @@ -180,47 +318,189 @@ begin using errcode = 'insufficient_privilege'; end if; + -- The keys above are distinct, but two inputs with DIFFERENT keys can still + -- resolve to the same EXISTING row: an unnamed {id: X} and a {tree, name} + -- whose slot already holds id X. Status is attributed by stored id, so that + -- would mark both inputs from one write — breaking one-status-per-input (and + -- two CTEs would touch the same row). Reject it. The id arm is restricted to + -- UNNAMED explicit-id inputs (the id-keyed partition); a named row's explicit + -- id is not a key (name wins), so a single named input whose id equals its own + -- stored id is not a false positive. + -- + -- The collision needs BOTH an explicit-id input and a named input, so skip + -- this two-join probe against `memory` entirely when either is absent (e.g. a + -- name-only or all-anonymous batch). + if _names is not null + and cardinality(array_remove(_ids, null)) > 0 + and exists + ( + select 1 from + ( + select m.id + from unnest(_ids, _names) u(id, name) + join {{schema}}.memory m on m.id = u.id + where u.id is not null and u.name is null + union all + select m.id + from unnest(_trees, _names) u(tree, name) + join {{schema}}.memory m on m.tree = u.tree and m.name = u.name + where u.name is not null + ) x + group by x.id having count(*) > 1 + ) then + raise exception 'batch inputs target the same existing memory via different keys (explicit id and (tree, name))' + using errcode = 'invalid_parameter_value'; + end if; + return query with r as ( select - coalesce(u.id, uuidv7()) as id + u.id as explicit_id -- null = no client-supplied id + , coalesce(u.id, uuidv7()) as id -- the row's identity (generated if absent) , u.tree , coalesce(nullif(e.meta, 'null'::jsonb), '{}'::jsonb) as meta , u.temporal , u.content + , u.name , u.ord - from unnest(_ids, _trees, _contents, _temporals) - with ordinality u(id, tree, content, temporal, ord) + from unnest + ( _ids + , _trees + , _contents + , _temporals + , coalesce(_names, array_fill(null::text, array[cardinality(_ids)])) + ) + with ordinality u(id, tree, content, temporal, name, ord) join jsonb_array_elements(_metas) with ordinality e(meta, ord) on e.ord = u.ord ) - , d as + -- Unnamed + explicit id → keyed on the id. (Within-batch id dups already + -- raised, so no dedup is needed here.) + , with_id as ( - -- First occurrence wins when a batch repeats an explicit id. - select distinct on (r.id) r.* - from r - order by r.id, r.ord + select r.* from r where r.explicit_id is not null and r.name is null + ) + -- Named (with OR without an id) → keyed on (tree, name); a name takes + -- precedence over the id as the dedup key. (Within-batch (tree, name) dups + -- already raised.) + , named as + ( + select r.* from r where r.name is not null + ) + -- No id, no name → anonymous; nothing to dedup. + , anon as + ( + select r.* from r where r.explicit_id is null and r.name is null + ) + -- Unnamed explicit-id rows dedup on the id, so the row keeps it (import/export + -- identity). An explicit id can collide with an existing row in a tree the + -- caller can't write (the up-front check only covers the INPUT trees), so the + -- conflict modes diverge there: 'error' always raises CONFLICT (so it never + -- silently skips → INTERNAL_ERROR on read-back); 'ignore' skips; 'replace' + -- raises (it can't perform the replace on an unwritable tree). + , ins_id as + ( + insert into {{schema}}.memory as m + ( id, tree, meta, temporal, content, name ) + select w.id, w.tree, w.meta, w.temporal, w.content, w.name + from with_id w + on conflict (id) do update set + tree = excluded.tree + , meta = excluded.meta + , temporal = excluded.temporal + , content = excluded.content + , name = excluded.name + where case + when _on_conflict = 'error' then {{schema}}.raise_conflict() + when _on_conflict = 'ignore' then false + -- only 'replace' remains + when not {{schema}}.has_tree_access(_tree_access, m.tree, 2) + then {{schema}}.raise_no_write_access() + -- an id-keyed replace can move/rename, so compare every updated field + else m.tree is distinct from excluded.tree + or m.name is distinct from excluded.name + or m.content is distinct from excluded.content + or m.meta is distinct from excluded.meta + or m.temporal is distinct from excluded.temporal + end + returning m.id as id, (m.xmax = 0) as inserted + ) + -- Named rows (with OR without an explicit id) dedup on (tree, name). On a + -- fresh insert the row uses its explicit id when given (e.g. an importer's + -- timestamp-prefixed v7), else a generated one; on a (tree, name) conflict the + -- existing row's id is kept. A stray pk collision on a given id raises. + , ins_named as + ( + insert into {{schema}}.memory as m + ( id, tree, meta, temporal, content, name ) + select n.id, n.tree, n.meta, n.temporal, n.content, n.name + from named n + on conflict (tree, name) where name is not null do update set + meta = excluded.meta + , temporal = excluded.temporal + , content = excluded.content + where case + when _on_conflict = 'replace' + then m.content is distinct from excluded.content + or m.meta is distinct from excluded.meta + or m.temporal is distinct from excluded.temporal + when _on_conflict = 'ignore' then false + else {{schema}}.raise_conflict() + end + returning m.id as id, (m.xmax = 0) as inserted ) - insert into {{schema}}.memory as m - ( id - , tree - , meta - , temporal - , content + -- Anonymous rows always insert (their generated id is unique). + , ins_anon as + ( + insert into {{schema}}.memory as m + ( id, tree, meta, temporal, content, name ) + select a.id, a.tree, a.meta, a.temporal, a.content, a.name + from anon a + returning m.id as id, true as inserted + ) + -- Rows actually written this statement: (id, inserted=fresh-insert vs replace). + , acted as + ( + select id, inserted from ins_id + union all + select id, inserted from ins_named + union all + select id, inserted from ins_anon + ) + -- The stored id per input. For a named row it is resolved by (tree, name): + -- this subquery reads the PRE-statement snapshot (data-modifying CTEs aren't + -- visible here), so an EXISTING row (update/skip) yields its kept id, while a + -- fresh insert yields null → fall back to the row's own (minted) id. Unnamed + -- and anonymous rows always keep their own id. + , resolved as + ( + select + r.ord + , case + when r.name is not null then coalesce + ( ( select mm.id + from {{schema}}.memory mm + where mm.tree = r.tree and mm.name = r.name ) + , r.id + ) + else r.id + end as id + from r ) - select d.id, d.tree, d.meta, d.temporal, d.content - from d - on conflict (id) do update set - tree = excluded.tree - , meta = excluded.meta - , temporal = excluded.temporal - , content = excluded.content - where _replace_if_meta_differs is not null - and m.meta->>_replace_if_meta_differs - is distinct from excluded.meta->>_replace_if_meta_differs - and {{schema}}.has_tree_access(_tree_access, m.tree, 2) - returning m.id, (m.xmax = 0) + -- One row per input, in order: present in `acted` → inserted/updated; absent + -- → skipped (onConflict ignore, or a replace no-op). + select + res.ord + , res.id + , case + when a.id is null then 'skipped' + when a.inserted then 'inserted' + else 'updated' + end as status + from resolved res + left join acted a on a.id = res.id + order by res.ord ; end; $func$ language plpgsql volatile security invoker @@ -231,12 +511,34 @@ set search_path to pg_catalog, {{schema}}, public, pg_temp -- create memory -- -- One-row wrapper over batch_create_memory — see there for the conflict --- semantics (insert / replace-if-meta-differs / skip) and the return shape. +-- semantics (insert / content-aware replace / skip). Returns exactly one row, +-- (id, status), mirroring the batch shape: id is the row's stored id (the kept +-- existing id on a (tree, name) update/skip — so callers can read it back even +-- on a skip), status is 'inserted' | 'updated' | 'skipped'. -- --- The drop covers the pre-upsert 6-arg signature — without it, create would --- add an ambiguous overload (and the return type changed). No-op on re-runs. +-- The return type changed from (id, inserted) to (id, status), which +-- create-or-replace cannot make (42P13). Drop a prior definition only when it +-- lacks `status` among its columns (guarded so it doesn't churn the live +-- function every boot) — a no-op on fresh schemas and once current. The plain +-- drops cover older arg-signature overloads (pre-upsert 6-arg, pre-name 7-arg, +-- _replace_if_meta_differs 9-arg). ------------------------------------------------------------------------------- drop function if exists {{schema}}.create_memory(jsonb, ltree, text, uuid, jsonb, tstzrange); +drop function if exists {{schema}}.create_memory(jsonb, ltree, text, uuid, jsonb, tstzrange, text); +drop function if exists {{schema}}.create_memory(jsonb, ltree, text, uuid, jsonb, tstzrange, text, text, text); +do $$ begin + if exists + ( + select 1 + from pg_proc p + join pg_namespace n on n.oid = p.pronamespace + where n.nspname = '{{schema}}' + and p.proname = 'create_memory' + and not ('status' = any(coalesce(p.proargnames, array[]::text[]))) + ) then + drop function {{schema}}.create_memory(jsonb, ltree, text, uuid, jsonb, tstzrange, text, text); + end if; +end $$; create or replace function {{schema}}.create_memory ( _tree_access jsonb , _tree ltree @@ -244,11 +546,12 @@ create or replace function {{schema}}.create_memory , _id uuid default null , _meta jsonb default '{}' , _temporal tstzrange default null -, _replace_if_meta_differs text default null +, _name text default null +, _on_conflict text default 'error' ) -returns table (id uuid, inserted boolean) +returns table (id uuid, status text) as $func$ - select b.id, b.inserted + select b.id, b.status from {{schema}}.batch_create_memory( _tree_access, array[_id]::uuid[], @@ -256,7 +559,8 @@ as $func$ array[_content], jsonb_build_array(coalesce(_meta, '{}'::jsonb)), array[_temporal], - _replace_if_meta_differs + array[_name]::text[], + _on_conflict ) b; $func$ language sql volatile security invoker set search_path to pg_catalog, {{schema}}, public, pg_temp @@ -278,7 +582,7 @@ declare _ok bool; begin -- at least one valid field must be present - select count(*) filter (where k in ('meta', 'tree', 'temporal', 'content')) > 0 + select count(*) filter (where k in ('meta', 'tree', 'temporal', 'content', 'name')) > 0 into strict _ok from jsonb_each(_patch) o(k, v) ; @@ -340,11 +644,15 @@ begin using errcode = 'insufficient_privilege'; end if; + -- A rename or a move into a tree that already has this name violates the + -- (tree, name) unique index; that 23505 propagates and is mapped to CONFLICT + -- at the RPC boundary. Setting name to JSON null clears it. update {{schema}}.memory m set tree = case when _patch ? 'tree' then (_patch->>'tree')::ltree else m.tree end , meta = case when _patch ? 'meta' then _patch->'meta' else m.meta end , temporal = case when _patch ? 'temporal' then (_patch->>'temporal')::tstzrange else m.temporal end , content = case when _patch ? 'content' then _patch->>'content' else m.content end + , name = case when _patch ? 'name' then (_patch->>'name') else m.name end where id = _id returning id into _id ; @@ -494,6 +802,7 @@ begin ( insert into {{schema}}.memory ( meta + , name -- preserved; a (dst, name) clash raises 23505 → CONFLICT , tree , temporal , content @@ -502,6 +811,7 @@ begin ) select m.meta + , m.name , case when nlevel(m.tree) = nlevel(_src) then _dst else _dst || subpath(m.tree, nlevel(_src), nlevel(m.tree) - nlevel(_src)) diff --git a/packages/database/space/migrate/idempotent/002_search.sql b/packages/database/space/migrate/idempotent/002_search.sql index d9504e7e..14e7b34b 100644 --- a/packages/database/space/migrate/idempotent/002_search.sql +++ b/packages/database/space/migrate/idempotent/002_search.sql @@ -1,6 +1,20 @@ ------------------------------------------------------------------------------- -- search_memory ------------------------------------------------------------------------------- +-- search_memory gained a `name` return column (a return-type change, which +-- create-or-replace cannot make → 42P13 on an existing function). Drop a prior +-- definition only when it lacks `name` among its columns; a no-op on fresh +-- schemas and once current. +do $$ begin + if exists ( + select 1 from pg_proc p + join pg_namespace n on n.oid = p.pronamespace + where n.nspname = '{{schema}}' and p.proname = 'search_memory' + and not ('name' = any(coalesce(p.proargnames, array[]::text[]))) + ) then + drop function {{schema}}.search_memory(jsonb, bm25query, halfvec, float8, ltree, lquery, ltxtquery, jsonb, tstzrange, tstzrange, timestamptz, timestamptz, text, bigint, text); + end if; +end $$; create or replace function {{schema}}.search_memory ( _tree_access jsonb , _bm25 bm25query default null @@ -24,6 +38,7 @@ returns table , tree ltree , temporal tstzrange , content text +, name text , has_embedding bool , created_at timestamptz , updated_at timestamptz @@ -189,6 +204,7 @@ begin , m.tree , m.temporal , m.content + , m.name , m.embedding is not null , m.created_at , m.updated_at @@ -225,6 +241,17 @@ set search_path to pg_catalog, {{schema}}, public, pg_temp ------------------------------------------------------------------------------- -- hybrid_search_memory ------------------------------------------------------------------------------- +-- Same `name` return-column addition as search_memory; same guarded drop. +do $$ begin + if exists ( + select 1 from pg_proc p + join pg_namespace n on n.oid = p.pronamespace + where n.nspname = '{{schema}}' and p.proname = 'hybrid_search_memory' + and not ('name' = any(coalesce(p.proargnames, array[]::text[]))) + ) then + drop function {{schema}}.hybrid_search_memory(jsonb, bm25query, halfvec, float8, ltree, lquery, ltxtquery, jsonb, tstzrange, tstzrange, timestamptz, timestamptz, text, float8, bigint, float8, float8, bigint); + end if; +end $$; create or replace function {{schema}}.hybrid_search_memory ( _tree_access jsonb , _bm25 bm25query @@ -251,6 +278,7 @@ returns table , tree ltree , temporal tstzrange , content text +, name text , has_embedding bool , created_at timestamptz , updated_at timestamptz @@ -286,6 +314,7 @@ begin , coalesce(x1.tree, x2.tree) as tree , coalesce(x1.temporal, x2.temporal) as temporal , coalesce(x1.content, x2.content) as content + , coalesce(x1.name, x2.name) as name , coalesce(x1.has_embedding, x2.has_embedding) as has_embedding , coalesce(x1.created_at, x2.created_at) as created_at , coalesce(x1.updated_at, x2.updated_at) as updated_at diff --git a/packages/database/space/migrate/incremental/005_memory_name.sql b/packages/database/space/migrate/incremental/005_memory_name.sql new file mode 100644 index 00000000..7c6c0998 --- /dev/null +++ b/packages/database/space/migrate/incremental/005_memory_name.sql @@ -0,0 +1,26 @@ +------------------------------------------------------------------------------- +-- memory.name +-- +-- An optional, mutable, human-chosen leaf name, unique within a tree path. The +-- UUID stays the immutable identity (embeddings, audit, links survive +-- rename/move); `name` is additive addressing (`/share/auth/jwt-rotation`) and +-- the idempotency key for `(tree, name)` upserts. Filename-like and allowed to +-- contain dots — it is never an ltree label, so a dotted name cannot collide +-- with the tree separator. +------------------------------------------------------------------------------- +alter table {{schema}}.memory add column name text; + +-- Unique within the exact tree path. Partial (where name is not null) so any +-- number of unnamed memories coexist under one tree, and two different trees +-- may reuse a name. +create unique index memory_tree_name_uidx on {{schema}}.memory (tree, name) +where name is not null; + +-- Defensive format check (the application validates the same shape): a +-- filename-like slug that must start alphanumeric, so `.`/`..`/hidden names are +-- rejected, no slashes or spaces, <= 128 chars. +alter table {{schema}}.memory add constraint memory_name_format check +( + name is null + or (name operator(pg_catalog.~) '^[A-Za-z0-9][A-Za-z0-9._-]*$' and length(name) <= 128) +); diff --git a/packages/database/space/migrate/migrate.integration.test.ts b/packages/database/space/migrate/migrate.integration.test.ts index 3693fe9d..e928fcf3 100644 --- a/packages/database/space/migrate/migrate.integration.test.ts +++ b/packages/database/space/migrate/migrate.integration.test.ts @@ -43,6 +43,7 @@ const EXPECTED_MIGRATIONS = [ "002_embedding_queue", "003_embedding_fk_idx", "004_count_tree", + "005_memory_name", ]; const EXPECTED_MEMORY_FUNCTIONS = [ @@ -58,6 +59,7 @@ const EXPECTED_MEMORY_FUNCTIONS = [ "list_tree", "move_tree", "patch_memory", + "resolve_memory_id", "search_memory", "tree_access", ]; @@ -74,6 +76,7 @@ const EXPECTED_MEMORY_INDEXES = [ "memory_meta_gin_idx", "memory_temporal_gist_idx", "memory_tree_gist_idx", + "memory_tree_name_uidx", ]; let sql: SQL; @@ -305,8 +308,51 @@ describe("provisioned schema is functional", () => { expect(updated?.updated_at).not.toBeNull(); }); - // create_memory's conditional upsert: (treeAccess, tree, content, id, meta, - // temporal, replaceIfMetaDiffers) → zero rows (skip) or (id, inserted). + test("name: (tree,name) unique, nulls coexist, format enforced", async () => { + const t = "namecol"; + await sql.unsafe( + `insert into ${canonical.schema}.memory (content, tree, name) + values ('first', '${t}', 'doc')`, + ); + // Same (tree, name) collides on the partial unique index. + await expectReject(() => + sql.unsafe( + `insert into ${canonical.schema}.memory (content, tree, name) + values ('second', '${t}', 'doc')`, + ), + ); + // The same name under a different tree is fine. + const [other] = await sql.unsafe( + `insert into ${canonical.schema}.memory (content, tree, name) + values ('elsewhere', '${t}.sub', 'doc') returning id`, + ); + expect(other?.id).toBeDefined(); + // Any number of unnamed (null) memories coexist under one tree. + await sql.unsafe( + `insert into ${canonical.schema}.memory (content, tree) + values ('a', '${t}'), ('b', '${t}')`, + ); + // Filename-like names (dots allowed) pass; leading-dot / spaces rejected. + await sql.unsafe( + `insert into ${canonical.schema}.memory (content, tree, name) + values ('cfg', '${t}', 'config.yaml')`, + ); + await expectReject(() => + sql.unsafe( + `insert into ${canonical.schema}.memory (content, tree, name) + values ('bad', '${t}', '.hidden')`, + ), + ); + await expectReject(() => + sql.unsafe( + `insert into ${canonical.schema}.memory (content, tree, name) + values ('bad', '${t}', 'has space')`, + ), + ); + }); + + // create_memory: (treeAccess, tree, content, id, meta, temporal, name, + // onConflict) → zero rows (skip) or (id, inserted). const OWNER = `'[{"tree_path": "", "access": 3}]'::jsonb`; const createMemory = (args: string) => sql.unsafe(`select * from ${canonical.schema}.create_memory(${args})`); @@ -327,80 +373,88 @@ describe("provisioned schema is functional", () => { expect(Number(capped?.n)).toBe(2); }); - test("create_memory skips a duplicate explicit id by default", async () => { - // Deterministic-id importers re-submit existing ids; with no replace key - // the second create must be a zero-row no-op leaving the row intact. + test("create_memory raises on a bare duplicate explicit id", async () => { + // A conflict on the id key under the default onConflict ('error') is a hard + // error; importers pass onConflict 'replace' (next test) to stay idempotent. const id = "01941000-0000-7000-8000-000000000001"; const [first] = await createMemory( `${OWNER}, 'a.dup'::ltree, 'original', '${id}'::uuid`, ); expect(first?.id).toBe(id); - expect(first?.inserted).toBe(true); + expect(first?.status).toBe("inserted"); - const second = await createMemory( - `${OWNER}, 'a.dup'::ltree, 'replacement', '${id}'::uuid`, + await expectReject(() => + createMemory(`${OWNER}, 'a.dup'::ltree, 'replacement', '${id}'::uuid`), ); - expect(second.length).toBe(0); const [row] = await sql.unsafe( `select content from ${canonical.schema}.memory where id = '${id}'`, ); - expect(row?.content).toBe("original"); + expect(row?.content).toBe("original"); // untouched }); - test("create_memory replaces a duplicate when the meta key differs, skips when it matches", async () => { + test("create_memory id-keyed 'replace' is content-aware: replaces when a field differs, no-op when identical", async () => { const id = "01941000-0000-7000-8000-000000000002"; await createMemory( `${OWNER}, 'a.ver'::ltree, 'render v1', '${id}'::uuid, '{"v": "1"}'::jsonb`, ); - // Same version → skip, content untouched. - const same = await createMemory( - `${OWNER}, 'a.ver'::ltree, 'render v1 again', '${id}'::uuid, '{"v": "1"}'::jsonb, null, 'v'`, + // Identical content+meta → content-aware replace is a no-op (skipped). + const [same] = await createMemory( + `${OWNER}, 'a.ver'::ltree, 'render v1', '${id}'::uuid, '{"v": "1"}'::jsonb, null, null, 'replace'`, ); - expect(same.length).toBe(0); + expect(same?.id).toBe(id); + expect(same?.status).toBe("skipped"); - // Bumped version → replaced in place, inserted = false. + // Meta differs (same content) → replaced in place (updated). This is how an + // importer_version bump propagates: the version lives in meta. const [bumped] = await createMemory( - `${OWNER}, 'a.ver'::ltree, 'render v2', '${id}'::uuid, '{"v": "2"}'::jsonb, null, 'v'`, + `${OWNER}, 'a.ver'::ltree, 'render v1', '${id}'::uuid, '{"v": "2"}'::jsonb, null, null, 'replace'`, ); expect(bumped?.id).toBe(id); - expect(bumped?.inserted).toBe(false); + expect(bumped?.status).toBe("updated"); const [row] = await sql.unsafe( - `select content, meta->>'v' as v, updated_at + `select meta->>'v' as v, updated_at from ${canonical.schema}.memory where id = '${id}'`, ); - expect(row?.content).toBe("render v2"); expect(row?.v).toBe("2"); expect(row?.updated_at).not.toBeNull(); - - // A key absent on the stored row but present on the new record counts as - // "differs" (legacy rows written before the version key existed). - const [legacy] = await createMemory( - `${OWNER}, 'a.ver'::ltree, 'render v3', '${id}'::uuid, '{"v": "2", "legacy_v": "1"}'::jsonb, null, 'legacy_v'`, - ); - expect(legacy?.inserted).toBe(false); - const [afterLegacy] = await sql.unsafe( - `select content from ${canonical.schema}.memory where id = '${id}'`, - ); - expect(afterLegacy?.content).toBe("render v3"); }); - test("create_memory replace requires write access on the existing row's tree", async () => { - // Row lives under a.secret; the caller's grant covers only a.open — the - // insert-arm check passes (target a.open) but the replace arm must skip. + test("create_memory explicit-id collision with an unwritable tree: error raises, replace raises, ignore skips", async () => { + // The row lives under a.secret; the caller's grant covers only a.open, so + // the INPUT tree (a.open) passes the up-front check while the EXISTING row's + // tree (a.secret) is unwritable. 'error' raises CONFLICT and 'replace' + // raises insufficient_privilege (it can't perform the replace) — neither + // silently skips, which used to surface as INTERNAL_ERROR on read-back. + // 'ignore' skips, leaving the existing row alone. const id = "01941000-0000-7000-8000-000000000003"; await createMemory( `${OWNER}, 'a.secret'::ltree, 'guarded', '${id}'::uuid, '{"v": "1"}'::jsonb`, ); - const limited = `'[{"tree_path": "a.open", "access": 3}]'::jsonb`; - const res = await createMemory( - `${limited}, 'a.open'::ltree, 'hijack', '${id}'::uuid, '{"v": "2"}'::jsonb, null, 'v'`, + + // error (default) → raise. + await expectReject(() => + createMemory( + `${limited}, 'a.open'::ltree, 'hijack', '${id}'::uuid, '{"v": "2"}'::jsonb`, + ), ); - expect(res.length).toBe(0); + // replace → raise (can't replace a row in an unwritable tree). + await expectReject(() => + createMemory( + `${limited}, 'a.open'::ltree, 'hijack', '${id}'::uuid, '{"v": "2"}'::jsonb, null, null, 'replace'`, + ), + ); + // ignore → skip, returning the existing stored id. + const [ignored] = await createMemory( + `${limited}, 'a.open'::ltree, 'hijack', '${id}'::uuid, '{"v": "2"}'::jsonb, null, null, 'ignore'`, + ); + expect(ignored?.id).toBe(id); + expect(ignored?.status).toBe("skipped"); + // The existing row is untouched in every case. const [row] = await sql.unsafe( `select content, tree::text as tree from ${canonical.schema}.memory where id = '${id}'`, ); @@ -418,24 +472,35 @@ describe("provisioned schema is functional", () => { `${OWNER}, 'a.batch'::ltree, 'current', '${fresh}'::uuid, '{"v": "2"}'::jsonb`, ); - // One call carrying: a stale row (update), a current row (skip), a brand - // new row (insert), and a no-id row (insert with generated id). + // One call carrying: a stale row (changed content → update), a current row + // (identical content+meta → skip), a brand new row (insert), and a no-id + // row (insert with generated id) — all under content-aware 'replace'. const rows = await sql.unsafe( - `select * from ${canonical.schema}.batch_create_memory( + `select ord, id, status from ${canonical.schema}.batch_create_memory( ${OWNER}, array['${stale}', '${fresh}', '01941000-0000-7000-8000-00000000b003', null]::uuid[], array['a.batch', 'a.batch', 'a.batch', 'a.batch']::ltree[], - array['new render', 'untouched', 'added', 'generated']::text[], + array['new render', 'current', 'added', 'generated']::text[], '[{"v": "2"}, {"v": "2"}, {"v": "2"}, {"v": "2"}]'::jsonb, array[null, null, null, null]::tstzrange[], - 'v' - )`, - ); - const byId = new Map(rows.map((r) => [r.id as string, r.inserted])); - expect(byId.get(stale)).toBe(false); // replaced - expect(byId.has(fresh)).toBe(false); // skipped → absent - expect(byId.get("01941000-0000-7000-8000-00000000b003")).toBe(true); - expect(rows).toHaveLength(3); // 2 inserts + 1 update + null, + 'replace' + ) order by ord`, + ); + // One row per input, in order, with a per-row status. + expect(rows.map((r) => Number(r.ord))).toEqual([1, 2, 3, 4]); + expect(rows.map((r) => r.status)).toEqual([ + "updated", // stale: content changed + "skipped", // fresh: identical → content-aware no-op + "inserted", // b003: brand new + "inserted", // generated id + ]); + // Returned ids map back to the inputs (the explicit ones, and a fresh + // uuid for the no-id row). + expect(rows[0]?.id).toBe(stale); + expect(rows[1]?.id).toBe(fresh); + expect(rows[2]?.id).toBe("01941000-0000-7000-8000-00000000b003"); + expect(rows[3]?.id).toMatch(/^[0-9a-f-]{36}$/); const [updated] = await sql.unsafe( `select content from ${canonical.schema}.memory where id = '${stale}'`, @@ -447,23 +512,105 @@ describe("provisioned schema is functional", () => { expect(skipped?.content).toBe("current"); }); - test("batch_create_memory collapses an id repeated within the batch (first wins)", async () => { + test("batch_create_memory rejects a duplicate explicit id within the batch", async () => { const id = "01941000-0000-7000-8000-00000000b010"; - const rows = await sql.unsafe( - `select * from ${canonical.schema}.batch_create_memory( - ${OWNER}, - array['${id}', '${id}']::uuid[], - array['a.batchdup', 'a.batchdup']::ltree[], - array['first', 'second']::text[], - '[{}, {}]'::jsonb, - array[null, null]::tstzrange[] - )`, + await expectReject(() => + sql.unsafe( + `select * from ${canonical.schema}.batch_create_memory( + ${OWNER}, + array['${id}', '${id}']::uuid[], + array['a.batchdup', 'a.batchdup']::ltree[], + array['first', 'second']::text[], + '[{}, {}]'::jsonb, + array[null, null]::tstzrange[] + )`, + ), ); - expect(rows).toHaveLength(1); + // Nothing was written — the whole batch is rejected up front. const [row] = await sql.unsafe( - `select content from ${canonical.schema}.memory where id = '${id}'`, + `select count(*)::int as n from ${canonical.schema}.memory where id = '${id}'`, + ); + expect(row?.n).toBe(0); + }); + + test("batch_create_memory rejects a duplicate (tree, name) within the batch", async () => { + await expectReject(() => + sql.unsafe( + `select * from ${canonical.schema}.batch_create_memory( + ${OWNER}, + array[null, null]::uuid[], + array['a.bdupname', 'a.bdupname']::ltree[], + array['first', 'second']::text[], + '[{}, {}]'::jsonb, + array[null, null]::tstzrange[], + array['doc', 'doc']::text[] + )`, + ), + ); + }); + + test("batch_create_memory catches an id shared across a named and an unnamed row", async () => { + // The two rows aren't (tree, name) duplicates and land in different + // partitions (one keyed on id, one on (tree, name)), but they DO collide on + // the explicit id — the duplicate-id check must catch it. + const id = "01941000-0000-7000-8000-00000000b011"; + await expectReject(() => + sql.unsafe( + `select * from ${canonical.schema}.batch_create_memory( + ${OWNER}, + array['${id}', '${id}']::uuid[], + array['a.bmixed', 'a.bmixed']::ltree[], + array['by-id', 'by-name']::text[], + '[{}, {}]'::jsonb, + array[null, null]::tstzrange[], + array[null, 'doc']::text[] + )`, + ), ); - expect(row?.content).toBe("first"); + }); + + test("batch_create_memory rejects two inputs targeting the same existing row via different keys", async () => { + // Existing named row at (n.cross, doc) with id X. + const x = "01941000-0000-7000-8000-00000000e001"; + await createMemory( + `${OWNER}, 'n.cross'::ltree, 'v1', '${x}'::uuid, '{}'::jsonb, null, 'doc'`, + ); + + // input1 {id: X} (unnamed, id-keyed) and input2 {n.cross, doc} (name-keyed) + // both resolve to the SAME stored row X — distinct keys, so the per-key dup + // checks miss it; the cross-key check must reject it (else one write would + // attribute a status to both inputs). + await expectReject(() => + sql.unsafe( + `select * from ${canonical.schema}.batch_create_memory( + ${OWNER}, + array['${x}', null]::uuid[], + array['n.cross', 'n.cross']::ltree[], + array['by-id', 'by-name']::text[], + '[{}, {}]'::jsonb, + array[null, null]::tstzrange[], + array[null, 'doc']::text[], + 'replace' + )`, + ), + ); + + // A single NAMED input whose explicit id equals its own stored id is fine + // (name wins; not a cross-key collision) — identical content+meta → skip. + const [same] = await sql.unsafe( + `select ord, id, status from ${canonical.schema}.batch_create_memory( + ${OWNER}, + array['${x}']::uuid[], + array['n.cross']::ltree[], + array['v1']::text[], + '[{}]'::jsonb, + array[null]::tstzrange[], + array['doc']::text[], + 'replace' + )`, + ); + expect(same?.id).toBe(x); + expect(same?.status).toBe("skipped"); }); test("batch_create_memory rejects misaligned arrays and bad target access", async () => { @@ -518,7 +665,7 @@ describe("provisioned schema is functional", () => { // Meta-only replace (identical content): embedding survives, no re-enqueue. await createMemory( - `${OWNER}, 'a.emb'::ltree, 'stable content', '${id}'::uuid, '{"v": "2"}'::jsonb, null, 'v'`, + `${OWNER}, 'a.emb'::ltree, 'stable content', '${id}'::uuid, '{"v": "2"}'::jsonb, null, null, 'replace'`, ); const [afterMeta] = await sql.unsafe( `select (embedding is not null) as has_embedding, @@ -531,7 +678,7 @@ describe("provisioned schema is functional", () => { // Content replace: embedding invalidated and re-enqueued. await createMemory( - `${OWNER}, 'a.emb'::ltree, 'new content', '${id}'::uuid, '{"v": "3"}'::jsonb, null, 'v'`, + `${OWNER}, 'a.emb'::ltree, 'new content', '${id}'::uuid, '{"v": "3"}'::jsonb, null, null, 'replace'`, ); const [afterContent] = await sql.unsafe( `select (embedding is null) as embedding_cleared, @@ -562,6 +709,157 @@ describe("provisioned schema is functional", () => { ), ); }); + + // create_memory args: (treeAccess, tree, content, id, meta, temporal, name, + // onConflict). + // onConflict: a bare named conflict errors; 'ignore' skips; 'replace' is + // content-aware (no-op when identical, replaces when something differs). + test("create_memory onConflict: error | ignore | replace(content-aware)", async () => { + const [first] = await createMemory( + `${OWNER}, 'n.dir'::ltree, 'v1', null, '{}'::jsonb, null, 'note'`, + ); + expect(first?.status).toBe("inserted"); + const id = first?.id; + + // default 'error' → a hard conflict (raise). + await expectReject(() => + createMemory( + `${OWNER}, 'n.dir'::ltree, 'v2', null, '{}'::jsonb, null, 'note'`, + ), + ); + + // 'ignore' → skip (status 'skipped', existing id), existing row untouched. + const [ignored] = await createMemory( + `${OWNER}, 'n.dir'::ltree, 'v2', null, '{}'::jsonb, null, 'note', 'ignore'`, + ); + expect(ignored?.id).toBe(id); + expect(ignored?.status).toBe("skipped"); + expect( + ( + await sql.unsafe( + `select content from ${canonical.schema}.memory where id = '${id}'`, + ) + )[0]?.content, + ).toBe("v1"); + + // 'replace' with differing content → replaced in place, same id. + const [up] = await createMemory( + `${OWNER}, 'n.dir'::ltree, 'v2', null, '{}'::jsonb, null, 'note', 'replace'`, + ); + expect(up?.id).toBe(id); + expect(up?.status).toBe("updated"); + + // 'replace' with identical content/meta → no-op (content-aware, skipped). + const [noop] = await createMemory( + `${OWNER}, 'n.dir'::ltree, 'v2', null, '{}'::jsonb, null, 'note', 'replace'`, + ); + expect(noop?.id).toBe(id); + expect(noop?.status).toBe("skipped"); + + const [row] = await sql.unsafe( + `select content, name from ${canonical.schema}.memory where id = '${id}'`, + ); + expect(row?.content).toBe("v2"); + expect(row?.name).toBe("note"); + }); + + test("create_memory: id-keyed replace applies a tree-only move (not a no-op)", async () => { + const id = "01941000-0000-7000-8000-0000000000a0"; + await createMemory(`${OWNER}, 'mv.from'::ltree, 'body', '${id}'::uuid`); + // Same id + content, new tree → content-aware replace must still move it. + const [moved] = await createMemory( + `${OWNER}, 'mv.to'::ltree, 'body', '${id}'::uuid, '{}'::jsonb, null, null, 'replace'`, + ); + expect(moved?.id).toBe(id); + expect(moved?.status).toBe("updated"); + const [row] = await sql.unsafe( + `select tree::text as tree from ${canonical.schema}.memory where id = '${id}'`, + ); + expect(row?.tree).toBe("mv.to"); + }); + + test("named create 'replace' is content-aware; a bare batch named collision raises", async () => { + await createMemory( + `${OWNER}, 'n.imp'::ltree, 'r1', null, '{"v":"1"}'::jsonb, null, 'doc'`, + ); + // Identical content+meta → idempotent no-op (no raise, status 'skipped'). + const [same] = await createMemory( + `${OWNER}, 'n.imp'::ltree, 'r1', null, '{"v":"1"}'::jsonb, null, 'doc', 'replace'`, + ); + expect(same?.status).toBe("skipped"); + // Meta differs (importer-version bump) → replace in place (no raise). + const [diff] = await createMemory( + `${OWNER}, 'n.imp'::ltree, 'r1', null, '{"v":"2"}'::jsonb, null, 'doc', 'replace'`, + ); + expect(diff?.status).toBe("updated"); + // A batch with a bare (default-error) named collision raises, aborting it. + await expectReject(() => + sql.unsafe( + `select * from ${canonical.schema}.batch_create_memory( + ${OWNER}, + array[null]::uuid[], + array['n.imp']::ltree[], + array['dupe']::text[], + '[{}]'::jsonb, + array[null]::tstzrange[], + array['doc']::text[] + )`, + ), + ); + }); + + test("a named row dedups on (tree, name) even with an explicit id (name wins)", async () => { + // First insert carries an explicit id — used as the row's identity. + const id1 = "01941000-0000-7000-8000-00000000d101"; + const [first] = await createMemory( + `${OWNER}, 'n.idname'::ltree, 'v1', '${id1}'::uuid, '{}'::jsonb, null, 'doc'`, + ); + expect(first?.id).toBe(id1); + expect(first?.status).toBe("inserted"); + + // Re-submit the SAME (tree, name) with a DIFFERENT explicit id + 'replace'. + // Dedup is on (tree, name), so it replaces in place and KEEPS id1 — the new + // id is ignored (name wins over id). + const id2 = "01941000-0000-7000-8000-00000000d102"; + const [second] = await createMemory( + `${OWNER}, 'n.idname'::ltree, 'v2', '${id2}'::uuid, '{}'::jsonb, null, 'doc', 'replace'`, + ); + expect(second?.id).toBe(id1); // not id2 + expect(second?.status).toBe("updated"); + + const [row] = await sql.unsafe( + `select id, content from ${canonical.schema}.memory + where tree = 'n.idname' and name = 'doc'`, + ); + expect(row?.id).toBe(id1); + expect(row?.content).toBe("v2"); + // id2 was never inserted. + const [ghost] = await sql.unsafe( + `select count(*)::int as n from ${canonical.schema}.memory where id = '${id2}'`, + ); + expect(ghost?.n).toBe(0); + }); + + test("get_memory and resolve_memory_id surface the name", async () => { + const [m] = await createMemory( + `${OWNER}, 'n.resolve'::ltree, 'body', null, '{}'::jsonb, null, 'doc'`, + ); + const [got] = await sql.unsafe( + `select name from ${canonical.schema}.get_memory(${OWNER}, '${m?.id}'::uuid)`, + ); + expect(got?.name).toBe("doc"); + + const [resolved] = await sql.unsafe( + `select ${canonical.schema}.resolve_memory_id(${OWNER}, 'n.resolve'::ltree, 'doc') as id`, + ); + expect(resolved?.id).toBe(m?.id); + + // No read access → null, so a non-reader can't probe existence. + const [denied] = await sql.unsafe( + `select ${canonical.schema}.resolve_memory_id('[]'::jsonb, 'n.resolve'::ltree, 'doc') as id`, + ); + expect(denied?.id).toBeNull(); + }); }); describe("bootstrapSpaceDatabase", () => { diff --git a/packages/database/space/migrate/migrate.ts b/packages/database/space/migrate/migrate.ts index c4461933..8c52e97c 100644 --- a/packages/database/space/migrate/migrate.ts +++ b/packages/database/space/migrate/migrate.ts @@ -29,6 +29,9 @@ import incremental003 from "./incremental/003_embedding_fk_idx.sql" with { import incremental004 from "./incremental/004_count_tree.sql" with { type: "text", }; +import incremental005 from "./incremental/005_memory_name.sql" with { + type: "text", +}; import provisionSql from "./provision.sql" with { type: "text" }; const DIR = "packages/database/space/migrate"; @@ -54,6 +57,11 @@ const incrementals: Migration[] = [ file: "incremental/004_count_tree.sql", sql: incremental004, }, + { + name: "005_memory_name", + file: "incremental/005_memory_name.sql", + sql: incremental005, + }, ]; const idempotents: Migration[] = [ diff --git a/packages/database/space/path.test.ts b/packages/database/space/path.test.ts index dff6c5fc..60aebe83 100644 --- a/packages/database/space/path.test.ts +++ b/packages/database/space/path.test.ts @@ -156,32 +156,44 @@ describe("homePrefix", () => { }); describe("denormalizeTreePath", () => { - test("reverse-maps the caller's home to ~ with the canonical dot separator", () => { + test("reverse-maps the caller's home to ~ (no leading slash; ~ is the anchor)", () => { expect(denormalizeTreePath(HOME, { home: ID })).toBe("~"); - expect(denormalizeTreePath(`${HOME}.bar`, { home: ID })).toBe("~.bar"); - expect(denormalizeTreePath(`${HOME}.a.b`, { home: ID })).toBe("~.a.b"); + expect(denormalizeTreePath(`${HOME}.bar`, { home: ID })).toBe("~/bar"); + expect(denormalizeTreePath(`${HOME}.a.b`, { home: ID })).toBe("~/a/b"); }); - test("leaves non-home paths (and other principals' homes) unchanged", () => { + test("renders non-home paths (and other principals' homes) as absolute /paths", () => { expect(denormalizeTreePath("work.projects", { home: ID })).toBe( - "work.projects", + "/work/projects", ); expect(denormalizeTreePath("home.deadbeef.x", { home: ID })).toBe( - "home.deadbeef.x", + "/home/deadbeef/x", ); - expect(denormalizeTreePath(`${HOME}.bar`)).toBe(`${HOME}.bar`); // no home opt + // no home opt → still an absolute slash path + expect(denormalizeTreePath(`${HOME}.bar`)).toBe( + `/${HOME.replace(/\./g, "/")}/bar`, + ); + }); + + test("the empty root renders as /", () => { + expect(denormalizeTreePath("", { home: ID })).toBe("/"); + expect(denormalizeTreePath("")).toBe("/"); }); test("round-trips with normalizeTreePath", () => { - const display = denormalizeTreePath(`${HOME}.a.b`, { home: ID }); // ~.a.b - expect(normalizeTreePath(display, { home: ID })).toBe(`${HOME}.a.b`); + const home = denormalizeTreePath(`${HOME}.a.b`, { home: ID }); // ~/a/b + expect(normalizeTreePath(home, { home: ID })).toBe(`${HOME}.a.b`); + const abs = denormalizeTreePath("work.projects"); // /work/projects + expect(normalizeTreePath(abs)).toBe("work.projects"); // leading slash stripped }); test("reverse-maps an agent's nested home (homeOwner) to ~", () => { const opts = { home: AGENT, homeOwner: ID }; expect(denormalizeTreePath(AGENT_HOME, opts)).toBe("~"); - expect(denormalizeTreePath(`${AGENT_HOME}.a.b`, opts)).toBe("~.a.b"); + expect(denormalizeTreePath(`${AGENT_HOME}.a.b`, opts)).toBe("~/a/b"); // the owner's own home (one level up) is NOT the agent's ~ - expect(denormalizeTreePath(HOME, opts)).toBe(HOME); + expect(denormalizeTreePath(HOME, opts)).toBe( + `/${HOME.replace(/\./g, "/")}`, + ); }); }); diff --git a/packages/database/space/path.ts b/packages/database/space/path.ts index fd0d55e6..70819b21 100644 --- a/packages/database/space/path.ts +++ b/packages/database/space/path.ts @@ -197,21 +197,27 @@ export function classifyTreeFilter( } /** - * Reverse of the home expansion, for display. A path under the given - * principal's home is shown with a leading `~`, keeping the canonical dot - * separator (`home.` → `~`, `home..a.b` → `~.a.b`); everything else - * (including other principals' homes) is returned unchanged. Dot is the - * canonical output separator throughout. + * Reverse of the home expansion, for display, in canonical **slash** form. The + * caller's home is shown with a leading `~` (`home.` → `~`, + * `home..a.b` → `~/a/b`); every other path is rendered as an absolute, + * slash-separated path with a leading `/` (`share.auth` → `/share/auth`), and + * the root (empty path) is `/`. So `~` anchors home and `/` anchors the root, + * shell-style. ltree storage and the SQL layer stay dot-native, and + * `normalizeTreePath` strips a leading separator and accepts both `/` and `.`, + * so a displayed path fed back in round-trips. */ export function denormalizeTreePath( path: string, opts: TreePathOptions = {}, ): string { - if (opts.home === undefined) return path; - const prefix = homePrefix(opts.home, opts.homeOwner); - if (path === prefix) return "~"; - if (path.startsWith(`${prefix}.`)) { - return `~${path.slice(prefix.length)}`; // home..a.b → ~.a.b + if (opts.home !== undefined) { + const prefix = homePrefix(opts.home, opts.homeOwner); + if (path === prefix) return "~"; + if (path.startsWith(`${prefix}.`)) { + // home..a.b → ~/a/b — `~` is the anchor, so no leading slash + return `~${path.slice(prefix.length).replace(/\./g, "/")}`; + } } - return path; + // Absolute path: leading `/`, dots → slashes. Root ("") → "/". + return `/${path.replace(/\./g, "/")}`; } diff --git a/packages/database/space/version.ts b/packages/database/space/version.ts index 4d6445bf..d7bdc8b5 100644 --- a/packages/database/space/version.ts +++ b/packages/database/space/version.ts @@ -1 +1 @@ -export const SPACE_SCHEMA_VERSION = "0.0.3"; +export const SPACE_SCHEMA_VERSION = "0.0.4"; diff --git a/packages/engine/space/db.integration.test.ts b/packages/engine/space/db.integration.test.ts index f8a6446c..bb41c7f2 100644 --- a/packages/engine/space/db.integration.test.ts +++ b/packages/engine/space/db.integration.test.ts @@ -57,8 +57,9 @@ async function mustCreate( params: Parameters[1], ): Promise { const created = await db.createMemory(access, params); - if (created === null) throw new Error("unexpected duplicate-id skip"); - if (!created.inserted) throw new Error("unexpected replace"); + if (created.status !== "inserted") { + throw new Error(`unexpected status: ${created.status}`); + } return created.id; } @@ -74,28 +75,57 @@ test("createMemory + getMemory round-trips", async () => { expect(m?.content).toBe("hello world"); expect(m?.meta).toEqual({ kind: "note" }); expect(m?.hasEmbedding).toBe(false); + expect(m?.name).toBeNull(); }); -test("createMemory returns null for a duplicate explicit id", async () => { +test("name: create / getMemory / resolveMemoryId; onConflict ignore skips", async () => { + const id = await mustCreate(FULL, { + tree: "work.named", + content: "body", + name: "doc.md", + }); + expect((await db.getMemory(FULL, id))?.name).toBe("doc.md"); + expect(await db.resolveMemoryId(FULL, "work.named", "doc.md")).toBe(id); + expect(await db.resolveMemoryId(FULL, "work.named", "missing")).toBeNull(); + // resolve is read-gated (level 1), so readonly access still resolves. + expect(await db.resolveMemoryId(READONLY, "work.named", "doc.md")).toBe(id); + + // A bare (tree, name) collision raises; onConflict 'ignore' skips it. + await expect( + db.createMemory(FULL, { tree: "work.named", content: "x", name: "doc.md" }), + ).rejects.toThrow(); + const skipped = await db.createMemory(FULL, { + tree: "work.named", + content: "x", + name: "doc.md", + onConflict: "ignore", + }); + // Skip still reports the existing row's id so the caller can read it back. + expect(skipped).toEqual({ id, status: "skipped" }); + expect((await db.getMemory(FULL, id))?.content).toBe("body"); // untouched +}); + +test("createMemory raises on a bare duplicate explicit id", async () => { const id = "01900000-0000-7000-8000-0000000000d0"; const first = await db.createMemory(FULL, { id, tree: "work.dup", content: "original", }); - expect(first).toEqual({ id, inserted: true }); + expect(first).toEqual({ id, status: "inserted" }); - // Re-submitting the same id is a no-op skip, not an error. - const second = await db.createMemory(FULL, { - id, - tree: "work.dup", - content: "replacement", - }); - expect(second).toBeNull(); + // Re-submitting the same id with no upsert / replace key is a hard conflict. + await expect( + db.createMemory(FULL, { + id, + tree: "work.dup", + content: "replacement", + }), + ).rejects.toThrow(); expect((await db.getMemory(FULL, id))?.content).toBe("original"); }); -test("createMemory with replaceIfMetaDiffers rewrites stale rows in place", async () => { +test("createMemory onConflict 'replace' rewrites only when a field differs", async () => { const id = "01900000-0000-7000-8000-0000000000d1"; await db.createMemory(FULL, { id, @@ -104,26 +134,27 @@ test("createMemory with replaceIfMetaDiffers rewrites stale rows in place", asyn meta: { importer_version: "1" }, }); - // Same version → skip. + // Identical re-submit → content-aware replace is a no-op (skipped). const same = await db.createMemory(FULL, { id, tree: "work.upsert", - content: "render v1 again", + content: "render v1", meta: { importer_version: "1" }, - replaceIfMetaDiffers: "importer_version", + onConflict: "replace", }); - expect(same).toBeNull(); + expect(same).toEqual({ id, status: "skipped" }); expect((await db.getMemory(FULL, id))?.content).toBe("render v1"); - // Bumped version → replaced, reported as an update (inserted: false). + // Bumped version re-render → meta + content differ → replaced, reported as an + // update. The importer_version stamp drives this via meta. const bumped = await db.createMemory(FULL, { id, tree: "work.upsert", content: "render v2", meta: { importer_version: "2" }, - replaceIfMetaDiffers: "importer_version", + onConflict: "replace", }); - expect(bumped).toEqual({ id, inserted: false }); + expect(bumped).toEqual({ id, status: "updated" }); const after = await db.getMemory(FULL, id); expect(after?.content).toBe("render v2"); expect(after?.meta).toEqual({ importer_version: "2" }); @@ -140,19 +171,21 @@ test("batchCreateMemories upserts a batch in one call", async () => { const rows = await db.batchCreateMemories( FULL, [ + // changed content → replaced { id: stale, tree: "work.batch", content: "new", meta: { v: "2" } }, - { id: fresh, tree: "work.batch", content: "untouched", meta: { v: "2" } }, + // identical content+meta → content-aware replace no-op (skipped) + { id: fresh, tree: "work.batch", content: "current", meta: { v: "2" } }, { tree: "work.batch", content: "generated id" }, ], - "v", + "replace", ); - const byId = new Map(rows.map((r) => [r.id, r.inserted])); - expect(rows).toHaveLength(2); // fresh skipped → absent - expect(byId.get(stale)).toBe(false); + // One row per input, in input order, each with a status. + expect(rows.map((r) => r.status)).toEqual(["updated", "skipped", "inserted"]); + expect(rows[0]?.id).toBe(stale); + expect(rows[1]?.id).toBe(fresh); expect((await db.getMemory(FULL, stale))?.content).toBe("new"); expect((await db.getMemory(FULL, fresh))?.content).toBe("current"); - const generated = rows.find((r) => r.id !== stale); - expect(generated?.inserted).toBe(true); + const generated = rows[2]; expect((await db.getMemory(FULL, generated?.id as string))?.content).toBe( "generated id", ); @@ -287,3 +320,21 @@ test("copyTree copies a subtree without removing the source", async () => { expect(await db.countTree(FULL, { tree: "work.copy_src" }, 1)).toBe(2); expect(await db.countTree(FULL, { tree: "work.copy_dst" }, 1)).toBe(2); }); + +test("copyTree preserves the name on copied memories", async () => { + await db.createMemory(FULL, { + tree: "work.cpname_src", + name: "doc.md", + content: "named", + }); + + await db.copyTree(FULL, "work.cpname_src", "work.cpname_dst"); + + // The copy lands at the new path under the SAME name (a fresh id), so it is + // addressable by (tree, name) there — not silently nulled out. + const srcId = await db.resolveMemoryId(FULL, "work.cpname_src", "doc.md"); + const dstId = await db.resolveMemoryId(FULL, "work.cpname_dst", "doc.md"); + expect(srcId).not.toBeNull(); + expect(dstId).not.toBeNull(); + expect(dstId).not.toBe(srcId); // distinct row, same name +}); diff --git a/packages/engine/space/db.ts b/packages/engine/space/db.ts index 10d146fe..031f783e 100644 --- a/packages/engine/space/db.ts +++ b/packages/engine/space/db.ts @@ -5,12 +5,20 @@ import type { HybridSearchOptions, Memory, MemoryPatch, + OnConflict, SearchOptions, SearchResultItem, TreeAccess, TreeListEntry, + WriteStatus, } from "./types"; +/** One row's outcome: its stored id + what happened. */ +export interface WriteResult { + id: string; + status: WriteStatus; +} + /** * The space data-plane layer for one space schema (me_). * @@ -20,29 +28,34 @@ import type { */ export interface SpaceStore { /** - * Insert one memory. When an explicit `params.id` already exists the - * outcome depends on `params.replaceIfMetaDiffers`: unset → skip (null); - * set to a meta key → the existing row is replaced when its value for that - * key differs from the new record's (`inserted: false`), else skipped. - * Deterministic-id importers use this to re-submit idempotently and push - * version-bump re-renders in the same call. + * Insert one memory. When the idempotency key (a named row's (tree, name), + * else the explicit `params.id`) already exists the outcome depends on + * `params.onConflict`: 'error' (default) raises, 'replace' overwrites in place + * when a field differs, 'ignore' skips. Always returns the row's stored id + * (the kept existing id on an update/skip, readable even when skipped) plus + * its status. */ createMemory( treeAccess: TreeAccess, params: CreateMemoryParams, - ): Promise<{ id: string; inserted: boolean } | null>; + ): Promise; /** - * Set-based createMemory for a whole batch: one statement, one round - * trip, same per-row conflict semantics. Returns one row per - * insert/replace — skipped rows are absent — and an explicit id repeated - * within the batch collapses to its first occurrence. Atomic. + * Set-based createMemory for a whole batch: one statement, one round trip, + * same per-row conflict semantics. Returns one {id, status} per input in + * input order (atomic). A duplicate idempotency key within the batch raises. */ batchCreateMemories( treeAccess: TreeAccess, memories: CreateMemoryParams[], - replaceIfMetaDiffers?: string, - ): Promise>; + onConflict?: OnConflict, + ): Promise; getMemory(treeAccess: TreeAccess, id: string): Promise; + /** Resolve a (tree, name) reference to its memory id (read-gated), or null. */ + resolveMemoryId( + treeAccess: TreeAccess, + tree: string, + name: string, + ): Promise; patchMemory( treeAccess: TreeAccess, id: string, @@ -92,6 +105,7 @@ function mapMemory(row: Record): Memory { return { id: row.id as string, tree: row.tree as string, + name: (row.name as string | null) ?? null, meta: (row.meta as Record) ?? {}, temporal: (row.temporal as string | null) ?? null, content: row.content as string, @@ -132,54 +146,67 @@ export function spaceStore(sql: Sql, schema: string): SpaceStore { return { async createMemory(treeAccess, p) { + // create_memory returns exactly one (id, status) row — the stored id (the + // kept existing id on an update/skip) and what happened. const [row] = await sql` - select id, inserted from ${sch}.create_memory( + select id, status from ${sch}.create_memory( ${jb(treeAccess)}, ${p.tree}::ltree, ${p.content}, ${p.id ?? null}, ${jb(p.meta)}, ${p.temporal ?? null}::tstzrange, - ${p.replaceIfMetaDiffers ?? null} + ${p.name ?? null}, + ${p.onConflict ?? "error"} )`; - // Zero rows = the explicit id already exists and was skipped (version - // match, no replace key, or no write access on the existing row's tree). - if (!row) return null; - return { id: row.id as string, inserted: Boolean(row.inserted) }; + return { + id: (row as { id: string }).id, + status: (row as { status: WriteStatus }).status, + }; }, - async batchCreateMemories(treeAccess, memories, replaceIfMetaDiffers) { + async batchCreateMemories(treeAccess, memories, onConflict) { if (memories.length === 0) return []; // Parallel arrays aligned by position. Metas travel as ONE jsonb array // via sql.json — a jsonb[] parameter would double-encode each element - // into a string scalar (see the jb() note above). + // into a string scalar (see the jb() note above). batch_create_memory + // returns one (ord, id, status) row per input in input order. const rows = await sql` - select id, inserted from ${sch}.batch_create_memory( + select id, status from ${sch}.batch_create_memory( ${jb(treeAccess)}, ${memories.map((m) => m.id ?? null)}::uuid[], ${memories.map((m) => m.tree)}::ltree[], ${memories.map((m) => m.content)}::text[], ${jb(memories.map((m) => m.meta ?? {}))}, ${memories.map((m) => m.temporal ?? null)}::tstzrange[], - ${replaceIfMetaDiffers ?? null} - )`; + ${memories.map((m) => m.name ?? null)}::text[], + ${onConflict ?? "error"} + ) + order by ord`; return rows.map((r) => ({ id: r.id as string, - inserted: Boolean(r.inserted), + status: r.status as WriteStatus, })); }, async getMemory(treeAccess, id) { const [row] = await sql` - select id, tree::text as tree, meta, temporal::text as temporal, + select id, tree::text as tree, name, meta, temporal::text as temporal, content, has_embedding, created_at, updated_at from ${sch}.get_memory(${jb(treeAccess)}, ${id})`; return row ? mapMemory(row) : null; }, + async resolveMemoryId(treeAccess, tree, name) { + const [row] = await sql` + select ${sch}.resolve_memory_id(${jb(treeAccess)}, ${tree}::ltree, ${name}) as id`; + return (row?.id as string | null) ?? null; + }, + async patchMemory(treeAccess, id, patch) { const obj: Record = {}; if (patch.tree !== undefined) obj.tree = patch.tree; + if (patch.name !== undefined) obj.name = patch.name; // null clears it if (patch.meta !== undefined) obj.meta = patch.meta; if (patch.temporal !== undefined) obj.temporal = patch.temporal; if (patch.content !== undefined) obj.content = patch.content; @@ -243,7 +270,7 @@ export function spaceStore(sql: Sql, schema: string): SpaceStore { const o = options; const rows = await sql` select id, meta, tree::text as tree, temporal::text as temporal, - content, has_embedding, created_at, updated_at, score + content, name, has_embedding, created_at, updated_at, score from ${sch}.search_memory( ${jb(treeAccess)}, ${bm25(o.bm25)}, @@ -268,7 +295,7 @@ export function spaceStore(sql: Sql, schema: string): SpaceStore { const o = options; const rows = await sql` select id, meta, tree::text as tree, temporal::text as temporal, - content, has_embedding, created_at, updated_at, score + content, name, has_embedding, created_at, updated_at, score from ${sch}.hybrid_search_memory( ${jb(treeAccess)}, ${bm25(o.bm25)}, diff --git a/packages/engine/space/types.ts b/packages/engine/space/types.ts index 215c17fe..e4840350 100644 --- a/packages/engine/space/types.ts +++ b/packages/engine/space/types.ts @@ -13,9 +13,17 @@ export type { TreeAccess }; /** tstzrange rendered as its text form, e.g. "[2024-01-01,2024-01-02)". */ export type TemporalRange = string; +/** Conflict action on the idempotency key (named rows: (tree, name); else id). */ +export type OnConflict = "error" | "replace" | "ignore"; + +/** What a create/batchCreate did to one row. */ +export type WriteStatus = "inserted" | "updated" | "skipped"; + export interface Memory { id: string; tree: string; + /** Optional, mutable leaf name; null for unnamed memories. */ + name: string | null; meta: Record; temporal: TemporalRange | null; content: string; @@ -31,19 +39,26 @@ export interface SearchResultItem extends Memory { export interface CreateMemoryParams { tree: string; content: string; + /** Optional explicit id (preserves identity for import/export). */ id?: string; + /** Optional leaf name; the (tree, name) idempotency key when no id is given. */ + name?: string; meta?: Record; temporal?: TemporalRange; /** - * Meta key for conditional replace: when an explicit `id` already exists, - * replace the row iff its meta value for this key differs from the new - * record (e.g. importer_version). Default: duplicates are skipped. + * Action when the idempotency key conflicts: 'error' (default) raises, + * 'replace' overwrites in place (a no-op unless content/meta/temporal differ), + * 'ignore' skips. Returns null when the row is skipped (ignore, or replace + * no-op). Deterministic-id importers pass 'replace' and stamp + * meta.importer_version, so a version bump makes meta differ and re-renders. */ - replaceIfMetaDiffers?: string; + onConflict?: OnConflict; } export interface MemoryPatch { tree?: string; + /** null clears the name; a string sets/renames; undefined leaves it. */ + name?: string | null; meta?: Record; temporal?: TemporalRange | null; content?: string; diff --git a/packages/protocol/fields.ts b/packages/protocol/fields.ts index 1e93f4e7..d25be931 100644 --- a/packages/protocol/fields.ts +++ b/packages/protocol/fields.ts @@ -61,6 +61,49 @@ export const treePathSchema = z */ export const SHARE_NAMESPACE = "share"; +/** + * Memory name (leaf) — an optional, mutable, filename-like slug, unique within + * its tree path. Must start alphanumeric, then `[A-Za-z0-9._-]` (no slashes or + * spaces; dots are fine because a name is never an ltree label). Mirrors the + * `memory.name` CHECK in the space schema. + */ +export const memoryNameSchema = z + .string() + .min(1, "name must not be empty") + .max(128, "name must be at most 128 characters") + .regex( + /^[A-Za-z0-9][A-Za-z0-9._-]*$/, + "name must be a filename-like slug: start alphanumeric, then [A-Za-z0-9._-]", + ); + +/** + * A `tree/name` address (for getByPath/deleteByPath), split at the final `/`: + * the leaf is the name, the rest is the tree. The tree part is lenient (as + * elsewhere), but the leaf must be a valid memory name — so a trailing `/` + * (empty leaf) or a leaf with name-illegal chars (a leading `.`/`-`/`~`) fails + * fast as VALIDATION_ERROR rather than masquerading as NOT_FOUND. + */ +export const memoryPathSchema = treePathSchema + .min(1, "path is required") + .refine( + (p) => memoryNameSchema.safeParse(p.slice(p.lastIndexOf("/") + 1)).success, + "path must end in a valid memory name (filename-like slug; no trailing '/')", + ); + +/** + * What a create/batchCreate row does when it conflicts with the existing memory + * on its idempotency key — a named row's `(tree, name)` slot (name takes + * precedence), else the explicit id: `error` (default) raises CONFLICT; + * `replace` overwrites in place but is a no-op when nothing changed; `ignore` + * skips, leaving the existing row. Note this governs the idempotency-key + * conflict only — a row whose explicit id collides with a *different* existing + * row still raises a pk violation regardless of `ignore`/`replace`. + */ +export const onConflictSchema = z.enum(["error", "replace", "ignore"]); + +/** What a create/batchCreate did to one row. */ +export const writeStatusSchema = z.enum(["inserted", "updated", "skipped"]); + /** * Tree filter schema (ltree, lquery, or ltxtquery). * More permissive than treePathSchema since it allows query operators. diff --git a/packages/protocol/memory.test.ts b/packages/protocol/memory.test.ts new file mode 100644 index 00000000..7161966f --- /dev/null +++ b/packages/protocol/memory.test.ts @@ -0,0 +1,104 @@ +import { describe, expect, test } from "bun:test"; +import { + memoryNameSchema, + memoryPathSchema, + onConflictSchema, +} from "./fields.ts"; +import { memoryCreateParams, memoryGetByPathParams } from "./memory.ts"; + +describe("memoryNameSchema", () => { + test("accepts filename-like slugs (dots, hyphens, underscores, mixed case)", () => { + for (const ok of [ + "jwt-rotation", + "config.yaml", + "README.md", + "v1.2_notes", + "a", + ]) { + expect(memoryNameSchema.safeParse(ok).success).toBe(true); + } + }); + + test("rejects slashes, spaces, leading dot/hyphen, and > 128 chars", () => { + for (const bad of [ + "", + "a/b", + "has space", + ".hidden", + "..", + "-x", + "a".repeat(129), + ]) { + expect(memoryNameSchema.safeParse(bad).success).toBe(false); + } + }); +}); + +describe("memoryPathSchema", () => { + test("accepts a path whose leaf is a valid memory name", () => { + for (const ok of [ + "share/auth/jwt-rotation", + "/share/auth/jwt-rotation", + "~/notes/todo", + "share/config.yaml", + "jwt-rotation", // no slash → leaf is the whole string + ]) { + expect(memoryPathSchema.safeParse(ok).success).toBe(true); + } + }); + + test("rejects a trailing slash (empty leaf) or a leaf with name-illegal chars", () => { + for (const bad of [ + "", // empty + "share/auth/", // trailing slash → empty leaf + "share/.hidden", // leaf starts with '.' + "share/-x", // leaf starts with '-' + "~", // leaf is '~' (not a valid name) + "share/has space", + ]) { + expect(memoryPathSchema.safeParse(bad).success).toBe(false); + } + }); + + test("getByPath params reject an invalid path (VALIDATION_ERROR, not NOT_FOUND)", () => { + expect( + memoryGetByPathParams.safeParse({ path: "share/auth/" }).success, + ).toBe(false); + expect( + memoryGetByPathParams.safeParse({ path: "share/auth/jwt-rotation" }) + .success, + ).toBe(true); + }); +}); + +describe("onConflictSchema", () => { + test("accepts error|replace|ignore, rejects others", () => { + for (const ok of ["error", "replace", "ignore"]) { + expect(onConflictSchema.safeParse(ok).success).toBe(true); + } + expect(onConflictSchema.safeParse("upsert").success).toBe(false); + }); +}); + +describe("memoryCreateParams", () => { + test("name + onConflict are optional and validated", () => { + expect( + memoryCreateParams.safeParse({ content: "x", tree: "share" }).success, + ).toBe(true); + expect( + memoryCreateParams.safeParse({ + content: "x", + tree: "share/auth", + name: "jwt-rotation", + onConflict: "replace", + }).success, + ).toBe(true); + expect( + memoryCreateParams.safeParse({ + content: "x", + tree: "share", + name: "bad name", + }).success, + ).toBe(false); + }); +}); diff --git a/packages/protocol/memory.ts b/packages/protocol/memory.ts index ab466cee..10ea9756 100644 --- a/packages/protocol/memory.ts +++ b/packages/protocol/memory.ts @@ -3,13 +3,17 @@ */ import { z } from "zod"; import { + memoryNameSchema, + memoryPathSchema, metaSchema, + onConflictSchema, searchWeightsSchema, temporalFilterSchema, temporalSchema, treeFilterSchema, treePathSchema, uuidv7Schema, + writeStatusSchema, } from "./fields.ts"; // ============================================================================= @@ -18,13 +22,21 @@ import { /** * memory.create params. + * + * `id` is optional — supply it to preserve identity (import/export, deterministic + * importers); omit it for a server-generated uuidv7. `name` is the optional leaf + * slug. `onConflict` governs a clash on the idempotency key (a named row's + * (tree, name) slot, which takes precedence over id; else the explicit id): + * default `error`. */ export const memoryCreateParams = z.object({ id: uuidv7Schema.optional().nullable(), content: z.string().min(1, "content is required"), meta: metaSchema.optional().nullable(), tree: treePathSchema.min(1, "tree path is required"), + name: memoryNameSchema.optional().nullable(), temporal: temporalSchema.optional().nullable(), + onConflict: onConflictSchema.optional().nullable(), }); export type MemoryCreateParams = z.infer; @@ -32,11 +44,13 @@ export type MemoryCreateParams = z.infer; /** * memory.batchCreate params. * - * `replaceIfMetaDiffers` names a meta key for conditional replace: a memory - * whose explicit id already exists is rewritten in place when the stored - * row's value for that key differs from the submitted one (deterministic-id - * importers pass e.g. "importer_version" so version bumps re-render existing - * rows), and skipped when it matches. Without it, duplicates are skipped. + * `onConflict` governs a clash on each row's idempotency key (a named row's + * (tree, name) slot, which takes precedence over id; else its explicit id): + * `error` (default) raises, `replace` overwrites in place when + * content/meta/temporal differ (a no-op when identical), `ignore` skips. + * Deterministic-id importers pass `replace` and stamp + * `meta.importer_version`, so an unchanged re-import is a no-op while a version + * bump makes meta differ and re-renders. */ export const memoryBatchCreateParams = z.object({ memories: z @@ -46,18 +60,20 @@ export const memoryBatchCreateParams = z.object({ content: z.string().min(1, "content is required"), meta: metaSchema.optional().nullable(), tree: treePathSchema.min(1, "tree path is required"), + name: memoryNameSchema.optional().nullable(), temporal: temporalSchema.optional().nullable(), }), ) .min(1, "at least one memory required") .max(1000, "maximum 1000 memories per batch"), - replaceIfMetaDiffers: z.string().min(1).optional().nullable(), + onConflict: onConflictSchema.optional().nullable(), }); export type MemoryBatchCreateParams = z.infer; /** - * memory.get params. + * memory.get params — by id. To address by the `tree/name` form use + * memory.getByPath. */ export const memoryGetParams = z.object({ id: uuidv7Schema, @@ -65,6 +81,19 @@ export const memoryGetParams = z.object({ export type MemoryGetParams = z.infer; +/** + * memory.getByPath params — address a named memory by its `tree/name` path + * (e.g. "share/auth/jwt-rotation"). The server splits at the final `/`: the + * last segment is the name, the rest is the tree (with `~`/separators + * normalized). The leaf must be a valid memory name (VALIDATION_ERROR + * otherwise); NOT_FOUND when no such named memory exists. + */ +export const memoryGetByPathParams = z.object({ + path: memoryPathSchema, +}); + +export type MemoryGetByPathParams = z.infer; + /** * memory.update params. */ @@ -73,13 +102,16 @@ export const memoryUpdateParams = z.object({ content: z.string().min(1).optional().nullable(), meta: metaSchema.optional().nullable(), tree: treePathSchema.optional().nullable(), + // null clears the name; a string sets/renames; omitted leaves it unchanged. + name: memoryNameSchema.optional().nullable(), temporal: temporalSchema.optional().nullable(), }); export type MemoryUpdateParams = z.infer; /** - * memory.delete params. + * memory.delete params — delete one memory by id. (Address a named memory by + * its path with memory.deleteByPath; delete a whole subtree with deleteTree.) */ export const memoryDeleteParams = z.object({ id: uuidv7Schema, @@ -87,6 +119,17 @@ export const memoryDeleteParams = z.object({ export type MemoryDeleteParams = z.infer; +/** + * memory.deleteByPath params — delete one named memory by its `tree/name` + * path (split like memory.getByPath). The leaf must be a valid memory name + * (VALIDATION_ERROR otherwise); NOT_FOUND when it doesn't resolve. + */ +export const memoryDeleteByPathParams = z.object({ + path: memoryPathSchema, +}); + +export type MemoryDeleteByPathParams = z.infer; + /** * memory.search params. */ @@ -170,6 +213,7 @@ export const memoryResponse = z.object({ content: z.string(), meta: z.record(z.string(), z.unknown()), tree: z.string(), + name: z.string().nullable(), temporal: z .object({ start: z.string(), @@ -193,17 +237,30 @@ export const memoryWithScoreResponse = memoryResponse.extend({ export type MemoryWithScoreResponse = z.infer; +/** + * One row's outcome from create/batchCreate: its stored `id` (the kept existing + * id on a (tree, name) update/skip — readable even when skipped) and `status` + * ('inserted' | 'updated' | 'skipped'). + */ +export const memoryWriteResult = z.object({ + id: z.string(), + status: writeStatusSchema, +}); + +export type MemoryWriteResult = z.infer; + /** * memory.batchCreate result. * - * `ids` are the freshly inserted memories; `updatedIds` are existing rows - * rewritten in place via `replaceIfMetaDiffers`. A submitted explicit id in - * neither array (and not in a failed request) was skipped — it already - * existed at the same meta-key value. + * `results` carries one entry per submitted memory, in request order (so + * `results[i]` is the outcome of `memories[i]`). Each is `{ id, status }`: + * `inserted` (new row), `updated` (existing row rewritten by `onConflict: + * 'replace'`), or `skipped` (already existed and nothing differed, or + * `onConflict: 'ignore'`). Derive inserted/updated/skipped sets by filtering on + * `status`. */ export const memoryBatchCreateResult = z.object({ - ids: z.array(z.string()), - updatedIds: z.array(z.string()), + results: z.array(memoryWriteResult), }); export type MemoryBatchCreateResult = z.infer; diff --git a/packages/server/rpc/memory/management.integration.test.ts b/packages/server/rpc/memory/management.integration.test.ts index eaf20cc8..794f38d7 100644 --- a/packages/server/rpc/memory/management.integration.test.ts +++ b/packages/server/rpc/memory/management.integration.test.ts @@ -284,7 +284,7 @@ test("grant: set / list / remove", async () => { grants: { principalId: string; treePath: string; access: number }[]; }>("grant.list", { principalId: other }); expect(grants.grants).toHaveLength(1); - expect(grants.grants[0]?.treePath).toBe("docs"); + expect(grants.grants[0]?.treePath).toBe("/docs"); expect(grants.grants[0]?.access).toBe(1); expect( @@ -616,7 +616,7 @@ test("grant authority is path-scoped: a subtree owner delegates within it", asyn const underProj = await call<{ grants: { treePath: string }[]; }>("grant.list", { treePath: "proj" }, as); - expect(underProj.grants.some((g) => g.treePath === "proj.sub")).toBe(true); + expect(underProj.grants.some((g) => g.treePath === "/proj/sub")).toBe(true); await expectAppError(call("grant.list", {}, as), "FORBIDDEN"); }); diff --git a/packages/server/rpc/memory/memory.integration.test.ts b/packages/server/rpc/memory/memory.integration.test.ts index 4a7ecc94..5669a4b6 100644 --- a/packages/server/rpc/memory/memory.integration.test.ts +++ b/packages/server/rpc/memory/memory.integration.test.ts @@ -133,14 +133,14 @@ test("~ home + lenient separators normalize on input, reverse-map on output", as const home = `home.${principalId.replace(/-/g, "")}`; // `~/notes` (slash accepted on input) stores under the caller's home and - // reads back in canonical dot form as `~.notes`. + // reads back in canonical slash form as `~/notes`. const a = await call<{ id: string; tree: string }>("memory.create", { content: "home note", tree: "~/notes", }); - expect(a.tree).toBe("~.notes"); + expect(a.tree).toBe("~/notes"); expect((await call<{ tree: string }>("memory.get", { id: a.id })).tree).toBe( - "~.notes", + "~/notes", ); // The raw stored ltree is home..notes — proves real expansion, not just display. @@ -155,14 +155,14 @@ test("~ home + lenient separators normalize on input, reverse-map on output", as content: "slashy", tree: "/work/projects/", }); - expect(b.tree).toBe("work.projects"); + expect(b.tree).toBe("/work/projects"); - // memory.tree with a `~` base finds the home node, reverse-mapped to `~.notes`. + // memory.tree with a `~` base finds the home node, reverse-mapped to `~/notes`. const tree = await call<{ nodes: { path: string; count: number }[] }>( "memory.tree", { tree: "~" }, ); - expect(tree.nodes.some((n) => n.path === "~.notes")).toBe(true); + expect(tree.nodes.some((n) => n.path === "~/notes")).toBe(true); // An illegal label is a validation error. await expectAppError( @@ -196,7 +196,7 @@ test("create → get round-trips content/tree/meta and createdBy is null", async hasEmbedding: boolean; }>("memory.get", { id: created.id }); expect(got.content).toBe("hello world"); - expect(got.tree).toBe("share.notes.work"); + expect(got.tree).toBe("/share/notes/work"); expect(got.meta).toEqual({ tag: "a" }); expect(got.hasEmbedding).toBe(false); }); @@ -217,6 +217,103 @@ test("create with temporal round-trips as {start,end}", async () => { }); }); +test("name: getByPath / deleteByPath address a named memory; response carries name", async () => { + const created = await call<{ id: string; name: string | null }>( + "memory.create", + { content: "rotation notes", tree: "share/auth", name: "jwt-rotation" }, + ); + expect(created.name).toBe("jwt-rotation"); + + // getByPath splits the final segment as the name and resolves it. + const got = await call<{ id: string; tree: string; name: string | null }>( + "memory.getByPath", + { path: "share/auth/jwt-rotation" }, + ); + expect(got.id).toBe(created.id); + expect(got.tree).toBe("/share/auth"); + expect(got.name).toBe("jwt-rotation"); + + // a path that doesn't resolve is NOT_FOUND + await expectAppError( + call("memory.getByPath", { path: "share/auth/missing" }), + "NOT_FOUND", + ); + + const del = await call<{ deleted: boolean }>("memory.deleteByPath", { + path: "share/auth/jwt-rotation", + }); + expect(del.deleted).toBe(true); + await expectAppError(call("memory.get", { id: created.id }), "NOT_FOUND"); +}); + +test("name: create onConflict error (default) / ignore / replace", async () => { + const first = await call<{ id: string }>("memory.create", { + content: "v1", + tree: "share/conf", + name: "doc", + }); + + // default → CONFLICT + await expectAppError( + call("memory.create", { content: "v2", tree: "share/conf", name: "doc" }), + "CONFLICT", + ); + + // ignore → returns the existing memory (idempotent), unchanged + const ignored = await call<{ id: string; content: string }>("memory.create", { + content: "v2", + tree: "share/conf", + name: "doc", + onConflict: "ignore", + }); + expect(ignored.id).toBe(first.id); + expect(ignored.content).toBe("v1"); + + // replace → overwrites in place, same id + const replaced = await call<{ id: string; content: string }>( + "memory.create", + { content: "v2", tree: "share/conf", name: "doc", onConflict: "replace" }, + ); + expect(replaced.id).toBe(first.id); + expect(replaced.content).toBe("v2"); +}); + +test("update can rename and clear a name", async () => { + const created = await call<{ id: string }>("memory.create", { + content: "body", + tree: "share/ren", + name: "old", + }); + const renamed = await call<{ name: string | null }>("memory.update", { + id: created.id, + name: "new", + }); + expect(renamed.name).toBe("new"); + const cleared = await call<{ name: string | null }>("memory.update", { + id: created.id, + name: null, + }); + expect(cleared.name).toBeNull(); +}); + +test("update adds a name to a previously-unnamed memory", async () => { + const created = await call<{ id: string; name: string | null }>( + "memory.create", + { content: "body", tree: "share/addname" }, + ); + expect(created.name).toBeNull(); + const named = await call<{ name: string | null }>("memory.update", { + id: created.id, + name: "added", + }); + expect(named.name).toBe("added"); + // The new name is now resolvable as a path. + const byPath = await call<{ id: string }>("memory.getByPath", { + path: "share/addname/added", + }); + expect(byPath.id).toBe(created.id); +}); + test("update patches fields", async () => { const created = await call<{ id: string }>("memory.create", { content: "before", @@ -227,7 +324,7 @@ test("update patches fields", async () => { { id: created.id, content: "after", tree: "share.a.b" }, ); expect(updated.content).toBe("after"); - expect(updated.tree).toBe("share.a.b"); + expect(updated.tree).toBe("/share/a/b"); }); test("delete removes; get then NOT_FOUND", async () => { @@ -249,7 +346,7 @@ test("get / delete unknown id → NOT_FOUND", async () => { }); test("batchCreate inserts all and is retrievable", async () => { - const res = await call<{ ids: string[]; updatedIds: string[] }>( + const res = await call<{ results: { id: string; status: string }[] }>( "memory.batchCreate", { memories: [ @@ -259,8 +356,8 @@ test("batchCreate inserts all and is retrievable", async () => { ], }, ); - expect(res.ids).toHaveLength(3); - expect(res.updatedIds).toHaveLength(0); + expect(res.results).toHaveLength(3); + expect(res.results.every((r) => r.status === "inserted")).toBe(true); const count = await call<{ count: number }>("memory.countTree", { tree: "share.batch", }); @@ -303,22 +400,24 @@ test("create with a duplicate explicit id → CONFLICT", async () => { ); }); -test("batchCreate without replaceIfMetaDiffers skips duplicates", async () => { +test("batchCreate with a bare duplicate id raises CONFLICT", async () => { const id = "01941000-0000-7000-8000-00000000c0f2"; await call("memory.batchCreate", { memories: [{ id, content: "original", tree: "share.skip" }], }); - const res = await call<{ ids: string[]; updatedIds: string[] }>( - "memory.batchCreate", - { memories: [{ id, content: "replacement", tree: "share.skip" }] }, + // Default onConflict ('error') → the duplicate id is a hard conflict that + // aborts the batch (importers pass onConflict 'replace' to stay idempotent). + await expectAppError( + call("memory.batchCreate", { + memories: [{ id, content: "replacement", tree: "share.skip" }], + }), + "CONFLICT", ); - expect(res.ids).toHaveLength(0); - expect(res.updatedIds).toHaveLength(0); const got = await call<{ content: string }>("memory.get", { id }); expect(got.content).toBe("original"); }); -test("batchCreate with replaceIfMetaDiffers splits insert/update/skip", async () => { +test("batchCreate onConflict 'replace' splits insert/update/skip", async () => { const stale = "01941000-0000-7000-8000-00000000c0f3"; const fresh = "01941000-0000-7000-8000-00000000c0f4"; const brandNew = "01941000-0000-7000-8000-00000000c0f5"; @@ -329,24 +428,30 @@ test("batchCreate with replaceIfMetaDiffers splits insert/update/skip", async () ], }); - const res = await call<{ ids: string[]; updatedIds: string[] }>( + const res = await call<{ results: { id: string; status: string }[] }>( "memory.batchCreate", { memories: [ + // changed content → replaced { id: stale, content: "new render", tree: "share.up", meta: { v: "2" }, }, - { id: fresh, content: "untouched", tree: "share.up", meta: { v: "2" } }, + // identical content+meta → content-aware replace no-op (skipped) + { id: fresh, content: "current", tree: "share.up", meta: { v: "2" } }, { id: brandNew, content: "added", tree: "share.up", meta: { v: "2" } }, ], - replaceIfMetaDiffers: "v", + onConflict: "replace", }, ); - expect(res.ids).toEqual([brandNew]); - expect(res.updatedIds).toEqual([stale]); + // One row per input, in input order: replaced / skipped / inserted. + expect(res.results).toEqual([ + { id: stale, status: "updated" }, + { id: fresh, status: "skipped" }, + { id: brandNew, status: "inserted" }, + ]); const updated = await call<{ content: string }>("memory.get", { id: stale }); expect(updated.content).toBe("new render"); @@ -367,11 +472,11 @@ test("tree returns descendant node counts under a path", async () => { { tree: "share.root" }, ); const byPath = Object.fromEntries(res.nodes.map((n) => [n.path, n.count])); - expect(byPath["share.root.a"]).toBe(2); - expect(byPath["share.root.a.deep"]).toBe(1); - expect(byPath["share.root.b"]).toBe(1); + expect(byPath["/share/root/a"]).toBe(2); + expect(byPath["/share/root/a/deep"]).toBe(1); + expect(byPath["/share/root/b"]).toBe(1); // the base path itself is excluded - expect(byPath["share.root"]).toBeUndefined(); + expect(byPath["/share/root"]).toBeUndefined(); }); test("tree respects levels depth limit", async () => { @@ -383,8 +488,8 @@ test("tree respects levels depth limit", async () => { levels: 1, }); const paths = res.nodes.map((n) => n.path); - expect(paths).toContain("share.t.a"); - expect(paths).not.toContain("share.t.a.b"); + expect(paths).toContain("/share/t/a"); + expect(paths).not.toContain("/share/t/a/b"); }); test("move relocates a subtree (dryRun counts without moving)", async () => { @@ -519,7 +624,7 @@ test("search: tree filter only (no ranking) returns matches", async () => { tree: "share.scope", }); expect(res.results.length).toBe(1); - expect(res.results[0]?.tree).toBe("share.scope.a"); + expect(res.results[0]?.tree).toBe("/share/scope/a"); }); test("search: tree lquery wildcard matches descendants", async () => { @@ -539,7 +644,7 @@ test("search: tree lquery wildcard matches descendants", async () => { tree: "share.proj.*", }); const trees = res.results.map((r) => r.tree).sort(); - expect(trees).toEqual(["share.proj", "share.proj.a", "share.proj.a.deep"]); + expect(trees).toEqual(["/share/proj", "/share/proj/a", "/share/proj/a/deep"]); }); test("search: tree ltxtquery (label boolean) matches by label", async () => { @@ -553,7 +658,7 @@ test("search: tree ltxtquery (label boolean) matches by label", async () => { const res = await call<{ results: { tree: string }[] }>("memory.search", { tree: "alpha & beta", }); - expect(res.results.map((r) => r.tree)).toEqual(["share.alpha.beta"]); + expect(res.results.map((r) => r.tree)).toEqual(["/share/alpha/beta"]); }); test("search: grep alone is rejected", async () => { diff --git a/packages/server/rpc/memory/memory.ts b/packages/server/rpc/memory/memory.ts index e0d476c3..541d9b72 100644 --- a/packages/server/rpc/memory/memory.ts +++ b/packages/server/rpc/memory/memory.ts @@ -23,10 +23,12 @@ import type { MemoryCountTreeParams, MemoryCountTreeResult, MemoryCreateParams, + MemoryDeleteByPathParams, MemoryDeleteParams, MemoryDeleteResult, MemoryDeleteTreeParams, MemoryDeleteTreeResult, + MemoryGetByPathParams, MemoryGetParams, MemoryMoveParams, MemoryMoveResult, @@ -42,8 +44,10 @@ import { memoryCopyParams, memoryCountTreeParams, memoryCreateParams, + memoryDeleteByPathParams, memoryDeleteParams, memoryDeleteTreeParams, + memoryGetByPathParams, memoryGetParams, memoryMoveParams, memorySearchParams, @@ -77,6 +81,14 @@ function mapSpaceError(e: unknown): never { e instanceof Error ? e.message : "Invalid parameter", ); } + // unique_violation — a duplicate id, or a (tree, name) clash (create with no + // upsert/replace directive, or a rename/move into a taken name). + if (code === "23505") { + throw new AppError( + "CONFLICT", + "Memory already exists (id or tree/name conflict)", + ); + } throw e instanceof Error ? e : new Error(String(e)); } @@ -135,6 +147,7 @@ function toMemoryResponse( content: m.content, meta: m.meta, tree: displayTreePath(ctx, m.tree), + name: m.name, temporal: parseTemporal(m.temporal), hasEmbedding: m.hasEmbedding, createdAt: m.createdAt.toISOString(), @@ -171,6 +184,38 @@ function mapTemporalFilter(tf: MemorySearchParams["temporal"]): { // Method Handlers // ============================================================================= +/** + * Split a `folder/name` path at its final `/`: the last segment is the name, + * the rest is the tree. A path with no `/` is a root-level name. + */ +function splitPath(path: string): { tree: string; name: string } { + const i = path.lastIndexOf("/"); + return i === -1 + ? { tree: "", name: path } + : { tree: path.slice(0, i), name: path.slice(i + 1) }; +} + +/** + * Resolve a `folder/name` path to a memory id, expanding `~` and normalizing + * the tree. NOT_FOUND when no such named memory exists (or it's unreadable). + */ +async function resolvePath( + ctx: SpaceRpcContext, + path: string, +): Promise { + const { tree, name } = splitPath(path); + if (name === "") { + throw new AppError("VALIDATION_ERROR", "path must end in a name"); + } + const id = await guard(() => + ctx.store.resolveMemoryId(ctx.treeAccess, inputTreePath(ctx, tree), name), + ); + if (id == null) { + throw new AppError("NOT_FOUND", `Memory not found: ${path}`); + } + return id; +} + /** memory.create */ async function memoryCreate( params: MemoryCreateParams, @@ -180,21 +225,23 @@ async function memoryCreate( const ctx = context as SpaceRpcContext; const { store, treeAccess } = ctx; - const created = await guard(() => + const tree = inputTreePath(ctx, params.tree); + // createMemory returns the row's STORED id for every outcome — including a + // skip ('ignore'/'replace' no-op), where for a named row that's the existing + // row's id (which may differ from a submitted id; name wins over id). A bare + // conflict (default onConflict 'error') raises 23505 → CONFLICT via guard. + const { id } = await guard(() => store.createMemory(treeAccess, { id: params.id ?? undefined, content: params.content, meta: params.meta ?? undefined, - tree: inputTreePath(ctx, params.tree), + tree, + name: params.name ?? undefined, temporal: formatTemporal(params.temporal), + onConflict: params.onConflict ?? undefined, }), ); - if (created === null) { - // The store skips an explicit id that already exists (no replace key is - // passed here). For a single create that's a caller error, not a skip. - throw new AppError("CONFLICT", `Memory already exists: ${params.id}`); - } - const memory = await store.getMemory(treeAccess, created.id); + const memory = await store.getMemory(treeAccess, id); if (!memory) { throw new AppError("INTERNAL_ERROR", "Created memory could not be read"); } @@ -205,11 +252,11 @@ async function memoryCreate( * memory.batchCreate — atomic across the batch (one set-based statement, * `batch_create_memory`). * - * `ids` carries the inserted memories; `updatedIds` the existing rows - * rewritten via `replaceIfMetaDiffers` (conditional upsert). A submitted - * explicit id in neither array was skipped — deterministic-id importers - * re-submit freely and classify the missing ids as already imported. An id - * repeated within one batch collapses to its first occurrence. + * Returns one `{ id, status }` per submitted memory, in request order, so the + * caller can map each result back to its input and see whether it was inserted, + * updated (rewritten by `onConflict: 'replace'`), or skipped (already current, + * or `onConflict: 'ignore'`). A duplicate idempotency key within one batch + * raises. */ async function memoryBatchCreate( params: MemoryBatchCreateParams, @@ -227,17 +274,13 @@ async function memoryBatchCreate( content: m.content, meta: m.meta ?? undefined, tree: inputTreePath(ctx, m.tree), + name: m.name ?? undefined, temporal: formatTemporal(m.temporal), })), - params.replaceIfMetaDiffers ?? undefined, + params.onConflict ?? undefined, ), ); - const ids: string[] = []; - const updatedIds: string[] = []; - for (const r of rows) { - (r.inserted ? ids : updatedIds).push(r.id); - } - return { ids, updatedIds }; + return { results: rows.map((r) => ({ id: r.id, status: r.status })) }; } /** memory.get */ @@ -256,6 +299,23 @@ async function memoryGet( return toMemoryResponse(memory, ctx); } +/** memory.getByPath — address a named memory by its folder/name path. */ +async function memoryGetByPath( + params: MemoryGetByPathParams, + context: HandlerContext, +): Promise { + assertSpaceRpcContext(context); + const ctx = context as SpaceRpcContext; + const { store, treeAccess } = ctx; + + const id = await resolvePath(ctx, params.path); + const memory = await guard(() => store.getMemory(treeAccess, id)); + if (!memory) { + throw new AppError("NOT_FOUND", `Memory not found: ${params.path}`); + } + return toMemoryResponse(memory, ctx); +} + /** memory.update */ async function memoryUpdate( params: MemoryUpdateParams, @@ -269,6 +329,7 @@ async function memoryUpdate( content?: string; meta?: Record; tree?: string; + name?: string | null; temporal?: string | null; } = {}; if (params.content !== undefined && params.content !== null) { @@ -280,6 +341,10 @@ async function memoryUpdate( if (params.tree !== undefined && params.tree !== null) { patch.tree = inputTreePath(ctx, params.tree); } + // null clears the name; a string sets/renames; undefined leaves it unchanged. + if (params.name !== undefined) { + patch.name = params.name; + } if (params.temporal !== undefined) { patch.temporal = params.temporal === null @@ -313,6 +378,23 @@ async function memoryDelete( return { deleted }; } +/** memory.deleteByPath — delete one named memory by its folder/name path. */ +async function memoryDeleteByPath( + params: MemoryDeleteByPathParams, + context: HandlerContext, +): Promise { + assertSpaceRpcContext(context); + const ctx = context as SpaceRpcContext; + const { store, treeAccess } = ctx; + + const id = await resolvePath(ctx, params.path); + const deleted = await guard(() => store.deleteMemory(treeAccess, id)); + if (!deleted) { + throw new AppError("NOT_FOUND", `Memory not found: ${params.path}`); + } + return { deleted }; +} + /** memory.search — hybrid (fulltext+semantic) or single-arm / filter-only. */ async function memorySearch( params: MemorySearchParams, @@ -544,8 +626,10 @@ export const memoryDataMethods = buildRegistry() .register("memory.create", memoryCreateParams, memoryCreate) .register("memory.batchCreate", memoryBatchCreateParams, memoryBatchCreate) .register("memory.get", memoryGetParams, memoryGet) + .register("memory.getByPath", memoryGetByPathParams, memoryGetByPath) .register("memory.update", memoryUpdateParams, memoryUpdate) .register("memory.delete", memoryDeleteParams, memoryDelete) + .register("memory.deleteByPath", memoryDeleteByPathParams, memoryDeleteByPath) .register("memory.search", memorySearchParams, memorySearch) .register("memory.tree", memoryTreeParams, memoryTree) .register("memory.copy", memoryCopyParams, memoryCopy) diff --git a/packages/web/src/api/queries.ts b/packages/web/src/api/queries.ts index b3de3096..ba04f072 100644 --- a/packages/web/src/api/queries.ts +++ b/packages/web/src/api/queries.ts @@ -19,6 +19,19 @@ import { memoryClient } from "./client.ts"; const SEARCH_LIMIT = 1000; +// The RPC wire speaks canonical slash paths (`/share/auth`, `~/a/b`, root `/`); +// the navigation/tree logic in this app is ltree-dot-native (sentinels, lquery +// building, path splitting), so we convert incoming paths to dots at the fetch +// boundary: drop the leading `/` (the absolute anchor), then swap `/` → `.`. +// `~/a/b` → `~.a.b`, `/share/auth` → `share.auth`, `/` → ``. Display rendering +// of slashes is handled separately in the view layer. +function wirePathToDot(path: string): string { + return path.replace(/^\//, "").replace(/\//g, "."); +} +function memoryToDot(m: T): T { + return { ...m, tree: wirePathToDot(m.tree) }; +} + /** * Convert an exact ltree path to an lquery pattern that matches only that * path (no descendants). The engine's tree filter auto-detects lquery vs @@ -49,7 +62,10 @@ export function useMemories(params: MemorySearchParams, enabled = true) { return useQuery({ enabled, queryKey: ["memories", normalized], - queryFn: () => memoryClient.memory.search(normalized), + queryFn: () => + memoryClient.memory + .search(normalized) + .then((r) => ({ ...r, results: r.results.map(memoryToDot) })), }); } @@ -62,7 +78,11 @@ export function useMemories(params: MemorySearchParams, enabled = true) { export function useTree() { return useQuery({ queryKey: ["memory-tree"], - queryFn: () => memoryClient.memory.tree(), + queryFn: () => + memoryClient.memory.tree().then((r) => ({ + ...r, + nodes: r.nodes.map((n) => ({ ...n, path: wirePathToDot(n.path) })), + })), }); } @@ -78,10 +98,12 @@ export function useMemoriesAtExactPath(path: string, enabled: boolean) { enabled, queryKey: ["memories-at-exact-path", path], queryFn: () => - memoryClient.memory.search({ - tree: exactTreeLquery(path), - limit: SEARCH_LIMIT, - }), + memoryClient.memory + .search({ + tree: exactTreeLquery(path), + limit: SEARCH_LIMIT, + }) + .then((r) => ({ ...r, results: r.results.map(memoryToDot) })), }); } @@ -94,7 +116,8 @@ export function useMemory(id: string | null) { return useQuery({ enabled: id !== null, queryKey: ["memory", id], - queryFn: () => memoryClient.memory.get({ id: id as string }), + queryFn: () => + memoryClient.memory.get({ id: id as string }).then(memoryToDot), }); } @@ -104,7 +127,7 @@ export function useMemory(id: string | null) { export function useUpdateMemory(queryClient: QueryClient) { return useMutation({ mutationFn: (params: MemoryUpdateParams) => - memoryClient.memory.update(params), + memoryClient.memory.update(params).then(memoryToDot), onSuccess: (memory) => { invalidateTreeQueries(queryClient); queryClient.setQueryData(["memory", memory.id], memory); diff --git a/packages/web/src/components/SearchResultRow.tsx b/packages/web/src/components/SearchResultRow.tsx index 8d8bea32..d826066b 100644 --- a/packages/web/src/components/SearchResultRow.tsx +++ b/packages/web/src/components/SearchResultRow.tsx @@ -25,7 +25,7 @@ export function SearchResultRow({ openContextMenu({ x: event.clientX, y: event.clientY, - target: { kind: "memory", id: memory.id, title: fragment }, + target: { kind: "memory", id: memory.id, title: memory.name ?? fragment }, }); }; @@ -49,6 +49,14 @@ export function SearchResultRow({ > {memory.tree || "(root)"} + {memory.name && ( + + {memory.name} + + )} {formatScore(memory.score)} diff --git a/packages/web/src/components/editor/EditorPane.tsx b/packages/web/src/components/editor/EditorPane.tsx index cb56fe85..400774bd 100644 --- a/packages/web/src/components/editor/EditorPane.tsx +++ b/packages/web/src/components/editor/EditorPane.tsx @@ -91,10 +91,12 @@ export function EditorPane({ memory, onRequestDelete }: Props) { if (!parsed.ok) return; const fm = parsed.value; try { - // Send the diff: server accepts null to clear a field. + // Send the diff: server accepts null to clear a field. Omitting `name` + // from the frontmatter clears it (parsed as null); a slug sets/renames. await update.mutateAsync({ id: memory.id, content: fm.body, + name: fm.name, tree: fm.tree, meta: fm.meta, temporal: fm.temporal, diff --git a/packages/web/src/components/viewer/FrontmatterBlock.tsx b/packages/web/src/components/viewer/FrontmatterBlock.tsx index 8a058211..1dbba44a 100644 --- a/packages/web/src/components/viewer/FrontmatterBlock.tsx +++ b/packages/web/src/components/viewer/FrontmatterBlock.tsx @@ -1,13 +1,13 @@ /** * Collapsible frontmatter display for view mode. * - * Renders tree / meta / temporal as a compact inspector panel. Metadata rows - * include a small filter button that merges the value into the current + * Renders name / tree / meta / temporal as a compact inspector panel. Metadata + * rows include a small filter button that merges the value into the current * advanced meta JSON filter, then switches search into advanced mode so the * filter takes effect immediately. * - * Returns `null` when there is nothing to show (no tree, empty meta, no - * temporal) so the view-mode pane stays uncluttered for bare memories. + * Returns `null` when there is nothing to show (no name, no tree, empty meta, + * no temporal) so the view-mode pane stays uncluttered for bare memories. */ import type { ReactNode } from "react"; @@ -20,7 +20,10 @@ import { useFilter } from "../../store/filter.ts"; import { useLayout } from "../../store/layout.ts"; import { pushToast } from "../toast/Toast.tsx"; -type Frontmatter = Pick; +type Frontmatter = Pick< + ParsedFrontmatter, + "name" | "tree" | "meta" | "temporal" +>; interface Props { frontmatter: Frontmatter; @@ -33,7 +36,13 @@ export function FrontmatterBlock({ frontmatter }: Props) { const setSearchCollapsed = useLayout((s) => s.setSearchCollapsed); const hasMeta = Object.keys(frontmatter.meta).length > 0; - if (!frontmatter.tree && !hasMeta && !frontmatter.temporal) return null; + if ( + !frontmatter.name && + !frontmatter.tree && + !hasMeta && + !frontmatter.temporal + ) + return null; function handleApplyMetaFilter(path: string[], value: unknown) { applyMetaJsonFilter(buildMetaFilter(path, value)); @@ -64,6 +73,14 @@ export function FrontmatterBlock({ frontmatter }: Props) {
+ {frontmatter.name && ( + + + {frontmatter.name} + + + )} + {frontmatter.tree && ( diff --git a/packages/web/src/lib/frontmatter.test.ts b/packages/web/src/lib/frontmatter.test.ts index 48968943..08993e29 100644 --- a/packages/web/src/lib/frontmatter.test.ts +++ b/packages/web/src/lib/frontmatter.test.ts @@ -12,6 +12,7 @@ function mkMemory(partial: Partial): MemoryResponse { content: partial.content ?? "body", meta: partial.meta ?? {}, tree: partial.tree ?? "", + name: partial.name ?? null, temporal: partial.temporal ?? null, hasEmbedding: partial.hasEmbedding ?? false, createdAt: partial.createdAt ?? "2026-01-01T00:00:00Z", @@ -26,10 +27,11 @@ describe("memoryToEditorText", () => { expect(text).toBe("hello"); }); - test("emits tree, meta, and temporal when present", () => { + test("emits name, tree, meta, and temporal when present", () => { const text = memoryToEditorText( mkMemory({ content: "body text", + name: "jwt-rotation", tree: "work.projects", meta: { priority: "high" }, temporal: { @@ -38,16 +40,25 @@ describe("memoryToEditorText", () => { }, }), ); + expect(text).toContain("name: jwt-rotation"); expect(text).toContain("tree: work.projects"); expect(text).toContain("priority: high"); expect(text).toContain("start: '2026-01-01T00:00:00Z'"); expect(text.trimEnd().endsWith("body text")).toBe(true); }); + + test("omits name when the memory is unnamed", () => { + const text = memoryToEditorText( + mkMemory({ content: "body", tree: "work" }), + ); + expect(text).not.toContain("name:"); + }); }); describe("parseEditorText", () => { test("body-only input parses as empty frontmatter", () => { const parsed = parseEditorText("no frontmatter here"); + expect(parsed.name).toBeNull(); expect(parsed.tree).toBe(""); expect(parsed.meta).toEqual({}); expect(parsed.temporal).toBeNull(); @@ -57,6 +68,7 @@ describe("parseEditorText", () => { test("standard object-form frontmatter round-trips", () => { const original = mkMemory({ content: "hello world", + name: "jwt-rotation", tree: "work.projects", meta: { a: 1, b: "two" }, temporal: { @@ -66,12 +78,24 @@ describe("parseEditorText", () => { }); const text = memoryToEditorText(original); const parsed = parseEditorText(text); + expect(parsed.name).toBe("jwt-rotation"); expect(parsed.tree).toBe("work.projects"); expect(parsed.meta).toEqual({ a: 1, b: "two" }); expect(parsed.temporal).toEqual(original.temporal); expect(parsed.body).toBe("hello world"); }); + test("omitting name parses as null (clears the name on save)", () => { + const parsed = parseEditorText("---\ntree: work\n---\nbody"); + expect(parsed.name).toBeNull(); + }); + + test("invalid name (slash) throws", () => { + expect(() => parseEditorText("---\nname: a/b\n---\nbody")).toThrow( + /name.*slug/, + ); + }); + test("accepts array-form temporal", () => { const source = "---\ntemporal:\n - '2026-01-01T00:00:00Z'\n - '2026-06-30T00:00:00Z'\n---\nbody"; diff --git a/packages/web/src/lib/frontmatter.ts b/packages/web/src/lib/frontmatter.ts index b042ed30..78063789 100644 --- a/packages/web/src/lib/frontmatter.ts +++ b/packages/web/src/lib/frontmatter.ts @@ -4,6 +4,8 @@ * Mirrors the Markdown import schema in `docs/formats.md`, scoped to the * fields the editor allows the user to change: * + * - `name` — optional filename-like leaf slug (unique within the tree); + * omit it to clear the name * - `tree` — ltree path string * - `meta` — arbitrary JSON object * - `temporal`— `{ start, end? }` object (same shape the server returns) @@ -15,8 +17,13 @@ import type { MemoryResponse, Temporal } from "@memory.build/client"; import yaml from "js-yaml"; +// Mirrors `memoryNameSchema` in @memory.build/protocol: a filename-like leaf +// slug, 1–128 chars, no slashes. Server-validated too; this is fast feedback. +const NAME_RE = /^[A-Za-z0-9][A-Za-z0-9._-]*$/; + export interface ParsedFrontmatter { - /** Editable fields extracted from frontmatter. Missing → null. */ + /** Editable fields extracted from frontmatter. An absent or empty name → null. */ + name: string | null; tree: string; meta: Record; temporal: Temporal | null; @@ -29,6 +36,7 @@ export interface ParsedFrontmatter { * * ``` * --- + * name: my-note * tree: work.projects.me * meta: * priority: high @@ -43,6 +51,7 @@ export interface ParsedFrontmatter { */ export function memoryToEditorText(memory: MemoryResponse): string { const frontmatter: Record = {}; + if (memory.name) frontmatter.name = memory.name; if (memory.tree) frontmatter.tree = memory.tree; if (memory.meta && Object.keys(memory.meta).length > 0) { frontmatter.meta = memory.meta; @@ -68,6 +77,7 @@ export function parseEditorText(source: string): ParsedFrontmatter { if (!match) { // No frontmatter — everything is body. return { + name: null, tree: "", meta: {}, temporal: null, @@ -88,7 +98,7 @@ export function parseEditorText(source: string): ParsedFrontmatter { } if (parsed === null || parsed === undefined) { - return { tree: "", meta: {}, temporal: null, body }; + return { name: null, tree: "", meta: {}, temporal: null, body }; } if (typeof parsed !== "object" || Array.isArray(parsed)) { throw new Error("Frontmatter must be a YAML mapping"); @@ -96,6 +106,7 @@ export function parseEditorText(source: string): ParsedFrontmatter { const obj = parsed as Record; return { + name: coerceName(obj.name), tree: coerceTree(obj.tree), meta: coerceMeta(obj.meta), temporal: coerceTemporal(obj.temporal), @@ -103,6 +114,21 @@ export function parseEditorText(source: string): ParsedFrontmatter { }; } +// Absent/empty → null (clears the name on save); a string must be a valid +// filename-like slug. Throwing here keeps the Save button disabled until fixed. +function coerceName(value: unknown): string | null { + if (value === undefined || value === null || value === "") return null; + if (typeof value !== "string") { + throw new Error("`name` must be a string"); + } + if (value.length > 128 || !NAME_RE.test(value)) { + throw new Error( + "`name` must be a filename-like slug (letters, digits, '.', '-', '_'; no leading '.'/'-'), ≤128 chars", + ); + } + return value; +} + function coerceTree(value: unknown): string { if (value === undefined || value === null) return ""; if (typeof value !== "string") { diff --git a/packages/web/src/lib/tree-build.test.ts b/packages/web/src/lib/tree-build.test.ts index b2b28514..67978d83 100644 --- a/packages/web/src/lib/tree-build.test.ts +++ b/packages/web/src/lib/tree-build.test.ts @@ -30,6 +30,7 @@ function mkMemory( content: partial.content ?? "placeholder content", meta: partial.meta ?? {}, tree: partial.tree ?? "", + name: partial.name ?? null, temporal: partial.temporal ?? null, hasEmbedding: partial.hasEmbedding ?? false, createdAt: partial.createdAt ?? new Date().toISOString(), @@ -259,6 +260,14 @@ describe("memoryToLeaf + sortLeaves + titleForMemory", () => { expect(leaf.depth).toBe(2); }); + test("memoryToLeaf prefers the name as the leaf title when present", () => { + const leaf = memoryToLeaf( + mkMemory({ name: "jwt-rotation", content: "# Hello World\n\nbody" }), + 0, + ); + expect(leaf.title).toBe("jwt-rotation"); + }); + test("sortLeaves: newest temporal first, nulls last, title tiebreak", () => { const leaves = [ memoryToLeaf( diff --git a/packages/web/src/lib/tree-build.ts b/packages/web/src/lib/tree-build.ts index a8cd934f..b3748b79 100644 --- a/packages/web/src/lib/tree-build.ts +++ b/packages/web/src/lib/tree-build.ts @@ -193,7 +193,9 @@ export function memoryToLeaf( return { kind: "memory", id: memory.id, - title: titleForMemory(memory.content, memory.id), + // A named memory shows its name (the filename-like leaf); otherwise fall + // back to the first content line, then the id tail. + title: memory.name ?? titleForMemory(memory.content, memory.id), tree: memory.tree, temporalStart: memory.temporal?.start ?? null, depth,