diff --git a/adr/README.md b/adr/README.md deleted file mode 100644 index d91de133..00000000 --- a/adr/README.md +++ /dev/null @@ -1,22 +0,0 @@ -# Architecture Decision Records - -This directory records significant architectural decisions for Memory Engine. An ADR explains the context that led to a decision, the decision itself, and its consequences. ADRs are historical records: when a decision changes, add a new ADR that supersedes the old one instead of rewriting the old decision. - -ADRs use numbered filenames and a lightweight Nygaard format: - -- **Context** describes the problem and constraints. -- **Decision** states the chosen approach. -- **Consequences** records benefits, costs, and limitations. -- **Alternatives considered** explains why other approaches were not chosen. - -The `status` field progresses from `Proposed` to `Accepted`, `Deprecated`, or `Superseded`. The `supersedes` field contains ADR numbers when a decision replaces earlier records. The `tickets` field contains only publicly accessible issue references. - -## Decisions - -| ADR | Title | Status | -| --- | --- | --- | -| [0001](0001-semantic-search-scoring.md) | Use cosine similarity for semantic search scores | Accepted | -| [0002](0002-hnsw-filtered-recall.md) | Use strict-order HNSW iterative scans for filtered semantic search | Accepted | -| [0003](0003-fulltext-bm25-scoring.md) | Return only positive BM25 matches from fulltext search | Accepted | -| [0004](0004-hybrid-rrf-scoring.md) | Use Reciprocal Rank Fusion without a fused-score threshold | Accepted | -| [0005](0005-jsonpath-metadata-predicates.md) | Expose PostgreSQL JSONPath predicates for advanced metadata filtering | Accepted | diff --git a/design/README.md b/design/README.md new file mode 100644 index 00000000..4c72009c --- /dev/null +++ b/design/README.md @@ -0,0 +1,64 @@ +# Design + +This directory describes Memory Engine's implemented design. These are living +documents: update them in place when the design changes. Focus on the model, +interfaces, invariants, constraints, and operational tradeoffs that future work +must preserve. + +## Harnesses + +| Document | Scope | +| --- | --- | +| [Harness integrations](harness-integrations.md) | Shared policy, lifecycle, runtime contract, and security boundaries | +| [Claude Code adapter](harnesses/claude.md) | Plugin, environment injection, MCP, and capture | +| [OpenCode adapter](harnesses/opencode.md) | MCP entry, plugin, environment injection, and capture | +| [Codex CLI adapter](harnesses/codex.md) | MCP entry, hooks, command rewrite, and capture | + +## Database + +| Document | Scope | +| --- | --- | +| [Database migrations and versioning](database-migrations-and-versioning.md) | Schema evolution, deployment reconciliation, and SQL function safety | + +## Embeddings + +| Document | Scope | +| --- | --- | +| [Embedding queue and worker](embedding-queue-and-worker.md) | Asynchronous vector generation and backlog processing | + +## Memory Model + +| Document | Scope | +| --- | --- | +| [Memory schema](memory-schema.md) | Flexible memory records, identity, temporal modeling, metadata, and retrieval indexes | + +## Spaces + +| Document | Scope | +| --- | --- | +| [Spaces and provisioning](spaces.md) | Space isolation, lifecycle, and custom defaults | + +## Authorization + +| Document | Scope | +| --- | --- | +| [Authentication](authentication.md) | Human login, OAuth, sessions, device authorization, and API-key boundaries | +| [Principal model](principal-model.md) | Users, groups, service accounts, and agent removal | +| [Memory authorization](memory-authorization.md) | Space admission, tree grants, and effective access | +| [Restricted API keys](restricted-api-keys.md) | Scoped personal and service-account credentials | + +## Presentation + +| Document | Scope | +| --- | --- | +| [Field projection and format selection](field-projection-and-format-selection.md) | CLI and MCP memory-read presentation | + +## Search + +| Document | Scope | +| --- | --- | +| [Semantic search scoring](semantic-search-scoring.md) | Cosine similarity scores and thresholds | +| [Filtered semantic search](hnsw-filtered-semantic-search.md) | HNSW recall and rank ordering under filters | +| [Fulltext BM25 scoring](fulltext-bm25-scoring.md) | Positive lexical-match invariant and score semantics | +| [Hybrid RRF scoring](hybrid-rrf-scoring.md) | Fusion behavior, candidate windows, and tuning | +| [Metadata predicates](jsonpath-metadata-predicates.md) | Advanced JSONPath metadata filtering | diff --git a/design/authentication.md b/design/authentication.md new file mode 100644 index 00000000..bb66cc79 --- /dev/null +++ b/design/authentication.md @@ -0,0 +1,114 @@ +--- +title: Authentication +tags: [authentication, oauth, sessions, api-keys, device-flow] +--- + +# Authentication + +Authentication establishes the credential holder's identity. Authorization then +decides what that identity may do: the user endpoint applies its method-level +rules, while the memory endpoint additionally requires direct membership in the +selected space and passes effective tree grants to the data plane. API keys are +described in more detail in [Restricted API Keys](restricted-api-keys.md). + +## Credential classes + +| Credential | Holder | Primary use | Validation | +| --- | --- | --- | --- | +| Browser session cookie | User | Hosted web UI | better-auth session lookup | +| OAuth access token | User | CLI and MCP | Hashed token lookup in the auth schema | +| Signed session bearer | User | `me login --device` | better-auth bearer plugin session lookup | +| API key | User or service account | Headless automation and explicit key use | Core API-key validation | + +Human identity and OAuth issuance use better-auth. API keys remain in the core +control plane, so service accounts never need a social-login identity. A bearer +is classified by its `me..` structure: a matching value is an +API key; every other bearer is tried as an OAuth access token and then as a +signed session bearer. This explicit dispatch prevents a cookie from being +mistaken for an API key. + +## Web sessions + +GitHub and Google social login create better-auth sessions. A provider-verified +email is required before a session is created. This is the human-login front +door: OAuth authorization and device approval both depend on a valid web +session, so they inherit the same verification requirement. + +Sessions have a seven-day rolling lifetime and refresh at most daily. They are +stored in the auth schema because better-auth needs to round-trip the session +token. HTTPS deployments use secure, same-site cookies; browser requests using +the ambient cookie must also pass the server's allowed-origin check. A supplied +Bearer credential is never allowed to fall back to a cookie session, so an +invalid bearer cannot bypass that CSRF boundary. + +## OAuth for CLI and MCP + +`me login` uses OAuth 2.1 authorization code with PKCE and an RFC 8252 loopback +redirect. The first-party `me-cli` client is public, requires PKCE, and skips +consent. The authorization server issues opaque access and refresh tokens. Both +are hashed before storage; a request is validated by hashing its presented +access token, finding an unexpired row, and resolving the bound user. + +The CLI refreshes access tokens proactively shortly before expiry and reacts to +an unexpected 401 with one forced refresh. Refreshes are serialized across +processes sharing a credential store, preventing concurrent reuse of a rotated +refresh token. An injected `ME_SESSION_TOKEN` is treated as a static bearer and +is never refreshed. + +OAuth credentials are user-bound. Client-credentials tokens without a user do +not authenticate the RPC APIs; independently managed automation uses a service +account API key instead. + +## Device authorization + +`me login --device` supports headless environments through RFC 8628. The CLI +requests a device code, presents the human with a verification URL and user +code, then polls at the server-provided interval. Issuance is restricted to the +first-party CLI, codes expire after 15 minutes, and unauthenticated code +issuance is rate-limited. + +Approval creates a normal better-auth session rather than an OAuth token pair. +The returned session token is converted to a signed bearer form before it leaves +the server. The bearer plugin requires that signature, so a raw session-table +token exposed from storage cannot authenticate an API request. Device sessions +have no refresh token; they slide while used and require another device login +after expiry. + +## API keys + +API keys are global credentials for a user or service account and are stored +only as hashes. They select a space through `X-Me-Space`; the key itself does +not encode a space. The memory endpoint checks the holder's direct membership +in that space, then computes the holder's live tree access. Restricted keys add +a server-enforced ceiling to that access. + +The user endpoint accepts both user and service-account keys, but handlers +limit service accounts to safe reads. Key-authenticated callers cannot mint or +revoke keys, preventing a compromised key from creating a replacement. Legacy +space-scoped API keys are rejected with a migration-specific error rather than +silently accepting an obsolete credential format. + +## Endpoint boundaries + +`/api/v1/user/rpc` authenticates a principal for account and cross-space +operations. OAuth tokens and sessions always represent users; API keys may +represent users or service accounts. The RPC method gate imposes the remaining +credential-specific restrictions. + +`/api/v1/memory/rpc` also requires `X-Me-Space`. Authentication resolves the +credential holder, then requires a direct roster entry in the named space. +Membership is separate from data authority: a member with no tree grants may +reach the endpoint but receives no data from the space SQL functions. + +## Operations and invariants + +- Auth rows are swept on a server cron: expired sessions, verifications, OAuth + access and refresh tokens, and device codes are removed. +- OAuth access and refresh tokens, and core API keys, are hashed at rest. +- Raw browser session tokens are intentionally not valid bearer credentials; + only the signed device-flow form is accepted as a session bearer. +- Email verification is checked when a new human session is created. Existing + credentials remain valid until their normal expiry or revocation. +- Auth is identity resolution, not permission granting. Space membership, + admin authority, tree grants, and restricted-key ceilings are evaluated for + every applicable request. diff --git a/design/database-migrations-and-versioning.md b/design/database-migrations-and-versioning.md new file mode 100644 index 00000000..ae7ae038 --- /dev/null +++ b/design/database-migrations-and-versioning.md @@ -0,0 +1,94 @@ +--- +title: Database Migrations and Versioning +tags: [database, migrations, versioning, postgres, spaces] +--- + +# Database Migrations and Versioning + +Migrations keep the database schema and database-resident behavior compatible +with the running server. They run for three schema families in one PostgreSQL +database: + +| Schema family | Role | +| --- | --- | +| `auth` | Authentication and OAuth state. | +| `core` | Spaces, principals, grants, invitations, and API keys. | +| `me_` | One memory data schema for each space. | + +Every schema carries a singleton `version` row and a `migration` ledger. The +version records the latest schema-version migrator that touched the schema; the +ledger records exactly which incremental migrations have been applied. + +## Migration kinds + +Migrations are ordered by filename and divided into two categories: + +| Kind | Purpose | Execution | +| --- | --- | --- | +| Incremental | One-time schema or data transitions, including backfills. | Applied once and recorded in the migration ledger. | +| Idempotent | Current definitions of functions, triggers, and other replaceable database behavior. | Re-applied on every migration pass. | + +Incremental files are immutable history: never edit an incremental that may have +run against an existing schema. Add a new incremental instead. Idempotent files +are living definitions and should be updated in place as their behavior changes. + +This split is necessary because much of Memory Engine's behavior lives in SQL +functions. Tracking only versioned one-time files would leave existing spaces +with stale function bodies after a server deployment. + +## Provisioning and migration + +New space provisioning creates the schema plus empty `version` and `migration` +tracking tables, then runs the same migration sequence used for existing spaces. +When provisioning is composed into a caller's transaction, the new schema and +the related control-plane changes succeed or roll back together. + +Standalone migration acquires a transaction-scoped advisory lock per schema and +requires the current database user to own that schema. This serializes concurrent +server replicas and prevents accidental migration by an unprivileged connection. +The migration transaction also verifies the supported PostgreSQL and extension +versions before applying schema-specific SQL. + +Space migrations allow a long-running backfill statement, but cap the whole +migration transaction at 20 minutes. This lets a legitimate table-sized backfill +complete while still failing a genuinely stuck deployment. Lock and idle +transaction timeouts remain short to avoid prolonged contention. + +## Startup reconciliation + +Server startup migrates `core` and `auth`, then enumerates every existing space +and runs its space migration. Incrementals already present in the ledger are +cheap no-ops; idempotent definitions are refreshed. + +Every space is attempted so failures are individually reported. If any space +fails to migrate, startup fails rather than serving a deployment whose database +behavior may be stale. The per-space advisory lock makes this safe when multiple +server replicas start concurrently. + +## Schema versions and downgrade protection + +Each schema family has an independent semantic schema version in code. Before a +migration runs, the runner compares that version with the schema's stored +version. A server older than the stored schema version refuses to run, preventing +older SQL definitions from being reapplied to a newer database. + +Migration runs proceed even when the versions are equal because idempotent SQL +must be refreshed. The incremental ledger, not the version row, is the source of +truth for which one-time migrations have completed. + +## SQL templating and function signatures + +Migration SQL is templated so the same files can target production schemas and +isolated test schemas. Space migrations also substitute database configuration +such as embedding dimensions and index parameters. + +PostgreSQL cannot use `create or replace function` to change a function's return +type or input parameter names, and a changed argument type can leave an obsolete +overload behind. Function definitions whose signatures may change use a +`{{fn name(args) returns result}}` header. The migration runner expands it into +a pre-create stale-signature drop and a post-create signature assertion. + +This makes signature drift fail during a fresh test migration instead of only +failing when an existing production schema is upgraded. Parameter defaults are +not part of that signature contract; removing a default may require an explicit +guarded drop before recreating the function. diff --git a/design/embedding-queue-and-worker.md b/design/embedding-queue-and-worker.md new file mode 100644 index 00000000..9b210ad2 --- /dev/null +++ b/design/embedding-queue-and-worker.md @@ -0,0 +1,92 @@ +--- +title: Embedding Queue and Worker +tags: [embeddings, queue, worker, semantic-search, spaces] +--- + +# Embedding Queue and Worker + +Embedding generation is asynchronous so memory writes do not wait for an +embedding provider. A new or changed memory is stored immediately and remains +available to fulltext and filter-based search. It becomes eligible for semantic +search after the worker writes its vector. + +Every space owns its own embedding queue alongside its memory table. This keeps +queue work, lifecycle, and cleanup within the same isolation boundary as the +memories it represents. + +## Enqueueing + +The queue records a memory ID and its `content_version`. Database triggers enqueue +work when: + +- A memory is inserted without an embedding. +- A memory's content changes and its embedding is reset. + +An insert that already supplies an embedding does not enqueue work. Content +versions make a queue entry refer to one exact memory state, rather than merely +to a memory ID. + +## Claiming and staleness + +Workers claim visible queue rows in batches. A claim increments the attempt count +and moves a row's visibility time into the future, preventing concurrent workers +from processing it. Claims use `FOR UPDATE SKIP LOCKED`, so workers can drain the +same queue without blocking one another. + +Before claiming, the queue cancels rows that are no longer useful: + +- A newer pending content version exists for the same memory. +- The memory was deleted or its content version no longer matches. + +The worker embeds outside the claim transaction. If it crashes before write-back, +the visibility timeout expires and another worker can claim the row. A sweep marks +rows that exhausted their attempt budget as failed, including rows stranded by a +crash after their final claim. + +## Write-back and retries + +Write-back is version-guarded and atomic. A completed embedding is written only +when the memory still has the claimed content version. Otherwise the queue row is +cancelled and no stale vector is stored. Deleting a memory also removes its queue +rows through the foreign-key cascade. + +Ordinary provider or write-back failures record the last error while leaving the +row pending. It becomes eligible for another claim after its visibility timeout. +This makes the queue, rather than the embedding SDK, the primary retry authority. + +Rate limits are treated separately from ordinary failures. The worker releases +claimed rows, refunds their attempt count, and defers their visibility by the +same backoff interval that the worker observes. Workers in one pool also share a +rate-limit gate, so one provider `429` pauses new claims across the pool. The +database visibility delay prevents another worker or process from immediately +reclaiming the same rows. + +## Worker operation + +Workers periodically discover space schemas and poll their queues in shuffled +round-robin order. They process immediately while work exists, sleep while idle, +and apply bounded exponential backoff for consecutive non-rate-limit failures. +An optional drain timeout lets a worker exit after sustained idleness. + +Queue functions own claiming, completion, failure, release, and pruning. The +worker calls those functions rather than issuing inline queue mutations, keeping +the queue state machine in the database. + +Terminal queue rows are retained for seven days by default, then pruned +opportunistically when a worker finds no claimable work in that space. This +preserves short-term failure and cancellation visibility without unbounded queue +growth. + +## Operational visibility + +The embedding status surface reports aggregate queue state for the active space: + +| State | Meaning | +| --- | --- | +| Pending | Queue rows without a terminal outcome. | +| In flight | Pending rows whose visibility timeout is still in the future. | +| Waiting | Pending rows that are claimable now. | +| Failed | Terminal failures retained before pruning. | + +It also reports the oldest pending enqueue time. These are space-wide operational +counts, not tree-scoped memory results. diff --git a/design/field-projection-and-format-selection.md b/design/field-projection-and-format-selection.md new file mode 100644 index 00000000..dabe0b60 --- /dev/null +++ b/design/field-projection-and-format-selection.md @@ -0,0 +1,98 @@ +--- +title: Field Projection and Format Selection +tags: [cli, mcp, presentation, memory] +--- + +# Field Projection and Format Selection + +`select` and MCP `format` control how memory reads are presented by the CLI and +MCP server. They are not part of the memory data API. + +## Motivation + +Memory search is commonly a discovery step: agents first identify the few +memories relevant to a task, then retrieve their complete content. Returning +every full memory while searching can consume enough context to crowd out the +work the agent is trying to do. + +Use `select` to return a compact search result with identifying fields and a +content preview, then retrieve the selected memory in full with `get` or +`getByPath`. This reduces model-context consumption without changing search +semantics or withholding data from callers that need complete records. + +## Boundary + +The TypeScript client calls the existing `memory.get`, `memory.getByPath`, and +`memory.search` RPC methods and receives complete responses. The CLI and MCP +server project those responses locally, immediately before terminal rendering or +MCP serialization. + +The JSON-RPC protocol, server handlers, engine, database functions, and general +TypeScript client do not accept `select` or `format` parameters. Projection does +not reduce data transferred from the server, change authorization, or change how +search results are formed. A server-side projection API requires a separate +design because it would make otherwise full, strongly typed client responses +partial. + +## Selectors + +CLI `memory get` and `memory search` accept a comma-separated `--select` value. +MCP accepts a `select` array on `me_memory_get`, `me_memory_get_by_path`, and +`me_memory_search`. A selection must contain at least one valid selector. + +The available response fields are: + +```text +id, content, meta, tree, name, temporal, score, hasEmbedding, +createdAt, createdBy, updatedAt, version, versionHash +``` + +Metadata selectors use `meta.`. The suffix is any nonempty metadata key, +including keys containing punctuation or `$`. Missing metadata keys are omitted +from the projection. + +Content may be selected in full or sliced with zero-based, end-exclusive bounds: + +```text +content:N # [0, N) +content:M:N # [M, N) +content:M: # [M, end) +``` + +Slice bounds must be non-negative JavaScript safe integers. Slices use +JavaScript string indexing, so `contentLength` and offsets are measured in +UTF-16 code units. A slice adds `contentLength` to the response. At most one +distinct slice selector is permitted; duplicate instances of the same selector +are allowed. + +When selectors overlap, a content slice takes precedence over full `content`, +and bare `meta` takes precedence over individual `meta.` selectors. The +projected field order is fixed by the presentation implementation, not by the +selector order. + +`score` is a valid selector for every read tool, but get responses have no score +field and therefore omit it. Search projections retain the `results`, `total`, +and `limit` envelope while projecting each result row. + +## Formats + +MCP supports a presentation-only `format` argument on its three retrieval and +search tools. It defaults to `yaml`; `json` and `compact` both produce compact +one-line JSON. + +CLI does not have a per-command format option. Its existing global `--json` and +`--yaml` options control structured output; otherwise commands render text. +`memory get --raw` cannot be combined with `--select`. + +Without an explicit `--select`, CLI text search still fetches complete results +but locally presents `id`, `tree`, `content:120`, and `score`. Structured CLI +output and MCP calls without `select` present complete responses. + +## Constraints + +- Selector validation and projection remain shared CLI-package behavior so CLI + and MCP output stays consistent. +- MCP retrieval/search tools validate `select` before rendering and accept + omitted or `null` presentation options as the default full YAML response. +- Presentation options stay limited to the three retrieval/search MCP tools; + they are not generic options for every read tool. diff --git a/adr/0003-fulltext-bm25-scoring.md b/design/fulltext-bm25-scoring.md similarity index 53% rename from adr/0003-fulltext-bm25-scoring.md rename to design/fulltext-bm25-scoring.md index b4a2d14a..6efc2d1f 100644 --- a/adr/0003-fulltext-bm25-scoring.md +++ b/design/fulltext-bm25-scoring.md @@ -1,16 +1,11 @@ --- -title: Return only positive BM25 matches from fulltext search -status: Accepted -date: 2026-08-07 -deciders: [jgpruitt] +title: Fulltext BM25 Scoring tags: [search, fulltext-search, bm25, pg-textsearch] -tickets: [] -supersedes: [] --- -# ADR 0003: Return only positive BM25 matches from fulltext search +# Fulltext BM25 Scoring -## Context +## Score semantics Memory Engine uses pg_textsearch's BM25 operator, `<@>`, for fulltext ranking. The operator returns negative BM25 so that an ascending PostgreSQL index scan visits the most relevant rows first. Memory Engine negates that value when returning it, producing a positive, higher-is-better BM25 score. @@ -27,8 +22,6 @@ When fewer genuine matches exist than the requested limit, PostgreSQL can fill t Zero-score rows are especially harmful in hybrid search. Once assigned a rank, a non-match receives a positive Reciprocal Rank Fusion contribution despite having no lexical match. -## Decision - Fulltext search enforces an unconditional positive-BM25 invariant: ```sql @@ -41,28 +34,11 @@ The predicate is applied while preserving the required `ORDER BY content <@> que Memory Engine does not expose a `fulltextThreshold` parameter. The returned fulltext score remains positive, unnormalized BM25 and is meaningful for ranking results within a query, not as a stable absolute relevance scale. -## Consequences +## Invariants and constraints - Fulltext results contain genuine lexical matches only; the result limit is a maximum rather than a target padded with non-matches. -- Hybrid search no longer gives zero-score lexical non-matches a positive fused contribution merely because they occupied a candidate slot. +- Hybrid search never gives zero-score lexical non-matches a fused contribution merely because they occupied a candidate slot. - Queries containing only stop words produce no processed terms and return no results. -- Returned scores remain unbounded and depend on query terms, term frequency, document length, and corpus-wide document frequency. -- Callers cannot set an absolute BM25 quality threshold. They can raise the result limit to retrieve more genuine lexical matches, up to the search result cap. - -## Alternatives considered - -### Preserve zero-score rows - -Returning exactly the requested number of rows can appear convenient, but rows with no query stems are not fulltext matches. Including them misrepresents relevance and contaminates hybrid ranking. - -### Expose an absolute `fulltextThreshold` - -BM25 has no corpus- and query-independent scale. A value that is selective for one query can reject everything or almost nothing for another, and corpus changes can move scores over time. Exposing such a threshold as an intuitive public quality control would be misleading. - -### Use a relative threshold based on the top result - -A relative cutoff could adapt to each result set, but it would define a different feature with behavior tied to the top match and candidate window. It is not needed to enforce the correctness invariant that non-matches must be excluded. - -### Filter after retrieving the top-k rows - -Filtering outside the ranked query would still prevent non-matches from reaching callers, but it could leave fewer useful rows without letting the index continue to consider later matches. Keeping the predicate in the ranked query lets PostgreSQL apply it as part of candidate selection while preserving index eligibility. +- Returned scores are unbounded and depend on query terms, term frequency, document length, and corpus-wide document frequency. +- There is no `fulltextThreshold`: BM25 has no corpus- and query-independent quality scale. Raising the result limit retrieves more genuine lexical matches, up to the search result cap. +- The positive-match predicate remains in the ranked query so PostgreSQL can consider later matches while preserving the BM25 index ordering. diff --git a/design/harness-integrations.md b/design/harness-integrations.md new file mode 100644 index 00000000..597c3269 --- /dev/null +++ b/design/harness-integrations.md @@ -0,0 +1,104 @@ +--- +title: Harness Integrations +tags: [harnesses, mcp, capture, configuration, cli] +--- + +# Harness Integrations + +Memory Engine integrates with Claude Code, OpenCode, and Codex CLI. The +integration model keeps provider-specific installation separate from activation +and credentials. Installing support for a harness does not enable Memory Engine +in every project or create a new execution identity. + +## Layers + +Each integration has two independent layers: + +| Layer | Scope | Responsibility | +| --- | --- | --- | +| Installation | User-global | Register provider-specific MCP, hook, and plugin plumbing owned by Memory Engine. | +| Activation | Machine-local and directory-scoped | Select which harnesses receive MCP, capture, and CLI-routing behavior. | + +Installers create dormant artifacts without a server, space, credential, or +project-specific policy. They record the artifacts they own. Uninstall removes +only recorded, unchanged artifacts; `--purge` additionally removes the harness +from activation profiles. + +## Machine-local policy + +Harness policy lives in the normal non-secret configuration file, +`~/.config/me/config.yaml` (or its XDG equivalent). It is not repository +configuration and never persists API keys. + +The policy has fallback defaults and canonical absolute directory profiles. The +longest matching directory profile wins. A matched directory profile replaces +defaults as a complete profile: a surface omitted from that profile is disabled, +not inherited. This prevents a broad default from unexpectedly activating a +harness behavior in a more specific project. + +Each profile configures three independent surfaces: + +| Surface | Purpose | +| --- | --- | +| MCP | Enable managed Memory Engine tools for selected harnesses and select their server and optional space. | +| Capture | Import selected harness sessions into a configured space and tree. | +| CLI routing | Select the server and optional space for `me` commands run by selected harnesses. | + +Every enabled surface explicitly selects its harnesses. Capture also requires a +server, space, and destination tree. `me init` writes these profiles; `me doctor` +shows the resolved profile, active surfaces, and their source. + +## Runtime contract + +Every provider adapter injects the same inert context into shell commands run by +its harness: + +```text +AI_AGENT= +ME_PROJECT_DIR= +``` + +`AI_AGENT` identifies a recognized harness for policy lookup. `ME_PROJECT_DIR` +is the directory anchor used to resolve the machine-local profile. Neither value +authenticates a principal, changes credentials, or enables repository +configuration discovery. + +Only a plain `me` command running under this recognized harness contract applies +the CLI-routing surface. Commands run in a user's ordinary shell continue using +normal flags, `ME_*` environment variables, and global CLI configuration. + +## MCP and capture + +Managed MCP registrations run `me mcp --harness `. The MCP command resolves +the matching directory policy before resolving credentials. A missing, disabled, +or unselected managed harness starts without Memory Engine tools. A bare `me mcp` +invocation remains the separate manual mode and is not policy-gated. + +Capture is opt-in and best-effort. Hooks resolve the capture surface from the +harness project directory, return successfully when capture is disabled or +fails, and import through the normal idempotent session-import path. Capture must +not interrupt an interactive coding session. + +## Credentials and boundaries + +Harnesses run as the credential they actually receive. There is no agent +impersonation, `--as-agent` mode, or access header that changes the principal. +Credential selection remains normal: explicit flags and `ME_*` environment values +override saved targeting, and API keys are supplied explicitly rather than stored +in harness policy. + +For a deliberately restricted harness, run it in an environment that does not +also contain the user's broader credentials and provide a restricted personal API +key or a service-account key. Policy routing is convenience and activation +control, not a credential-security boundary. + +## Provider adapters + +| Harness | Adapter document | MCP | Capture | Shell contract | +| --- | --- | --- | --- | --- | +| Claude Code | [Claude adapter](harnesses/claude.md) | Plugin-provided | Stop and SessionEnd hooks | SessionStart writes the sourced Claude environment file. | +| OpenCode | [OpenCode adapter](harnesses/opencode.md) | Managed local entry | Idle and deleted session events | Plugin `shell.env` hook. | +| Codex CLI | [Codex adapter](harnesses/codex.md) | Managed local entry | Stop and SessionEnd hooks | PreToolUse Bash command rewrite. | + +Provider adapters own only the mechanics needed to satisfy this shared contract. +Provider setup instructions and command reference remain in `docs/`. diff --git a/design/harnesses/claude.md b/design/harnesses/claude.md new file mode 100644 index 00000000..08e87690 --- /dev/null +++ b/design/harnesses/claude.md @@ -0,0 +1,39 @@ +--- +title: Claude Code Harness Adapter +tags: [harnesses, claude, mcp, capture] +--- + +# Claude Code Harness Adapter + +The Claude adapter is a user-scoped Memory Engine plugin installed through the +Memory Engine marketplace. The plugin provides the managed MCP registration and +the hooks needed for the shared harness contract and optional capture. + +## Session environment + +The plugin's `SessionStart` hook runs `me claude env`. The hook reads Claude's +session `cwd` from its event payload and writes `AI_AGENT=claude` plus +`ME_PROJECT_DIR` into Claude's sourced environment file. The write is +marker-delimited and idempotent because SessionStart can run again on resume or +after a session reset. + +This is an environment-discovery mechanism only. It does not select credentials, +enable MCP, or enable capture. + +## MCP and capture + +The plugin starts the managed MCP process, which resolves the Claude MCP policy +for the session directory. Claude's project-directory environment can be used as +a fallback for managed MCP discovery, but an existing directory profile at the +process working directory takes precedence to avoid worktree ambiguity. + +The plugin's `Stop` and `SessionEnd` hooks invoke `me claude hook`. Each hook +reads the transcript path and resolves capture policy from the session directory. +Capture runs asynchronously and exits successfully on errors so it cannot block +Claude's session lifecycle. + +## Installation boundary + +Installation registers the marketplace and plugin at user scope. Restarting +Claude Code loads changed plugin assets. Policy remains external to the plugin, +so one installation can be active in some directories and dormant in others. diff --git a/design/harnesses/codex.md b/design/harnesses/codex.md new file mode 100644 index 00000000..0627381f --- /dev/null +++ b/design/harnesses/codex.md @@ -0,0 +1,35 @@ +--- +title: Codex CLI Harness Adapter +tags: [harnesses, codex, mcp, capture, hooks] +--- + +# Codex CLI Harness Adapter + +The Codex adapter installs a user-global managed MCP registration and hooks in +Codex's hook configuration. It also allows non-secret `ME_API_KEY`, `ME_SERVER`, +and `ME_SPACE` environment variables to reach the managed MCP process without +writing their values into Codex configuration. + +## Session environment + +Codex does not expose a shell-environment hook. Its `PreToolUse` hook may rewrite +a Bash tool command, so `me codex env-hook` reads the hook payload and prepends a +shell-quoted export of `AI_AGENT=codex` and the payload's `cwd` as +`ME_PROJECT_DIR`. + +The rewrite is deliberately narrow and fail-open. Non-Bash calls are ignored. An +unrecognized payload produces no rewrite and records only a sanitized payload +shape for `me doctor`; it never logs command content or blocks a Codex turn. + +## MCP and capture + +The managed MCP registration runs `me mcp --harness codex` and is gated by the +Codex MCP policy. `Stop` and `SessionEnd` hooks invoke `me codex hook`, which +imports the supplied transcript only when capture policy selects Codex. Capture +is best-effort and always exits successfully. + +## Provider constraint + +Codex requires the user to approve hooks through `/hooks`. Installation cannot +and must not bypass that provider trust decision. This also prevents a safe, +fully non-interactive live smoke test of Codex hook installation. diff --git a/design/harnesses/opencode.md b/design/harnesses/opencode.md new file mode 100644 index 00000000..6ce637b6 --- /dev/null +++ b/design/harnesses/opencode.md @@ -0,0 +1,35 @@ +--- +title: OpenCode Harness Adapter +tags: [harnesses, opencode, mcp, capture] +--- + +# OpenCode Harness Adapter + +The OpenCode adapter installs two user-global artifacts: a local managed MCP +entry and a generated Memory Engine plugin. The MCP entry runs +`me mcp --harness opencode`; the plugin supplies runtime environment and capture +hooks. + +## Session environment + +OpenCode exposes a `shell.env` plugin hook. The generated plugin sets +`AI_AGENT=opencode` and `ME_PROJECT_DIR` to OpenCode's session directory on every +harness shell command. No command rewriting is needed. + +## MCP and capture + +The managed MCP entry is dormant until the matching MCP policy enables OpenCode. +The generated plugin listens for `session.idle` and `session.deleted`, then calls +`me opencode hook` with the session ID and project directory. That command loads +the session from OpenCode storage and imports it only when capture policy selects +OpenCode. + +The plugin awaits the capture attempt but swallows failures. Session capture is +therefore idempotent and best-effort rather than a condition for OpenCode to +continue. + +## Installation boundary + +The installer changes only its named MCP entry and generated plugin file. It +refuses to claim an unrecorded or user-modified artifact, and uninstall preserves +such artifacts for manual review. diff --git a/adr/0002-hnsw-filtered-recall.md b/design/hnsw-filtered-semantic-search.md similarity index 57% rename from adr/0002-hnsw-filtered-recall.md rename to design/hnsw-filtered-semantic-search.md index 5dfc0c68..1db01499 100644 --- a/adr/0002-hnsw-filtered-recall.md +++ b/design/hnsw-filtered-semantic-search.md @@ -1,16 +1,11 @@ --- -title: Use strict-order HNSW iterative scans for filtered semantic search -status: Accepted -date: 2026-08-07 -deciders: [jgpruitt] +title: Filtered Semantic Search tags: [search, semantic-search, pgvector, hnsw] -tickets: [] -supersedes: [] --- -# ADR 0002: Use strict-order HNSW iterative scans for filtered semantic search +# Filtered Semantic Search -## Context +## HNSW behavior under filters Memory Engine combines semantic ranking with access-control, tree, metadata, temporal, regular-expression, and similarity filters. pgvector's HNSW index normally explores a candidate window controlled by `hnsw.ef_search`, while PostgreSQL applies additional predicates to those candidates. If filters reject many candidates, a query can return fewer than its requested limit even when more qualifying memories exist beyond the initial HNSW search horizon. @@ -20,8 +15,6 @@ pgvector supports iterative HNSW scans, which continue exploring the graph when The setting is a pgvector custom PostgreSQL parameter. Such parameters are registered lazily when pgvector loads in a database connection. A function-level `SET hnsw.iterative_scan ...` clause is validated when the function is created. During an existing-database migration, pgvector may not yet be loaded on that connection, and a non-superuser migration role can be denied permission to set the unrecognized parameter. -## Decision - Semantic search enables pgvector iterative scans in `strict_order` mode before executing its vector query: ```sql @@ -34,33 +27,12 @@ The call is made inside the vector branch of the search function. Its third argu `hnsw.ef_search`, `hnsw.max_scan_tuples`, and `hnsw.scan_mem_multiplier` remain at their configured defaults. With iterative scanning enabled, `ef_search` is primarily a performance tuning parameter rather than the initial candidate window becoming a correctness boundary. Further tuning requires workload-specific benchmarks. -## Consequences +## Invariants and constraints - Filtered semantic searches continue scanning for qualifying candidates instead of stopping after filters shrink the initial HNSW candidate set. - Direct semantic search and the semantic arm of hybrid search share the same recall behavior. -- Returned candidates preserve strict distance order, which keeps hybrid rank assignment deterministic for a given candidate set. +- Candidates preserve strict distance order, which keeps hybrid rank assignment deterministic for a given candidate set. - Iterative scanning can inspect more graph tuples and increase query latency when filters are selective. -- Search remains bounded by the requested result limit and pgvector's scan limits. This improves filtered recall but does not turn approximate nearest-neighbor search into an exhaustive scan. -- Setting the parameter at query time avoids coupling function creation to whether pgvector has already registered its custom parameters on the migration connection. - -## Alternatives considered - -### Function-level `SET` clause - -A function-level clause is concise and automatically applies on every call, but it can make an existing-database migration fail when pgvector has not loaded on the connection and the migration role is not a superuser. Query-time `set_config` applies the setting where it is needed without that creation-time dependency. - -### `relaxed_order` iterative scans - -Relaxed ordering can offer different performance tradeoffs, but RRF depends on the rank assigned by each search arm. Preserving strict distance order is more important than the potential performance gain. - -### Increase `hnsw.ef_search` without iterative scanning - -A larger fixed candidate window reduces the problem but cannot guarantee that enough rows survive arbitrary filters. It also imposes the higher search cost on queries whose filters do not need it. - -### Set a database-wide default - -A database-wide setting requires privileged operational configuration and applies to unrelated vector queries. Keeping the decision in the search function makes the behavior portable and scoped to Memory Engine's query. - -### Derive `hnsw.ef_search` from the requested limit - -This may improve latency for some workloads, but the appropriate relationship depends on data distribution and filter selectivity. It is deferred until supported by benchmarks. +- Search remains bounded by the requested result limit and pgvector's scan limits. It improves filtered recall but does not make approximate nearest-neighbor search exhaustive. +- The parameter is set at query time because a function-level `SET` clause can fail migration when pgvector has not registered its custom parameters on that connection. +- `hnsw.ef_search`, `hnsw.max_scan_tuples`, and `hnsw.scan_mem_multiplier` retain their configured defaults until workload-specific benchmarks justify tuning them. diff --git a/adr/0004-hybrid-rrf-scoring.md b/design/hybrid-rrf-scoring.md similarity index 54% rename from adr/0004-hybrid-rrf-scoring.md rename to design/hybrid-rrf-scoring.md index 09d64467..5a6abec5 100644 --- a/adr/0004-hybrid-rrf-scoring.md +++ b/design/hybrid-rrf-scoring.md @@ -1,16 +1,11 @@ --- -title: Use Reciprocal Rank Fusion without a fused-score threshold -status: Accepted -date: 2026-08-07 -deciders: [jgpruitt] +title: Hybrid RRF Scoring tags: [search, hybrid-search, reciprocal-rank-fusion, scoring] -tickets: [] -supersedes: [] --- -# ADR 0004: Use Reciprocal Rank Fusion without a fused-score threshold +# Hybrid RRF Scoring -## Context +## Fusion model Semantic search returns bounded cosine similarity, while fulltext search returns unbounded BM25 scores whose scale depends on the query and corpus. Adding the raw scores would allow whichever scoring system produces larger numbers to dominate, even when that scale does not represent greater relevance. @@ -21,12 +16,14 @@ score = fulltext_weight / (k + fulltext_rank) + semantic_weight / (k + semantic_rank) ``` -A missing arm contributes zero. The defaults are `k = 60`, equal weights, and 30 candidates per arm. Candidate limits are always at least the requested result limit and are bounded by the search result cap. +A missing arm contributes zero. Each arm assigns ranks by score, using memory ID as +a deterministic tie-break. The defaults are `k = 60`, equal weights, and 30 +candidates per arm. Candidate limits are always at least the requested result +limit and are bounded by the search result cap. `k` is clamped to zero or above; +each weight is clamped to `[0, 1]`. RRF deliberately discards raw scores and uses rank. This normalizes the incomparable semantic and BM25 scales and rewards results that rank well in both arms. It also means the fused score is determined by position within the candidate windows, not by an absolute relevance measurement. -## Decision - Hybrid search uses weighted RRF over the semantic and fulltext candidate rankings. The fused score is exposed for ordering and inspection, but Memory Engine does not expose an `rrfThreshold` parameter. Search quality is controlled before fusion: @@ -39,29 +36,12 @@ Search quality is controlled before fusion: Hybrid search is defined as a fused top-k operation, not an exhaustive retrieval interface. Callers that need as many results as possible above a quality bar should use a single search mode with a larger limit: semantic search with `semanticThreshold`, or fulltext search with its positive-match invariant. Truly exhaustive retrieval requires a separate paginated design. -## Consequences +## Invariants and constraints -- Hybrid search can combine semantic and lexical rankings without normalizing or calibrating their raw score distributions. +- Hybrid search combines semantic and lexical rankings without normalizing or calibrating their raw score distributions. - Results appearing in both arms generally receive more weight than results appearing in only one arm. This cross-arm agreement is intentional. - The hybrid score is a small positive rank-fusion value. It is comparable only within the same result set and is not an absolute relevance score. +- The ranking in each arm is deterministic for equal scores because memory ID is the tie-break. - Changing `k`, weights, candidate limits, or competing candidates can change a memory's fused score even when its content and query are unchanged. -- There is no intuitive fixed threshold for "good" hybrid results. Callers tune the input arms and result limits instead. -- Raising `candidateLimit` broadens the fusion pool and can increase query cost; it still does not make hybrid search exhaustive. - -## Alternatives considered - -### Add or average raw semantic and BM25 scores - -The raw score scales are incompatible. Cosine similarity is bounded, while BM25 is unbounded and corpus-dependent. A numeric combination would require calibration that changes with queries and corpus composition. - -### Expose an `rrfThreshold` - -An RRF threshold would filter rank position within a particular candidate window, not absolute relevance. The useful numeric range shifts with `k`, weights, candidate limits, and whether a result appears in one or both arms. Such a parameter would be difficult to set by intuition and would not satisfy requests for every result above a relevance bar. - -### Make hybrid search exhaustive when a threshold is present - -RRF is computed after each arm has been truncated to its candidate set. A post-fusion threshold cannot recover records that were never candidates. Making the candidate sets unbounded would introduce a substantially different and potentially expensive query model that needs pagination and explicit resource controls. - -### Prefer one arm when scores disagree - -Always preferring semantic or fulltext ranking would discard the benefit of hybrid retrieval. Adjustable weights already let callers express a preference while retaining evidence from both modes. +- There is no `rrfThreshold`: a fused score represents rank position in particular candidate windows, not absolute relevance. Callers tune the input arms and result limits instead. +- Raising `candidateLimit` broadens the fusion pool and can increase query cost; it does not make hybrid search exhaustive. diff --git a/adr/0005-jsonpath-metadata-predicates.md b/design/jsonpath-metadata-predicates.md similarity index 65% rename from adr/0005-jsonpath-metadata-predicates.md rename to design/jsonpath-metadata-predicates.md index 5e041e9d..e4bc938b 100644 --- a/adr/0005-jsonpath-metadata-predicates.md +++ b/design/jsonpath-metadata-predicates.md @@ -1,16 +1,11 @@ --- -title: Expose PostgreSQL JSONPath predicates for advanced metadata filtering -status: Accepted -date: 2026-08-09 -deciders: [jgpruitt] +title: Metadata Predicates tags: [search, metadata, jsonb, jsonpath, gin] -tickets: [] -supersedes: [] --- -# ADR 0005: Expose PostgreSQL JSONPath predicates for advanced metadata filtering +# Metadata Predicates -## Context +## Metadata filters Memory Engine stores each memory's metadata as a `jsonb` object and exposes a structured `meta` search filter. That filter uses PostgreSQL containment: @@ -89,9 +84,9 @@ Relevant PostgreSQL 18 documentation: - [Boolean predicate check expressions](https://www.postgresql.org/docs/18/functions-json.html#FUNCTIONS-SQLJSON-CHECK-EXPRESSIONS) - [GIN built-in operator classes](https://www.postgresql.org/docs/18/gin.html#GIN-BUILTIN-OPCLASSES) -## Decision +## `metaPredicate` interface -Add an optional `metaPredicate` search filter containing a PostgreSQL `jsonpath` +`metaPredicate` is an optional search filter containing a PostgreSQL `jsonpath` predicate check expression. Evaluate it with the `@@` operator: ```ts @@ -129,7 +124,7 @@ metadata. Callers may prefix a path with `strict` when they need exact structura matching instead of lax-mode array wrapping and unwrapping; the operator still suppresses the documented evaluation errors. -Expose the same filter on public surfaces that provide the complete memory +The same filter is available on public surfaces that provide the complete memory search contract, including the TypeScript client, CLI, MCP search and export tools, and advanced web search. Their descriptions and documentation include PostgreSQL-specific examples for equality, numeric comparison, Boolean @@ -146,7 +141,7 @@ index strategy for `@@`, not for the function call. Do not promise that every valid predicate is index-backed: documentation must distinguish extractable equality predicates from predicates that may scan. -## Consequences +## Behavior and constraints - Callers can express metadata comparisons, ranges, OR and negation, missing-key checks, regexes, array cardinality, arithmetic, cross-field comparisons, and @@ -162,66 +157,9 @@ equality predicates from predicates that may scan. - Missing or differently typed metadata generally produces a non-match rather than failing the whole search because `@@` suppresses common structural and evaluation errors. -- The public API becomes coupled to PostgreSQL's SQL/JSON path dialect. This is - intentional: exposing the database-native predicate avoids inventing and - maintaining a separate expression language. -- Adding the predicate to the database search functions changes their signatures - and requires the guarded function-signature migration pattern. The new - argument should be trailing and defaulted so existing positional callers stay - valid during a rolling deployment. -- The space schema version must advance because an older application does not - understand the new search-function contract. -- A new client talking to an old server must not silently lose the predicate and - return broader results. Client/server compatibility bounds must make the - required server version explicit. - -## Alternatives considered - -### Document containment and add no new filter - -This is sufficient for the allow-list example and should be documented -regardless. It does not address numeric comparisons, OR, negation, regexes, -cardinality, arithmetic, or cross-field predicates. - -### Name the field `metaQuery` - -The name is approachable but ambiguous. PostgreSQL distinguishes paths that -select items from predicate check expressions that return a Boolean. A public -name that omits that distinction makes incorrect `@?`/`@@` usage more likely. - -### Use `@?` or `jsonb_path_exists` - -Path-existence semantics work for expressions such as: - -```text -$.allowList[*] ? (@ == "tom") -``` - -They are less natural for a general search filter, where callers expect to -write Boolean expressions. Predicate expressions are also unsafe with `@?` -because both `true` and `false` are returned items. If path-existence semantics -are added later, they should use a distinct name such as `metaPathExists`. - -### Use `jsonb_path_match` instead of `@@` - -The function has the desired Boolean semantics and supports a separate variables -object, but the existing GIN index advertises an index strategy for `@@`, not the -function. Using the operator preserves index eligibility for extractable -equality clauses. - -### Define a structured metadata expression language - -A custom tree of operators could be easier to validate and generate safely, but -it would duplicate PostgreSQL's expression model, require an expanding public -schema, and need a compiler with well-defined null, missing-field, array, type, -and error semantics. The structured `meta` filter already covers the common -simple case; the advanced escape hatch should remain database-native. - -### Replace the index with `jsonb_path_ops` - -`jsonb_path_ops` can be smaller and more selective for supported containment and -JSONPath searches, but it supports fewer operators and creates no entries for -value-less structures such as `{"pack":{}}`. Replacing the existing index could -regress current containment and future key-existence workloads. A second index -would add storage and write amplification. Either change requires workload -evidence and is outside this decision. +- The public API uses PostgreSQL's SQL/JSON path dialect rather than a custom + expression language. +- Database search-function signatures retain `metaPredicate` as a trailing, + defaulted argument so existing positional callers remain valid. +- Client/server compatibility requires a server that understands the filter; a + request must never silently drop it and return broader results. diff --git a/design/memory-authorization.md b/design/memory-authorization.md new file mode 100644 index 00000000..676cb718 --- /dev/null +++ b/design/memory-authorization.md @@ -0,0 +1,126 @@ +--- +title: Memory Authorization +tags: [authorization, access-control, trees, spaces, grants] +--- + +# Memory Authorization + +Memory authorization is space-scoped and tree-scoped. Each memory has an `ltree` +path, and a caller may access a memory only when its effective grant set covers +that path at the required level. This permits shared and private subtrees in the +same space without making the full space visible to every member. + +## Membership and data access + +Direct membership in a space admits a user or service account to its memory RPC +endpoint. It does not grant access to any memory. A direct member with no tree +grants can authenticate successfully but has an empty effective grant set and +cannot read or mutate memory data. + +Groups are principals that hold raw grants, but they are not executable callers. +Their grants become effective for a user or service account only when that member +is also directly rostered in the same space. This allows group membership to be +prepared before a member joins without granting access prematurely. + +The conventional tree roots are `/share` for shared data and a private home tree +for each user. `~` is input and display shorthand for the current user's home. +Joining users normally receive `owner` access to their home; service accounts do +not have a home grant and receive access only through explicit or group grants. +Custom space provisioning can disable the automatic home grant. + +## Grant model + +Grants are stored against a principal and a tree path in a space. They are +hierarchical and additive: a grant applies to its path and every descendant. +There are three levels: + +| Level | Name | Effect | +| --- | --- | --- | +| 1 | `read` | Read and discover memories in the covered subtree. | +| 2 | `write` | Create, update, move, and delete memories in the covered subtree. | +| 3 | `owner` | `write`, plus delegation of grants within the covered subtree. | + +There are no deny entries or action-specific permissions. Removing a grant only +removes that grant; it cannot override another direct or group-derived grant. +The root path is the empty `ltree` path, displayed as `/`; an `owner` grant there +covers the entire space. + +`write` is required at the affected tree. Cross-tree moves require write access +to both the source and destination. Read checks also protect named-memory +resolution, so callers cannot use a path and name to probe for inaccessible +memories. + +## Effective access + +At space-endpoint authentication, the server first verifies direct space +membership, then builds the caller's effective access from direct grants and +applicable group grants. The client never submits this grant set. + +The resulting JSON array has this shape: + +```json +[ + { "tree_path": "share.projects", "access": 2 }, + { "tree_path": "home.user_id", "access": 3 } +] +``` + +The server passes that array to memory data functions. Those functions +parse it and authorize an operation when at least one grant path is an +ancestor-or-self of the memory path and has an equal or higher access level. +There is no row-level security policy or caller-controlled database session +variable involved in this check. + +This separation keeps grant resolution in the control plane while making the +data plane enforce every read and mutation with the same request-bound access +set. `access.effective` exposes the resolved set for inspection; raw grant rows +remain available separately through `grant.*` operations. + +## Why not RLS + +Earlier versions used PostgreSQL row-level security (RLS) policies to call a +tree-access check for each memory row. RLS preserved authorization correctness, +but its security barrier can prevent non-`LEAKPROOF` user predicates from being +pushed below the policy check. That can make indexes for rich memory filters +unavailable. + +In manual testing with roughly 280,000 memories, a filter-only tree query using +the `ltree` GiST index took about 32 ms without RLS and about 239 ms with RLS, +where PostgreSQL selected a sequential scan. The same risk applies to other +filter dimensions that use extension operators, including JSONB containment, +temporal ranges, and regular expressions. + +This is not a general rejection of RLS. It is well suited to simple, row-local +predicates such as indexed tenant equality. Memory search combines dynamic +hierarchical grants with arbitrary filters and extension indexes, where keeping +those predicates visible to the planner is necessary. Passing the resolved, +request-bound grant set to memory SQL functions keeps authorization in the +database while avoiding the RLS planning barrier. + +## Administration and delegation + +Space administration and tree ownership are separate authorities: + +- A space admin manages structural concerns such as the roster, groups, and + invitations. An admin may grant or remove access anywhere in the space and can + self-grant `owner` at `/` when data ownership is needed. +- A tree owner may grant, remove, and list grants within the owned subtree even + when they are not a space admin. This is how data-access administration is + delegated without giving roster authority. + +Grant listing is scoped the same way: an owner may list grants under an owned +subtree, while a space admin can list the full space. Members may inspect their +own effective access without either authority. + +## API-key ceilings + +API keys authenticate as their owning user or service account. An unrestricted +key receives that principal's live effective access in a directly admitted +space. A restricted key declares the spaces, optional space-admin authority, and +optional tree grants it may use. + +Restricted-key declarations are ceilings, never additional grants. The server +intersects every declared path and level with the principal's current effective +access, retaining the narrower path and lower level. If the key, principal, or +space binding is inconsistent, the effective set is empty. Revoking or lowering +the principal's live grants therefore constrains existing keys immediately. diff --git a/design/memory-schema.md b/design/memory-schema.md new file mode 100644 index 00000000..6f1546b5 --- /dev/null +++ b/design/memory-schema.md @@ -0,0 +1,141 @@ +--- +title: Memory Schema +tags: [memory, schema, modeling, temporal, metadata, trees] +--- + +# Memory Schema + +Each space has one `memory` table. The table is a flexible substrate for context +engineering, not a prescribed ontology for facts, conversations, or skills. A +memory is content plus three independent annotations: a hierarchical tree, free +form metadata, and an optional temporal range. Callers choose the conventions +that fit their workflow; Memory Engine does not extract facts, infer entities, +or transform content into a hidden schema. + +This shape supports several common memory types without separate tables: + +| Type | Typical representation | +| --- | --- | +| Working memory | Kept by the caller in the active context window; retrieved memories may be added when relevant. | +| Episodic memory | Immutable or append-oriented content with a point-in-time temporal value. | +| Semantic memory | A discrete fact, preference, decision, or reference, optionally with a validity range. | +| Procedural memory | A runbook, workflow, or reusable instruction stored as content and organized by tree and metadata. | + +The storage model does not reserve one representation for any of these types. +`meta` conventions can distinguish them when an application needs to filter or +operate on a category. + +## Record fields + +| Field | Meaning | +| --- | --- | +| `id` | Immutable UUIDv7 identity. It supports chronological ordering and survives a move or rename. | +| `content` | Required, caller-provided text. It is the source for full-text and semantic search. | +| `tree` | Required `ltree` path that organizes the memory and defines its authorization boundary. | +| `meta` | Required JSON object, defaulting to `{}`. It holds caller-defined facets and source information. | +| `temporal` | Optional `tstzrange` describing when the memory happened or was valid. | +| `name` | Optional mutable filename-like leaf name, unique within one exact tree. | +| `embedding` | Optional vector derived asynchronously from `content`. | +| `created_at`, `updated_at` | Storage timestamps, distinct from the represented time in `temporal`. | +| `version`, `version_hash` | Server-maintained optimistic-concurrency state. | + +`id` is the canonical address. A named memory also has a human-friendly +`tree/name` address, such as `/share/auth/jwt-rotation`. The name is not part of +the `ltree`, so dots in a filename do not create a hierarchy. Multiple unnamed +memories may share a tree; a non-null name is unique only within that exact tree. + +## Orthogonal annotations + +### Tree + +The tree is a hierarchy, not just an organizational tag. It supports subtree +search and is the unit of data access: a grant covers its path and descendants. +Conventional roots are `/share` for collaboration and each user's private home +tree. See [Memory Authorization](memory-authorization.md) for access semantics. + +### Metadata + +`meta` is an object rather than a fixed column set. It can hold type, source, +importer, status, owner-defined facets, or any other workflow-specific data. +JSONB supports exact metadata filters and JSONPath predicates without forcing +all users into a global schema. Metadata is stored as supplied; callers own the +meaning and lifecycle of their keys. + +### Temporal range + +`temporal` models the time represented by the memory, not its database creation +time. A point event uses equal inclusive bounds, such as a message, commit, or +deployment. A period uses an inclusive start and exclusive end, such as the +validity of a fact, a project phase, or an outage. The database enforces these +two conventions. A temporal range can be queried by containment and overlap, +making time both a modeling and retrieval dimension. + +## Search substrate + +The model exposes six composable retrieval dimensions. A caller selects the +dimensions that match the question instead of passing every request through a +fixed extraction or retrieval pipeline. + +| Dimension | Record field or index | Use | +| --- | --- | --- | +| Semantic | HNSW over `embedding` | Find related meaning when the wording differs. | +| Full-text | BM25 over `content` | Match exact identifiers, terms, and phrases. | +| Hierarchy | GiST over `tree` | Scope retrieval to a path or subtree. | +| Temporal | GiST over `temporal` | Find information that contains, overlaps, precedes, or follows a time window. | +| Metadata | GIN over `meta` | Filter caller-defined facets with structured metadata or JSONPath. | +| Regex | Case-insensitive POSIX expression over `content` | Apply an exact content pattern alongside an indexed query or filter. | + +Regex is deliberately required to accompany semantic or full-text search, a +tree, structured metadata, or a temporal filter. Used alone, it could force an +unbounded scan; it is a precision filter, not a broad retrieval primitive. + +Hybrid is not a seventh independent dimension. It is an optional ranking mode +that combines semantic and full-text result sets through Reciprocal Rank Fusion +(RRF). The public product description calls this one of six *search modes*: +semantic, keyword, temporal, metadata, hierarchy, and hybrid. That +mode-oriented wording emphasizes how people initiate a search; the API-oriented +model retains regex as the sixth composable dimension and treats hybrid as the +ranking composition of two dimensions. + +For example, an agent can use semantic search plus a project tree to explore a +concept, BM25 plus a temporal range to investigate a change, regex plus metadata +to find a precise format within a source corpus, or hybrid ranking followed by +any of those filters. The schema makes all of these query-time choices possible. + +## Writes, identity, and concurrency + +Creating a memory requires an explicit target tree. The idempotency key depends +on the record shape: + +- A named memory is keyed by `(tree, name)`; name takes precedence over a + supplied `id` for deduplication. +- An unnamed memory with an explicit `id` is keyed by that id. +- An unnamed memory without an `id` is anonymous and always inserts. + +`onConflict: error` rejects a collision, `replace` updates only when the stored +content, metadata, or temporal value differs, and `ignore` leaves the existing +record unchanged. A named replacement keeps the existing record identity. This +makes importer reruns predictable while preserving links to a named record. + +Updates use the current `version_hash`. Any change to tree, name, metadata, +temporal value, or content increments `version` and computes a new hash. A patch +with an old hash fails rather than silently overwriting a concurrent edit. + +## Embedding lifecycle + +Embeddings are derived data, not the memory's source of truth. A content change +clears the current embedding and increments the content version so the embedding +worker can regenerate it. Metadata, tree, name, and temporal changes do not +require re-embedding because they do not change the text being represented. A +memory remains usable for all non-semantic retrieval while embedding is pending. + +## Consequences + +- The database contains the content the caller wrote, with explicit annotations; + there is no hidden fact-extraction or summarization layer. +- A record can be moved, renamed, or reclassified without changing its immutable + id. +- Created and updated timestamps answer when storage changed; `temporal` answers + when the represented information occurred or applied. +- New use cases should normally begin with tree and metadata conventions, not a + new memory table or a new fixed record type. diff --git a/design/principal-model.md b/design/principal-model.md new file mode 100644 index 00000000..a18ee48b --- /dev/null +++ b/design/principal-model.md @@ -0,0 +1,96 @@ +--- +title: Principal Model +tags: [authorization, principals, users, groups, service-accounts] +--- + +# Principal Model + +The authorization model has three principal kinds: + +| Kind | Purpose | Credential-bearing | +| --- | --- | --- | +| User (`u`) | A global human identity. | Yes | +| Group (`g`) | A space-scoped collection for grants and delegated administration. | No | +| Service account (`s`) | A space-scoped non-human identity for independent automation. | Yes | + +`principal` is the common unit used by the space roster and tree grants. +`member` is deliberately narrower: it means only a user or service account, the +two kinds that can belong to groups and hold API keys. + +## Users + +Users are global principals whose IDs match their authentication identities. They +can be directly rostered in multiple spaces and authenticate through user +credentials, including personal API keys. A user's name is its global identity +handle, so it is not renamed through space-management operations. + +Joining a space makes a user a direct member and normally grants ownership of +that user's home tree. The user receives no other data access unless it is +granted directly or through a group. + +## Groups + +Groups belong to one space and are rostered into that space when created, making +them resolvable grant recipients. A group can receive tree grants and can be +made an admin group, but it cannot authenticate, hold an API key, or be a group +member itself. Groups cannot nest; only users and service accounts can be group +members. + +Group membership is space-scoped and non-transitive. It does not admit a user or +service account to the space: group-derived tree grants and admin authority take +effect only after that member has its own direct roster entry. This permits +preparing group membership before a user joins without prematurely granting +access. + +A group's `admin` roster flag makes it an admin group, whose authority flows to +its direct user members that are also direct space members. This is distinct from +the group-member admin flag, which controls administration of that group itself. + +## Service accounts + +Service accounts are independent, non-human principals for automation. They are +created in exactly one space, are directly rostered there, and use API keys to +authenticate. A service account starts with no tree grants, no home tree, and no +default-group membership; it receives access only through explicit grants or +ordinary group membership. + +Creating a service account also creates a bound, name-derived admin group. Space +admins, or direct user members of that bound group who are also direct space +members, may manage the service account and its API keys. The bound group is an +administration mechanism, not an automatic data-access grant. + +Service accounts may be ordinary group members and group administrators. They do +not inherit space-admin authority from admin-group membership; only a direct +space-admin roster entry can make a service account a space admin. + +## Why there is no agent principal + +The retired `agent` principal kind represented a harness-specific subordinate +identity. Its purpose was valuable: let a user give a coding harness only the +memories relevant to its task. A restricted scope reduces both data exposure and +model-context waste from unrelated memories. + +Its effective access depended on its own grants, group membership, owner +relationship, home path, and an additional runtime intersection with the owner's +access. In other words, the agent's authorization was capped by its owner's. +This was powerful, but difficult to reason about. + +The agent principal existed to restrict a coding harness. That required the CLI, +MCP server, hooks, and other integration surfaces to run as the agent whenever a +harness was active. Detecting that context and locking it to the correct agent was +complex, especially when projects on the same machine needed different agent +identities. In practice, a harness that could reach the user's credentials could +escape the intended scope and use those broader credentials. + +It also conflated two separate decisions: installing a local AI harness and +creating an independent identity with its own IAM relationship. Harness commands +now always run as the principal represented by their actual credential; there is +no agent impersonation or agent-specific access header. + +The reliable way to restrict a harness is to run it in an environment that never +receives the user's credentials. In that setting, a restricted personal API key +can enforce the desired space and tree-access ceiling with much less complexity +than a separate agent principal. Independent automation that needs its own +credential and permissions uses a service account. These approaches keep the +integration identity, grants, and administrators explicit and inspectable without +coupling harness configuration to a hidden subordinate-principal model. diff --git a/design/restricted-api-keys.md b/design/restricted-api-keys.md new file mode 100644 index 00000000..66f543f5 --- /dev/null +++ b/design/restricted-api-keys.md @@ -0,0 +1,82 @@ +--- +title: Restricted API Keys +tags: [authorization, api-keys, personal-access-tokens, sandboxing] +--- + +# Restricted API Keys + +API keys are credentials for a user or service account. They are global to that +principal rather than intrinsically bound to one space. An unrestricted key has +the same live authority as its holder in every space where that holder is directly +admitted. + +A restricted key adds an explicit, server-enforced ceiling. It is intended for +uses that need less authority than the user normally has, especially a coding +harness in a sandbox. A restricted personal access token lets the sandbox receive +only the spaces and trees relevant to its task, reducing both data exposure and +unnecessary model context. The sandbox must not also receive the user's broader +credentials, or it can bypass the intended restriction. + +## Key lifecycle + +A user can mint a personal access token for themselves or a key for a service +account they administer. The plaintext key is returned only at creation; the +server stores only its hash. Keys may expire, and deletion is revocation. + +Key creation and deletion require a user session. API-key-authenticated callers +cannot mint or revoke keys, preventing a leaked key from creating a replacement +that survives its own revocation. + +## Declarations + +Omitting access declarations creates an unrestricted key. Supplying one or more +per-space declarations creates a restricted key. Each declaration contains: + +| Field | Meaning | +| --- | --- | +| Space | A space where the key may be used. The holder must be a direct member. | +| Tree grants | Optional path and access-level ceilings: `read`, `write`, or `owner`. | +| Space admin | Optional permission to exercise the holder's existing space-admin authority. | + +A declaration with no tree grants allows the holder's full *live* tree access in +that declared space. It does not grant root ownership or any access the holder +does not already have. A declaration with tree grants limits the key to those +paths and levels. + +A restricted service-account key can declare only its service account's space. +The CLI exposes declarations with repeatable `--allow` options and optional +`--space-admin`; the latter requires an `--allow` declaration for the same space. + +## Effective authority + +The key declaration is a ceiling, not an independent grant. For each request, +the server first resolves the holder's current direct membership and effective +tree grants. It then intersects that live authority with the key's declaration: + +- The key must declare the selected space. +- For overlapping tree grants, the effective path is the narrower path and the + effective level is the lower level. +- A declared space-admin capability is effective only when the holder is also a + live space admin. + +An inconsistent key, principal, or space binding resolves to no tree access. Any +later loss of space membership, grant, or admin authority takes effect on the +next request without rotating the key. + +## Endpoint behavior + +Restricted keys authenticate as their holder; they do not impersonate an agent +or alter the principal model. On the memory endpoint, normal tree-access checks +consume the key-clamped effective grant set. + +For the user endpoint, a restricted personal access token may inspect its +identity and discover only its declared spaces. It cannot manage account +resources, including API keys. This prevents a scoped token from widening or +replacing its own authority. + +## Choosing an identity + +Use a restricted personal access token when a sandboxed harness needs a reduced +view of one user's existing access. Use a service account when automation needs +its own independently managed identity and grants. Neither model relies on a +harness-specific principal or credential impersonation. diff --git a/adr/0001-semantic-search-scoring.md b/design/semantic-search-scoring.md similarity index 55% rename from adr/0001-semantic-search-scoring.md rename to design/semantic-search-scoring.md index 74540a68..ff2c8e0f 100644 --- a/adr/0001-semantic-search-scoring.md +++ b/design/semantic-search-scoring.md @@ -1,16 +1,11 @@ --- -title: Use cosine similarity for semantic search scores -status: Accepted -date: 2026-08-07 -deciders: [jgpruitt] +title: Semantic Search Scoring tags: [search, semantic-search, pgvector, scoring] -tickets: [] -supersedes: [] --- -# ADR 0001: Use cosine similarity for semantic search scores +# Semantic Search Scoring -## Context +## Score semantics Memory Engine uses pgvector's cosine distance operator, `<=>`, to rank semantic search results. Cosine distance is `1 - cosine similarity`, so lower distance is better and its range is `[0, 2]`. Cosine similarity has the inverse interpretation: higher is better and its range is `[-1, 1]`. @@ -24,8 +19,6 @@ This preserved ranking, but produced values in `[-2, 0]` that were neither cosin The HNSW index adds another constraint. PostgreSQL can use the cosine HNSW index for an ascending `ORDER BY embedding <=> query LIMIT ...`. Applying a transformation to the ordering expression can make the index ineligible. -## Decision - Semantic search returns cosine similarity: ```text @@ -49,28 +42,11 @@ maximum distance = 1 - minimum similarity Values outside `[0, 1]` are rejected rather than clamped. Validation occurs at both the public protocol boundary and the database boundary so direct database callers receive the same contract. -## Consequences +## Invariants and constraints - Semantic scores have a standard mathematical meaning and agree with `semanticThreshold`. -- Existing ranking is unchanged because cosine similarity is a monotonic transformation of cosine distance. +- Ranking is unchanged because cosine similarity is a monotonic transformation of cosine distance. - The HNSW index remains eligible because only the projected score is transformed; ordering uses the raw distance operator. -- Callers can compare semantic scores within a result set and use an intuitive absolute quality threshold. Scores are not intended for comparison with BM25 or hybrid scores. -- Although returned cosine similarity can be negative, the public threshold deliberately accepts only `[0, 1]`. The threshold is a relevance filter, not a way to request results that point away from the query vector. - -## Alternatives considered - -### Return cosine distance - -Returning the raw distance would match pgvector's operator directly, but it would reverse the product's established "higher is better" score convention and the public minimum-similarity vocabulary. - -### Keep negated cosine distance - -Negated distance preserves ordering, but its `[-2, 0]` range has no standard interpretation and conflicts with the public threshold contract. - -### Transform the `ORDER BY` expression - -Ordering by the projected similarity would read naturally, but risks preventing PostgreSQL from selecting the HNSW index. Transforming only the projection provides the same result order without that risk. - -### Clamp invalid thresholds - -Clamping would hide caller errors and could silently turn a malformed request into a much broader or narrower search. Rejecting invalid values makes the contract explicit. +- Scores are comparable within a semantic result set, but not with BM25 or hybrid scores. +- Returned cosine similarity can be negative, but `semanticThreshold` accepts only `[0, 1]`. It is a relevance filter, not a way to request results that point away from the query vector. +- Invalid thresholds are rejected rather than clamped so malformed requests cannot silently broaden or narrow a search. diff --git a/design/spaces.md b/design/spaces.md new file mode 100644 index 00000000..800a598b --- /dev/null +++ b/design/spaces.md @@ -0,0 +1,88 @@ +--- +title: Spaces and Provisioning +tags: [spaces, provisioning, authorization, groups] +--- + +# Spaces and Provisioning + +A space is the unit of memory isolation, membership, and authorization. Each +space has a stable 12-character slug, a mutable display name, and its own memory +data schema. The slug is used for routing and is never renamed; renaming a space +changes only its display name. + +Space-local schemas keep each space's data movable as scale requires. The current +deployment uses one PostgreSQL database and one connection pool, but a space +schema can later be moved to a separate database or shard without changing the +space's authorization or data model. Sharding is not part of the current request +routing design. + +Spaces have no implicit relationship to one another. A user, group, or service +account must have its own direct roster entry in a space before it can act there. +Tree grants then determine which of that space's memories it may access. + +## Default provisioning + +Creating a standard space establishes collaborative defaults: + +- The creator becomes a space admin. +- The creator owns their home tree and `/share`, but not other members' homes or + the root of the space. +- Joining users automatically receive ownership of their own home tree. +- A memberless default group named `team` is created. +- The default group receives `read` access to `/share` and `write` access to + `/share/projects`. + +The default group's grants are ordinary tree grants. They become effective only +when a user is both directly rostered in the space and added to the group. +Service accounts never receive an automatic home tree or default-group +membership. + +The default group is identified by a per-space marker, not its name. This makes +renaming robust and lets invitations consistently target the configured group. + +## Custom provisioning + +Space creation can opt out of any default that does not match the desired +governance model: + +| Creation option | Effect | +| --- | --- | +| `--no-home-grants` | Disables automatic home ownership for every joining user. The creator instead receives admin plus `owner` at `/`, covering the whole space. | +| `--default-group ` | Uses a different name for the default invitation group. | +| `--no-default-group-grants` | Creates the default group without access grants, so an administrator configures its grants explicitly. | +| `--no-default-group` | Creates no default invitation group. | + +Automatic home ownership is a space-wide membership rule, not a creation-time +grant only for the creator. Direct additions and invitation redemption use the +same join path, so `--no-home-grants` applies consistently to every later user. + +The creator remains a space admin in either model. Standard spaces favor +least-privilege collaboration: the creator can administer the space but initially +sees only shared data and their own home. A no-home-grants space favors explicit, +central setup: the creator has root ownership to establish the access model, and +no user receives private ownership automatically. + +## Invitations + +Invitations are the normal way to add a user to a space with group-derived +access. Each invitation explicitly records the groups the recipient will join; +the server does not infer a group when an invitation is created. On acceptance or +link redemption, the user becomes a direct space member and is added to those +groups. + +CLI and web invitation flows normally select the space's marked default group, +which is `team` in a standard space. As a result, an invited user typically gains +the default group's shared-tree access when they redeem the invitation. Directly +adding a user to a space does not add them to the default group, and an +invitation can deliberately select different groups or none. + +## Lifecycle + +Creating a user identity does not immediately create a personal space. Onboarding +ensures a default personal space only when the user has no existing space +memberships, preventing users who join through an invitation from receiving an +unneeded extra space. + +Deleting a space removes its control-plane memberships, groups, and grants and +also drops its memory data schema. This is a full-space lifecycle operation, not +a way to remove one member or subtree. diff --git a/docs/cli/me-memory.md b/docs/cli/me-memory.md index a9125790..fec7dc65 100644 --- a/docs/cli/me-memory.md +++ b/docs/cli/me-memory.md @@ -103,7 +103,7 @@ me memory search [query] [options] | `--weight-semantic ` | Semantic weight, 0-1. | | `--weight-fulltext ` | Fulltext weight, 0-1. | | `--order-by ` | For filter-only searches, sort by recency: `desc` (default) or `asc`. | -| `--select ` | Comma-separated response fields to return for each result. Omit for full records in JSON/YAML output; the default text view requests an ID, tree, 120-code-unit content preview, and score. | +| `--select ` | Comma-separated response fields to return for each result. Omit for full records in JSON/YAML output; the default text view locally presents an ID, tree, 120-code-unit content preview, and score. | At least one search criterion is required. A positional `query` runs hybrid search by sending the same text to semantic and fulltext ranking. Use `--semantic` for pure vector search, `--fulltext` for pure keyword search, or both flags to provide different text for each mode.