docs: add living system design

design/README.md

@@ -0,0 +1,64 @@
+# Design
+
+This directory describes Memory Engine's implemented design. These are living
+documents: update them in place when the design changes. Focus on the model,
+interfaces, invariants, constraints, and operational tradeoffs that future work
+must preserve.
+
+## Harnesses
+
+| Document | Scope |
+| --- | --- |
+| [Harness integrations](harness-integrations.md) | Shared policy, lifecycle, runtime contract, and security boundaries |
+| [Claude Code adapter](harnesses/claude.md) | Plugin, environment injection, MCP, and capture |
+| [OpenCode adapter](harnesses/opencode.md) | MCP entry, plugin, environment injection, and capture |
+| [Codex CLI adapter](harnesses/codex.md) | MCP entry, hooks, command rewrite, and capture |
+
+## Database
+
+| Document | Scope |
+| --- | --- |
+| [Database migrations and versioning](database-migrations-and-versioning.md) | Schema evolution, deployment reconciliation, and SQL function safety |
+
+## Embeddings
+
+| Document | Scope |
+| --- | --- |
+| [Embedding queue and worker](embedding-queue-and-worker.md) | Asynchronous vector generation and backlog processing |
+
+## Memory Model
+
+| Document | Scope |
+| --- | --- |
+| [Memory schema](memory-schema.md) | Flexible memory records, identity, temporal modeling, metadata, and retrieval indexes |
+
+## Spaces
+
+| Document | Scope |
+| --- | --- |
+| [Spaces and provisioning](spaces.md) | Space isolation, lifecycle, and custom defaults |
+
+## Authorization
+
+| Document | Scope |
+| --- | --- |
+| [Authentication](authentication.md) | Human login, OAuth, sessions, device authorization, and API-key boundaries |
+| [Principal model](principal-model.md) | Users, groups, service accounts, and agent removal |
+| [Memory authorization](memory-authorization.md) | Space admission, tree grants, and effective access |
+| [Restricted API keys](restricted-api-keys.md) | Scoped personal and service-account credentials |
+
+## Presentation
+
+| Document | Scope |
+| --- | --- |
+| [Field projection and format selection](field-projection-and-format-selection.md) | CLI and MCP memory-read presentation |
+
+## Search
+
+| Document | Scope |
+| --- | --- |
+| [Semantic search scoring](semantic-search-scoring.md) | Cosine similarity scores and thresholds |
+| [Filtered semantic search](hnsw-filtered-semantic-search.md) | HNSW recall and rank ordering under filters |
+| [Fulltext BM25 scoring](fulltext-bm25-scoring.md) | Positive lexical-match invariant and score semantics |
+| [Hybrid RRF scoring](hybrid-rrf-scoring.md) | Fusion behavior, candidate windows, and tuning |
+| [Metadata predicates](jsonpath-metadata-predicates.md) | Advanced JSONPath metadata filtering |

design/authentication.md

@@ -0,0 +1,114 @@
+---
+title: Authentication
+tags: [authentication, oauth, sessions, api-keys, device-flow]
+---
+
+# Authentication
+
+Authentication establishes the credential holder's identity. Authorization then
+decides what that identity may do: the user endpoint applies its method-level
+rules, while the memory endpoint additionally requires direct membership in the
+selected space and passes effective tree grants to the data plane. API keys are
+described in more detail in [Restricted API Keys](restricted-api-keys.md).
+
+## Credential classes
+
+| Credential | Holder | Primary use | Validation |
+| --- | --- | --- | --- |
+| Browser session cookie | User | Hosted web UI | better-auth session lookup |
+| OAuth access token | User | CLI and MCP | Hashed token lookup in the auth schema |
+| Signed session bearer | User | `me login --device` | better-auth bearer plugin session lookup |
+| API key | User or service account | Headless automation and explicit key use | Core API-key validation |
+
+Human identity and OAuth issuance use better-auth. API keys remain in the core
+control plane, so service accounts never need a social-login identity. A bearer
+is classified by its `me.<lookup-id>.<secret>` structure: a matching value is an
+API key; every other bearer is tried as an OAuth access token and then as a
+signed session bearer. This explicit dispatch prevents a cookie from being
+mistaken for an API key.
+
+## Web sessions
+
+GitHub and Google social login create better-auth sessions. A provider-verified
+email is required before a session is created. This is the human-login front
+door: OAuth authorization and device approval both depend on a valid web
+session, so they inherit the same verification requirement.
+
+Sessions have a seven-day rolling lifetime and refresh at most daily. They are
+stored in the auth schema because better-auth needs to round-trip the session
+token. HTTPS deployments use secure, same-site cookies; browser requests using
+the ambient cookie must also pass the server's allowed-origin check. A supplied
+Bearer credential is never allowed to fall back to a cookie session, so an
+invalid bearer cannot bypass that CSRF boundary.
+
+## OAuth for CLI and MCP
+
+`me login` uses OAuth 2.1 authorization code with PKCE and an RFC 8252 loopback
+redirect. The first-party `me-cli` client is public, requires PKCE, and skips
+consent. The authorization server issues opaque access and refresh tokens. Both
+are hashed before storage; a request is validated by hashing its presented
+access token, finding an unexpired row, and resolving the bound user.
+
+The CLI refreshes access tokens proactively shortly before expiry and reacts to
+an unexpected 401 with one forced refresh. Refreshes are serialized across
+processes sharing a credential store, preventing concurrent reuse of a rotated
+refresh token. An injected `ME_SESSION_TOKEN` is treated as a static bearer and
+is never refreshed.
+
+OAuth credentials are user-bound. Client-credentials tokens without a user do
+not authenticate the RPC APIs; independently managed automation uses a service
+account API key instead.
+
+## Device authorization
+
+`me login --device` supports headless environments through RFC 8628. The CLI
+requests a device code, presents the human with a verification URL and user
+code, then polls at the server-provided interval. Issuance is restricted to the
+first-party CLI, codes expire after 15 minutes, and unauthenticated code
+issuance is rate-limited.
+
+Approval creates a normal better-auth session rather than an OAuth token pair.
+The returned session token is converted to a signed bearer form before it leaves
+the server. The bearer plugin requires that signature, so a raw session-table
+token exposed from storage cannot authenticate an API request. Device sessions
+have no refresh token; they slide while used and require another device login
+after expiry.
+
+## API keys
+
+API keys are global credentials for a user or service account and are stored
+only as hashes. They select a space through `X-Me-Space`; the key itself does
+not encode a space. The memory endpoint checks the holder's direct membership
+in that space, then computes the holder's live tree access. Restricted keys add
+a server-enforced ceiling to that access.
+
+The user endpoint accepts both user and service-account keys, but handlers
+limit service accounts to safe reads. Key-authenticated callers cannot mint or
+revoke keys, preventing a compromised key from creating a replacement. Legacy
+space-scoped API keys are rejected with a migration-specific error rather than
+silently accepting an obsolete credential format.
+
+## Endpoint boundaries
+
+`/api/v1/user/rpc` authenticates a principal for account and cross-space
+operations. OAuth tokens and sessions always represent users; API keys may
+represent users or service accounts. The RPC method gate imposes the remaining
+credential-specific restrictions.
+
+`/api/v1/memory/rpc` also requires `X-Me-Space`. Authentication resolves the
+credential holder, then requires a direct roster entry in the named space.
+Membership is separate from data authority: a member with no tree grants may
+reach the endpoint but receives no data from the space SQL functions.
+
+## Operations and invariants
+
+- Auth rows are swept on a server cron: expired sessions, verifications, OAuth
+  access and refresh tokens, and device codes are removed.
+- OAuth access and refresh tokens, and core API keys, are hashed at rest.
+- Raw browser session tokens are intentionally not valid bearer credentials;
+  only the signed device-flow form is accepted as a session bearer.
+- Email verification is checked when a new human session is created. Existing
+  credentials remain valid until their normal expiry or revocation.
+- Auth is identity resolution, not permission granting. Space membership,
+  admin authority, tree grants, and restricted-key ceilings are evaluated for
+  every applicable request.

design/database-migrations-and-versioning.md

@@ -0,0 +1,94 @@
+---
+title: Database Migrations and Versioning
+tags: [database, migrations, versioning, postgres, spaces]
+---
+
+# Database Migrations and Versioning
+
+Migrations keep the database schema and database-resident behavior compatible
+with the running server. They run for three schema families in one PostgreSQL
+database:
+
+| Schema family | Role |
+| --- | --- |
+| `auth` | Authentication and OAuth state. |
+| `core` | Spaces, principals, grants, invitations, and API keys. |
+| `me_<slug>` | One memory data schema for each space. |
+
+Every schema carries a singleton `version` row and a `migration` ledger. The
+version records the latest schema-version migrator that touched the schema; the
+ledger records exactly which incremental migrations have been applied.
+
+## Migration kinds
+
+Migrations are ordered by filename and divided into two categories:
+
+| Kind | Purpose | Execution |
+| --- | --- | --- |
+| Incremental | One-time schema or data transitions, including backfills. | Applied once and recorded in the migration ledger. |
+| Idempotent | Current definitions of functions, triggers, and other replaceable database behavior. | Re-applied on every migration pass. |
+
+Incremental files are immutable history: never edit an incremental that may have
+run against an existing schema. Add a new incremental instead. Idempotent files
+are living definitions and should be updated in place as their behavior changes.
+
+This split is necessary because much of Memory Engine's behavior lives in SQL
+functions. Tracking only versioned one-time files would leave existing spaces
+with stale function bodies after a server deployment.
+
+## Provisioning and migration
+
+New space provisioning creates the schema plus empty `version` and `migration`
+tracking tables, then runs the same migration sequence used for existing spaces.
+When provisioning is composed into a caller's transaction, the new schema and
+the related control-plane changes succeed or roll back together.
+
+Standalone migration acquires a transaction-scoped advisory lock per schema and
+requires the current database user to own that schema. This serializes concurrent
+server replicas and prevents accidental migration by an unprivileged connection.
+The migration transaction also verifies the supported PostgreSQL and extension
+versions before applying schema-specific SQL.
+
+Space migrations allow a long-running backfill statement, but cap the whole
+migration transaction at 20 minutes. This lets a legitimate table-sized backfill
+complete while still failing a genuinely stuck deployment. Lock and idle
+transaction timeouts remain short to avoid prolonged contention.
+
+## Startup reconciliation
+
+Server startup migrates `core` and `auth`, then enumerates every existing space
+and runs its space migration. Incrementals already present in the ledger are
+cheap no-ops; idempotent definitions are refreshed.
+
+Every space is attempted so failures are individually reported. If any space
+fails to migrate, startup fails rather than serving a deployment whose database
+behavior may be stale. The per-space advisory lock makes this safe when multiple
+server replicas start concurrently.
+
+## Schema versions and downgrade protection
+
+Each schema family has an independent semantic schema version in code. Before a
+migration runs, the runner compares that version with the schema's stored
+version. A server older than the stored schema version refuses to run, preventing
+older SQL definitions from being reapplied to a newer database.
+
+Migration runs proceed even when the versions are equal because idempotent SQL
+must be refreshed. The incremental ledger, not the version row, is the source of
+truth for which one-time migrations have completed.
+
+## SQL templating and function signatures
+
+Migration SQL is templated so the same files can target production schemas and
+isolated test schemas. Space migrations also substitute database configuration
+such as embedding dimensions and index parameters.
+
+PostgreSQL cannot use `create or replace function` to change a function's return
+type or input parameter names, and a changed argument type can leave an obsolete
+overload behind. Function definitions whose signatures may change use a
+`{{fn name(args) returns result}}` header. The migration runner expands it into
+a pre-create stale-signature drop and a post-create signature assertion.
+
+This makes signature drift fail during a fresh test migration instead of only
+failing when an existing production schema is upgraded. Parameter defaults are
+not part of that signature contract; removing a default may require an explicit
+guarded drop before recreating the function.

design/embedding-queue-and-worker.md

@@ -0,0 +1,92 @@
+---
+title: Embedding Queue and Worker
+tags: [embeddings, queue, worker, semantic-search, spaces]
+---
+
+# Embedding Queue and Worker
+
+Embedding generation is asynchronous so memory writes do not wait for an
+embedding provider. A new or changed memory is stored immediately and remains
+available to fulltext and filter-based search. It becomes eligible for semantic
+search after the worker writes its vector.
+
+Every space owns its own embedding queue alongside its memory table. This keeps
+queue work, lifecycle, and cleanup within the same isolation boundary as the
+memories it represents.
+
+## Enqueueing
+
+The queue records a memory ID and its `content_version`. Database triggers enqueue
+work when:
+
+- A memory is inserted without an embedding.
+- A memory's content changes and its embedding is reset.
+
+An insert that already supplies an embedding does not enqueue work. Content
+versions make a queue entry refer to one exact memory state, rather than merely
+to a memory ID.
+
+## Claiming and staleness
+
+Workers claim visible queue rows in batches. A claim increments the attempt count
+and moves a row's visibility time into the future, preventing concurrent workers
+from processing it. Claims use `FOR UPDATE SKIP LOCKED`, so workers can drain the
+same queue without blocking one another.
+
+Before claiming, the queue cancels rows that are no longer useful:
+
+- A newer pending content version exists for the same memory.
+- The memory was deleted or its content version no longer matches.
+
+The worker embeds outside the claim transaction. If it crashes before write-back,
+the visibility timeout expires and another worker can claim the row. A sweep marks
+rows that exhausted their attempt budget as failed, including rows stranded by a
+crash after their final claim.
+
+## Write-back and retries
+
+Write-back is version-guarded and atomic. A completed embedding is written only
+when the memory still has the claimed content version. Otherwise the queue row is
+cancelled and no stale vector is stored. Deleting a memory also removes its queue
+rows through the foreign-key cascade.
+
+Ordinary provider or write-back failures record the last error while leaving the
+row pending. It becomes eligible for another claim after its visibility timeout.
+This makes the queue, rather than the embedding SDK, the primary retry authority.
+
+Rate limits are treated separately from ordinary failures. The worker releases
+claimed rows, refunds their attempt count, and defers their visibility by the
+same backoff interval that the worker observes. Workers in one pool also share a
+rate-limit gate, so one provider `429` pauses new claims across the pool. The
+database visibility delay prevents another worker or process from immediately
+reclaiming the same rows.
+
+## Worker operation
+
+Workers periodically discover space schemas and poll their queues in shuffled
+round-robin order. They process immediately while work exists, sleep while idle,
+and apply bounded exponential backoff for consecutive non-rate-limit failures.
+An optional drain timeout lets a worker exit after sustained idleness.
+
+Queue functions own claiming, completion, failure, release, and pruning. The
+worker calls those functions rather than issuing inline queue mutations, keeping
+the queue state machine in the database.
+
+Terminal queue rows are retained for seven days by default, then pruned
+opportunistically when a worker finds no claimable work in that space. This
+preserves short-term failure and cancellation visibility without unbounded queue
+growth.
+
+## Operational visibility
+
+The embedding status surface reports aggregate queue state for the active space:
+
+| State | Meaning |
+| --- | --- |
+| Pending | Queue rows without a terminal outcome. |
+| In flight | Pending rows whose visibility timeout is still in the future. |
+| Waiting | Pending rows that are claimable now. |
+| Failed | Terminal failures retained before pruning. |
+
+It also reports the oldest pending enqueue time. These are space-wide operational
+counts, not tree-scoped memory results.