diff --git a/README.md b/README.md index ef60b67..ce0c9f4 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,28 @@ cp .env.example .env.local # Add your API keys bun run src/index.ts run -p supermemory -b locomo ``` +LongMemEval-V2 uses the build-aware workflow because multiple questions share +the same exact haystack and require multimodal, benchmark-owned evaluation: + +```bash +bun run src/index.ts lme-v2 --help +bun run src/index.ts lme-v2 dry-run \ + --run-id lme-v2-dry-run \ + --dataset data/benchmarks/longmemeval-v2 +``` + +See [docs/LONGMEMEVAL_V2.md](docs/LONGMEMEVAL_V2.md) for the architecture, +download/preparation commands, MemoryBuild semantics, live-service preflight, +resume behavior, and validation status. + +To use the browser workflow, run `bun run src/index.ts serve --no-open`, open +**New Run**, and choose **LongMemEval-V2** from the **Benchmark** dropdown. Its +guided form defaults to an offline Plan and exposes exact-haystack limits, +reader/evaluator models, a V2 memory-provider dropdown, and advanced +bounded-ingestion controls. Supermemory, Filesystem, and local RAG support live +V2 stages; Mem0 and Zep are clearly labelled Plan-only until their exact +interruption/reconciliation contracts are implemented. + ## Configuration ```bash diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md new file mode 100644 index 0000000..afe89c9 --- /dev/null +++ b/THIRD_PARTY_NOTICES.md @@ -0,0 +1,20 @@ +# Third-party notices + +## LongMemEval-V2 + +The LongMemEval-V2 integration is based on the public +[`xiaowu0162/longmemeval-v2`](https://huggingface.co/datasets/xiaowu0162/longmemeval-v2) +dataset and the LongMemEval-V2 benchmark repository. + +- Reference revision: `f152293e235517d504809563c833d7190b8c713b` +- Supermemory adapter oracle: `feat/supermemory` commit + `2fa6616dce77e0385d7e1c44510dfde8aa3c46e3` +- Structured Accessibility Converter oracle: + `supermemory_adapter/approaches/Approach_1.py` +- Oracle SHA-256: + `22cff05fafa9f882040afa8296439da0f911f800c107424de105ab3af5e69236` +- License: Apache License 2.0 + +MemoryBench downloads the dataset's own `LICENSE` file as part of the pinned, +checksum-verified snapshot. The TypeScript converter is a new implementation +whose behavior is tested against the cited reference. diff --git a/docs/LONGMEMEVAL_V2.md b/docs/LONGMEMEVAL_V2.md new file mode 100644 index 0000000..7c8b512 --- /dev/null +++ b/docs/LONGMEMEVAL_V2.md @@ -0,0 +1,901 @@ +# LongMemEval-V2 in MemoryBench + +Status: implementation complete on the `migration` branch. Local validation, +synthetic live Supermemory preflight, a real one-trajectory canary, and one real +complete-haystack GPT-5-high question pass. The user explicitly approved the +required data egress. The measured selected-question result is documented +below; it is not presented as a complete benchmark score. + +## 1. Decision and naming + +MemoryBench is the benchmark control plane. The UI can select Supermemory, +Filesystem, or local RAG as the live memory provider. LongMemEval-V2 owns its +dataset rules, reader prompt, answer parser, evaluator, and official +aggregation. Mem0 and Zep are visible as Plan-only choices until their remote +APIs can prove exact interrupted-ingestion reconciliation and cleanup. + +Only one conversion strategy is in scope: + +> **Structured Accessibility Converter** + +This is the new name for the former `Approach_1.py` behavior. “Parser” is not a +good name for it because parsing and conversion are different jobs: + +- The **dataset parser** reads JSONL, validates records, resolves haystacks, and + creates typed trajectory objects. +- The **Structured Accessibility Converter** turns one typed trajectory into a + deterministic set of provider-ready documents. + +There are no numbered approaches in this integration. + +## 2. What a MemoryBuild is + +A **MemoryBuild** is an immutable, reusable memory corpus made from: + +- one exact, ordered haystack; +- the byte hashes of its trajectories and screenshots; +- one converter name, version, and source hash; +- one provider and its content-changing build settings. + +Questions do not own memory ingestion. They reference a `buildId`. + +For example, all small-tier web questions share the same exact web haystack, so +MemoryBench ingests that haystack once and reuses its MemoryBuild. Small tier +therefore produces two builds: one web build and one enterprise build. + +Changing `topK`, the reader, or the evaluator does not rebuild memory. Changing +the haystack, trajectory bytes, screenshot bytes, converter, or provider build +settings produces a different build fingerprint and therefore a different +MemoryBuild. + +```mermaid +flowchart LR + H["Exact ordered haystack"] --> F["Build fingerprint"] + A["Trajectory and image hashes"] --> F + C["Structured Accessibility Converter"] --> F + P["Provider build settings"] --> F + F --> B["Immutable MemoryBuild"] + Q1["Question A"] --> B + Q2["Question B"] --> B + Q3["Question C"] --> B +``` + +## 3. Frozen source and branch state + +The implementation is grounded in these exact references: + +| Item | Frozen value | +| --- | --- | +| MemoryBench base | `origin/main` at `118209a746d97d0d85e5a7234267f0b6962857e9` | +| Working branch | `migration`, created from that exact main commit | +| LongMemEval-V2 adapter oracle | `feat/supermemory` at `2fa6616dce77e0385d7e1c44510dfde8aa3c46e3` | +| Dataset repository | `xiaowu0162/longmemeval-v2` | +| Dataset revision | `f152293e235517d504809563c833d7190b8c713b` | +| Converter oracle | `supermemory_adapter/approaches/Approach_1.py` | +| Converter oracle SHA-256 | `22cff05fafa9f882040afa8296439da0f911f800c107424de105ab3af5e69236` | + +MemoryBench `main` and `origin/main` were verified equal before the branch was +created. The LongMemEval-V2 checkout has an unrelated local change in +`evaluation/harness.py`; the implementation did not edit or use that local +change as source evidence. + +The dataset and upstream benchmark are Apache-2.0 licensed. Attribution is in +[`THIRD_PARTY_NOTICES.md`](../THIRD_PARTY_NOTICES.md), and the downloader now +includes the dataset's checksum-verified `LICENSE` file. + +## 4. Before and after + +### LongMemEval-V2 `feat/supermemory` + +The reference fork had the needed benchmark behavior, but it was a specialized +Python execution path: + +- question data and haystacks were loaded by the benchmark harness; +- each trajectory was converted into structured documents; +- Supermemory V3 batch ingestion used deterministic custom IDs, metadata, + `filterByMetadata`, and `dreaming: instant`; +- SQLite checkpoints tracked ingestion and resume; +- upload, polling, and question work used bounded concurrency; +- retrieval used Supermemory V4 and added retrieved screenshots to the reader; +- the reader produced boxed answers; +- `eval_function` selected deterministic or strict LLM grading; +- scripts supported preflight, dry run, canary, resume, and inspection. + +Important problems found during the audit were container-only retrieval +isolation, question-centric repeated work, weak separation of cache identities, +path-only media identity in places, and no generic MemoryBench representation +for a shared reusable memory corpus. + +### MemoryBench `origin/main` + +The baseline framework already had providers, benchmarks, judges, a staged +orchestrator, checkpoint JSON, reports, a server, and a UI. Its main execution +model was: + +```text +question -> sessions -> ingest -> search -> generic answer -> generic judge +``` + +That model was not enough for LongMemEval-V2 because the benchmark needs shared +haystacks, multimodal evidence, durable remote-document reconciliation, and +official benchmark-owned evaluation. + +### MemoryBench `migration` + +The migration adds a separate build-aware path without rewriting the existing +benchmark path: + +```text +dataset snapshot + -> parser and validator + -> exact haystack grouping + -> Structured Accessibility Converter + -> reusable MemoryBuild + -> filtered retrieval + -> multimodal GPT-5 reader + -> official LongMemEval-V2 evaluator + -> official report plus diagnostics +``` + +All existing providers now declare capabilities. Unsupported build-aware +features fail before remote work begins. Supermemory, Filesystem, and RAG have +V2 `BuildProvider` adapters; Mem0 and Zep remain Plan-only. + +## 5. End-to-end architecture + +```mermaid +flowchart TD + D["Pinned dataset files and archives"] --> P["Safe preparation"] + P --> V["Dataset parser and validator"] + V --> M["Versioned manifest and asset hashes"] + V --> G["Exact ordered haystack groups"] + G --> C["Structured Accessibility Converter"] + C --> DP["Validated document plans"] + DP --> MB["MemoryBuild plans"] + MB --> BE["Durable build engine"] + BE <--> SQ["SQLite WAL checkpoint"] + BE --> SM["Selected provider build"] + + V --> Q["Question plans"] + Q --> QR["Query runner"] + MB --> QR + SM --> QR + QR --> SR["Provider-scoped retrieval"] + SR --> RA["Raw and normalized retrieval artifacts"] + RA --> R["Multimodal GPT-5 reader"] + M --> R + R --> EA["Reader artifact and boxed answer"] + EA --> E["Official eval_function dispatcher"] + E --> OA["Official aggregate"] + OA --> UI["CLI, report, control API, and interactive UI"] + + BF["Build fingerprint"] -.-> MB + QF["Query fingerprint"] -.-> QR + RF["Reader fingerprint"] -.-> R + EF["Evaluator fingerprint"] -.-> E +``` + +## 6. Dataset ingestion and preparation + +### Download + +`source.ts` and `download.ts` pin the revision, core-file hashes and sizes, 29 +question-image hashes, two archive hashes and sizes, the checksum manifest, and +the Apache-2.0 dataset license. + +Downloads are streamed into a private staging directory. A file is published +only after its size and SHA-256 match. The complete dataset directory is renamed +atomically. Concurrent operations use an owner lock. A dead same-root lock can +be recovered; a live or unreadable lock is rejected. + +An existing incomplete dataset root is never overwritten. This protects local +operator files and makes partial failures visible. + +### Screenshot preparation + +`prepare.ts`: + +- verifies both archive hashes before extraction; +- lists every tar entry first; +- rejects absolute paths, traversal, links, and special files; +- extracts into temporary directories; +- validates that extracted files remain under the destination; +- creates one common `screenshots/` view using relative symlinks, with a copy + fallback where symlinks are not supported; +- validates all 48,609 referenced state screenshots before publishing; +- refuses to overwrite an incomplete existing screenshot view. + +### Parser and manifest + +`dataset.ts` parses and validates the pinned snapshot: + +| Property | Audited value | +| --- | ---: | +| Questions | 451 | +| Trajectories | 1,870 | +| Trajectory states | 48,609 | +| Question images | 29 | +| Total used image assets | 48,638 | +| Unique small-tier builds | 2 | +| Unique medium-tier builds | 447 | + +The parser validates IDs, domains, haystack membership, duplicate trajectory +references, cross-domain builds, `eval_function`, question order, and image +magic bytes. Asset paths are resolved through real paths and cannot escape the +dataset root. Every used image receives a stable SHA-256, MIME type, and byte +length. + +Selection is deterministic. Exact IDs, prefix limits, per-category sampling, +domain filters, and seeded replay are supported. The complete ordered haystack +is always used for an official selected question. + +## 7. Structured Accessibility Converter + +The converter receives only a typed trajectory. Question text, answer, and +ground truth are not in its function signature, so they cannot leak into +ingested memory. + +For a trajectory with `N` states it produces exactly `N + 2` independent +documents: + +1. one trajectory overview; +2. one document for every state, in state-index order; +3. one trajectory result/outcome document. + +Each state document contains structured visible evidence such as page titles, +landmarks, alerts, headings, controls, checkbox/radio state, options, tables, +and retained unparsed evidence. HTML entities, Unicode private-use characters, +spacing, repeated labels, table rows, and empty values are normalized +deterministically. The TypeScript cleaner is byte-equivalent to the Python +oracle on every state in the pinned dataset. It is not presented as a +general-purpose HTML5 decoder or full Unicode case-folding library: an +unobserved named entity or unusual case-fold character in a future dataset +revision must be treated as converter drift, covered by the source hash and +revalidated against the oracle before that revision is accepted. + +The plan is one independent V3 batch per trajectory. It has no previous- +trajectory dependency and no external entity context. Every state document +carries its screenshot reference and state/step provenance. + +The generic plan validator rejects: + +- empty plans or content; +- duplicate logical IDs; +- accidental duplicate content; +- missing attachments or screenshots; +- invalid or reserved metadata; +- missing dependencies, self-dependencies, and cycles; +- nondeterministic converter output; +- a batch document that would require splitting. + +Physical remote IDs are deterministic functions of build fingerprint, +trajectory ID, document ordinal, and part index. + +## 8. Durable MemoryBuild ingestion + +Each build has: + +```text +data/memory-builds-v2/// + plan.json + checkpoint.sqlite + summary.json +``` + +The SQLite database uses WAL, `synchronous=FULL`, foreign keys, and a busy +timeout. It records builds, ordered trajectories, every physical document, +attempt counts, leases, remote IDs, states, errors, and events. + +The document lifecycle is: + +```text +planned -> submitting -> accepted -> indexing -> ready + | | + +---- reconcile ----------+ + | + retryable / failed +``` + +Before a retry, ambiguous state is reconciled by deterministic `customId`. +This covers a process dying before submission, a response being lost after +remote success, a 409 conflict, or a local commit not happening after a +successful remote request. + +A build becomes ready only when every required trajectory and document is +ready. Partial indexing can never silently become a completed build. Reused +ready builds receive a remote health check before query. + +Readiness polling is finite. The default LongMemEval-V2 CLI gives each +trajectory a 30-minute indexing deadline and four non-timeout attempts. If the +deadline expires, the engine deletes the exact unresolved deterministic IDs, +marks those documents and their trajectory as skipped, and makes the build +`degraded`. The remaining benchmark work may continue, but its report is +automatically `officiallyComparable: false`. Exact cleanup must succeed; an +unverified remote document is never silently ignored. `--strict-ingestion` +instead stops the run at the same bounded deadline. + +`--force-build` first deletes only the exact enumerated documents whose remote +metadata proves they belong to the build, then resets the local checkpoint. +There is no container-wide destructive cleanup. + +## 9. Provider behavior + +| Provider | Live V2 stages | Durable resume and exact cleanup | Retrieval profile | +| --- | --- | --- | --- | +| Supermemory | Build through report | Remote custom-ID reconciliation, exact deletion, preflight gate | Hybrid, reranking on | +| Filesystem | Build through report | Atomic memory files plus exact metadata sidecars | Memory text matching, reranking off | +| Local RAG | Build through report | Per-container SQLite/WAL with atomic session replacement | Hybrid BM25/vector, reranking off | +| Mem0 | Plan only | Not yet proven for async event reconciliation and individual cleanup | Memories | +| Zep | Plan only | Not yet proven for episode provenance and individual cleanup | Memories | + +Filesystem and RAG use the existing MemoryBench extraction behavior, including +`gpt-4o-mini`; this extraction configuration is part of the build fingerprint. +RAG persists embeddings and chunks transactionally, removes stale chunks on a +session retry, and uses deterministic score tie-breaking. Both local adapters +require exactly one contributing custom ID before attaching a screenshot to a +retrieval result. Their OpenAI extraction/embedding calls receive an abort +signal and are bounded by the configured per-trajectory deadline. + +Mem0 and Zep are deliberately not advertised as live adapters. Their existing +legacy paths use asynchronous generated identities and cannot yet guarantee +that a timed-out remote job will never appear later or that one failed document +can be removed without disturbing the rest of the build. This prevents an +infinite wait without pretending that an unsafe degraded build is comparable. + +### Supermemory reference adapter + +The advanced provider supports: + +- custom base URLs; +- V3 single and batch document operations; +- deterministic `customId`; +- metadata and `filterByMetadata`; +- `dreaming: instant`; +- status reconciliation and readiness polling; +- V4 hybrid or memories search; +- reranking and query rewriting capability declarations; +- exact cleanup and remote health verification. + +Every remote document carries: + +```text +benchmark +buildId +buildFingerprint +runFingerprint +tier +domain +haystackHash +trajectoryId +trajectoryOrder +documentType +documentOrdinal +partIndex / partCount +contentHash +causalKey +stateIndex / step +screenshot path / asset ID / SHA-256 / MIME / byte length +``` + +Requests share one adaptive request budget. A 429 or retryable server error +reduces concurrency. `Retry-After` is honored. Sustained success recovers the +budget gradually. Secrets are redacted from errors and artifacts. + +An expensive build cannot start from CLI or the real advanced provider until a +passing preflight gate exists. The gate is scoped by normalized service URL, +must be fresh (24 hours by default), and must have tested at least the run's +configured top-K. Its report fingerprint, generation time, service URL, and +tested top-K are persisted in the run checkpoint. Missing, stale, mismatched, +or insufficient gates fail before the first dataset upload. + +## 10. Retrieval, top-K, and screenshots + +There is one authoritative retrieval `topK`. The reader has a separate +`evidenceTopK`, which must be less than or equal to retrieval `topK`. + +Every V4 search uses both: + +- the exact `containerTag`; and +- a mandatory logical metadata filter: + +```json +{ + "AND": [ + { + "key": "runFingerprint", + "value": "", + "filterType": "metadata" + } + ] +} +``` + +Extra filters are nested under that mandatory build boundary. A caller cannot +override the fingerprint. Invalid keys, types, numeric operands, and filters +nested deeper than the provider contract are rejected before search. + +Returned result and document metadata are checked again. A result with missing +or mismatched provenance fails the question instead of entering the reader. +Provider results above configured `topK` are rejected. + +Normalization preserves rank, score, memory text, summaries, chunks, document +IDs, trajectory/state provenance, and raw results. Duplicate chunks are removed +by content hash while first occurrence order is retained. + +Screenshots are not uploaded as Supermemory media. Their content identity is +stored in document metadata. When search returns a screenshot reference, +MemoryBench matches all of its path, asset ID, hash, MIME type, and byte length +against the selected MemoryBuild. The reader then loads the verified local +bytes and places the image immediately after that evidence unit. + +Raw request/response and normalized artifacts are immutable. A cache record is +accepted only if its identity, linked artifact hashes, normalized contents, +provenance, and result count all validate. Cached queries retain the original +remote duration but receive a new wall duration and `cacheHit: true`; reports do +not count cached remote time as a new live measurement. + +## 11. Reader and evaluation + +The reader uses benchmark-owned web or enterprise system prompts. Default +production settings are GPT-5 with high reasoning, a 200,000-token context +budget, and 20,000 maximum completion tokens. + +Context order is: + +```text +memory heading +retrieval rank 1 text +retrieval rank 1 verified screenshots +retrieval rank 2 text +retrieval rank 2 verified screenshots +... +question text +question image, when present +``` + +The conservative GPT-5 budget uses `o200k_base` for text and an explicit image +token allowance. Evidence units are removed from the end until the request +fits. The question and its image are never silently removed. Images also have +count and byte-size limits. + +The reader artifact stores the full typed request parts, exact sent asset IDs, +omission count, model settings, response, usage, raw attempts, parsed answer, +duration, and cache status. Image byte hashes participate in reader identity. + +Boxed-answer parsing uses the final `\boxed{...}` expression and supports nested +braces. A response without a box falls back to trimmed response text. + +`eval_function` is parsed and dispatched by LongMemEval-V2 code, not the generic +MemoryBench judge. Implemented official paths are: + +- normalized phrase-set match; +- ordered phrase-set match; +- single-choice match; +- multi-choice match; +- strict abstention judge; +- strict gotcha judge. + +Every evaluator specification present in all 451 questions is validated. LLM +judge artifacts preserve request, raw response, parsed binary verdict, and +rationale. Evaluator failures are saved separately. Failed or blocked questions +remain in the official full-set denominator. + +Official accuracy and category/abstention aggregates are a separate namespace +from MemoryBench diagnostics such as cache hits, search latency, and images +sent. + +## 12. Fingerprints and reuse + +| Identity | Includes | A change reruns | +| --- | --- | --- | +| Build | dataset and asset hashes, ordered trajectories, converter, document plans, provider build settings | ingestion | +| Query | build, question text/image hash, top-K, threshold, search mode, reranker, rewrite, metadata filters, normalizer | search | +| Reader | normalized retrieval, model/settings, prompt version, image hashes, budget algorithm | answer generation | +| Evaluator | answer, ground truth, exact `eval_function`, model/settings, prompt and implementation versions | grading | + +This dependency split allows experiment changes without unnecessary remote +ingestion. + +## 13. Concurrency and resume + +There are four bounded concurrency layers: + +- build concurrency: different unique MemoryBuilds; +- trajectory concurrency: trajectories inside a build; +- maximum provider requests in flight; +- question concurrency: query/read/evaluate work. + +All Supermemory operations still pass through the shared adaptive request +budget, so multiplying worker counts cannot bypass the account-level cap. + +Trajectory workers use renewable leases. Another worker cannot claim active +work; an expired lease can be recovered after a crash. Run checkpoint writes +are serialized and atomic. Resume reloads the same configuration, dataset +selection, question-to-build links, document plan, query artifact, reader +artifact, and evaluation artifact. + +Changing semantic configuration under the same run ID is rejected. The +machine-local dataset path is excluded from semantic identity so a moved but +byte-identical dataset can resume. + +## 14. CLI workflow + +```bash +# Inspect all commands. +bun run src/index.ts lme-v2 --help + +# Download and verify the pinned snapshot. +bun run src/index.ts lme-v2 download \ + --dataset data/benchmarks/longmemeval-v2 + +# Safely prepare the common screenshot view. +bun run src/index.ts lme-v2 prepare \ + --dataset data/benchmarks/longmemeval-v2 + +# Validate data, selection, assets, conversion, and cost shape without network. +bun run src/index.ts lme-v2 dry-run \ + --run-id lme-v2-dry-run \ + --dataset data/benchmarks/longmemeval-v2 + +# Probe the current Supermemory V3/V4 contract with synthetic records. +bun run src/index.ts lme-v2 preflight \ + --run-id lme-v2-preflight \ + --top-k 20 + +# Non-official, exactly one trajectory; build and query only. +bun run src/index.ts lme-v2 canary \ + --run-id lme-v2-canary \ + --dataset data/benchmarks/longmemeval-v2 \ + --question-id + +# Official selected-question run. Its complete exact haystack is always used. +bun run src/index.ts lme-v2 run \ + --run-id lme-v2-selected \ + --dataset data/benchmarks/longmemeval-v2 \ + --question-id \ + --reader-model gpt-5 \ + --reasoning-effort high \ + --indexing-timeout-ms 300000 \ + --max-trajectory-attempts 2 + +# Resume or inspect without changing semantic configuration. +bun run src/index.ts lme-v2 resume --run-id lme-v2-selected +bun run src/index.ts lme-v2 inspect --run-id lme-v2-selected +``` + +Other actions are `build`, `query`, and `evaluate`. Medium tier is rejected +unless `--allow-medium` is explicit. A one-trajectory canary is prevented from +reading, evaluating, or reporting an official score. + +`--preflight-max-age-hours` controls the maximum gate age for later build +commands. A successful preflight writes a service-scoped latest-passing gate; +a failed preflight never replaces it. + +Ingestion is bounded. `--indexing-timeout-ms` is a hard per-trajectory +readiness deadline, and `--max-trajectory-attempts` bounds non-timeout retries. +By default, documents still unresolved at the indexing deadline are deleted by +their exact deterministic IDs, recorded as skipped, and the run continues with +a `degraded` build. A degraded run is visibly marked `officiallyComparable: +false`; its score is diagnostic and cannot satisfy a parity gate. Pass +`--strict-ingestion` when any skipped document should fail the whole run +instead. + +## 15. Artifacts and inspection + +Run artifacts are under `data/runs-v2//`: + +```text +checkpoint.json +dataset-manifest.json +selection.json +preflight.json # when run with the same preflight run ID +builds/.plan.json +report.json +``` + +The reusable live-service gate is stored separately under: + +```text +data/preflights-v2/supermemory//latest-passed.json +``` + +Reusable content-addressed artifacts are under `data/artifacts-v2/`: + +```text +queries/// +readers//.json +evaluations//.json +assets/. +``` + +All artifact writes are immutable, atomic, path-contained, and secret-redacted. +Symlink escapes and hash mismatches are rejected. + +The server exposes a dedicated LongMemEval-V2 control API and safe inspection +routes. LongMemEval-V2 appears in the normal Benchmark dropdown. Its guided +form has a provider dropdown, complete-haystack selection, separate reader and +evaluator model choices, and an Advanced section for all four concurrency +limits, retry bounds, timeout behavior, force rebuild, and fresh retrieval. The +UI can create plan, build, retrieval, evaluation, or report runs; +stop an active run; resume the exact durable checkpoint target; and explicitly +continue a completed intermediate stage. It defaults to offline Plan and needs +an explicit confirmation before any unselected full-scope live run, including +every later resume or continuation beyond Plan. The UI can also run the bounded +synthetic Supermemory preflight with the server-side key; one preflight may run +at a time and its status is polled for at most eight minutes in the page. + +LongMemEval-V2 is selected from the normal Benchmark dropdown. Its guided form +keeps dataset scope, exact-haystack count, reader/evaluator models, and stopping +point visible; paths, reasoning effort, retrieval depth, concurrency, retry +bounds, timeouts, and cache controls stay under Setup or Advanced. A +`haystackLimit` keeps the first N exact builds in pinned question order and all +questions linked to those builds. It never truncates a haystack's ordered +trajectory list. The pinned counts shown by the UI are small: 2 total (1 web, +1 enterprise), 100 trajectories each; medium: 447 total (236 web, 211 +enterprise). + +Run and question pages show build identity/reuse, stage and lifecycle history, +paginated question summaries, raw/normalized/reader/evaluation artifacts, +retrieval provenance, the exact answer, official evaluator verdict, and the +separate official-versus-diagnostic namespaces. Screenshot assets are rendered +only after the server validates checkpoint membership, path containment, +SHA-256, byte length, MIME type, and file signature. A restarted UI-managed +process whose checkpoint still says `running` is presented as failed and +resumable instead of polling forever. An independently managed CLI checkpoint +is not relabeled from the UI's process-local state; operators should not run UI +and CLI control for the same run ID at the same time. + +## 16. Edge-case coverage + +| Area | Cases handled | +| --- | --- | +| Dataset | wrong revision; corrupt checksum; missing file; partial download; stale lock; concurrent operation; duplicate ID; duplicate haystack item; unknown trajectory; cross-domain trajectory | +| Archives | traversal; absolute path; symlink/hardlink/special entry; duplicate entry; partial extraction; existing incomplete destination | +| Images | missing file; escape through symlink; corrupt magic bytes; changed bytes at same path; duplicate asset; oversized reader image | +| Conversion | unstable output; empty content; duplicate ID/content; invalid metadata; missing attachment; dependency error/cycle; batch split | +| Remote ingestion | crash before request; response lost after success; unexpired crashed-worker lease; 409 conflict; missing/partial batch response; absent/pending/ready/failed state; zero-memory document; bounded stuck indexing with exact deletion and explicit degraded status | +| Limits | 429 and `Retry-After`; retryable server failures; indexing deadline; lease renewal during slow polling; shared request cap | +| Cleanup | wrong build metadata; absent remote document; empty target list; exact enumerated deletion only | +| Retrieval | top-K violation; fewer results; duplicate chunks; missing/wrong build fingerprint; invalid logical filter; mismatched screenshot metadata | +| Caches | changed top-K; changed question image; changed reader; changed evaluator; tampered record; missing normalized file; moved local asset root | +| Reader/evaluator | empty model response; nested/missing box; exact `UNKNOWN`; malformed evaluator spec/output; judge failure; failed question denominator | +| UI/API | run/question traversal; artifact traversal; symlink escape; corrupt artifact hash; secret fields and absolute paths in responses; duplicate run; immediate redirect race; stop/resume target drift; user-stopped ingestion must remain retryable; UI-owned stale `running` checkpoint; CLI ownership separation; accidental full run at start/resume/continue; deterministic complete-haystack limits; provider-specific capability gating; model-specific OpenAI reasoning controls; failed-evaluator artifact inspection; 451-question pagination | + +## 17. Main difficulties and operational cost + +| Difficulty | Impact | Current handling | +| --- | --- | --- | +| Shared haystacks do not fit legacy per-question ingestion | Critical correctness and cost issue | MemoryBuild separates reusable ingestion from questions | +| Remote success can be ambiguous after a crash | Duplicate or missing memory | Deterministic IDs plus SQLite reconciliation | +| Container reuse can mix memory | Invalid benchmark result | Container plus mandatory build filter plus returned provenance validation | +| Screenshots cross dataset, retrieval, cache, model, and UI boundaries | Easy stale/wrong-image bugs | Byte hashes, typed assets, content-addressed copies, ordered reader parts | +| Official evaluation differs from a generic judge | Score drift | Benchmark-owned dispatcher, prompts, raw verdicts, and denominator | +| Medium tier has 447 builds | High time and provider cost | Cost-visible dry run, explicit opt-in, bounded concurrency, independent reuse | +| Supermemory API contracts can change | Live failures after local tests | Synthetic preflight; the current logical V4 filter contract was discovered and fixed live | +| Full parity is expensive | Cannot infer parity from unit tests | Staged canary, selected exact-haystack question, then complete small tier | +| A future dataset may use text forms absent from the pinned corpus | Limited HTML-entity and Unicode case-fold helpers could diverge from Python | Revision is pinned; the complete current corpus has zero converter mismatches; any revision change requires rerunning the offline oracle | +| A few provider documents may remain queued/indexing indefinitely | A small tail can stall every later phase | Configurable hard deadline; exact-ID deletion; explicit degraded build; non-official score labeling | +| External data egress | Dataset text/screenshots leave the machine | Explicit operator approval was recorded before the real canary and selected-question run | +| Large browser payloads | A 451-question checkpoint contains large retrieval and reader artifacts | Run detail uses a compact payload and loads 25 artifact-free question summaries per page | + +## 18. Implementation map + +| Responsibility | Files | +| --- | --- | +| Dataset source/download/preparation | `src/benchmarks/longmemeval-v2/source.ts`, `download.ts`, `prepare.ts` | +| Dataset parser and grouping | `src/benchmarks/longmemeval-v2/dataset.ts`, `types.ts` | +| Converter and planner | `src/benchmarks/longmemeval-v2/converter.ts`, `planner.ts` | +| Reader and official evaluation | `src/benchmarks/longmemeval-v2/reader.ts`, `evaluation/` | +| Generic build-aware contracts | `src/types/migration.ts`, `src/types/build-aware.ts` | +| Fingerprints, plans, artifacts, build/query engines | `src/core/` | +| Advanced Supermemory provider | `src/providers/supermemory/advanced/` | +| Filesystem/RAG build-aware adapters | `src/providers/build-aware/`, `src/providers/filesystem/`, `src/providers/rag/` | +| Durable run orchestration | `src/orchestrator/longmemeval-v2.ts`, `build-aware-run-store.ts` | +| CLI | `src/cli/commands/longmemeval-v2.ts` | +| UI control and inspection API | `src/server/routes/longmemeval-v2-control.ts`, `src/server/routes/build-aware-inspection.ts` | +| UI launcher and inspection | `ui/components/longmemeval-v2-launcher.tsx`, `ui/components/build-aware-*` | + +## 19. Validation evidence + +Completed on 2026-07-27 and updated with UI verification on 2026-07-28: + +- Root `bunx tsc --noEmit`: passed. +- UI `bunx tsc --noEmit`: passed. +- `bun test`: 139 passed, 0 failed, 620 assertions. +- UI `bun run build`: passed with all 9 application routes generated. +- Pinned real snapshot validation: 451 questions, 1,870 trajectories, 48,609 + states, 29 question images, two small builds, and 447 medium builds. +- Full-small offline plan: 451 questions split into 240 web and 211 enterprise + questions. They reference two 100-trajectory builds: 1,937 web documents and + 3,558 enterprise documents. This plan made no external calls. +- Real-data dry run for question `01307e07`: full 100-trajectory haystack, + 3,558 planned documents, and 2,244 selected hashed images; no external calls. +- Offline converter-oracle comparison against + `feat/supermemory@2fa6616dce77e0385d7e1c44510dfde8aa3c46e3`: all 1,870 + trajectories, 48,609 states, and 52,349 logical documents produced identical + normalized document-plan hashes between Python `Approach_1.py` and the + TypeScript Structured Accessibility Converter (zero mismatches). Compared + fields included document order, logical IDs, content bytes, converter-owned + metadata, state/step, screenshot selection, dependencies, parallel-upload + flags, invariants, and notes. This proves converter-plan parity for the pinned + valid corpus; it does not prove equivalence for arbitrary synthetic input or + prove remote ingestion, retrieval, reader, or score parity. +- All 451 real `eval_function` strings parsed: 200 normalized phrase-set, 26 + ordered phrase-set, 128 abstention-judge, 68 single-choice, 28 gotcha-judge, + and 1 multi-choice question, across 8 unique specifications. +- Fake-provider full pipeline: build, query, multimodal reader, official + evaluation, report, failure denominator, and second-run zero extra remote + work. +- Synthetic live Supermemory preflight: all checks passed. This covered V3 + batch submission, `customId` idempotency, indexing readiness, memory + visibility, single and array metadata-filter acceptance, V4 search + visibility, requested top-K 20 acceptance, mandatory fingerprint filtering, + zero-memory documents, exact cleanup of all four probe documents, and + publication of the enforced service-scoped gate. +- Real one-trajectory canary for question `01307e07`: trajectory `f224a4eb`, + 12/12 documents ready, V4 top-K 20 returned in 1.769 seconds remote time, + all returned provenance valid, and screenshot references resolved. The + canary stopped before reader/evaluation as required. +- Real complete-haystack run for question `01307e07`: exact 100 ordered + enterprise trajectories and 3,558/3,558 documents ready in build + `mb-ff12c35021e74bfb1c258fb8`. Retrieval returned exactly 20 results in + 1.831 seconds remote time; all provenance passed; 15 unique verified + screenshots were sent to `gpt-5` with high reasoning; no evidence item was + omitted. GPT-5 returned `UNKNOWN`, so the deterministic official evaluator + scored this question `0`. This is one measured question, not an aggregate + small-tier score. +- Python reference comparison for the same question and settings + (`Approach_1`, top-K 20, `gpt-5`, high): it also returned `UNKNOWN` and scored + `0`. Its cached retrieval produced 20 text items and 18 deduplicated images; + MemoryBench's fresh retrieval produced 20 text items and 15 deduplicated + images. Prompt token counts were 22,301 and 19,379 respectively, showing that + live retrieval artifacts are not byte-identical even though the answer and + verdict matched. +- Identical MemoryBench replay: MemoryBuild `reused=true`; query, reader, and + evaluator were cache hits; no new search latency or model generation was + recorded; build attempts remained unchanged. +- Live interruption also exposed an unexpired-lease resume defect. The engine + now waits for another worker to finish or its lease to expire instead of + falsely failing a partial build; a regression test covers this behavior. +- A provider simulation that never reaches ready now proves the bounded path: + one finite attempt, exact-ID cleanup, explicit skipped counts, a reusable + `degraded` checkpoint, and no repeated upload or deletion on resume. +- In-app browser validation used the actual localhost UI and server. It proved: + the LongMemEval-V2 launcher; safe Plan default and prerequisites; + immediate start redirect; active Stop; durable `start -> stop-request -> + stopped -> resume(plan) -> completed` history; and a disabled full-tier + continuation until the fresh confirmation checkbox is selected. No full-tier + continuation was started. +- The later selector/layout refactor moved LongMemEval-V2 into the Benchmark + dropdown, added the guided haystack/model form, and moved technical settings + under Setup/Advanced. Per the repository instruction, this follow-up was + validated by source tests, both typechecks, and the production build without + another browser session; the browser evidence above remains for the same + launcher/control/inspection APIs before the layout-only refactor. +- Historical run `lme-v2-live-exact-01307e07-bounded-20260727` displayed the + 100-trajectory/3,558-document build, top-K 20 and 20 results, valid provenance, + parsed answer `UNKNOWN`, official score `0`, evaluator rationale, and all 15 + reader screenshots. All 15 images completed loading at 1280x720 through the + verified asset route. +- Historical replay `lme-v2-live-exact-01307e07-reuse-20260727` displayed build, + query, and reader reuse without presenting diagnostics as official scores. +- A new UI one-trajectory canary + `lme-v2-ui-canary-01307e07-20260728` completed through retrieval only. It + reused the cached build/query, displayed 20 provenance-valid results and + screenshot links, and correctly showed no reader result or official score. +- The 451-question offline plan displayed 25 compact rows per page and advanced + from page 1 to page 2 of 19. Raw, normalized, reader, and evaluation artifact + viewers displayed no API-key prefixes or absolute user paths. + +The live preflight is not a benchmark result. It used synthetic probe text and +produced no accuracy score. + +## 20. Phase and acceptance status + +### Implemented phases + +- [x] Phase 0: references, dataset revision, checksums, converter oracle, and + fixtures frozen. +- [x] Phase 1: manifests, MemoryBuilds, typed plans/results, capabilities, media, + and four fingerprint layers. +- [x] Phase 2: pinned LongMemEval-V2 dataset plugin, exact grouping, safe + download/preparation, selection, and image hashes. +- [x] Phase 3: Structured Accessibility Converter, deterministic golden tests, + plan validation, and lossless physical-document planning. +- [x] Phase 4: SQLite WAL build engine, leases, reconciliation, deadlines, + readiness barrier, resume, and exact force rebuild. +- [x] Phase 5: advanced Supermemory V3/V4 provider, adaptive budget, logical + filters, provenance validation, enforced fresh preflight gate, health checks, + and cleanup. +- [x] Phase 6: authoritative top-K, immutable raw/normalized artifacts, chunk + deduplication, verified screenshots, and cache-safe timing. +- [x] Phase 7: multimodal GPT-5 reader, boxed parsing, all dataset evaluator + specifications, strict judges, and official aggregation. +- [x] Phase 8: CLI, reusable artifacts, reports, UI start/stop/resume/continue, + safe screenshot and artifact inspection, stale-run recovery, and paginated + build-aware UI. +- [ ] Phase 9 release proof: real-data canary and selected exact-haystack live + question are complete; full small-tier parity comparison and fork retirement + remain. + +### Acceptance checklist + +#### Dataset + +- [x] Exact revision, license, file hashes, archive hashes, and sizes are pinned. +- [x] Counts and deterministic order match the audited snapshot. +- [x] Small maps to two exact builds. +- [x] Medium groups to 447 exact duplicate-haystack builds. +- [x] Every selected image has a stable byte hash and validated media type. + +#### Ingestion + +- [x] The only in-scope converter has deterministic golden tests. +- [x] Its normalized plans match the Python Approach 1 oracle for the complete + local 1,870-trajectory corpus. +- [x] Question and gold data cannot enter converter input. +- [x] Every remote document has a deterministic external ID. +- [x] Resume reconciles ambiguous remote state. +- [x] A ready build blocks query until all required documents are healthy. +- [x] Timed-out documents are exactly deleted and recorded in a degraded, + non-official build; strict mode fails at the same finite deadline. +- [x] Partial builds cannot silently become ready or officially comparable. +- [x] Force rebuild and cleanup are exact-build scoped. + +#### Retrieval + +- [x] One configured top-K is sent and violations are rejected. +- [x] Search uses container and mandatory build/run fingerprint. +- [x] Returned result/document provenance is validated. +- [x] Raw and normalized artifacts are immutable and integrity-checked. +- [x] Screenshot order follows evidence order. +- [x] Cached timings are separate from new live remote measurements. + +#### Reader and evaluation + +- [x] Question images and retrieved screenshots reach the typed reader. +- [x] Image bytes participate in cache identity. +- [x] Context budgeting is explicit and versioned for GPT-5. +- [x] Boxed-answer parsing matches nested and missing-box fixtures. +- [x] Every `eval_function` specification in the snapshot parses and dispatches. +- [x] Raw LLM judge output and rationale are stored. +- [x] Official aggregation uses all target questions, including failures. + +#### Framework quality + +- [x] Build, query, reader, and evaluator identities are separate. +- [x] Every provider declares capabilities. +- [x] Providers without durable split-phase behavior declare that limitation. +- [x] Crash, resume, ambiguous response, lease, and cache tests pass. +- [x] API/UI models explain build reuse and artifact provenance. +- [x] Official metrics and MemoryBench diagnostics are separate. +- [x] UI start, bounded stop, checkpoint resume, intermediate-stage continue, + fresh full-scope confirmation, UI-owned stale-process recovery, safe success + and failure artifacts, and 451-question pagination are validated. +- [x] Legacy framework code still typechecks and the complete repository test + suite passes. +- [ ] Existing external-provider/benchmark combinations have not all been + rerun live; that is a regression release gate, not a LongMemEval-V2 score. + +#### Live parity and retirement + +- [x] Synthetic live Supermemory service contract passes and cleans probes. +- [x] Real one-trajectory LongMemEval-V2 canary. +- [x] Real selected question with its complete exact haystack and GPT-5 high. +- [ ] Complete small-tier live pipeline and artifact-by-artifact comparison + with the Python oracle. Converter-plan parity alone is already proven above. +- [ ] Retire the official fork only after documented parity. + +## 21. Honest completion boundary + +The code path is implemented and has now passed both staged real-data gates. +The user approved the trajectory/screenshot upload to Supermemory and selected +evidence upload to OpenAI before those runs. + +The remaining release gate is expensive rather than architectural: run all 451 +small-tier questions, compare the two shared MemoryBuilds and per-question +artifacts with the Python reference, then retire the fork only if that audit +passes. The current approval established data-egress permission, but the user +requested a small live test. A complete 451-question GPT-5-high run can incur +material provider/model cost and should receive separate explicit cost +approval before execution. diff --git a/src/benchmarks/longmemeval-v2/converter.test.ts b/src/benchmarks/longmemeval-v2/converter.test.ts new file mode 100644 index 0000000..c24c1fe --- /dev/null +++ b/src/benchmarks/longmemeval-v2/converter.test.ts @@ -0,0 +1,391 @@ +import { describe, expect, test } from "bun:test" +import { fileURLToPath } from "node:url" +import { createPhysicalDocuments, validateDocumentPlan } from "../../core/document-plan" +import type { AssetRef } from "../../types/migration" +import { + cleanAccessibilityText, + LEGACY_APPROACH_1_SOURCE_SHA256, + parseStructuredAccessibilityTree, + STRUCTURED_ACCESSIBILITY_CONVERTER_NAME, + STRUCTURED_ACCESSIBILITY_CONVERTER_SOURCE_HASH, + STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + STRUCTURED_ACCESSIBILITY_INVARIANTS, + structuredAccessibilityConverter, +} from "./converter" +import type { PreparedTrajectory } from "./types" + +function screenshot(assetId: string): AssetRef { + return { + assetId, + kind: "trajectory-screenshot", + absolutePath: fileURLToPath(import.meta.url), + relativePath: `screenshots/${assetId}.png`, + mimeType: "image/png", + sha256: assetId.padEnd(64, "0").slice(0, 64), + byteLength: 100, + } +} + +function trajectory(overrides: Partial = {}): PreparedTrajectory { + return { + id: "trajectory-1", + domain: "enterprise", + goal: "Find the overdue invoice.", + startUrl: "https://example.test/start", + outcome: "Invoice 42 was found.", + states: [ + { + stateIndex: 0, + step: 3, + url: "https://example.test/invoices", + thoughts: "I should inspect the visible invoice list.", + action: "click('Invoices')", + accessibilityTree: [ + "[1] RootWebArea 'Admin & Billing'", + " [2] dialog 'Invoice editor'", + " [3] status 'Saved', live=polite", + " [4] heading 'Invoices', level=2", + " [5] option 'Open', selected=true", + " [6] option 'Open', selected=false", + " [7] checkbox 'Paid', checked=false", + " [8] radio 'Card', checked=true", + " [9] button 'Submit', disabled=false", + " [10] StaticText 'Invoice #42'", + ].join("\n"), + screenshot: screenshot("state-0"), + }, + ], + contentHash: "trajectory-content-hash", + ...overrides, + } +} + +describe("Structured Accessibility Converter", () => { + test("emits the exact independent overview, state, and result documents", () => { + const input = trajectory() + const original = structuredClone(input) + const plan = structuredAccessibilityConverter.convert(input, undefined) + + expect(input).toEqual(original) + expect(structuredAccessibilityConverter.name).toBe(STRUCTURED_ACCESSIBILITY_CONVERTER_NAME) + expect(structuredAccessibilityConverter.version).toBe(1) + expect(LEGACY_APPROACH_1_SOURCE_SHA256).toBe( + "22cff05fafa9f882040afa8296439da0f911f800c107424de105ab3af5e69236" + ) + expect(STRUCTURED_ACCESSIBILITY_CONVERTER_SOURCE_HASH).toBe( + "3e3d367fa0c691059f586f3c3ee65725dc678afaef98c8b854842bb9d7c5e716" + ) + expect(plan.trajectoryId).toBe("trajectory-1") + expect(plan.batchUpload).toBe(true) + expect(plan.declaredInvariants).toEqual([...STRUCTURED_ACCESSIBILITY_INVARIANTS]) + expect(plan.notes).toBe( + "Independent V3 batch per trajectory with structured accessibility documents and no state-level or cross-trajectory ingestion context." + ) + expect(plan.documents).toHaveLength(3) + expect( + plan.documents.every( + (document) => document.dependsOn.length === 0 && document.allowParallelUpload + ) + ).toBe(true) + + expect(plan.documents[0]).toEqual({ + logicalDocumentId: "overview", + content: [ + "# STATE_-1: TRAJECTORY OVERVIEW", + "Trajectory ID: trajectory-1", + "Domain: enterprise", + "Start URL: https://example.test/start", + "Document role: requested goal only; this is not proof that the task succeeded.", + "", + "## Requested goal", + "Find the overdue invoice.", + ].join("\n"), + metadata: { + evidenceFormat: STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + sequenceIndex: -1, + contentRole: "trajectory_goal", + }, + documentType: "overview", + sourceStateIndices: [], + localAttachmentPaths: [], + dependsOn: [], + allowParallelUpload: true, + allowDuplicateContent: false, + }) + + expect(plan.documents[1]).toEqual({ + logicalDocumentId: "state-0000", + content: [ + "# STATE_0: STRUCTURED UI OBSERVATION", + "Trajectory ID: trajectory-1", + "State index: 0", + "Step: 3", + "URL: https://example.test/invoices", + "Document role: UI observed at this state, followed by an unverified interpretation and attempted action.", + "", + "## Agent interpretation or next-step plan (unverified)", + "I should inspect the visible invoice list.", + "", + "## Action issued after this observation (attempted, not proof of success)", + "click('Invoices')", + "", + "## Observed accessibility evidence", + "Page titles: Admin & Billing", + "", + "## Page landmarks and dialogs", + "- dialog: Invoice editor", + "", + "## Alerts and status messages", + "- status: Saved [live=polite]", + "", + "## Headings", + "- heading: Invoices [level=2]", + "", + "## COMPLETE COLLECTION: options", + "Observed option count: 2", + "Completeness scope: all option roles in this captured snapshot.", + "- option: Open [selected=true]", + "- option: Open [selected=false]", + "", + "## COMPLETE COLLECTION: checkboxes", + "Observed checkbox count: 1", + "Completeness scope: all checkbox roles in this captured snapshot.", + "- checkbox: Paid [checked=false]", + "", + "## COMPLETE COLLECTION: radio choices", + "Observed radio count: 1", + "Completeness scope: all radio roles in this captured snapshot.", + "- radio: Card [checked=true]", + "", + "## Interactive controls", + "- button: Submit [disabled=false]", + "", + "## Other exact visible evidence", + "- StaticText: Invoice #42", + ].join("\n"), + metadata: { + evidenceFormat: STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + sequenceIndex: 0, + contentRole: "ui_state_transition", + }, + sourceStateIndices: [0], + documentType: "state", + stateIndex: 0, + step: 3, + screenshotRef: input.states[0].screenshot, + localAttachmentPaths: [], + dependsOn: [], + allowParallelUpload: true, + allowDuplicateContent: false, + }) + + expect(plan.documents[2]).toEqual({ + logicalDocumentId: "result", + content: [ + "# RESULT: TRAJECTORY OUTCOME", + "Trajectory ID: trajectory-1", + "Document role: final runner outcome only; it does not restate the goal or override observed UI facts.", + "Final outcome: Invoice 42 was found.", + ].join("\n"), + metadata: { + evidenceFormat: STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + sequenceIndex: 1, + contentRole: "trajectory_outcome", + }, + documentType: "result", + sourceStateIndices: [], + localAttachmentPaths: [], + dependsOn: [], + allowParallelUpload: true, + allowDuplicateContent: false, + }) + + expect(structuredAccessibilityConverter.convert(input, undefined)).toEqual(plan) + }) + + test("sorts states, keeps goal/outcome isolated, and uses empty-value fallbacks", () => { + const firstScreenshot = screenshot("first") + const secondScreenshot = screenshot("second") + const plan = structuredAccessibilityConverter.convert( + trajectory({ + outcome: null, + states: [ + { + stateIndex: 4, + step: 9, + url: "https://example.test/four", + thoughts: "", + action: "", + accessibilityTree: "generic 'container'", + screenshot: secondScreenshot, + }, + { + stateIndex: 2, + step: 5, + url: "https://example.test/two", + thoughts: null, + action: null, + accessibilityTree: "", + screenshot: firstScreenshot, + }, + ], + }), + undefined + ) + + expect(plan.documents.map((document) => document.logicalDocumentId)).toEqual([ + "overview", + "state-0002", + "state-0004", + "result", + ]) + expect(plan.documents[1].content).toContain("No agent interpretation was recorded.") + expect(plan.documents[1].content).toContain("No action was issued from this state.") + expect(plan.documents[1].content).toEndWith( + "No named accessibility evidence was captured in this snapshot." + ) + expect(plan.documents[2].content).toEndWith( + "No named accessibility evidence was captured in this snapshot." + ) + expect(plan.documents[3].metadata.sequenceIndex).toBe(5) + expect(plan.documents[3].content).toEndWith("Final outcome: unknown") + + const goal = "Find the overdue invoice." + expect(plan.documents[0].content).toContain(goal) + expect(plan.documents.slice(1).every((document) => !document.content.includes(goal))).toBe(true) + expect(plan.documents[3].content).toContain("Final outcome: unknown") + expect( + plan.documents.slice(0, -1).every((document) => !document.content.includes("Final outcome:")) + ).toBe(true) + }) + + test("cleans entities, Unicode private-use characters, and apostrophes in labels", () => { + expect(cleanAccessibilityText("<A€\uE000\\uE123 e\u0301>\t done")).toBe( + " done" + ) + + const evidence = parseStructuredAccessibilityTree( + "[1] button 'Owner's report', value='', placeholder='A B', checked=false" + ) + expect(evidence.controls).toEqual([ + { + role: "button", + label: "Owner's report", + attributes: ["value=(empty)", "placeholder=A B", "checked=false"], + indent: 0, + }, + ]) + }) + + test("preserves complete collection order and repeated labels", () => { + const evidence = parseStructuredAccessibilityTree( + [ + "option 'Same', selected=true", + "option 'Same', selected=true", + "option 'Other', selected=false", + "checkbox 'Flag', checked=false", + "checkbox 'Flag', checked=false", + "radio 'One', checked=true", + "radio 'One', checked=true", + ].join("\n") + ) + + expect(evidence.options.map((node) => node.label)).toEqual(["Same", "Same", "Other"]) + expect(evidence.checkboxes).toHaveLength(2) + expect(evidence.radios).toHaveLength(2) + }) + + test("preserves table rows and binds cells only when cardinalities match", () => { + const evidence = parseStructuredAccessibilityTree( + [ + "table 'Visible Users'", + " row ''", + " columnheader 'Name Name'", + " columnheader 'Role column options'", + " row ''", + " cell 'Ada'", + " cell 'Admin'", + " row ''", + " cell 'Grace'", + ].join("\n") + ) + + expect(evidence.tables).toEqual([ + { + title: "Visible Users", + headers: ["Name", "Role"], + rows: [["Ada", "Admin"], ["Grace"]], + }, + ]) + + const plan = structuredAccessibilityConverter.convert( + trajectory({ + states: [ + { + ...trajectory().states[0], + accessibilityTree: [ + "table 'Visible Users'", + " row ''", + " columnheader 'Name Name'", + " columnheader 'Role column options'", + " row ''", + " cell 'Ada'", + " cell 'Admin'", + " row ''", + " cell 'Grace'", + ].join("\n"), + }, + ], + }), + undefined + ) + const stateContent = plan.documents[1].content + expect(stateContent).toContain("- Name: Ada\n- Role: Admin") + expect(stateContent).toContain( + "Schema mismatch: 2 visible headers and 1 visible cells; values are preserved without inferred bindings.\n- Ordered cell 1: Grace" + ) + }) + + test("retains labelled unparsed evidence and deduplicates normalized repeats", () => { + const evidence = parseStructuredAccessibilityTree( + ["??? 'Unparsed value'", "??? 'unparsed value'", "not labelled"].join("\n") + ) + expect(evidence.unparsedLines).toEqual(["??? 'Unparsed value'"]) + expect(evidence.other).toEqual([]) + }) + + test("rejects empty and duplicate state-index inputs", () => { + expect(() => + structuredAccessibilityConverter.convert(trajectory({ states: [] }), undefined) + ).toThrow("must contain at least one state") + + const state = trajectory().states[0] + expect(() => + structuredAccessibilityConverter.convert( + trajectory({ states: [state, { ...state }] }), + undefined + ) + ).toThrow("duplicate stateIndex 0") + }) + + test("passes the generic deterministic-plan validator and rejects batch splitting", () => { + const input = trajectory() + const plan = structuredAccessibilityConverter.convert(input, undefined) + const validated = validateDocumentPlan({ + plan, + converter: structuredAccessibilityConverter, + trajectory: input, + context: undefined, + }) + + expect(validated.batchUpload).toBe(true) + expect(validated.documents.map((document) => document.dependsOnOrdinals)).toEqual([[], [], []]) + expect(() => + createPhysicalDocuments({ + plan: validated, + buildFingerprint: "build-fingerprint", + maxDocumentChars: 50, + }) + ).toThrow("a batch document cannot be split") + }) +}) diff --git a/src/benchmarks/longmemeval-v2/converter.ts b/src/benchmarks/longmemeval-v2/converter.ts new file mode 100644 index 0000000..15bee99 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/converter.ts @@ -0,0 +1,761 @@ +import { sha256 } from "../../core/canonical" +import type { TrajectoryConverter } from "../../core/document-plan" +import type { DocumentPlan, DocumentSpec } from "../../types/migration" +import type { PreparedTrajectory, PreparedTrajectoryState } from "./types" + +export const STRUCTURED_ACCESSIBILITY_CONVERTER_NAME = "Structured Accessibility Converter" +export const STRUCTURED_ACCESSIBILITY_CONVERTER_VERSION = 1 +export const STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT = "structured-accessibility-v1" + +// Source oracle: LongMemEval-V2 feat/supermemory@2fa6616, Approach_1.py. +export const LEGACY_APPROACH_1_SOURCE_SHA256 = + "22cff05fafa9f882040afa8296439da0f911f800c107424de105ab3af5e69236" + +const NODE_ID_PREFIX = /^\s*\[[^\]]+\]\s*/ +const PRIVATE_USE = /[\uE000-\uF8FF\u{F0000}-\u{FFFFD}\u{100000}-\u{10FFFD}]/gu +const ESCAPED_PRIVATE_USE = /\\u(?:[eEfF][0-9a-fA-F]{3})/g + +const STRUCTURAL_ROLES = new Set([ + "generic", + "group", + "main", + "row", + "rowgroup", + "list", + "listbox", + "menu", + "toolbar", +]) +const TABLE_ROLES = new Set(["table", "grid", "treegrid"]) +const TABLE_CELL_ROLES = new Set(["gridcell", "cell", "rowheader"]) +const LANDMARK_ROLES = new Set(["dialog", "form", "navigation", "region", "tabpanel", "tablist"]) +const ALERT_ROLES = new Set(["alert", "status", "log", "marquee", "timer"]) +const CONTROL_ROLES = new Set([ + "button", + "link", + "menuitem", + "tab", + "textbox", + "searchbox", + "combobox", + "switch", + "slider", + "spinbutton", +]) +const STATE_ATTRIBUTE_NAMES = [ + "value", + "placeholder", + "checked", + "selected", + "expanded", + "disabled", + "required", + "pressed", + "level", + "live", + "hasPopup", + "autocomplete", +] as const + +const NAMED_HTML_ENTITIES: Readonly> = { + amp: "&", + AMP: "&", + apos: "'", + bull: "•", + copy: "©", + divide: "÷", + euro: "€", + gt: ">", + GT: ">", + hellip: "…", + ldquo: "“", + lsquo: "‘", + lt: "<", + LT: "<", + mdash: "—", + middot: "·", + nbsp: "\u00a0", + ndash: "–", + quot: '"', + QUOT: '"', + rdquo: "”", + reg: "®", + rsquo: "’", + times: "×", + trade: "™", +} + +const WINDOWS_1252_NUMERIC_REFERENCES: Readonly> = { + 0x80: 0x20ac, + 0x82: 0x201a, + 0x83: 0x0192, + 0x84: 0x201e, + 0x85: 0x2026, + 0x86: 0x2020, + 0x87: 0x2021, + 0x88: 0x02c6, + 0x89: 0x2030, + 0x8a: 0x0160, + 0x8b: 0x2039, + 0x8c: 0x0152, + 0x8e: 0x017d, + 0x91: 0x2018, + 0x92: 0x2019, + 0x93: 0x201c, + 0x94: 0x201d, + 0x95: 0x2022, + 0x96: 0x2013, + 0x97: 0x2014, + 0x98: 0x02dc, + 0x99: 0x2122, + 0x9a: 0x0161, + 0x9b: 0x203a, + 0x9c: 0x0153, + 0x9e: 0x017e, + 0x9f: 0x0178, +} + +const HTML_ENTITY_EXPRESSION = new RegExp( + `&(#(?:[xX][0-9a-fA-F]+|[0-9]+)|${Object.keys(NAMED_HTML_ENTITIES) + .sort((left, right) => right.length - left.length) + .join("|")});?`, + "g" +) + +export interface StructuredAccessibilityNode { + role: string + label: string + attributes: string[] + indent: number +} + +export interface StructuredAccessibilityTable { + title: string + headers: string[] + rows: string[][] +} + +export interface StructuredAccessibilityEvidence { + pageTitles: string[] + landmarks: StructuredAccessibilityNode[] + alerts: StructuredAccessibilityNode[] + headings: StructuredAccessibilityNode[] + options: StructuredAccessibilityNode[] + checkboxes: StructuredAccessibilityNode[] + radios: StructuredAccessibilityNode[] + controls: StructuredAccessibilityNode[] + other: StructuredAccessibilityNode[] + tables: StructuredAccessibilityTable[] + unparsedLines: string[] +} + +function decodeNumericReference(raw: string): string { + const hexadecimal = raw[1] === "x" || raw[1] === "X" + const parsed = Number.parseInt(raw.slice(hexadecimal ? 2 : 1), hexadecimal ? 16 : 10) + if (!Number.isFinite(parsed)) return `&${raw};` + const codePoint = WINDOWS_1252_NUMERIC_REFERENCES[parsed] ?? parsed + if (codePoint === 0 || codePoint > 0x10ffff || (codePoint >= 0xd800 && codePoint <= 0xdfff)) { + return "\ufffd" + } + return String.fromCodePoint(codePoint) +} + +function unescapeHtml(text: string): string { + return text.replace(HTML_ENTITY_EXPRESSION, (_match, entity: string) => + entity.startsWith("#") ? decodeNumericReference(entity) : NAMED_HTML_ENTITIES[entity] + ) +} + +export function cleanAccessibilityText(value: unknown): string { + const text = unescapeHtml(String(value || "")) + .replace(PRIVATE_USE, "") + .replace(ESCAPED_PRIVATE_USE, "") + .normalize("NFC") + return text.replace(/\s+/gu, " ").trim() +} + +function caseFoldCharacter(character: string): string { + if (character === "ß" || character === "ẞ") return "ss" + if (character === "ς") return "σ" + return character.toLowerCase() +} + +function normalizationKey(value: string): string { + const output: string[] = [] + for (const character of value) { + output.push(/[\p{L}\p{N}]/u.test(character) ? caseFoldCharacter(character) : " ") + } + return output.join("").split(/\s+/u).filter(Boolean).join(" ") +} + +function orderedUniqueStrings(values: string[]): string[] { + const seen = new Set() + const output: string[] = [] + for (const value of values) { + const key = normalizationKey(value) + if (!key || seen.has(key)) continue + seen.add(key) + output.push(value) + } + return output +} + +function orderedUniqueNodes(nodes: StructuredAccessibilityNode[]): StructuredAccessibilityNode[] { + const seen = new Set() + const output: StructuredAccessibilityNode[] = [] + for (const node of nodes) { + const key = JSON.stringify([ + node.role.toLowerCase(), + normalizationKey(node.label), + node.attributes, + ]) + if (seen.has(key)) continue + seen.add(key) + output.push(node) + } + return output +} + +function quotedValue(text: string, quoteIndex: number): [string, number] | undefined { + for (let index = quoteIndex + 1; index < text.length; index += 1) { + if (text[index] !== "'") continue + const suffix = text.slice(index + 1) + if (!suffix || suffix.startsWith(",")) { + return [text.slice(quoteIndex + 1, index), index] + } + } + return undefined +} + +function attributeValue(line: string, name: string): string | undefined { + const match = new RegExp(`\\b${name}=`).exec(line) + if (!match || match.index === undefined) return undefined + const start = match.index + match[0].length + if (start >= line.length) return "" + if (line[start] === "'") { + const quoted = quotedValue(line, start) + return quoted ? quoted[0] : line.slice(start + 1) + } + let end = start + while (end < line.length && ![",", " ", "\t"].includes(line[end])) { + end += 1 + } + return line.slice(start, end) +} + +function expandedIndent(prefix: string): number { + let column = 0 + for (const character of prefix) { + if (character === "\t") { + column += 2 - (column % 2) + } else { + column += 1 + } + } + return column +} + +function parseNode(rawLine: string): StructuredAccessibilityNode | undefined { + const prefix = /^[\t ]*/.exec(rawLine)?.[0] ?? "" + const indent = expandedIndent(prefix) + const line = rawLine.replace(NODE_ID_PREFIX, "").trim() + if (!line) return undefined + const roleMatch = /^([A-Za-z][A-Za-z0-9_-]*)\b/.exec(line) + if (!roleMatch) return undefined + const role = roleMatch[1] + const quoteIndex = line.indexOf("'", roleMatch[0].length) + if (quoteIndex < 0) return undefined + const quoted = quotedValue(line, quoteIndex) + if (!quoted) return undefined + const attributes: string[] = [] + for (const name of STATE_ATTRIBUTE_NAMES) { + const value = attributeValue(line, name) + if (value === undefined) continue + const cleaned = cleanAccessibilityText(value) + if (cleaned || value === "") { + attributes.push(`${name}=${cleaned || "(empty)"}`) + } + } + return { + role, + label: cleanAccessibilityText(quoted[0]), + attributes, + indent, + } +} + +function cleanHeader(label: string): string { + const cleaned = cleanAccessibilityText(label).replace(/\s+column options$/i, "") + const words = cleaned.split(" ") + for (let splitAt = 1; splitAt < words.length; splitAt += 1) { + const left = words.slice(0, splitAt).join(" ") + const right = words.slice(splitAt).join(" ") + if (normalizationKey(left) === normalizationKey(right)) return left + } + return cleaned || "(unnamed column)" +} + +function tableCell(label: string): string { + return cleanAccessibilityText(label) || "(empty)" +} + +export function parseStructuredAccessibilityTree(tree: string): StructuredAccessibilityEvidence { + const generalNodes: StructuredAccessibilityNode[] = [] + const unparsedLines: string[] = [] + const tables: StructuredAccessibilityTable[] = [] + let activeRowIndent: number | undefined + let activeHeaders: string[] = [] + let activeCells: string[] = [] + let currentHeaders: string[] = [] + let currentTableTitle = "Visible table" + + const flushRow = (): void => { + if (activeHeaders.length > 0) { + currentHeaders = activeHeaders.map(cleanHeader) + } + if (activeCells.length > 0) { + const row = activeCells.map(tableCell) + const last = tables.at(-1) + if ( + !last || + JSON.stringify(last.headers) !== JSON.stringify(currentHeaders) || + last.title !== currentTableTitle + ) { + tables.push({ + title: currentTableTitle, + headers: [...currentHeaders], + rows: [], + }) + } + tables.at(-1)!.rows.push(row) + } + activeRowIndent = undefined + activeHeaders = [] + activeCells = [] + } + + for (const rawLine of tree.split(/\r\n|[\n\r\v\f\x1c-\x1e\x85\u2028\u2029]/u)) { + const node = parseNode(rawLine) + if (!node) { + const stripped = cleanAccessibilityText(rawLine.replace(NODE_ID_PREFIX, "")) + if (stripped && (rawLine.includes("'") || rawLine.includes("value="))) { + unparsedLines.push(stripped) + } + continue + } + + if (activeRowIndent !== undefined && node.indent <= activeRowIndent) { + flushRow() + } + + const role = node.role.toLowerCase() + if (role === "row") { + flushRow() + activeRowIndent = node.indent + continue + } + + if (activeRowIndent !== undefined && node.indent > activeRowIndent) { + if (role === "columnheader") { + activeHeaders.push(node.label) + } else if (TABLE_CELL_ROLES.has(role)) { + activeCells.push(node.label) + } + continue + } + + if (TABLE_ROLES.has(role)) { + flushRow() + currentHeaders = [] + currentTableTitle = node.label || "Visible table" + continue + } + if (role === "rowgroup") continue + generalNodes.push(node) + } + flushRow() + + const pageTitles = orderedUniqueStrings( + generalNodes + .filter((node) => node.role.toLowerCase() === "rootwebarea" && node.label) + .map((node) => node.label) + ) + const landmarks: StructuredAccessibilityNode[] = [] + const alerts: StructuredAccessibilityNode[] = [] + const headings: StructuredAccessibilityNode[] = [] + const options: StructuredAccessibilityNode[] = [] + const checkboxes: StructuredAccessibilityNode[] = [] + const radios: StructuredAccessibilityNode[] = [] + const controls: StructuredAccessibilityNode[] = [] + const otherNonStatic: StructuredAccessibilityNode[] = [] + const staticNodes: StructuredAccessibilityNode[] = [] + + for (const node of generalNodes) { + const role = node.role.toLowerCase() + if (role === "rootwebarea") { + continue + } else if (LANDMARK_ROLES.has(role)) { + if (node.label || node.attributes.length > 0) landmarks.push(node) + } else if ( + ALERT_ROLES.has(role) || + node.attributes.some((value) => value.startsWith("live=")) + ) { + if (node.label || node.attributes.length > 0) alerts.push(node) + } else if (role === "heading") { + headings.push(node) + } else if (role === "option") { + options.push(node) + } else if (role === "checkbox") { + checkboxes.push(node) + } else if (role === "radio") { + radios.push(node) + } else if (CONTROL_ROLES.has(role)) { + controls.push(node) + } else if (role === "statictext") { + if (node.label) staticNodes.push(node) + } else if (!STRUCTURAL_ROLES.has(role) && (node.label || node.attributes.length > 0)) { + otherNonStatic.push(node) + } + } + + const representedLabels = new Set( + [ + ...landmarks, + ...alerts, + ...headings, + ...options, + ...checkboxes, + ...radios, + ...controls, + ...otherNonStatic, + ] + .filter((node) => node.label) + .map((node) => normalizationKey(node.label)) + ) + const other = [...otherNonStatic] + for (const node of staticNodes) { + const key = normalizationKey(node.label) + if (key && !representedLabels.has(key)) { + representedLabels.add(key) + other.push(node) + } + } + + return { + pageTitles, + landmarks: orderedUniqueNodes(landmarks), + alerts: orderedUniqueNodes(alerts), + headings: orderedUniqueNodes(headings), + options, + checkboxes, + radios, + controls: orderedUniqueNodes(controls), + other: orderedUniqueNodes(other), + tables, + unparsedLines: orderedUniqueStrings(unparsedLines), + } +} + +function formatNode(node: StructuredAccessibilityNode): string { + const label = node.label || "(unnamed)" + const suffix = node.attributes.length > 0 ? ` [${node.attributes.join(", ")}]` : "" + return `- ${node.role}: ${label}${suffix}` +} + +function appendNodes(lines: string[], heading: string, nodes: StructuredAccessibilityNode[]): void { + if (nodes.length === 0) return + lines.push("", heading, ...nodes.map(formatNode)) +} + +function appendCompleteCollection( + lines: string[], + heading: string, + collectionName: string, + nodes: StructuredAccessibilityNode[] +): void { + if (nodes.length === 0) return + lines.push( + "", + heading, + `Observed ${collectionName} count: ${nodes.length}`, + `Completeness scope: all ${collectionName} roles in this captured snapshot.`, + ...nodes.map(formatNode) + ) +} + +function appendTables(lines: string[], tables: StructuredAccessibilityTable[]): void { + tables.forEach((table, tableIndex) => { + lines.push("", `## TABLE ${tableIndex + 1}: ${table.title}`) + if (table.headers.length > 0) { + lines.push(`Visible columns (${table.headers.length}) in order: ${table.headers.join(" | ")}`) + } else { + lines.push("Visible column labels were unavailable for this table.") + } + lines.push( + `Visible row count: ${table.rows.length}`, + "Completeness scope: visible rows in this captured snapshot, not the full paginated dataset." + ) + table.rows.forEach((row, rowIndex) => { + lines.push(`### Record ${rowIndex + 1}`) + if (table.headers.length > 0 && table.headers.length === row.length) { + table.headers.forEach((header, index) => { + lines.push(`- ${header}: ${row[index]}`) + }) + } else { + lines.push( + `Schema mismatch: ${table.headers.length} visible headers and ${row.length} visible cells; values are preserved without inferred bindings.` + ) + row.forEach((value, index) => { + lines.push(`- Ordered cell ${index + 1}: ${value}`) + }) + } + }) + }) +} + +function baseDocument( + input: Omit< + DocumentSpec, + | "sourceStateIndices" + | "localAttachmentPaths" + | "dependsOn" + | "allowParallelUpload" + | "allowDuplicateContent" + > & + Partial> +): DocumentSpec { + return { + ...input, + sourceStateIndices: input.sourceStateIndices ?? [], + localAttachmentPaths: [], + dependsOn: [], + allowParallelUpload: true, + allowDuplicateContent: false, + } +} + +function overviewDocument(trajectory: PreparedTrajectory): DocumentSpec { + const content = [ + "# STATE_-1: TRAJECTORY OVERVIEW", + `Trajectory ID: ${trajectory.id}`, + `Domain: ${trajectory.domain}`, + `Start URL: ${trajectory.startUrl}`, + "Document role: requested goal only; this is not proof that the task succeeded.", + "", + "## Requested goal", + trajectory.goal, + ].join("\n") + return baseDocument({ + logicalDocumentId: "overview", + content, + metadata: { + evidenceFormat: STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + sequenceIndex: -1, + contentRole: "trajectory_goal", + }, + documentType: "overview", + }) +} + +function stateDocument( + trajectory: PreparedTrajectory, + state: PreparedTrajectoryState +): DocumentSpec { + const evidence = parseStructuredAccessibilityTree(state.accessibilityTree || "") + const lines = [ + `# STATE_${state.stateIndex}: STRUCTURED UI OBSERVATION`, + `Trajectory ID: ${trajectory.id}`, + `State index: ${state.stateIndex}`, + `Step: ${state.step}`, + `URL: ${state.url}`, + "Document role: UI observed at this state, followed by an unverified interpretation and attempted action.", + "", + "## Agent interpretation or next-step plan (unverified)", + state.thoughts || "No agent interpretation was recorded.", + "", + "## Action issued after this observation (attempted, not proof of success)", + state.action || "No action was issued from this state.", + "", + "## Observed accessibility evidence", + `Page titles: ${evidence.pageTitles.length > 0 ? evidence.pageTitles.join(" | ") : "unknown"}`, + ] + appendNodes(lines, "## Page landmarks and dialogs", evidence.landmarks) + appendNodes(lines, "## Alerts and status messages", evidence.alerts) + appendNodes(lines, "## Headings", evidence.headings) + appendCompleteCollection(lines, "## COMPLETE COLLECTION: options", "option", evidence.options) + appendCompleteCollection( + lines, + "## COMPLETE COLLECTION: checkboxes", + "checkbox", + evidence.checkboxes + ) + appendCompleteCollection(lines, "## COMPLETE COLLECTION: radio choices", "radio", evidence.radios) + appendNodes(lines, "## Interactive controls", evidence.controls) + appendTables(lines, evidence.tables) + appendNodes(lines, "## Other exact visible evidence", evidence.other) + if (evidence.unparsedLines.length > 0) { + lines.push( + "", + "## Residual labelled accessibility evidence", + "These source lines were preserved because they contained labelled or valued evidence that was not structurally classified.", + ...evidence.unparsedLines.map((line) => `- ${line}`) + ) + } + if ( + [ + evidence.pageTitles, + evidence.landmarks, + evidence.alerts, + evidence.headings, + evidence.options, + evidence.checkboxes, + evidence.radios, + evidence.controls, + evidence.tables, + evidence.other, + evidence.unparsedLines, + ].every((items) => items.length === 0) + ) { + lines.push("No named accessibility evidence was captured in this snapshot.") + } + + return baseDocument({ + logicalDocumentId: `state-${state.stateIndex.toString().padStart(4, "0")}`, + content: lines.join("\n"), + metadata: { + evidenceFormat: STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + sequenceIndex: state.stateIndex, + contentRole: "ui_state_transition", + }, + sourceStateIndices: [state.stateIndex], + documentType: "state", + stateIndex: state.stateIndex, + step: state.step, + screenshotRef: state.screenshot, + }) +} + +function resultDocument(trajectory: PreparedTrajectory): DocumentSpec { + const finalStateIndex = Math.max(...trajectory.states.map((state) => state.stateIndex)) + const content = [ + "# RESULT: TRAJECTORY OUTCOME", + `Trajectory ID: ${trajectory.id}`, + "Document role: final runner outcome only; it does not restate the goal or override observed UI facts.", + `Final outcome: ${trajectory.outcome ?? "unknown"}`, + ].join("\n") + return baseDocument({ + logicalDocumentId: "result", + content, + metadata: { + evidenceFormat: STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + sequenceIndex: finalStateIndex + 1, + contentRole: "trajectory_outcome", + }, + documentType: "result", + }) +} + +export const STRUCTURED_ACCESSIBILITY_INVARIANTS = [ + "the goal appears only in the overview document", + "the final outcome appears only in the result document", + "one logical document is produced per observed state", + "agent thoughts, attempted actions, and observed UI evidence are explicitly separated", + "complete option, checkbox, and radio collections preserve displayed order and count", + "table values are bound to headers only when their cardinalities match", + "documents inside one trajectory have no causal dependencies", + "all documents for one trajectory are submitted in one V3 batch", + "no trajectory receives ingestion context from any other trajectory", + "independent trajectories may be processed concurrently", + "no cross-state persistent-shell deduplication is performed", + "the benchmark question and gold answer are never used", +] as const + +function convertPreparedTrajectory(trajectory: PreparedTrajectory): DocumentPlan { + if (!trajectory.id) throw new Error("trajectory id must not be empty") + if (trajectory.states.length === 0) { + throw new Error(`trajectory ${trajectory.id} must contain at least one state`) + } + const states = [...trajectory.states].sort((left, right) => left.stateIndex - right.stateIndex) + const seenStateIndexes = new Set() + for (const state of states) { + if (!Number.isInteger(state.stateIndex) || state.stateIndex < 0) { + throw new Error(`trajectory ${trajectory.id} stateIndex must be an integer >= 0`) + } + if (seenStateIndexes.has(state.stateIndex)) { + throw new Error(`trajectory ${trajectory.id} has duplicate stateIndex ${state.stateIndex}`) + } + seenStateIndexes.add(state.stateIndex) + } + + return { + trajectoryId: trajectory.id, + documents: [ + overviewDocument(trajectory), + ...states.map((state) => stateDocument(trajectory, state)), + resultDocument(trajectory), + ], + batchUpload: true, + declaredInvariants: [...STRUCTURED_ACCESSIBILITY_INVARIANTS], + notes: + "Independent V3 batch per trajectory with structured accessibility documents and no state-level or cross-trajectory ingestion context.", + } +} + +const IMPLEMENTATION_FUNCTIONS = [ + decodeNumericReference, + unescapeHtml, + cleanAccessibilityText, + caseFoldCharacter, + normalizationKey, + orderedUniqueStrings, + orderedUniqueNodes, + quotedValue, + attributeValue, + expandedIndent, + parseNode, + cleanHeader, + tableCell, + parseStructuredAccessibilityTree, + formatNode, + appendNodes, + appendCompleteCollection, + appendTables, + baseDocument, + overviewDocument, + stateDocument, + resultDocument, + convertPreparedTrajectory, +] + +export const STRUCTURED_ACCESSIBILITY_CONVERTER_SOURCE_HASH = sha256( + [ + LEGACY_APPROACH_1_SOURCE_SHA256, + STRUCTURED_ACCESSIBILITY_EVIDENCE_FORMAT, + JSON.stringify(NAMED_HTML_ENTITIES), + JSON.stringify(WINDOWS_1252_NUMERIC_REFERENCES), + JSON.stringify(STATE_ATTRIBUTE_NAMES), + JSON.stringify([...STRUCTURAL_ROLES]), + JSON.stringify([...TABLE_ROLES]), + JSON.stringify([...TABLE_CELL_ROLES]), + JSON.stringify([...LANDMARK_ROLES]), + JSON.stringify([...ALERT_ROLES]), + JSON.stringify([...CONTROL_ROLES]), + JSON.stringify(STRUCTURED_ACCESSIBILITY_INVARIANTS), + ...IMPLEMENTATION_FUNCTIONS.map((implementation) => implementation.toString()), + ].join("\0") +) + +export class StructuredAccessibilityConverter implements TrajectoryConverter< + PreparedTrajectory, + void +> { + readonly name = STRUCTURED_ACCESSIBILITY_CONVERTER_NAME + readonly version = STRUCTURED_ACCESSIBILITY_CONVERTER_VERSION + readonly sourceHash = STRUCTURED_ACCESSIBILITY_CONVERTER_SOURCE_HASH + + convert(trajectory: PreparedTrajectory, _context: void): DocumentPlan { + return convertPreparedTrajectory(trajectory) + } +} + +export const structuredAccessibilityConverter = new StructuredAccessibilityConverter() diff --git a/src/benchmarks/longmemeval-v2/dataset.test.ts b/src/benchmarks/longmemeval-v2/dataset.test.ts new file mode 100644 index 0000000..ef1dd52 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/dataset.test.ts @@ -0,0 +1,612 @@ +import { afterEach, describe, expect, test } from "bun:test" +import { mkdtemp, mkdir, readdir, readFile, rm, stat, symlink, writeFile } from "node:fs/promises" +import { tmpdir } from "node:os" +import { basename, join, relative, resolve } from "node:path" +import { sha256 } from "../../core/canonical" +import { LongMemEvalV2Dataset } from "./dataset" +import { downloadDatasetSnapshotForTesting, LONGMEMEVAL_V2_COMPLETION_MARKER } from "./download" +import { + type ArchiveAdapter, + prepareLongMemEvalV2ScreenshotsForTesting, + validatePreparedScreenshotLayout, +} from "./prepare" +import { + LONGMEMEVAL_V2_PINNED_REVISION, + type LongMemEvalV2SnapshotSpec, + type PinnedDatasetFile, + parseChecksumManifest, + validateDatasetRelativePath, + verifyDatasetSnapshot, +} from "./source" + +const temporaryDirectories: string[] = [] +const PNG_HEADER = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]) + +afterEach(async () => { + await Promise.all( + temporaryDirectories.splice(0).map((path) => rm(path, { recursive: true, force: true })) + ) +}) + +async function temporaryDirectory(): Promise { + const path = await mkdtemp(join(tmpdir(), "memorybench-lme2-dataset-")) + temporaryDirectories.push(path) + return path +} + +function jsonLine(value: unknown): string { + return `${JSON.stringify(value)}\n` +} + +function fixtureSnapshot(): { + spec: LongMemEvalV2SnapshotSpec + files: Map +} { + const payloads = new Map([ + ["questions.jsonl", Buffer.from('{"id":"q1"}\n')], + ["trajectories.jsonl", Buffer.from('{"id":"t1"}\n')], + ["haystacks/lme_v2_small.json", Buffer.from('{"q1":["t1"]}\n')], + ["haystacks/lme_v2_medium.json", Buffer.from('{"q1":["t1"]}\n')], + ["question_screenshots/q1.png", PNG_HEADER], + ["README.md", Buffer.from("fixture\n")], + ["trajectory_screenshots/web.tar.gz", Buffer.from("web archive")], + ["trajectory_screenshots/enterprise.tar.gz", Buffer.from("enterprise archive")], + ]) + const requiredFiles = Object.fromEntries( + [ + "questions.jsonl", + "trajectories.jsonl", + "haystacks/lme_v2_small.json", + "haystacks/lme_v2_medium.json", + ].map((relativePath) => { + const bytes = payloads.get(relativePath)! + return [relativePath, { sha256: sha256(bytes), byteLength: bytes.byteLength }] + }) + ) + const manifestPaths = [...Object.keys(requiredFiles), "question_screenshots/q1.png", "README.md"] + const manifest = `${manifestPaths + .map((relativePath) => { + const bytes = payloads.get(relativePath)! + return `${sha256(bytes)} ${relativePath}` + }) + .join("\n")}\n` + payloads.set("checksums.sha256", Buffer.from(manifest)) + const archives = [ + "trajectory_screenshots/web.tar.gz", + "trajectory_screenshots/enterprise.tar.gz", + ].map((relativePath) => { + const bytes = payloads.get(relativePath)! + return { + relativePath, + sha256: sha256(bytes), + byteLength: bytes.byteLength, + } + }) + return { + spec: { + repository: "fixture/longmemeval-v2", + revision: "fixture-revision", + checksumManifest: { + relativePath: "checksums.sha256", + sha256: sha256(manifest), + byteLength: Buffer.byteLength(manifest), + }, + requiredFiles, + archives, + questionImageCount: 1, + }, + files: payloads, + } +} + +function fixtureFetch( + spec: LongMemEvalV2SnapshotSpec, + files: Map, + overrides: Map = new Map() +): typeof fetch { + return (async (input: string | URL | Request) => { + const url = new URL(typeof input === "string" || input instanceof URL ? input : input.url) + const marker = `/resolve/${encodeURIComponent(spec.revision)}/` + const markerIndex = url.pathname.indexOf(marker) + if (markerIndex < 0) return new Response("bad fixture URL", { status: 400 }) + const relativePath = url.pathname + .slice(markerIndex + marker.length) + .split("/") + .map(decodeURIComponent) + .join("/") + const bytes = overrides.get(relativePath) ?? files.get(relativePath) + return bytes + ? new Response(bytes as unknown as BodyInit, { status: 200 }) + : new Response("missing", { status: 404 }) + }) as typeof fetch +} + +async function writeDatasetFixture(dataRoot: string): Promise { + await mkdir(resolve(dataRoot, "haystacks"), { recursive: true }) + await mkdir(resolve(dataRoot, "screenshots/t1"), { recursive: true }) + await mkdir(resolve(dataRoot, "screenshots/t2"), { recursive: true }) + await mkdir(resolve(dataRoot, "question_screenshots"), { recursive: true }) + await writeFile(resolve(dataRoot, "screenshots/t1/0.png"), PNG_HEADER) + await writeFile( + resolve(dataRoot, "screenshots/t2/0.png"), + Buffer.concat([PNG_HEADER, Buffer.from([2])]) + ) + await writeFile( + resolve(dataRoot, "question_screenshots/q.png"), + Buffer.concat([PNG_HEADER, Buffer.from([3])]) + ) + + const questions = [ + { + id: "q1", + domain: "web", + environment: "browser", + question_type: "static-environment", + question: "Question one?", + image: "question_screenshots/q.png", + answer: "one", + eval_function: "[]", + }, + { + id: "q2", + domain: "web", + environment: "browser", + question_type: "static-environment", + question: "Question two?", + image: null, + answer: "two", + eval_function: "[]", + }, + { + id: "q3", + domain: "web", + environment: "browser", + question_type: "procedure", + question: "Question three?", + image: null, + answer: "three", + eval_function: "[]", + }, + { + id: "q4", + domain: "web", + environment: "browser", + question_type: "procedure", + question: "Question four?", + image: null, + answer: "four", + eval_function: "[]", + }, + ] + await writeFile(resolve(dataRoot, "questions.jsonl"), questions.map(jsonLine).join("")) + const trajectories = [ + { + id: "t1", + domain: "web", + goal: "Goal one", + start_url: "https://example.test/one", + outcome: "done", + states: [ + { + state_index: 7, + url: "https://example.test/one", + action: null, + thought: null, + accessibility_tree: "heading 'One'", + screenshot: "screenshots/t1/0.png", + }, + ], + }, + { + id: "t2", + domain: "web", + goal: "Goal two", + start_url: "https://example.test/two", + outcome: null, + states: [ + { + step: 4, + url: "https://example.test/two", + action: "click", + thought: "inspect", + accessibility_tree: "heading 'Two'", + screenshot: "screenshots/t2/0.png", + }, + ], + }, + ] + await writeFile(resolve(dataRoot, "trajectories.jsonl"), trajectories.map(jsonLine).join("")) + await writeFile( + resolve(dataRoot, "haystacks/lme_v2_small.json"), + `${JSON.stringify({ + q1: ["t1"], + q2: ["t1"], + q3: ["t2"], + q4: ["t2"], + })}\n` + ) +} + +describe("LongMemEval-V2 pinned snapshot source", () => { + test("parses only safe, complete checksum manifests", () => { + const { spec, files } = fixtureSnapshot() + const manifest = Buffer.from(files.get("checksums.sha256")!).toString("utf8") + expect(parseChecksumManifest(manifest, spec).map((file) => file.relativePath)).toEqual([ + "questions.jsonl", + "trajectories.jsonl", + "haystacks/lme_v2_small.json", + "haystacks/lme_v2_medium.json", + "question_screenshots/q1.png", + "README.md", + ]) + expect(() => validateDatasetRelativePath("../escape")).toThrow("Unsafe dataset path") + expect(() => validateDatasetRelativePath("folder\\escape")).toThrow("backslash") + expect(() => parseChecksumManifest(`${"0".repeat(64)} ../escape\n`, spec)).toThrow( + "Unsafe dataset path" + ) + }) + + test("downloads into an atomic staging directory and safely reuses a verified snapshot", async () => { + const parent = await temporaryDirectory() + const dataRoot = resolve(parent, "dataset") + const stale = resolve(parent, ".dataset.memorybench-partial-stale") + await mkdir(stale) + await writeFile(resolve(stale, "partial"), "old") + const staleLock = resolve(parent, ".dataset.memorybench-download.lock") + await mkdir(staleLock) + await writeFile( + resolve(staleLock, "owner.json"), + `${JSON.stringify({ + pid: 2147483647, + dataRoot, + operation: "dataset download", + })}\n` + ) + const { spec, files } = fixtureSnapshot() + + const result = await downloadDatasetSnapshotForTesting({ + dataRoot, + spec, + fetchImplementation: fixtureFetch(spec, files), + maxAttempts: 1, + }) + expect(result.status).toBe("downloaded") + expect(await stat(resolve(dataRoot, "questions.jsonl"))).toBeTruthy() + expect(await stat(resolve(dataRoot, LONGMEMEVAL_V2_COMPLETION_MARKER))).toBeTruthy() + expect(await verifyDatasetSnapshot(dataRoot, spec)).toEqual({ + repository: spec.repository, + revision: spec.revision, + files: result.files, + }) + expect( + (await readdir(parent)).some((name) => name.startsWith(".dataset.memorybench-partial-")) + ).toBe(false) + + const reused = await downloadDatasetSnapshotForTesting({ + dataRoot, + spec, + fetchImplementation: (async () => { + throw new Error("verified reuse must not fetch") + }) as unknown as typeof fetch, + }) + expect(reused.status).toBe("already-present") + }) + + test("removes partial work after a checksum failure and never publishes the destination", async () => { + const parent = await temporaryDirectory() + const dataRoot = resolve(parent, "dataset") + const { spec, files } = fixtureSnapshot() + const corrupt = new Map([["questions.jsonl", Buffer.from("corrupt")]]) + + await expect( + downloadDatasetSnapshotForTesting({ + dataRoot, + spec, + fetchImplementation: fixtureFetch(spec, files, corrupt), + maxAttempts: 1, + }) + ).rejects.toThrow("Could not download questions.jsonl") + await expect(stat(dataRoot)).rejects.toMatchObject({ code: "ENOENT" }) + expect( + (await readdir(parent)).filter( + (name) => name.includes("memorybench-partial") || name.includes("memorybench-download.lock") + ) + ).toEqual([]) + }) + + test("refuses to overwrite an existing incomplete destination", async () => { + const parent = await temporaryDirectory() + const dataRoot = resolve(parent, "dataset") + await mkdir(dataRoot) + await writeFile(resolve(dataRoot, "user-file"), "preserve") + const { spec, files } = fixtureSnapshot() + + await expect( + downloadDatasetSnapshotForTesting({ + dataRoot, + spec, + fetchImplementation: fixtureFetch(spec, files), + }) + ).rejects.toThrow("refusing to overwrite") + expect(await readFile(resolve(dataRoot, "user-file"), "utf8")).toBe("preserve") + }) +}) + +function screenshotArchivesFixture(): { + archives: PinnedDatasetFile[] + archiveAdapter: ArchiveAdapter +} { + const contents = new Map([ + ["web_screenshots.tar.gz", Buffer.from("web screenshots")], + ["enterprise_screenshots_base.tar.gz", Buffer.from("enterprise screenshots")], + ]) + const archives = [...contents].map(([name, bytes]) => ({ + relativePath: `trajectory_screenshots/${name}`, + sha256: sha256(bytes), + byteLength: bytes.byteLength, + })) + const archiveAdapter: ArchiveAdapter = { + async list(archivePath) { + return basename(archivePath).startsWith("web_") + ? [ + { path: "./web-trajectory/", type: "directory" }, + { path: "./web-trajectory/0.png", type: "file" }, + ] + : [ + { path: "./enterprise-trajectory/", type: "directory" }, + { path: "./enterprise-trajectory/0.png", type: "file" }, + ] + }, + async extract(archivePath, destination) { + const trajectory = basename(archivePath).startsWith("web_") + ? "web-trajectory" + : "enterprise-trajectory" + await mkdir(resolve(destination, trajectory), { recursive: true }) + await writeFile(resolve(destination, trajectory, "0.png"), PNG_HEADER) + }, + } + return { archives, archiveAdapter } +} + +async function writeScreenshotPreparationFixture( + dataRoot: string, + archives: PinnedDatasetFile[] +): Promise { + await mkdir(resolve(dataRoot, "trajectory_screenshots"), { recursive: true }) + const bytesByName: Record = { + "web_screenshots.tar.gz": Buffer.from("web screenshots"), + "enterprise_screenshots_base.tar.gz": Buffer.from("enterprise screenshots"), + } + for (const archive of archives) { + await writeFile( + resolve(dataRoot, archive.relativePath), + bytesByName[basename(archive.relativePath)] + ) + } + await writeFile( + resolve(dataRoot, "trajectories.jsonl"), + [ + { + id: "web-trajectory", + states: [{ screenshot: "screenshots/web-trajectory/0.png" }], + }, + { + id: "enterprise-trajectory", + states: [{ screenshot: "screenshots/enterprise-trajectory/0.png" }], + }, + ] + .map(jsonLine) + .join("") + ) +} + +describe("LongMemEval-V2 screenshot preparation", () => { + test("extracts safely, builds one atomic runtime view, validates it, and reuses it", async () => { + const dataRoot = await temporaryDirectory() + const { archives, archiveAdapter } = screenshotArchivesFixture() + await writeScreenshotPreparationFixture(dataRoot, archives) + await mkdir(resolve(dataRoot, ".screenshots.memorybench-partial-stale")) + await mkdir( + resolve(dataRoot, "trajectory_screenshots/.web_screenshots.memorybench-partial-stale") + ) + + const prepared = await prepareLongMemEvalV2ScreenshotsForTesting({ + dataRoot, + archives, + archiveAdapter, + mode: "symlink", + }) + expect(prepared.status).toBe("prepared") + expect(prepared.trajectoryDirectories).toBe(2) + expect(prepared.stateScreenshotsValidated).toBe(2) + expect(prepared.symlinked + prepared.copied).toBe(2) + expect(await validatePreparedScreenshotLayout(dataRoot)).toBe(2) + expect((await readdir(dataRoot)).some((name) => name.includes("memorybench-partial"))).toBe( + false + ) + + const reused = await prepareLongMemEvalV2ScreenshotsForTesting({ + dataRoot, + archives, + archiveAdapter, + mode: "symlink", + }) + expect(reused.status).toBe("already-prepared") + expect(reused.stateScreenshotsValidated).toBe(2) + }) + + test("rejects archive traversal before extraction and cleans partial state", async () => { + const dataRoot = await temporaryDirectory() + const { archives } = screenshotArchivesFixture() + await writeScreenshotPreparationFixture(dataRoot, archives) + let extracted = false + const unsafeAdapter: ArchiveAdapter = { + async list() { + return [{ path: "../escape", type: "file" }] + }, + async extract() { + extracted = true + }, + } + + await expect( + prepareLongMemEvalV2ScreenshotsForTesting({ + dataRoot, + archives: [archives[0]], + archiveAdapter: unsafeAdapter, + }) + ).rejects.toThrow("Unsafe dataset path") + expect(extracted).toBe(false) + expect( + (await readdir(resolve(dataRoot, "trajectory_screenshots"))).some((name) => + name.includes("memorybench-partial") + ) + ).toBe(false) + expect( + (await readdir(dataRoot)).some((name) => name.includes("screenshot-preparation.lock")) + ).toBe(false) + }) + + test("does not overwrite an existing incomplete screenshot view", async () => { + const dataRoot = await temporaryDirectory() + const { archives, archiveAdapter } = screenshotArchivesFixture() + await writeScreenshotPreparationFixture(dataRoot, archives) + await mkdir(resolve(dataRoot, "screenshots")) + await writeFile(resolve(dataRoot, "screenshots/user-file"), "preserve") + + await expect( + prepareLongMemEvalV2ScreenshotsForTesting({ + dataRoot, + archives, + archiveAdapter, + }) + ).rejects.toThrow("refusing to overwrite") + expect(await readFile(resolve(dataRoot, "screenshots/user-file"), "utf8")).toBe("preserve") + }) +}) + +describe("LongMemEval-V2 dataset selection and image identity", () => { + test("requires the pinned revision, groups exact ordered haystacks, and samples deterministically", async () => { + const dataRoot = await temporaryDirectory() + await writeDatasetFixture(dataRoot) + expect( + () => + new LongMemEvalV2Dataset({ + dataRoot, + tier: "small", + revision: "moving-main", + }) + ).toThrow(`must be pinned to ${LONGMEMEVAL_V2_PINNED_REVISION}`) + + const dataset = new LongMemEvalV2Dataset({ + dataRoot, + tier: "small", + revision: LONGMEMEVAL_V2_PINNED_REVISION, + }) + await dataset.load() + const selectedA = dataset.selectQuestions({ + perCategory: 1, + seed: "repeatable-seed", + }) + const selectedB = dataset.selectQuestions({ + perCategory: 1, + seed: "repeatable-seed", + }) + expect(selectedA.map((question) => question.id)).toEqual( + selectedB.map((question) => question.id) + ) + const planned = dataset.planQuestions(dataset.getQuestions()) + expect(planned.builds).toHaveLength(2) + expect(planned.builds.map((build) => build.questionIds)).toEqual([ + ["q1", "q2"], + ["q3", "q4"], + ]) + }) + + test("hashes question/state image bytes and invalidates trajectory content when bytes change", async () => { + const dataRoot = await temporaryDirectory() + await writeDatasetFixture(dataRoot) + const dataset = new LongMemEvalV2Dataset({ + dataRoot, + tier: "small", + revision: LONGMEMEVAL_V2_PINNED_REVISION, + }) + await dataset.load() + const planned = dataset.planQuestions([dataset.getQuestions()[0]]) + await dataset.resolveQuestionImages(planned.questions) + expect(planned.questions[0].questionImage?.sha256).toBe( + sha256(Buffer.concat([PNG_HEADER, Buffer.from([3])])) + ) + + const first = (await dataset.loadTrajectories(["t1"])).get("t1")! + await writeFile( + resolve(dataRoot, "screenshots/t1/0.png"), + Buffer.concat([PNG_HEADER, Buffer.from([9])]) + ) + const second = (await dataset.loadTrajectories(["t1"])).get("t1")! + expect(second.states[0].screenshot.sha256).not.toBe(first.states[0].screenshot.sha256) + expect(second.contentHash).not.toBe(first.contentHash) + }) + + test("rejects duplicate haystack members, escaping symlinks, and corrupt image bytes", async () => { + const parent = await temporaryDirectory() + const dataRoot = resolve(parent, "dataset") + await mkdir(dataRoot) + await writeDatasetFixture(dataRoot) + await writeFile( + resolve(dataRoot, "haystacks/lme_v2_small.json"), + `${JSON.stringify({ + q1: ["t1", "t1"], + q2: ["t1"], + q3: ["t2"], + q4: ["t2"], + })}\n` + ) + const duplicated = new LongMemEvalV2Dataset({ + dataRoot, + tier: "small", + revision: LONGMEMEVAL_V2_PINNED_REVISION, + }) + await expect(duplicated.load()).rejects.toThrow("Duplicate trajectory in haystack for q1") + + await writeFile( + resolve(dataRoot, "haystacks/lme_v2_small.json"), + `${JSON.stringify({ + q1: ["t1"], + q2: ["t1"], + q3: ["t2"], + q4: ["t2"], + })}\n` + ) + const outsideImage = resolve(parent, "outside.png") + await writeFile(outsideImage, PNG_HEADER) + await rm(resolve(dataRoot, "question_screenshots/q.png")) + await symlink( + relative(resolve(dataRoot, "question_screenshots"), outsideImage), + resolve(dataRoot, "question_screenshots/q.png") + ) + const escaping = new LongMemEvalV2Dataset({ + dataRoot, + tier: "small", + revision: LONGMEMEVAL_V2_PINNED_REVISION, + }) + await escaping.load() + const escapingPlan = escaping.planQuestions([escaping.getQuestions()[0]]) + await expect(escaping.resolveQuestionImages(escapingPlan.questions)).rejects.toThrow( + "Asset escapes dataset root" + ) + + await rm(resolve(dataRoot, "question_screenshots/q.png")) + await writeFile(resolve(dataRoot, "question_screenshots/q.png"), "not a png") + const corrupt = new LongMemEvalV2Dataset({ + dataRoot, + tier: "small", + revision: LONGMEMEVAL_V2_PINNED_REVISION, + }) + await corrupt.load() + const corruptPlan = corrupt.planQuestions([corrupt.getQuestions()[0]]) + await expect(corrupt.resolveQuestionImages(corruptPlan.questions)).rejects.toThrow( + "Corrupt or mismatched image/png" + ) + }) +}) diff --git a/src/benchmarks/longmemeval-v2/dataset.ts b/src/benchmarks/longmemeval-v2/dataset.ts new file mode 100644 index 0000000..dfd6fc0 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/dataset.ts @@ -0,0 +1,618 @@ +import { createReadStream } from "node:fs" +import { access, lstat, open, readFile, realpath, stat } from "node:fs/promises" +import { createInterface } from "node:readline" +import { extname, isAbsolute, relative, resolve, sep } from "node:path" +import type { AssetRef, DatasetFileManifest, DatasetManifest } from "../../types/migration" +import { stableHash } from "../../core/canonical" +import { + LONGMEMEVAL_V2_QUESTION_IMAGE_COUNT, + LONGMEMEVAL_V2_PINNED_REVISION, + LONGMEMEVAL_V2_REQUIRED_FILES, + sha256FileStreaming, +} from "./source" +import type { + LongMemEvalV2BuildGroup, + LongMemEvalV2Domain, + LongMemEvalV2Question, + LongMemEvalV2QuestionPlan, + LongMemEvalV2Tier, + LongMemEvalV2Trajectory, + PreparedTrajectory, +} from "./types" + +const EXPECTED_QUESTIONS = 451 +const EXPECTED_TRAJECTORIES = 1870 +const EXPECTED_STATES = 48609 +const EXPECTED_ASSETS = EXPECTED_STATES + LONGMEMEVAL_V2_QUESTION_IMAGE_COUNT +const EXPECTED_UNIQUE_BUILDS: Record = { + small: 2, + medium: 447, +} + +export interface LongMemEvalV2DatasetValidationProfile { + expectedCounts: { + questions: number + trajectories: number + states: number + assets: number + uniqueBuilds: Readonly> + } + requiredFiles: Readonly> +} + +export const AUDITED_LONGMEMEVAL_V2_DATASET_VALIDATION: LongMemEvalV2DatasetValidationProfile = { + expectedCounts: { + questions: EXPECTED_QUESTIONS, + trajectories: EXPECTED_TRAJECTORIES, + states: EXPECTED_STATES, + assets: EXPECTED_ASSETS, + uniqueBuilds: EXPECTED_UNIQUE_BUILDS, + }, + requiredFiles: LONGMEMEVAL_V2_REQUIRED_FILES, +} + +const MIME_TYPES: Record = { + ".png": "image/png", + ".jpg": "image/jpeg", + ".jpeg": "image/jpeg", + ".webp": "image/webp", + ".gif": "image/gif", +} + +async function validateImageSignature(path: string, mimeType: string): Promise { + const handle = await open(path, "r") + const header = Buffer.alloc(12) + let bytesRead = 0 + try { + ;({ bytesRead } = await handle.read(header, 0, header.length, 0)) + } finally { + await handle.close() + } + const bytes = header.subarray(0, bytesRead) + const valid = + (mimeType === "image/png" && + bytes.length >= 8 && + bytes.subarray(0, 8).equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]))) || + (mimeType === "image/jpeg" && + bytes.length >= 3 && + bytes[0] === 0xff && + bytes[1] === 0xd8 && + bytes[2] === 0xff) || + (mimeType === "image/webp" && + bytes.length >= 12 && + bytes.subarray(0, 4).toString("ascii") === "RIFF" && + bytes.subarray(8, 12).toString("ascii") === "WEBP") || + (mimeType === "image/gif" && + bytes.length >= 6 && + ["GIF87a", "GIF89a"].includes(bytes.subarray(0, 6).toString("ascii"))) + requireValue(valid, `Corrupt or mismatched ${mimeType} image: ${path}`) +} + +function requireValue(condition: unknown, message: string): asserts condition { + if (!condition) throw new Error(message) +} + +function validateQuestion(value: unknown, lineNumber: number): LongMemEvalV2Question { + requireValue(value && typeof value === "object", `Question line ${lineNumber} is not an object`) + const row = value as Record + requireValue( + typeof row.id === "string" && row.id.length > 0, + `Invalid question id at line ${lineNumber}` + ) + requireValue( + row.domain === "web" || row.domain === "enterprise", + `Invalid question domain for ${row.id}` + ) + requireValue(typeof row.environment === "string", `Invalid environment for ${row.id}`) + requireValue( + typeof row.question_type === "string" && row.question_type.length > 0, + `Invalid question type for ${row.id}` + ) + requireValue( + typeof row.question === "string" && row.question.trim().length > 0, + `Invalid question text for ${row.id}` + ) + requireValue( + row.image === null || typeof row.image === "string", + `Invalid question image for ${row.id}` + ) + requireValue(typeof row.answer === "string", `Invalid answer for ${row.id}`) + requireValue( + typeof row.eval_function === "string" && row.eval_function.length > 0, + `Invalid eval function for ${row.id}` + ) + return row as unknown as LongMemEvalV2Question +} + +function validateTrajectory(value: unknown, lineNumber: number): LongMemEvalV2Trajectory { + requireValue(value && typeof value === "object", `Trajectory line ${lineNumber} is not an object`) + const row = value as Record + requireValue( + typeof row.id === "string" && row.id.length > 0, + `Invalid trajectory id at line ${lineNumber}` + ) + requireValue( + row.domain === "web" || row.domain === "enterprise", + `Invalid trajectory domain for ${row.id}` + ) + requireValue(typeof row.goal === "string", `Invalid goal for ${row.id}`) + requireValue( + typeof row.start_url === "string" && row.start_url.length > 0, + `Invalid start_url for ${row.id}` + ) + requireValue( + row.outcome === null || typeof row.outcome === "string", + `Invalid outcome for ${row.id}` + ) + requireValue(Array.isArray(row.states) && row.states.length > 0, `Invalid states for ${row.id}`) + return row as unknown as LongMemEvalV2Trajectory +} + +function parseJsonLines(text: string, validator: (value: unknown, line: number) => T): T[] { + const output: T[] = [] + for (const [index, line] of text.split(/\r?\n/).entries()) { + if (!line.trim()) continue + output.push(validator(JSON.parse(line), index + 1)) + } + return output +} + +function seededRandom(seed: string): () => number { + let state = Number.parseInt(stableHash(seed).slice(0, 8), 16) >>> 0 + return () => { + state += 0x6d2b79f5 + let value = state + value = Math.imul(value ^ (value >>> 15), value | 1) + value ^= value + Math.imul(value ^ (value >>> 7), value | 61) + return ((value ^ (value >>> 14)) >>> 0) / 4294967296 + } +} + +export interface LongMemEvalV2DatasetOptions { + dataRoot: string + tier: LongMemEvalV2Tier + revision: string + /** + * Explicit validation profile for deterministic fixtures. Production callers + * omit this and remain pinned to the complete audited snapshot. + */ + validationProfile?: LongMemEvalV2DatasetValidationProfile +} + +export class LongMemEvalV2Dataset { + readonly dataRoot: string + readonly tier: LongMemEvalV2Tier + readonly revision: string + private questions: LongMemEvalV2Question[] = [] + private questionsById = new Map() + private haystacks = new Map() + private rootRealPath = "" + private readonly validationProfile: LongMemEvalV2DatasetValidationProfile + + constructor(options: LongMemEvalV2DatasetOptions) { + requireValue( + options.revision === LONGMEMEVAL_V2_PINNED_REVISION, + `LongMemEval-V2 revision must be pinned to ${LONGMEMEVAL_V2_PINNED_REVISION}` + ) + this.dataRoot = resolve(options.dataRoot) + this.tier = options.tier + this.revision = options.revision + this.validationProfile = options.validationProfile ?? AUDITED_LONGMEMEVAL_V2_DATASET_VALIDATION + } + + async load(): Promise { + this.rootRealPath = await realpath(this.dataRoot) + const questionsPath = resolve(this.dataRoot, "questions.jsonl") + const haystackPath = resolve(this.dataRoot, "haystacks", `lme_v2_${this.tier}.json`) + await Promise.all([ + access(questionsPath), + access(haystackPath), + access(resolve(this.dataRoot, "trajectories.jsonl")), + ]) + + this.questions = parseJsonLines(await readFile(questionsPath, "utf8"), validateQuestion) + this.questionsById = new Map() + for (const question of this.questions) { + requireValue(!this.questionsById.has(question.id), `Duplicate question id ${question.id}`) + this.questionsById.set(question.id, question) + } + const rawHaystacks = JSON.parse(await readFile(haystackPath, "utf8")) as unknown + requireValue( + rawHaystacks && typeof rawHaystacks === "object" && !Array.isArray(rawHaystacks), + "Haystack file must be an object" + ) + this.haystacks = new Map() + for (const [questionId, value] of Object.entries(rawHaystacks as Record)) { + requireValue( + this.questionsById.has(questionId), + `Haystack contains unknown question ${questionId}` + ) + requireValue( + Array.isArray(value) && value.every((id) => typeof id === "string" && id.length > 0), + `Invalid haystack for ${questionId}` + ) + const ids = value as string[] + requireValue( + new Set(ids).size === ids.length, + `Duplicate trajectory in haystack for ${questionId}` + ) + this.haystacks.set(questionId, [...ids]) + } + for (const question of this.questions) { + requireValue(this.haystacks.has(question.id), `Missing haystack for question ${question.id}`) + } + } + + getQuestions(domain?: LongMemEvalV2Domain): LongMemEvalV2Question[] { + return this.questions.filter((question) => !domain || question.domain === domain) + } + + selectQuestions( + options: { + domain?: LongMemEvalV2Domain + ids?: string[] + limit?: number + perCategory?: number + seed?: string + } = {} + ): LongMemEvalV2Question[] { + let selected = this.getQuestions(options.domain) + if (options.ids) { + const requested = new Set(options.ids) + selected = selected.filter((question) => requested.has(question.id)) + const found = new Set(selected.map((question) => question.id)) + const missing = [...requested].filter((id) => !found.has(id)) + requireValue(missing.length === 0, `Unknown question ids: ${missing.join(", ")}`) + } + if (options.perCategory !== undefined) { + requireValue( + Number.isInteger(options.perCategory) && options.perCategory > 0, + "perCategory must be a positive integer" + ) + const groups = new Map() + for (const question of selected) { + const group = groups.get(question.question_type) ?? [] + group.push(question) + groups.set(question.question_type, group) + } + selected = [] + const random = seededRandom(options.seed ?? "memorybench-longmemeval-v2") + for (const category of [...groups.keys()].sort()) { + const group = [...groups.get(category)!] + if (options.seed !== undefined) { + for (let index = group.length - 1; index > 0; index -= 1) { + const swap = Math.floor(random() * (index + 1)) + ;[group[index], group[swap]] = [group[swap], group[index]] + } + } + selected.push(...group.slice(0, options.perCategory)) + } + const originalOrder = new Map(this.questions.map((question, index) => [question.id, index])) + selected.sort((left, right) => originalOrder.get(left.id)! - originalOrder.get(right.id)!) + } + if (options.limit !== undefined) { + requireValue( + Number.isInteger(options.limit) && options.limit > 0, + "limit must be a positive integer" + ) + selected = selected.slice(0, options.limit) + } + requireValue(selected.length > 0, "No LongMemEval-V2 questions selected") + return selected + } + + planQuestions(questions: LongMemEvalV2Question[]): { + questions: LongMemEvalV2QuestionPlan[] + builds: LongMemEvalV2BuildGroup[] + } { + const buildMap = new Map() + const questionPlans = questions.map((question) => { + const orderedTrajectoryIds = this.haystacks.get(question.id) + requireValue(orderedTrajectoryIds, `Missing haystack for ${question.id}`) + const haystackHash = stableHash(orderedTrajectoryIds) + const buildKey = `${this.tier}:${question.domain}:${haystackHash}` + const existing = buildMap.get(buildKey) + if (existing) { + existing.questionIds.push(question.id) + } else { + buildMap.set(buildKey, { + buildKey, + domain: question.domain, + tier: this.tier, + orderedTrajectoryIds: [...orderedTrajectoryIds], + questionIds: [question.id], + }) + } + return { question, orderedTrajectoryIds: [...orderedTrajectoryIds], haystackHash, buildKey } + }) + return { questions: questionPlans, builds: [...buildMap.values()] } + } + + async resolveQuestionImages(plans: LongMemEvalV2QuestionPlan[]): Promise { + for (const plan of plans) { + if (plan.question.image) { + plan.questionImage = await this.resolveAsset(plan.question.image, "question-image") + } + } + } + + async resolveAssetReference(pathValue: string, kind: AssetRef["kind"]): Promise { + return this.resolveAsset(pathValue, kind) + } + + async loadTrajectories(ids: string[]): Promise> { + const requested = new Set(ids) + requireValue(requested.size === ids.length, "Requested trajectory ids contain duplicates") + const found = new Map() + const stream = createReadStream(resolve(this.dataRoot, "trajectories.jsonl"), { + encoding: "utf8", + }) + const lines = createInterface({ input: stream, crlfDelay: Infinity }) + let lineNumber = 0 + try { + for await (const line of lines) { + lineNumber += 1 + if (!line.trim()) continue + const idMatch = line.match(/^\s*\{\s*"id"\s*:\s*"([^"]+)"/) + if (idMatch && !requested.has(idMatch[1])) continue + const trajectory = validateTrajectory(JSON.parse(line), lineNumber) + if (!requested.has(trajectory.id)) continue + requireValue(!found.has(trajectory.id), `Duplicate trajectory id ${trajectory.id}`) + found.set(trajectory.id, await this.prepareTrajectory(trajectory)) + if (found.size === requested.size) break + } + } finally { + lines.close() + stream.destroy() + } + const missing = ids.filter((id) => !found.has(id)) + requireValue(missing.length === 0, `Unknown trajectories: ${missing.slice(0, 10).join(", ")}`) + return found + } + + async validateSnapshot(options: { hashAllAssets?: boolean } = {}): Promise<{ + questions: number + trajectories: number + states: number + assets: number + uniqueBuilds: number + }> { + const expected = this.validationProfile.expectedCounts + requireValue( + this.questions.length === expected.questions, + `Expected ${expected.questions} questions, found ${this.questions.length}` + ) + let trajectoryCount = 0 + let stateCount = 0 + let assetCount = 0 + const domains = new Map() + const seen = new Set() + const stream = createReadStream(resolve(this.dataRoot, "trajectories.jsonl"), { + encoding: "utf8", + }) + const lines = createInterface({ input: stream, crlfDelay: Infinity }) + let lineNumber = 0 + for await (const line of lines) { + lineNumber += 1 + if (!line.trim()) continue + const trajectory = validateTrajectory(JSON.parse(line), lineNumber) + requireValue(!seen.has(trajectory.id), `Duplicate trajectory id ${trajectory.id}`) + seen.add(trajectory.id) + domains.set(trajectory.id, trajectory.domain) + trajectoryCount += 1 + stateCount += trajectory.states.length + for (const state of trajectory.states) { + requireValue( + typeof state.screenshot === "string" && state.screenshot.length > 0, + `Missing screenshot for ${trajectory.id}` + ) + if (options.hashAllAssets) { + await this.resolveAsset(state.screenshot, "trajectory-screenshot") + } else { + await access(await this.resolveAssetPath(state.screenshot)) + } + assetCount += 1 + } + } + for (const question of this.questions) { + if (question.image) { + if (options.hashAllAssets) await this.resolveAsset(question.image, "question-image") + else await access(await this.resolveAssetPath(question.image)) + assetCount += 1 + } + for (const trajectoryId of this.haystacks.get(question.id)!) { + requireValue(seen.has(trajectoryId), `Unknown trajectory ${trajectoryId} in ${question.id}`) + requireValue( + domains.get(trajectoryId) === question.domain, + `Cross-domain trajectory ${trajectoryId} in ${question.id}` + ) + } + } + const uniqueBuilds = new Set( + this.questions.map((question) => stableHash(this.haystacks.get(question.id)!)) + ).size + requireValue( + trajectoryCount === expected.trajectories, + `Expected ${expected.trajectories} trajectories, found ${trajectoryCount}` + ) + requireValue( + stateCount === expected.states, + `Expected ${expected.states} states, found ${stateCount}` + ) + requireValue( + assetCount === expected.assets, + `Expected ${expected.assets} assets, found ${assetCount}` + ) + requireValue( + uniqueBuilds === expected.uniqueBuilds[this.tier], + `Expected ${expected.uniqueBuilds[this.tier]} ${this.tier} builds, found ${uniqueBuilds}` + ) + return { + questions: this.questions.length, + trajectories: trajectoryCount, + states: stateCount, + assets: assetCount, + uniqueBuilds, + } + } + + async createManifest(): Promise { + const counts = await this.validateSnapshot({ hashAllAssets: false }) + const filePaths = [ + ...("LICENSE" in this.validationProfile.requiredFiles ? ["LICENSE"] : []), + "questions.jsonl", + "trajectories.jsonl", + `haystacks/lme_v2_${this.tier}.json`, + ] + const files: DatasetFileManifest[] = [] + for (const relativePath of filePaths) { + const absolutePath = resolve(this.dataRoot, relativePath) + const fileStat = await stat(absolutePath) + const hash = await sha256FileStreaming(absolutePath) + const expected = this.validationProfile.requiredFiles[relativePath] + requireValue(expected, `No pinned checksum for ${relativePath}`) + requireValue(hash === expected.sha256, `Pinned checksum mismatch for ${relativePath}`) + if (expected.byteLength !== undefined) { + requireValue( + fileStat.size === expected.byteLength, + `Pinned size mismatch for ${relativePath}` + ) + } + files.push({ + relativePath, + sha256: hash, + byteLength: fileStat.size, + }) + } + const trajectoryOrder: string[] = [] + const stream = createReadStream(resolve(this.dataRoot, "trajectories.jsonl"), { + encoding: "utf8", + }) + const lines = createInterface({ input: stream, crlfDelay: Infinity }) + for await (const line of lines) { + const match = line.match(/^\s*\{\s*"id"\s*:\s*"([^"]+)"/) + if (match) trajectoryOrder.push(match[1]) + } + const payload = { + schemaVersion: 1, + benchmark: "longmemeval-v2", + source: "xiaowu0162/longmemeval-v2", + revision: this.revision, + dataRoot: this.dataRoot, + tier: this.tier, + files, + assets: [] as AssetRef[], + questionOrder: this.questions.map((question) => question.id), + trajectoryOrder, + expectedCounts: counts, + } + return { + ...payload, + fingerprint: stableHash({ + ...payload, + dataRoot: undefined, + assets: payload.assets.map((asset) => ({ ...asset, absolutePath: undefined })), + }), + } + } + + private async prepareTrajectory( + trajectory: LongMemEvalV2Trajectory + ): Promise { + const states = [] + for (const [index, state] of trajectory.states.entries()) { + requireValue( + typeof state.url === "string" && state.url.trim().length > 0, + `Invalid URL for ${trajectory.id} state ${index}` + ) + requireValue( + state.action === null || typeof state.action === "string", + `Invalid action for ${trajectory.id} state ${index}` + ) + const thoughts = state.thought ?? state.thoughts ?? null + requireValue( + thoughts === null || typeof thoughts === "string", + `Invalid thought for ${trajectory.id} state ${index}` + ) + const accessibilityTree = state.accessibility_tree ?? state.text + requireValue( + typeof accessibilityTree === "string", + `Invalid accessibility tree for ${trajectory.id} state ${index}` + ) + requireValue( + typeof state.screenshot === "string" && state.screenshot.length > 0, + `Invalid screenshot for ${trajectory.id} state ${index}` + ) + states.push({ + stateIndex: index, + step: Number.isInteger(state.step) + ? state.step! + : Number.isInteger(state.state_index) + ? state.state_index! + : index, + url: state.url, + action: state.action, + thoughts, + accessibilityTree, + screenshot: await this.resolveAsset(state.screenshot, "trajectory-screenshot"), + }) + } + const contentHash = stableHash({ + id: trajectory.id, + domain: trajectory.domain, + goal: trajectory.goal, + startUrl: trajectory.start_url, + outcome: trajectory.outcome, + states: states.map((state) => ({ + stateIndex: state.stateIndex, + step: state.step, + url: state.url, + action: state.action, + thoughts: state.thoughts, + accessibilityTree: state.accessibilityTree, + screenshotHash: state.screenshot.sha256, + })), + }) + return { + id: trajectory.id, + domain: trajectory.domain, + goal: trajectory.goal, + startUrl: trajectory.start_url, + outcome: trajectory.outcome, + states, + contentHash, + } + } + + private async resolveAssetPath(pathValue: string): Promise { + const candidate = isAbsolute(pathValue) ? pathValue : resolve(this.dataRoot, pathValue) + const resolvedPath = await realpath(candidate) + const relativePath = relative(this.rootRealPath, resolvedPath) + requireValue( + relativePath !== ".." && !relativePath.startsWith(`..${sep}`) && !isAbsolute(relativePath), + `Asset escapes dataset root: ${pathValue}` + ) + const fileStat = await lstat(resolvedPath) + requireValue(fileStat.isFile(), `Asset is not a regular file: ${pathValue}`) + return resolvedPath + } + + private async resolveAsset(pathValue: string, kind: AssetRef["kind"]): Promise { + const absolutePath = await this.resolveAssetPath(pathValue) + const fileStat = await stat(absolutePath) + const hash = await sha256FileStreaming(absolutePath) + const extension = extname(absolutePath).toLowerCase() + const mimeType = MIME_TYPES[extension] + requireValue(mimeType, `Unsupported image type ${extension || "(none)"} for ${pathValue}`) + await validateImageSignature(absolutePath, mimeType) + return { + assetId: `asset-${hash.slice(0, 24)}`, + kind, + absolutePath, + relativePath: relative(this.rootRealPath, absolutePath), + mimeType, + sha256: hash, + byteLength: fileStat.size, + } + } +} diff --git a/src/benchmarks/longmemeval-v2/download.ts b/src/benchmarks/longmemeval-v2/download.ts new file mode 100644 index 0000000..b7fcad7 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/download.ts @@ -0,0 +1,275 @@ +import { randomUUID } from "node:crypto" +import { once } from "node:events" +import { createWriteStream } from "node:fs" +import { lstat, mkdir, readdir, rename, rm, stat, writeFile } from "node:fs/promises" +import { basename, dirname, join, resolve } from "node:path" +import { finished } from "node:stream/promises" +import { + LONGMEMEVAL_V2_SNAPSHOT, + type LongMemEvalV2SnapshotSpec, + type PinnedDatasetFile, + acquireDatasetOperationLock, + parseChecksumManifest, + selectRuntimeSnapshotFiles, + sha256FileStreaming, + validateDatasetRelativePath, + verifyDatasetSnapshot, +} from "./source" + +export const LONGMEMEVAL_V2_COMPLETION_MARKER = ".memorybench-longmemeval-v2-snapshot.json" + +type FetchImplementation = (input: string | URL | Request, init?: RequestInit) => Promise + +export interface DownloadLongMemEvalV2Options { + dataRoot: string + fetchImplementation?: FetchImplementation + maxAttempts?: number +} + +export interface DownloadLongMemEvalV2Result { + status: "downloaded" | "already-present" + dataRoot: string + repository: string + revision: string + files: Array> +} + +interface DownloadSnapshotOptions extends DownloadLongMemEvalV2Options { + spec: LongMemEvalV2SnapshotSpec +} + +function requireValue(condition: unknown, message: string): asserts condition { + if (!condition) throw new Error(message) +} + +function datasetUrl(repository: string, revision: string, relativePath: string): string { + const encodedRepository = repository.split("/").map(encodeURIComponent).join("/") + const encodedRevision = encodeURIComponent(revision) + const encodedPath = validateDatasetRelativePath(relativePath) + .split("/") + .map(encodeURIComponent) + .join("/") + return `https://huggingface.co/datasets/${encodedRepository}/resolve/${encodedRevision}/${encodedPath}?download=true` +} + +async function streamResponseToFile(response: Response, path: string): Promise { + requireValue(response.body, "Download response has no body") + const output = createWriteStream(path, { + flags: "wx", + mode: 0o600, + }) + try { + for await (const chunk of response.body as unknown as AsyncIterable) { + if (!output.write(chunk)) { + await once(output, "drain") + } + } + output.end() + await finished(output) + } catch (error) { + output.destroy() + throw error + } +} + +async function downloadFile(input: { + dataRoot: string + file: PinnedDatasetFile + repository: string + revision: string + fetchImplementation: FetchImplementation + maxAttempts: number +}): Promise> { + const relativePath = validateDatasetRelativePath(input.file.relativePath) + const finalPath = resolve(input.dataRoot, relativePath) + const partialPath = `${finalPath}.download` + await mkdir(dirname(finalPath), { recursive: true }) + await rm(partialPath, { force: true }) + + let lastError: unknown + for (let attempt = 1; attempt <= input.maxAttempts; attempt += 1) { + try { + const response = await input.fetchImplementation( + datasetUrl(input.repository, input.revision, relativePath), + { redirect: "follow" } + ) + if (!response.ok) { + await response.body?.cancel() + throw new Error(`Download failed for ${relativePath}: HTTP ${response.status}`) + } + await streamResponseToFile(response, partialPath) + const fileStat = await stat(partialPath) + if (input.file.byteLength !== undefined) { + requireValue( + fileStat.size === input.file.byteLength, + `Size mismatch for ${relativePath}: expected ${input.file.byteLength}, got ${fileStat.size}` + ) + } + const actualHash = await sha256FileStreaming(partialPath) + requireValue( + actualHash === input.file.sha256, + `Checksum mismatch for ${relativePath}: expected ${input.file.sha256}, got ${actualHash}` + ) + await rename(partialPath, finalPath) + return { + relativePath, + sha256: input.file.sha256, + byteLength: fileStat.size, + } + } catch (error) { + lastError = error + await rm(partialPath, { force: true }) + if (attempt === input.maxAttempts) break + } + } + throw new Error(`Could not download ${relativePath} after ${input.maxAttempts} attempt(s)`, { + cause: lastError, + }) +} + +async function cleanupPartialDirectories(parent: string, prefix: string): Promise { + const entries = await readdir(parent, { withFileTypes: true }) + for (const entry of entries) { + if (entry.isDirectory() && entry.name.startsWith(prefix)) { + await rm(join(parent, entry.name), { recursive: true, force: true }) + } + } +} + +async function writeCompletionMarker( + dataRoot: string, + result: Omit +): Promise { + const markerPath = resolve(dataRoot, LONGMEMEVAL_V2_COMPLETION_MARKER) + const partialPath = `${markerPath}.partial` + await writeFile( + partialPath, + `${JSON.stringify( + { + schemaVersion: 1, + repository: result.repository, + revision: result.revision, + files: result.files, + }, + null, + 2 + )}\n`, + { encoding: "utf8", mode: 0o600 } + ) + await rename(partialPath, markerPath) +} + +async function downloadSnapshot( + options: DownloadSnapshotOptions +): Promise { + const dataRoot = resolve(options.dataRoot) + const parent = dirname(dataRoot) + const directoryName = basename(dataRoot) + requireValue(directoryName !== "." && directoryName !== "", "Invalid data root") + const maxAttempts = options.maxAttempts ?? 3 + requireValue( + Number.isInteger(maxAttempts) && maxAttempts >= 1, + "maxAttempts must be an integer >= 1" + ) + const fetchImplementation = options.fetchImplementation ?? fetch + await mkdir(parent, { recursive: true }) + + const lockPath = join(parent, `.${directoryName}.memorybench-download.lock`) + const releaseLock = await acquireDatasetOperationLock({ + lockPath, + dataRoot, + operation: "dataset download", + }) + const partialPrefix = `.${directoryName}.memorybench-partial-` + let stagingPath: string | undefined + try { + await cleanupPartialDirectories(parent, partialPrefix) + let dataRootExists = false + try { + const existing = await lstat(dataRoot) + dataRootExists = true + requireValue(existing.isDirectory(), `Data root is not a directory: ${dataRoot}`) + const verified = await verifyDatasetSnapshot(dataRoot, options.spec) + await writeCompletionMarker(dataRoot, verified) + return { + status: "already-present", + dataRoot, + ...verified, + } + } catch (error) { + if (dataRootExists || (error as NodeJS.ErrnoException).code !== "ENOENT") { + throw new Error( + `Existing data root is not the pinned complete snapshot; refusing to overwrite ${dataRoot}`, + { cause: error } + ) + } + } + + stagingPath = join(parent, `${partialPrefix}${process.pid}-${randomUUID()}`) + await mkdir(stagingPath, { mode: 0o700 }) + + const checksumManifest = await downloadFile({ + dataRoot: stagingPath, + file: options.spec.checksumManifest, + repository: options.spec.repository, + revision: options.spec.revision, + fetchImplementation, + maxAttempts, + }) + const checksumText = await Bun.file( + resolve(stagingPath, options.spec.checksumManifest.relativePath) + ).text() + const manifestFiles = selectRuntimeSnapshotFiles( + parseChecksumManifest(checksumText, options.spec), + options.spec + ).map((file) => ({ + ...file, + byteLength: options.spec.requiredFiles[file.relativePath]?.byteLength, + })) + const downloaded: Array> = [checksumManifest] + for (const file of [...manifestFiles, ...options.spec.archives]) { + downloaded.push( + await downloadFile({ + dataRoot: stagingPath, + file, + repository: options.spec.repository, + revision: options.spec.revision, + fetchImplementation, + maxAttempts, + }) + ) + } + + const verified = { + repository: options.spec.repository, + revision: options.spec.revision, + files: downloaded, + } + await writeCompletionMarker(stagingPath, verified) + await rename(stagingPath, dataRoot) + stagingPath = undefined + return { + status: "downloaded", + dataRoot, + ...verified, + } + } finally { + if (stagingPath) { + await rm(stagingPath, { recursive: true, force: true }) + } + await releaseLock() + } +} + +export async function downloadLongMemEvalV2Dataset( + options: DownloadLongMemEvalV2Options +): Promise { + return downloadSnapshot({ ...options, spec: LONGMEMEVAL_V2_SNAPSHOT }) +} + +/** A fixture seam for checksum/atomicity tests; production must use the pinned wrapper. */ +export async function downloadDatasetSnapshotForTesting( + options: DownloadSnapshotOptions +): Promise { + return downloadSnapshot(options) +} diff --git a/src/benchmarks/longmemeval-v2/evaluation/aggregate.ts b/src/benchmarks/longmemeval-v2/evaluation/aggregate.ts new file mode 100644 index 0000000..064ccd9 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/aggregate.ts @@ -0,0 +1,167 @@ +import { parseEvaluationSpec } from "./specs" + +export const LONGMEMEVAL_V2_CATEGORY_MAP = { + "static-environment": "static", + "static-environment-abs": "static-abs", + "dynamic-environment": "dynamic", + "dynamic-environment-abs": "dynamic-abs", + procedure: "procedure", + "procedure-abs": "procedure-abs", + "errors-gotchas": "gotchas", +} as const + +export type LongMemEvalV2Category = + (typeof LONGMEMEVAL_V2_CATEGORY_MAP)[keyof typeof LONGMEMEVAL_V2_CATEGORY_MAP] +export type EvaluationStatus = "completed" | "failed" | "pending" | "blocked" + +export interface LongMemEvalV2AggregateRecord { + questionId: string + questionType: string + evalFunction: string + status: EvaluationStatus + score?: 0 | 1 | boolean + isUnknown?: boolean +} + +export interface AccuracyBreakdown { + count: number + pct_correct: number | null + pct_answered_wrong: number | null + pct_unknown: number | null + count_failed_or_incomplete: number +} + +export interface LongMemEvalV2OfficialAggregate { + overall: { + overall_full_set: number + overall_non_abstention_only: number | null + overall_abstention_only: number | null + count_all_questions: number + count_non_abstention: number + count_abstention: number + } + non_abstention_by_category: Record< + "static" | "dynamic" | "procedure" | "gotchas", + AccuracyBreakdown + > + abstention_by_category: Record<"static-abs" | "dynamic-abs" | "procedure-abs", AccuracyBreakdown> + combined_abstention_by_category: Record<"static" | "dynamic" | "procedure", AccuracyBreakdown> + abstention_overall: AccuracyBreakdown + execution: Record +} + +export function categoryFromQuestionType(questionType: string): LongMemEvalV2Category { + const category = + LONGMEMEVAL_V2_CATEGORY_MAP[questionType as keyof typeof LONGMEMEVAL_V2_CATEGORY_MAP] + if (!category) throw new Error(`Unexpected question_type: ${questionType}`) + return category +} + +function numericScore(record: LongMemEvalV2AggregateRecord): 0 | 1 { + if (record.status !== "completed") return 0 + if (record.score === true || record.score === 1) return 1 + if (record.score === false || record.score === 0) return 0 + throw new Error(`Completed question ${record.questionId} is missing a binary score`) +} + +function meanScore(records: LongMemEvalV2AggregateRecord[]): number | null { + if (records.length === 0) return null + return records.reduce((total, record) => total + numericScore(record), 0) / records.length +} + +function breakdown(records: LongMemEvalV2AggregateRecord[]): AccuracyBreakdown { + const count = records.length + if (count === 0) { + return { + count: 0, + pct_correct: null, + pct_answered_wrong: null, + pct_unknown: null, + count_failed_or_incomplete: 0, + } + } + + const unknownCount = records.filter((record) => record.isUnknown === true).length + const correctCount = records.filter( + (record) => numericScore(record) === 1 && record.isUnknown !== true + ).length + const wrongCount = count - correctCount - unknownCount + return { + count, + pct_correct: correctCount / count, + pct_answered_wrong: wrongCount / count, + pct_unknown: unknownCount / count, + count_failed_or_incomplete: records.filter((record) => record.status !== "completed").length, + } +} + +/** + * Aggregate the complete selected question set. Failed, pending, and blocked + * rows remain in the denominator with score zero instead of disappearing from + * the official accuracy. + */ +export function aggregateLongMemEvalV2( + records: LongMemEvalV2AggregateRecord[] +): LongMemEvalV2OfficialAggregate { + if (records.length === 0) throw new Error("No records to aggregate") + const seen = new Set() + for (const record of records) { + if (seen.has(record.questionId)) { + throw new Error(`Duplicate aggregate question id: ${record.questionId}`) + } + seen.add(record.questionId) + categoryFromQuestionType(record.questionType) + } + + const enriched = records.map((record) => ({ + record, + category: categoryFromQuestionType(record.questionType), + isAbstention: parseEvaluationSpec(record.evalFunction).name === "llm_abstention_checker", + })) + const nonAbstention = enriched.filter((item) => !item.isAbstention).map((item) => item.record) + const abstention = enriched.filter((item) => item.isAbstention).map((item) => item.record) + + const categoryRows = (categories: LongMemEvalV2Category[]) => + enriched.filter((item) => categories.includes(item.category)).map((item) => item.record) + + const nonAbstentionByCategory = { + static: breakdown(categoryRows(["static"])), + dynamic: breakdown(categoryRows(["dynamic"])), + procedure: breakdown(categoryRows(["procedure"])), + gotchas: breakdown(categoryRows(["gotchas"])), + } + const abstentionByCategory = { + "static-abs": breakdown(categoryRows(["static-abs"])), + "dynamic-abs": breakdown(categoryRows(["dynamic-abs"])), + "procedure-abs": breakdown(categoryRows(["procedure-abs"])), + } + const combined = { + static: breakdown(categoryRows(["static", "static-abs"])), + dynamic: breakdown(categoryRows(["dynamic", "dynamic-abs"])), + procedure: breakdown(categoryRows(["procedure", "procedure-abs"])), + } + + const execution: Record = { + completed: 0, + failed: 0, + pending: 0, + blocked: 0, + } + for (const record of records) execution[record.status] += 1 + + return { + overall: { + overall_full_set: meanScore(records) ?? 0, + overall_non_abstention_only: meanScore(nonAbstention), + overall_abstention_only: meanScore(abstention), + count_all_questions: records.length, + count_non_abstention: nonAbstention.length, + count_abstention: abstention.length, + }, + non_abstention_by_category: nonAbstentionByCategory, + abstention_by_category: abstentionByCategory, + combined_abstention_by_category: combined, + abstention_overall: breakdown(abstention), + execution, + } +} diff --git a/src/benchmarks/longmemeval-v2/evaluation/answer.ts b/src/benchmarks/longmemeval-v2/evaluation/answer.ts new file mode 100644 index 0000000..a586f50 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/answer.ts @@ -0,0 +1,35 @@ +/** + * LongMemEval-V2 asks readers to put their final answer in the last + * `\boxed{...}` expression. The parser deliberately mirrors the reference + * Python implementation, including nested braces and its fallback behavior. + */ +export function extractBoxedAnswer(text: string): string { + const marker = "\\boxed{" + const markerIndex = text.lastIndexOf(marker) + if (markerIndex === -1) return text.trim() + + let index = markerIndex + marker.length + let depth = 1 + let parsed = "" + while (index < text.length && depth > 0) { + const character = text[index] + if (character === "{") { + depth += 1 + parsed += character + } else if (character === "}") { + depth -= 1 + if (depth === 0) break + parsed += character + } else { + parsed += character + } + index += 1 + } + + const trimmed = parsed.trim() + return trimmed.length > 0 ? trimmed : text.trim() +} + +export function isUnknownAnswer(answer: string): boolean { + return answer.trim().toLowerCase() === "unknown" +} diff --git a/src/benchmarks/longmemeval-v2/evaluation/evaluate.ts b/src/benchmarks/longmemeval-v2/evaluation/evaluate.ts new file mode 100644 index 0000000..58c229f --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/evaluate.ts @@ -0,0 +1,202 @@ +import type { EvaluationArtifact } from "../../../types/migration" +import { stableHash } from "../../../core/canonical" +import { extractBoxedAnswer, isUnknownAnswer } from "./answer" +import { + buildStrictJudgeMessages, + parseStrictJudgeResponse, + StrictJudgeError, + type StrictJudgeCallback, + type StrictJudgeRequest, +} from "./judges" +import { evaluateDeterministicSpec, parseEvaluationSpec, type LlmEvaluatorName } from "./specs" + +export const LONGMEMEVAL_V2_EVALUATOR_IMPLEMENTATION_VERSION = + "longmemeval-v2-official-evaluator-v1" +export const DETERMINISTIC_PROMPT_VERSION = "deterministic-no-prompt-v1" + +export interface LongMemEvalV2EvaluationInput { + questionId: string + questionType: string + question: string + responseText: string + groundTruth: string + evalFunction: string + evaluatorModel?: string + evaluatorSettings?: { + reasoningEffort?: string + maxCompletionTokens?: number + temperature?: number + topP?: number + } + judge?: StrictJudgeCallback + createdAt?: string +} + +function requireNonEmpty(options: Record): boolean { + const value = options.require_non_empty + if (value === undefined) return true + if (typeof value !== "boolean") throw new Error("require_non_empty must be a boolean") + return value +} + +function validateQuestionTypeForJudge(evaluatorName: LlmEvaluatorName, questionType: string): void { + if (evaluatorName === "llm_abstention_checker" && !questionType.includes("-abs")) { + throw new Error( + `llm_abstention_checker question must use an -abs question_type: ${questionType}` + ) + } + if (evaluatorName === "llm_gotchas_checker" && questionType !== "errors-gotchas") { + throw new Error( + `llm_gotchas_checker question must use errors-gotchas question_type: ${questionType}` + ) + } +} + +export function longMemEvalV2EvaluatorFingerprint(input: { + questionId: string + responseText: string + groundTruth: string + evalFunction: string + evaluatorModel?: string + evaluatorSettings?: LongMemEvalV2EvaluationInput["evaluatorSettings"] + promptVersion: string +}): string { + return stableHash({ + ...input, + implementationVersion: LONGMEMEVAL_V2_EVALUATOR_IMPLEMENTATION_VERSION, + }) +} + +function artifact(input: { + evaluation: LongMemEvalV2EvaluationInput + parsedAnswer: string + score: 0 | 1 + promptVersion: string + durationMs: number + request?: StrictJudgeRequest + rawResponse?: unknown + rationale?: string +}): EvaluationArtifact { + const { evaluation } = input + return { + schemaVersion: 1, + questionId: evaluation.questionId, + evaluatorFingerprint: longMemEvalV2EvaluatorFingerprint({ + questionId: evaluation.questionId, + responseText: evaluation.responseText, + groundTruth: evaluation.groundTruth, + evalFunction: evaluation.evalFunction, + evaluatorModel: evaluation.evaluatorModel, + evaluatorSettings: evaluation.evaluatorSettings, + promptVersion: input.promptVersion, + }), + evalFunction: evaluation.evalFunction, + answer: input.parsedAnswer, + groundTruth: evaluation.groundTruth, + score: input.score, + label: input.score === 1 ? "correct" : "incorrect", + evaluatorModel: evaluation.evaluatorModel, + promptVersion: input.promptVersion, + implementationVersion: LONGMEMEVAL_V2_EVALUATOR_IMPLEMENTATION_VERSION, + request: input.request ? { ...input.request } : undefined, + rawResponse: input.rawResponse, + rationale: input.rationale, + durationMs: input.durationMs, + createdAt: evaluation.createdAt ?? new Date().toISOString(), + } +} + +/** + * Dispatch one reader response through the exact evaluator named by the + * dataset row. LLM judging is injected so this module never owns credentials + * or performs an implicit network call. + */ +export async function evaluateLongMemEvalV2( + input: LongMemEvalV2EvaluationInput +): Promise { + const startedAt = performance.now() + const spec = parseEvaluationSpec(input.evalFunction) + const parsedAnswer = extractBoxedAnswer(input.responseText) + const unknown = isUnknownAnswer(parsedAnswer) + + if (spec.name !== "llm_abstention_checker" && spec.name !== "llm_gotchas_checker") { + const evaluatedScore = evaluateDeterministicSpec(spec, parsedAnswer, input.groundTruth) ? 1 : 0 + return artifact({ + evaluation: input, + parsedAnswer, + score: unknown ? 0 : evaluatedScore, + promptVersion: DETERMINISTIC_PROMPT_VERSION, + durationMs: performance.now() - startedAt, + rationale: unknown + ? "Exact UNKNOWN answers are forced incorrect by the benchmark protocol." + : undefined, + }) + } + + validateQuestionTypeForJudge(spec.name, input.questionType) + const prompt = buildStrictJudgeMessages(spec.name, { + question: input.question, + referenceAnswer: input.groundTruth, + modelFullResponse: input.responseText, + modelFinalAnswer: parsedAnswer, + }) + const request: StrictJudgeRequest = { + kind: prompt.kind, + evaluatorName: spec.name, + promptVersion: prompt.promptVersion, + messages: prompt.messages, + model: input.evaluatorModel, + reasoningEffort: input.evaluatorSettings?.reasoningEffort, + maxCompletionTokens: input.evaluatorSettings?.maxCompletionTokens, + temperature: input.evaluatorSettings?.temperature, + topP: input.evaluatorSettings?.topP, + } + + if (requireNonEmpty(spec.options) && (!input.responseText.trim() || !input.groundTruth.trim())) { + return artifact({ + evaluation: input, + parsedAnswer, + score: 0, + promptVersion: prompt.promptVersion, + durationMs: performance.now() - startedAt, + request, + rationale: "Prediction and reference answer must both be non-empty.", + }) + } + if (!input.judge) { + throw new StrictJudgeError(`${spec.name} requires an injected strict judge callback`, request) + } + + let callbackResult: Awaited> + try { + callbackResult = await input.judge(request) + } catch (cause) { + throw new StrictJudgeError("Strict evaluator callback failed", request, { cause }) + } + + const rawResponse = callbackResult.rawResponse ?? callbackResult.text + let parsed + try { + parsed = parseStrictJudgeResponse(callbackResult.text) + } catch (cause) { + throw new StrictJudgeError("Strict evaluator response could not be parsed", request, { + rawResponse, + cause, + }) + } + + return artifact({ + evaluation: input, + parsedAnswer, + score: unknown ? 0 : parsed.label, + promptVersion: prompt.promptVersion, + durationMs: performance.now() - startedAt, + request, + rawResponse, + rationale: unknown + ? `${parsed.rationale}${ + parsed.rationale ? " " : "" + }Exact UNKNOWN answers are forced incorrect by the benchmark protocol.` + : parsed.rationale, + }) +} diff --git a/src/benchmarks/longmemeval-v2/evaluation/evaluation.test.ts b/src/benchmarks/longmemeval-v2/evaluation/evaluation.test.ts new file mode 100644 index 0000000..3c6fb58 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/evaluation.test.ts @@ -0,0 +1,446 @@ +import { describe, expect, test } from "bun:test" +import { + aggregateLongMemEvalV2, + buildStrictJudgeMessages, + evaluateDeterministicSpec, + evaluateLongMemEvalV2, + extractBoxedAnswer, + isUnknownAnswer, + multipleChoiceMatch, + multipleChoiceSetMatch, + normalizePhrase, + normalizedPhraseSetMatch, + normalizedPhraseSetMatchOrdered, + parseEvaluationSpec, + parseStrictJudgeResponse, + splitPhrases, + StrictJudgeError, + type LongMemEvalV2AggregateRecord, + type StrictJudgeRequest, +} from "./index" + +describe("answer parsing", () => { + test("uses the final box and supports nested braces", () => { + expect(extractBoxedAnswer("first \\boxed{wrong}; final \\boxed{A {nested} value}")).toBe( + "A {nested} value" + ) + }) + + test("falls back exactly like the reference parser", () => { + expect(extractBoxedAnswer(" plain answer ")).toBe("plain answer") + expect(extractBoxedAnswer("prefix \\boxed{unfinished")).toBe("unfinished") + expect(extractBoxedAnswer("prefix \\boxed{}")).toBe("prefix \\boxed{}") + }) + + test("UNKNOWN requires an exact case-insensitive answer", () => { + expect(isUnknownAnswer(" UNKNOWN ")).toBeTrue() + expect(isUnknownAnswer("unknown")).toBeTrue() + expect(isUnknownAnswer("unknown because context is missing")).toBeFalse() + }) +}) + +describe("evaluation spec parsing and deterministic evaluators", () => { + const phraseSpec = + "norm_phrase_set_match|lower=true|normalize_hyphen=true|strip_punct=true|separators=,;|require_non_empty=true" + + test("parses every option form used by the dataset", () => { + expect(parseEvaluationSpec(phraseSpec)).toEqual({ + name: "norm_phrase_set_match", + options: { + lower: true, + normalize_hyphen: true, + strip_punct: true, + separators: [",", ";"], + require_non_empty: true, + }, + }) + expect( + parseEvaluationSpec('norm_phrase_set_match|separators=["::",";"]|require_non_empty=false') + .options + ).toEqual({ separators: ["::", ";"], require_non_empty: false }) + }) + + test("rejects empty, unknown, malformed, and duplicate specs", () => { + expect(() => parseEvaluationSpec("")).toThrow("non-empty") + expect(() => parseEvaluationSpec("not_a_real_evaluator")).toThrow("Unknown") + expect(() => parseEvaluationSpec("mc_choice_match|broken")).toThrow("Invalid") + expect(() => + parseEvaluationSpec("mc_choice_match|require_non_empty=true|require_non_empty=false") + ).toThrow("Duplicate") + expect(() => parseEvaluationSpec('norm_phrase_set_match|separators=[1,","]')).toThrow( + "array of strings" + ) + }) + + test("normalizes punctuation, Unicode, hyphens, underscores, and whitespace", () => { + expect(normalizePhrase(" CAFÉ_name—value; X-Y ")).toBe("café namevalue x y") + expect(splitPhrases("Alpha, beta;GAMMA", { separators: [",", ";"] })).toEqual([ + "alpha", + "beta", + "gamma", + ]) + expect(splitPhrases(" Alpha, Beta ", { separators: [] })).toEqual(["alpha beta"]) + }) + + test("normalized phrase-set match requires every whole phrase", () => { + expect(normalizedPhraseSetMatch("Beta then alpha-value!", "alpha value; beta")).toBeTrue() + expect(normalizedPhraseSetMatch("concatenate", "cat")).toBeFalse() + expect(normalizedPhraseSetMatch("café menu", "café")).toBeTrue() + expect(normalizedPhraseSetMatch("", "")).toBeFalse() + expect( + normalizedPhraseSetMatch("", "", { + require_non_empty: false, + }) + ).toBeTrue() + }) + + test("ordered phrase-set match preserves required order and repetition", () => { + expect(normalizedPhraseSetMatchOrdered("first then second", "first;second")).toBeTrue() + expect(normalizedPhraseSetMatchOrdered("second then first", "first;second")).toBeFalse() + expect(normalizedPhraseSetMatchOrdered("one and one again", "one;one")).toBeTrue() + }) + + test("matches single-choice answers with official cleanup", () => { + expect(multipleChoiceMatch("\\boxed{option b.}", "B")).toBeTrue() + expect(multipleChoiceMatch("Choice A.", "a")).toBeTrue() + expect(multipleChoiceMatch("The answer is A", "A")).toBeFalse() + expect(multipleChoiceMatch("", "")).toBeFalse() + }) + + test("matches multi-choice letters as a set and ignores official filler words", () => { + expect(multipleChoiceSetMatch("Final choices: C and A", "AC")).toBeTrue() + expect(multipleChoiceSetMatch("A, A, C", "CA")).toBeTrue() + expect(multipleChoiceSetMatch("A, B, C", "AC")).toBeFalse() + expect(multipleChoiceSetMatch("", "")).toBeFalse() + }) + + test("dispatches all four deterministic evaluator names", () => { + expect( + evaluateDeterministicSpec(parseEvaluationSpec(phraseSpec), "Alpha beta", "alpha;beta") + ).toBeTrue() + expect( + evaluateDeterministicSpec( + parseEvaluationSpec("norm_phrase_set_match_ordered|separators=;|require_non_empty=true"), + "alpha beta", + "alpha;beta" + ) + ).toBeTrue() + expect( + evaluateDeterministicSpec( + parseEvaluationSpec("mc_choice_match|require_non_empty=true"), + "option C.", + "C" + ) + ).toBeTrue() + expect( + evaluateDeterministicSpec( + parseEvaluationSpec("mc_choice_set_match|require_non_empty=true"), + "A and C", + "CA" + ) + ).toBeTrue() + }) +}) + +describe("strict LLM judge prompts and response parsing", () => { + test("builds the strict abstention and gotcha prompts", () => { + const common = { + question: "Question text", + referenceAnswer: "Reference text", + modelFullResponse: "Full response", + modelFinalAnswer: "Final answer", + } + const abstention = buildStrictJudgeMessages("llm_abstention_checker", common) + expect(abstention.kind).toBe("abstention") + expect(abstention.messages[0].content).toContain("flawed-premise") + expect(abstention.messages[1].content).toContain("generic UNKNOWN") + expect(abstention.messages[1].content).toContain("Question text") + + const gotcha = buildStrictJudgeMessages("llm_gotchas_checker", common) + expect(gotcha.kind).toBe("gotcha") + expect(gotcha.messages[0].content).toContain("at least one correct insight") + expect(gotcha.messages[1].content).toContain("any point") + }) + + test("parses strict JSON, fenced JSON, embedded JSON, and JSON-like fallbacks", () => { + expect(parseStrictJudgeResponse('{"label":1,"reason":"correct insight"}')).toEqual({ + label: 1, + rationale: "correct insight", + }) + expect( + parseStrictJudgeResponse('```json\n{"label":"0","reason":"contradiction"}\n```') + ).toEqual({ label: 0, rationale: "contradiction" }) + expect(parseStrictJudgeResponse('prefix {"label": 1, "reason": "ok"} suffix').label).toBe(1) + expect(parseStrictJudgeResponse("{'label': 0, 'reason': 'bad json'}")).toEqual({ + label: 0, + rationale: "{'label': 0, 'reason': 'bad json'}", + }) + expect(parseStrictJudgeResponse("label = 1 because it matches").label).toBe(1) + }) + + test("rejects empty, non-binary, and unparseable judge responses", () => { + expect(() => parseStrictJudgeResponse("")).toThrow("Empty") + expect(() => parseStrictJudgeResponse('{"label":2,"reason":"invalid"}')).toThrow( + "Could not parse" + ) + expect(() => parseStrictJudgeResponse("looks good")).toThrow("Could not parse") + }) +}) + +describe("official evaluator dispatch and artifacts", () => { + test("scores deterministic specs from the parsed boxed answer", async () => { + const result = await evaluateLongMemEvalV2({ + questionId: "q-det", + questionType: "static-environment", + question: "Which values?", + responseText: "Reasoning. \\boxed{Alpha; beta}", + groundTruth: "alpha,beta", + evalFunction: + "norm_phrase_set_match|lower=true|normalize_hyphen=true|strip_punct=true|separators=,;|require_non_empty=true", + createdAt: "2026-07-27T00:00:00.000Z", + }) + expect(result.answer).toBe("Alpha; beta") + expect(result.score).toBe(1) + expect(result.label).toBe("correct") + expect(result.request).toBeUndefined() + expect(result.evaluatorFingerprint).toHaveLength(64) + }) + + test("forces exact UNKNOWN incorrect after official judge dispatch", async () => { + let calls = 0 + const result = await evaluateLongMemEvalV2({ + questionId: "q-unknown", + questionType: "static-environment-abs", + question: "A flawed question", + responseText: "\\boxed{UNKNOWN}", + groundTruth: "The premise is wrong", + evalFunction: "llm_abstention_checker|require_non_empty=true", + judge: async () => { + calls += 1 + return { + text: '{"label":1,"reason":"judge label is overridden"}', + rawResponse: { id: "unknown-judge-response" }, + } + }, + }) + expect(calls).toBe(1) + expect(result.score).toBe(0) + expect(result.rationale).toContain("forced incorrect") + expect(result.rawResponse).toEqual({ id: "unknown-judge-response" }) + }) + + test("injects the strict judge and retains request, raw response, and rationale", async () => { + let captured: StrictJudgeRequest | undefined + const rawResponse = { + id: "judge-response-1", + choices: [{ message: { content: '{"label":1,"reason":"same core flaw"}' } }], + } + const result = await evaluateLongMemEvalV2({ + questionId: "q-abstention", + questionType: "dynamic-environment-abs", + question: "What impossible state occurred?", + responseText: "The premise is inconsistent. \\boxed{No such state occurred}", + groundTruth: "No such state occurred because the premise is inconsistent", + evalFunction: "llm_abstention_checker|require_non_empty=true", + evaluatorModel: "gpt-5.2", + evaluatorSettings: { + reasoningEffort: "medium", + maxCompletionTokens: 2048, + }, + judge: async (request) => { + captured = request + return { + text: '{"label":1,"reason":"same core flaw"}', + rawResponse, + } + }, + }) + expect(captured?.kind).toBe("abstention") + expect(captured?.messages[1].content).toContain("No such state occurred") + expect(result.score).toBe(1) + expect(result.request?.model).toBe("gpt-5.2") + expect(result.rawResponse).toEqual(rawResponse) + expect(result.rationale).toBe("same core flaw") + }) + + test("dispatches the gotcha judge independently", async () => { + const result = await evaluateLongMemEvalV2({ + questionId: "q-gotcha", + questionType: "errors-gotchas", + question: "What is the gotcha?", + responseText: "\\boxed{The visible toggle is read-only}", + groundTruth: "The toggle cannot be edited from this screen", + evalFunction: "llm_gotchas_checker|require_non_empty=true", + judge: async (request) => { + expect(request.kind).toBe("gotcha") + return { text: '{"label":0,"reason":"direction is wrong"}' } + }, + }) + expect(result.score).toBe(0) + expect(result.rationale).toBe("direction is wrong") + }) + + test("retains judge request and raw response on failures", async () => { + try { + await evaluateLongMemEvalV2({ + questionId: "q-bad-judge", + questionType: "errors-gotchas", + question: "What is the gotcha?", + responseText: "A non-empty answer", + groundTruth: "The reference", + evalFunction: "llm_gotchas_checker|require_non_empty=true", + judge: async () => ({ + text: "unparseable", + rawResponse: { provider: "raw-unparseable" }, + }), + }) + throw new Error("expected evaluateLongMemEvalV2 to throw") + } catch (error) { + expect(error).toBeInstanceOf(StrictJudgeError) + const judgeError = error as StrictJudgeError + expect(judgeError.request.kind).toBe("gotcha") + expect(judgeError.rawResponse).toEqual({ provider: "raw-unparseable" }) + } + }) + + test("requires an injected callback and validates judge question categories", async () => { + await expect( + evaluateLongMemEvalV2({ + questionId: "q-missing", + questionType: "procedure-abs", + question: "Question", + responseText: "Answer", + groundTruth: "Reference", + evalFunction: "llm_abstention_checker|require_non_empty=true", + }) + ).rejects.toBeInstanceOf(StrictJudgeError) + + await expect( + evaluateLongMemEvalV2({ + questionId: "q-wrong-category", + questionType: "procedure", + question: "Question", + responseText: "Answer", + groundTruth: "Reference", + evalFunction: "llm_abstention_checker|require_non_empty=true", + judge: async () => ({ text: '{"label":1,"reason":"x"}' }), + }) + ).rejects.toThrow("-abs") + }) +}) + +describe("official aggregation", () => { + const deterministic = + "norm_phrase_set_match|lower=true|normalize_hyphen=true|strip_punct=true|separators=,;|require_non_empty=true" + const records: LongMemEvalV2AggregateRecord[] = [ + { + questionId: "static", + questionType: "static-environment", + evalFunction: deterministic, + status: "completed", + score: 1, + }, + { + questionId: "dynamic", + questionType: "dynamic-environment", + evalFunction: deterministic, + status: "completed", + score: 0, + }, + { + questionId: "procedure", + questionType: "procedure", + evalFunction: deterministic, + status: "pending", + }, + { + questionId: "gotcha", + questionType: "errors-gotchas", + evalFunction: "llm_gotchas_checker|require_non_empty=true", + status: "completed", + score: 1, + }, + { + questionId: "static-abs", + questionType: "static-environment-abs", + evalFunction: "llm_abstention_checker|require_non_empty=true", + status: "completed", + score: 1, + }, + { + questionId: "dynamic-abs", + questionType: "dynamic-environment-abs", + evalFunction: "llm_abstention_checker|require_non_empty=true", + status: "failed", + }, + { + questionId: "procedure-abs", + questionType: "procedure-abs", + evalFunction: "llm_abstention_checker|require_non_empty=true", + status: "completed", + score: 0, + isUnknown: true, + }, + ] + + test("uses every target question in the official denominator", () => { + const aggregate = aggregateLongMemEvalV2(records) + expect(aggregate.overall).toEqual({ + overall_full_set: 3 / 7, + overall_non_abstention_only: 2 / 4, + overall_abstention_only: 1 / 3, + count_all_questions: 7, + count_non_abstention: 4, + count_abstention: 3, + }) + expect(aggregate.execution).toEqual({ + completed: 5, + failed: 1, + pending: 1, + blocked: 0, + }) + }) + + test("preserves category and abstention breakdown semantics", () => { + const aggregate = aggregateLongMemEvalV2(records) + expect(aggregate.non_abstention_by_category.static.pct_correct).toBe(1) + expect(aggregate.non_abstention_by_category.dynamic.pct_answered_wrong).toBe(1) + expect(aggregate.non_abstention_by_category.procedure.count_failed_or_incomplete).toBe(1) + expect(aggregate.non_abstention_by_category.gotchas.pct_correct).toBe(1) + expect(aggregate.abstention_overall).toEqual({ + count: 3, + pct_correct: 1 / 3, + pct_answered_wrong: 1 / 3, + pct_unknown: 1 / 3, + count_failed_or_incomplete: 1, + }) + expect(aggregate.combined_abstention_by_category.static.pct_correct).toBe(1) + expect(aggregate.combined_abstention_by_category.procedure.pct_unknown).toBe(0.5) + }) + + test("rejects invalid aggregate inputs", () => { + expect(() => aggregateLongMemEvalV2([])).toThrow("No records") + expect(() => aggregateLongMemEvalV2([records[0], records[0]])).toThrow("Duplicate") + expect(() => + aggregateLongMemEvalV2([ + { + questionId: "bad-type", + questionType: "unknown-category", + evalFunction: deterministic, + status: "completed", + score: 0, + }, + ]) + ).toThrow("Unexpected question_type") + expect(() => + aggregateLongMemEvalV2([ + { + questionId: "missing-score", + questionType: "static-environment", + evalFunction: deterministic, + status: "completed", + }, + ]) + ).toThrow("missing a binary score") + }) +}) diff --git a/src/benchmarks/longmemeval-v2/evaluation/index.ts b/src/benchmarks/longmemeval-v2/evaluation/index.ts new file mode 100644 index 0000000..5579d2c --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/index.ts @@ -0,0 +1,6 @@ +export * from "./answer" +export * from "./specs" +export * from "./judges" +export * from "./evaluate" +export * from "./aggregate" +export * from "./openai" diff --git a/src/benchmarks/longmemeval-v2/evaluation/judges.ts b/src/benchmarks/longmemeval-v2/evaluation/judges.ts new file mode 100644 index 0000000..eaec29b --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/judges.ts @@ -0,0 +1,214 @@ +import type { LlmEvaluatorName } from "./specs" + +export const ABSTENTION_JUDGE_PROMPT_VERSION = "longmemeval-v2-abstention-v1" +export const GOTCHA_JUDGE_PROMPT_VERSION = "longmemeval-v2-gotcha-v1" + +export interface JudgeMessage { + role: "system" | "user" + content: string +} + +export interface StrictJudgeRequest { + kind: "abstention" | "gotcha" + evaluatorName: LlmEvaluatorName + promptVersion: string + messages: JudgeMessage[] + model?: string + reasoningEffort?: string + maxCompletionTokens?: number + temperature?: number + topP?: number +} + +export interface StrictJudgeCallbackResult { + text: string + rawResponse?: unknown + durationMs?: number +} + +export type StrictJudgeCallback = ( + request: StrictJudgeRequest +) => Promise + +export interface ParsedJudgeResponse { + label: 0 | 1 + rationale: string +} + +const ABSTENTION_SYSTEM_PROMPT = + "You are a strict grader for flawed-premise (abstention) questions. " + + "Judge whether a model answer correctly identifies that the question premise is wrong, " + + "consistent with the reference answer. " + + "If the model follows the flawed premise and gives a concrete answer under that premise, " + + "it must be graded 0. " + + "If the model's final answer is just UNKNOWN / cannot determine without identifying the flaw, grade 0. " + + "If the model is contradictory (both rejects premise and also gives a concrete premise-following answer), grade 0. " + + "Paraphrases are allowed when they preserve the same core flaw described by the reference answer." + +const GOTCHA_SYSTEM_PROMPT = + "You are a strict grader for gotchas-style insight questions. " + + "The reference answer describes the key insight(s). " + + "Grade 1 if the model response includes at least one correct insight point from the reference answer " + + "(paraphrase allowed), and does not contradict any reference point. " + + "If the model's direction is wrong, or it contains contradictions against any reference point, grade 0. " + + "If the model gives multiple points, partial coverage is enough for 1 as long as no contradictions appear." + +function abstentionUserPrompt(input: JudgePromptInput): string { + return ( + "Evaluate whether the model correctly handled a flawed-premise question.\n\n" + + "Question:\n" + + `${input.question}\n\n` + + "Reference answer (the expected flaw-aware conclusion):\n" + + `${input.referenceAnswer}\n\n` + + "Model full response:\n" + + `${input.modelFullResponse}\n\n` + + "Model extracted final answer:\n" + + `${input.modelFinalAnswer}\n\n` + + "Scoring rubric:\n" + + "- Label 1 only if the model clearly identifies the flawed premise and reaches a conclusion " + + "equivalent to the reference answer.\n" + + "- Label 1 also if the model explicitly says it lacks access to the user's specific " + + "live environment/instance/configuration and therefore cannot verify, provided it does not " + + "give a concrete premise-following answer.\n" + + "- Label 0 if the model follows the flawed premise and gives a concrete answer under that premise.\n" + + "- Label 0 for generic UNKNOWN/insufficient-info replies that do not identify a flaw and do not " + + "make the explicit environment-access limitation clear.\n" + + "- Label 0 if contradictory.\n\n" + + "Output JSON only:\n" + + '{"label": 0 or 1, "reason": "short rationale"}' + ) +} + +function gotchaUserPrompt(input: JudgePromptInput): string { + return ( + "Evaluate whether the model answer captures the gotcha insight.\n\n" + + "Question:\n" + + `${input.question}\n\n` + + "Reference answer (insight points):\n" + + `${input.referenceAnswer}\n\n` + + "Model full response:\n" + + `${input.modelFullResponse}\n\n` + + "Model extracted final answer:\n" + + `${input.modelFinalAnswer}\n\n` + + "Scoring rubric:\n" + + "- Label 1 if the model includes at least one correct insight point from the reference answer " + + "(paraphrase acceptable), and does not contradict any reference point.\n" + + "- Label 1 even if only part of a multi-point reference answer is covered, as long as there is " + + "no contradiction.\n" + + "- Label 0 if direction is wrong (suggests opposite action/cause), even if some wording overlaps.\n" + + "- Label 0 if any point in the model response contradicts any reference point.\n" + + "- Label 0 if the response is irrelevant or generic without insight.\n\n" + + "Output JSON only:\n" + + '{"label": 0 or 1, "reason": "short rationale"}' + ) +} + +export interface JudgePromptInput { + question: string + referenceAnswer: string + modelFullResponse: string + modelFinalAnswer: string +} + +export function buildStrictJudgeMessages( + evaluatorName: LlmEvaluatorName, + input: JudgePromptInput +): { + kind: StrictJudgeRequest["kind"] + promptVersion: string + messages: JudgeMessage[] +} { + if (evaluatorName === "llm_abstention_checker") { + return { + kind: "abstention", + promptVersion: ABSTENTION_JUDGE_PROMPT_VERSION, + messages: [ + { role: "system", content: ABSTENTION_SYSTEM_PROMPT }, + { role: "user", content: abstentionUserPrompt(input) }, + ], + } + } + if (evaluatorName === "llm_gotchas_checker") { + return { + kind: "gotcha", + promptVersion: GOTCHA_JUDGE_PROMPT_VERSION, + messages: [ + { role: "system", content: GOTCHA_SYSTEM_PROMPT }, + { role: "user", content: gotchaUserPrompt(input) }, + ], + } + } + throw new Error(`Unsupported strict LLM evaluator: ${evaluatorName}`) +} + +function stripMarkdownCodeFence(text: string): string { + const stripped = text.trim() + if (stripped.startsWith("```") && stripped.endsWith("```")) { + const lines = stripped.split(/\r?\n/u) + if (lines.length >= 3) return lines.slice(1, -1).join("\n").trim() + } + return stripped +} + +function parseLabel(value: unknown): 0 | 1 | undefined { + if (value === 0 || value === "0") return 0 + if (value === 1 || value === "1") return 1 + return undefined +} + +export function parseStrictJudgeResponse(text: string): ParsedJudgeResponse { + const cleaned = stripMarkdownCodeFence(String(text ?? "").trim()) + if (!cleaned) throw new Error("Empty judgement response from evaluator model") + + const objectMatch = cleaned.match(/\{.*\}/su) + if (objectMatch) { + try { + const payload = JSON.parse(objectMatch[0]) as unknown + if (payload && typeof payload === "object" && !Array.isArray(payload)) { + const record = payload as Record + const label = parseLabel(record.label) + if (label !== undefined) { + return { + label, + rationale: + record.reason === null || record.reason === undefined + ? "" + : String(record.reason).trim(), + } + } + } + } catch { + // The reference implementation next accepts JSON-like label output. + } + } + + const patterns = [ + /"label"\s*:\s*([01])/iu, + /'label'\s*:\s*([01])/iu, + /\blabel\b\s*[:=]\s*([01])/iu, + ] + for (const pattern of patterns) { + const match = cleaned.match(pattern) + if (match) { + return { label: Number.parseInt(match[1], 10) as 0 | 1, rationale: cleaned } + } + } + + throw new Error(`Could not parse evaluator binary judgement: ${JSON.stringify(cleaned)}`) +} + +export class StrictJudgeError extends Error { + readonly request: StrictJudgeRequest + readonly rawResponse?: unknown + + constructor( + message: string, + request: StrictJudgeRequest, + options: { rawResponse?: unknown; cause?: unknown } = {} + ) { + super(message, { cause: options.cause }) + this.name = "StrictJudgeError" + this.request = request + this.rawResponse = options.rawResponse + } +} diff --git a/src/benchmarks/longmemeval-v2/evaluation/openai.ts b/src/benchmarks/longmemeval-v2/evaluation/openai.ts new file mode 100644 index 0000000..f89f2d4 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/openai.ts @@ -0,0 +1,77 @@ +import type { StrictJudgeCallback, StrictJudgeRequest } from "./judges" +import { openAICompletionControls } from "../openai-model" + +export interface OpenAIStrictJudgeOptions { + apiKey: string + baseUrl?: string + timeoutMs?: number + maxAttempts?: number +} + +export function createOpenAIStrictJudge(options: OpenAIStrictJudgeOptions): StrictJudgeCallback { + if (!options.apiKey) throw new Error("OPENAI_API_KEY is required for strict LLM evaluation") + return async (request: StrictJudgeRequest) => { + let lastError: Error | undefined + for (let attempt = 1; attempt <= (options.maxAttempts ?? 5); attempt += 1) { + const controller = new AbortController() + const timeout = setTimeout( + () => controller.abort(new Error("Evaluator request timed out")), + options.timeoutMs ?? 10 * 60 * 1000 + ) + try { + const response = await fetch( + `${(options.baseUrl ?? "https://api.openai.com").replace(/\/$/, "")}/v1/chat/completions`, + { + method: "POST", + headers: { + authorization: `Bearer ${options.apiKey}`, + "content-type": "application/json", + }, + body: JSON.stringify({ + model: request.model, + messages: request.messages, + ...openAICompletionControls( + request.model ?? "gpt-5", + request.maxCompletionTokens ?? 2048, + request.reasoningEffort + ), + ...(request.temperature !== undefined ? { temperature: request.temperature } : {}), + ...(request.topP !== undefined ? { top_p: request.topP } : {}), + }), + signal: controller.signal, + } + ) + const raw = (await response.json().catch(() => null)) as Record | null + if (!response.ok) { + const retryable = response.status === 429 || response.status >= 500 + if (!retryable) throw new Error(`Evaluator HTTP ${response.status}`) + lastError = new Error(`Evaluator HTTP ${response.status}`) + } else { + const choices = raw?.choices as Array> | undefined + const message = choices?.[0]?.message as Record | undefined + const content = message?.content + let text = typeof content === "string" ? content.trim() : "" + if (!text && Array.isArray(content)) { + text = content + .map((part) => + part && typeof part === "object" && typeof part.text === "string" ? part.text : "" + ) + .filter(Boolean) + .join("\n") + .trim() + } + if (text) return { text, rawResponse: raw } + lastError = new Error("Evaluator returned empty content") + } + } catch (error) { + lastError = error instanceof Error ? error : new Error(String(error)) + } finally { + clearTimeout(timeout) + } + if (attempt < (options.maxAttempts ?? 5)) { + await Bun.sleep(Math.min(1000 * 2 ** (attempt - 1), 8000)) + } + } + throw lastError ?? new Error("Evaluator failed") + } +} diff --git a/src/benchmarks/longmemeval-v2/evaluation/specs.ts b/src/benchmarks/longmemeval-v2/evaluation/specs.ts new file mode 100644 index 0000000..54228c6 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/evaluation/specs.ts @@ -0,0 +1,267 @@ +export const DETERMINISTIC_EVALUATOR_NAMES = [ + "norm_phrase_set_match", + "norm_phrase_set_match_ordered", + "mc_choice_match", + "mc_choice_set_match", +] as const + +export const LLM_EVALUATOR_NAMES = ["llm_abstention_checker", "llm_gotchas_checker"] as const + +export type DeterministicEvaluatorName = (typeof DETERMINISTIC_EVALUATOR_NAMES)[number] +export type LlmEvaluatorName = (typeof LLM_EVALUATOR_NAMES)[number] +export type LongMemEvalV2EvaluatorName = DeterministicEvaluatorName | LlmEvaluatorName + +export interface ParsedEvaluationSpec { + name: LongMemEvalV2EvaluatorName + options: Record +} + +const EVALUATOR_NAMES = new Set([...DETERMINISTIC_EVALUATOR_NAMES, ...LLM_EVALUATOR_NAMES]) + +const DEFAULT_SEPARATORS = [",", ";"] +const MULTI_SELECT_FILLER_WORDS = new Set([ + "AND", + "ANSWER", + "ANSWERS", + "CHOICE", + "CHOICES", + "FINAL", + "LETTER", + "LETTERS", + "OPTION", + "OPTIONS", +]) + +function parseOptionValue(key: string, value: string): unknown { + const lowered = value.toLowerCase() + if (lowered === "true" || lowered === "false") return lowered === "true" + if (lowered === "none" || lowered === "null") return null + + if (key === "separators" || key === "separator") { + if (value.length === 0) return [] + const stripped = value.trim() + if (stripped.startsWith("[") && stripped.endsWith("]")) { + const parsed = JSON.parse(stripped) as unknown + if (!Array.isArray(parsed) || !parsed.every((item) => typeof item === "string")) { + throw new Error(`${key} must be a JSON array of strings`) + } + return parsed + } + return [...value].filter((character) => character.trim().length > 0) + } + + if (/^[+-]?(?:\d+\.\d*|\.\d+)$/.test(value)) return Number.parseFloat(value) + if (/^[+-]?\d+$/.test(value)) return Number.parseInt(value, 10) + return value +} + +export function parseEvaluationSpec(spec: string): ParsedEvaluationSpec { + if (typeof spec !== "string" || spec.trim().length === 0) { + throw new Error("eval function spec must be a non-empty string") + } + + const parts = spec.split("|").map((part) => part.trim()) + const name = parts[0] + if (!name) throw new Error("eval function spec missing function name") + if (!EVALUATOR_NAMES.has(name)) throw new Error(`Unknown eval function: ${name}`) + + const options: Record = {} + for (const part of parts.slice(1)) { + if (!part) continue + const equalsIndex = part.indexOf("=") + if (equalsIndex === -1) throw new Error(`Invalid eval function option: ${part}`) + const key = part.slice(0, equalsIndex).trim() + const value = part.slice(equalsIndex + 1).trim() + if (!key) throw new Error(`Invalid eval function option: ${part}`) + if (Object.hasOwn(options, key)) throw new Error(`Duplicate eval function option: ${key}`) + options[key] = parseOptionValue(key, value) + } + + return { name: name as LongMemEvalV2EvaluatorName, options } +} + +function booleanOption( + options: Record, + key: string, + defaultValue: boolean +): boolean { + const value = options[key] + if (value === undefined) return defaultValue + if (typeof value !== "boolean") throw new Error(`${key} must be a boolean`) + return value +} + +function stringOption(options: Record, key: string, defaultValue: string): string { + const value = options[key] + if (value === undefined) return defaultValue + if (typeof value !== "string") throw new Error(`${key} must be a string`) + return value +} + +function separatorsOption(options: Record): string[] { + const value = options.separators ?? options.separator + if (value === undefined) return DEFAULT_SEPARATORS + if (!Array.isArray(value) || !value.every((item) => typeof item === "string")) { + throw new Error("separators must be an array of strings") + } + return value +} + +export function normalizePhrase(text: unknown, options: Record = {}): string { + if (text === null || text === undefined) return "" + let normalized = typeof text === "string" ? text : String(text) + if (booleanOption(options, "lower", true)) normalized = normalized.toLowerCase() + if (booleanOption(options, "normalize_hyphen", true)) { + normalized = normalized.replaceAll("-", " ").replaceAll("_", " ") + } + normalized = normalized.replace(/[,;]/gu, " ") + if (booleanOption(options, "strip_punct", true)) { + normalized = normalized.replace(/[^\p{L}\p{N}_\s]/gu, "") + } + return normalized.replace(/\s+/gu, " ").trim() +} + +export function splitPhrases(text: unknown, options: Record = {}): string[] { + if (text === null || text === undefined) return [] + const separators = separatorsOption(options) + if (separators.length === 0) { + const normalized = normalizePhrase(text, options) + return normalized ? [normalized] : [] + } + const escaped = separators.map((separator) => separator.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&")) + return String(text) + .split(new RegExp(escaped.join("|"), "gu")) + .map((part) => normalizePhrase(part, options)) + .filter(Boolean) +} + +function isWordCharacter(character: string | undefined): boolean { + return character !== undefined && /[\p{L}\p{N}_]/u.test(character) +} + +function findWholePhrase(text: string, phrase: string, start: number): number { + let index = text.indexOf(phrase, start) + while (index !== -1) { + const end = index + phrase.length + if (!isWordCharacter(text[index - 1]) && !isWordCharacter(text[end])) return end + index = text.indexOf(phrase, index + 1) + } + return -1 +} + +export function normalizedPhraseSetMatch( + prediction: unknown, + answer: unknown, + options: Record = {} +): boolean { + const normalizedPrediction = normalizePhrase(prediction, options) + const answerPhrases = splitPhrases(answer, options) + if ( + booleanOption(options, "require_non_empty", true) && + (!normalizedPrediction || answerPhrases.length === 0) + ) { + return false + } + for (const phrase of new Set(answerPhrases)) { + if (findWholePhrase(normalizedPrediction, phrase, 0) === -1) return false + } + return true +} + +export function normalizedPhraseSetMatchOrdered( + prediction: unknown, + answer: unknown, + options: Record = {} +): boolean { + const normalizedPrediction = normalizePhrase(prediction, options) + const answerPhrases = splitPhrases(answer, options) + if ( + booleanOption(options, "require_non_empty", true) && + (!normalizedPrediction || answerPhrases.length === 0) + ) { + return false + } + let start = 0 + for (const phrase of answerPhrases) { + const end = findWholePhrase(normalizedPrediction, phrase, start) + if (end === -1) return false + start = end + } + return true +} + +export function multipleChoiceMatch( + prediction: unknown, + answer: unknown, + options: Record = {} +): boolean { + if (prediction === null || prediction === undefined || answer === null || answer === undefined) { + return false + } + const predictionText = String(prediction) + const boxedMatch = predictionText.toLowerCase().match(/\\boxed\{([^}]*)\}/u) + let candidate = boxedMatch?.[1] ?? predictionText + candidate = candidate.replace(/\b(choice|option)\b/giu, "") + for (const character of stringOption(options, "strip_chars", ".")) { + candidate = candidate.replaceAll(character, "") + } + const cleaned = candidate.trim().toUpperCase() + const expected = String(answer).trim().toUpperCase() + if (booleanOption(options, "require_non_empty", true) && (!cleaned || !expected)) { + return false + } + return cleaned === expected +} + +function extractMultiSelectLetters(text: unknown): string[] { + if (text === null || text === undefined) return [] + const chunks = + String(text) + .toUpperCase() + .match(/[A-Z]+/gu) ?? [] + const letters: string[] = [] + for (const chunk of chunks) { + if (!MULTI_SELECT_FILLER_WORDS.has(chunk)) letters.push(...chunk) + } + return letters +} + +export function multipleChoiceSetMatch( + prediction: unknown, + answer: unknown, + options: Record = {} +): boolean { + const predictionLetters = extractMultiSelectLetters(prediction) + const answerLetters = extractMultiSelectLetters(answer) + if ( + booleanOption(options, "require_non_empty", true) && + (predictionLetters.length === 0 || answerLetters.length === 0) + ) { + return false + } + const predictionSet = new Set(predictionLetters) + const answerSet = new Set(answerLetters) + return ( + predictionSet.size === answerSet.size && + [...predictionSet].every((letter) => answerSet.has(letter)) + ) +} + +export function evaluateDeterministicSpec( + spec: ParsedEvaluationSpec, + prediction: unknown, + answer: unknown +): boolean { + switch (spec.name) { + case "norm_phrase_set_match": + return normalizedPhraseSetMatch(prediction, answer, spec.options) + case "norm_phrase_set_match_ordered": + return normalizedPhraseSetMatchOrdered(prediction, answer, spec.options) + case "mc_choice_match": + return multipleChoiceMatch(prediction, answer, spec.options) + case "mc_choice_set_match": + return multipleChoiceSetMatch(prediction, answer, spec.options) + default: + throw new Error(`Eval function ${spec.name} requires an LLM judge`) + } +} diff --git a/src/benchmarks/longmemeval-v2/index.ts b/src/benchmarks/longmemeval-v2/index.ts new file mode 100644 index 0000000..bdb0331 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/index.ts @@ -0,0 +1,9 @@ +export * from "./types" +export * from "./source" +export * from "./download" +export * from "./prepare" +export * from "./dataset" +export * from "./converter" +export * from "./planner" +export * from "./reader" +export * from "./evaluation" diff --git a/src/benchmarks/longmemeval-v2/openai-model.test.ts b/src/benchmarks/longmemeval-v2/openai-model.test.ts new file mode 100644 index 0000000..b52b530 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/openai-model.test.ts @@ -0,0 +1,19 @@ +import { describe, expect, test } from "bun:test" +import { openAICompletionControls, supportsOpenAIReasoning } from "./openai-model" + +describe("LongMemEval-V2 OpenAI model controls", () => { + test("sends reasoning controls only to reasoning models", () => { + expect(supportsOpenAIReasoning("gpt-5")).toBe(true) + expect(supportsOpenAIReasoning("gpt-5-mini")).toBe(true) + expect(supportsOpenAIReasoning("gpt-4.1")).toBe(false) + expect(supportsOpenAIReasoning("gpt-4o-mini")).toBe(false) + + expect(openAICompletionControls("gpt-5", 20_000, "high")).toEqual({ + max_completion_tokens: 20_000, + reasoning_effort: "high", + }) + expect(openAICompletionControls("gpt-4o", 8_000, "high")).toEqual({ + max_tokens: 8_000, + }) + }) +}) diff --git a/src/benchmarks/longmemeval-v2/openai-model.ts b/src/benchmarks/longmemeval-v2/openai-model.ts new file mode 100644 index 0000000..91e1339 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/openai-model.ts @@ -0,0 +1,19 @@ +export type LongMemEvalV2ReasoningEffort = "none" | "minimal" | "low" | "medium" | "high" | "xhigh" + +export function supportsOpenAIReasoning(model: string): boolean { + return /^(?:gpt-5|o1|o3|o4)/i.test(model.trim()) +} + +export function openAICompletionControls( + model: string, + maxCompletionTokens: number, + reasoningEffort?: string +): Record { + if (supportsOpenAIReasoning(model)) { + return { + max_completion_tokens: maxCompletionTokens, + ...(reasoningEffort ? { reasoning_effort: reasoningEffort } : {}), + } + } + return { max_tokens: maxCompletionTokens } +} diff --git a/src/benchmarks/longmemeval-v2/planner.ts b/src/benchmarks/longmemeval-v2/planner.ts new file mode 100644 index 0000000..14df177 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/planner.ts @@ -0,0 +1,149 @@ +import type { + DatasetManifest, + MemoryBuildPlan, + MetadataValue, + PhysicalDocument, +} from "../../types/migration" +import { + createPhysicalDocuments, + DOCUMENT_PLAN_VERSION, + SPLITTER_VERSION, + validateDocumentPlan, +} from "../../core/document-plan" +import { buildFingerprint } from "../../core/fingerprints" +import { stableHash } from "../../core/canonical" +import { structuredAccessibilityConverter } from "./converter" +import type { LongMemEvalV2BuildGroup, PreparedTrajectory } from "./types" + +export interface LongMemEvalV2BuildPlanningOptions { + provider: string + providerBuildConfig: { + dreaming: "instant" + rootFilterMode: "self" + maxDocumentChars: number + [key: string]: unknown + } + containerPrefix?: string +} + +function requireIdentifier(value: string, field: string): void { + if (!/^[A-Za-z0-9_-]+$/.test(value) || value.length > 40) { + throw new Error(`${field} must match [A-Za-z0-9_-]+ and be <= 40 characters`) + } +} + +export function planLongMemEvalV2Build(input: { + manifest: DatasetManifest + group: LongMemEvalV2BuildGroup + trajectories: Map + options: LongMemEvalV2BuildPlanningOptions +}): MemoryBuildPlan { + const { manifest, group, trajectories, options } = input + if (manifest.benchmark !== "longmemeval-v2") { + throw new Error(`Unexpected benchmark manifest: ${manifest.benchmark}`) + } + const orderedTrajectories = group.orderedTrajectoryIds.map((trajectoryId) => { + const trajectory = trajectories.get(trajectoryId) + if (!trajectory) throw new Error(`Missing prepared trajectory ${trajectoryId}`) + if (trajectory.domain !== group.domain) { + throw new Error(`Cross-domain trajectory ${trajectoryId} in ${group.domain} build`) + } + return trajectory + }) + const documentPlans = orderedTrajectories.map((trajectory) => { + const plan = structuredAccessibilityConverter.convert(trajectory, undefined) + return validateDocumentPlan({ + plan, + converter: structuredAccessibilityConverter, + trajectory, + context: undefined, + }) + }) + const sourceContentHashes = orderedTrajectories.map((trajectory) => trajectory.contentHash) + const fingerprint = buildFingerprint({ + benchmark: "longmemeval-v2", + datasetFingerprint: manifest.fingerprint, + tier: group.tier, + domain: group.domain, + orderedSourceIds: group.orderedTrajectoryIds, + sourceContentHashes, + converter: { + name: structuredAccessibilityConverter.name, + version: structuredAccessibilityConverter.version, + sourceHash: structuredAccessibilityConverter.sourceHash, + }, + validatedPlanHashes: documentPlans.map((plan) => plan.planHash), + provider: options.provider, + providerBuildConfig: options.providerBuildConfig, + documentPlanVersion: DOCUMENT_PLAN_VERSION, + splitterVersion: SPLITTER_VERSION, + }) + const prefix = options.containerPrefix ?? "lme-v2" + requireIdentifier(prefix, "containerPrefix") + const haystackHash = stableHash(group.orderedTrajectoryIds) + const containerTag = `${prefix}-${group.tier}-${group.domain}-${haystackHash.slice(0, 12)}-${fingerprint.slice(0, 12)}` + if (containerTag.length > 100) throw new Error("Computed container tag exceeds 100 characters") + const buildId = `mb-${fingerprint.slice(0, 24)}` + const trajectoryOrder = new Map( + group.orderedTrajectoryIds.map((trajectoryId, index) => [trajectoryId, index]) + ) + const physicalDocuments = documentPlans.flatMap((plan) => + createPhysicalDocuments({ + plan, + buildFingerprint: fingerprint, + maxDocumentChars: options.providerBuildConfig.maxDocumentChars, + }) + ) + const documents = physicalDocuments.map((document): PhysicalDocument => { + const infrastructure: Record = { + benchmark: "longmemeval-v2", + adapterSchemaVersion: 1, + buildFingerprint: fingerprint, + runFingerprint: fingerprint, + tier: group.tier, + domain: group.domain, + haystackHash, + trajectoryId: document.trajectoryId, + trajectoryOrder: trajectoryOrder.get(document.trajectoryId)!, + documentType: document.documentType, + documentOrdinal: document.documentOrdinal, + partIndex: document.partIndex, + partCount: document.partCount, + contentHash: document.contentHash, + logicalDocumentId: document.logicalDocumentId, + } + if (document.stateIndex !== undefined) infrastructure.stateIndex = document.stateIndex + if (document.step !== undefined) infrastructure.step = document.step + if (document.screenshotRef) { + infrastructure.screenshotPath = document.screenshotRef.relativePath + infrastructure.screenshotSha256 = document.screenshotRef.sha256 + infrastructure.screenshotMimeType = document.screenshotRef.mimeType + infrastructure.screenshotByteLength = document.screenshotRef.byteLength + } + return { + ...document, + metadata: { ...document.metadata, ...infrastructure }, + } + }) + return { + schemaVersion: 1, + buildId, + benchmark: "longmemeval-v2", + provider: options.provider, + datasetFingerprint: manifest.fingerprint, + tier: group.tier, + domain: group.domain, + orderedSourceIds: [...group.orderedTrajectoryIds], + sourceContentHashes, + converter: { + name: structuredAccessibilityConverter.name, + version: structuredAccessibilityConverter.version, + sourceHash: structuredAccessibilityConverter.sourceHash, + }, + providerBuildConfig: { ...options.providerBuildConfig }, + buildFingerprint: fingerprint, + containerTag, + documentPlans, + documents, + } +} diff --git a/src/benchmarks/longmemeval-v2/prepare.ts b/src/benchmarks/longmemeval-v2/prepare.ts new file mode 100644 index 0000000..d692072 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/prepare.ts @@ -0,0 +1,436 @@ +import { randomUUID } from "node:crypto" +import { createReadStream } from "node:fs" +import { cp, lstat, mkdir, readdir, realpath, rename, rm, stat, symlink } from "node:fs/promises" +import { createInterface } from "node:readline" +import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path" +import { + LONGMEMEVAL_V2_ARCHIVES, + type PinnedDatasetFile, + acquireDatasetOperationLock, + sha256FileStreaming, + validateDatasetRelativePath, +} from "./source" + +export type ScreenshotPreparationMode = "symlink" | "copy" + +export interface ArchiveEntry { + path: string + type: "file" | "directory" +} + +export interface ArchiveAdapter { + list(archivePath: string): Promise + extract(archivePath: string, destination: string): Promise +} + +export interface PrepareLongMemEvalV2ScreenshotsOptions { + dataRoot: string + mode?: ScreenshotPreparationMode + archiveAdapter?: ArchiveAdapter +} + +export interface PrepareLongMemEvalV2ScreenshotsResult { + status: "prepared" | "already-prepared" + dataRoot: string + screenshotsRoot: string + sourceDirectories: string[] + trajectoryDirectories: number + stateScreenshotsValidated: number + symlinked: number + copied: number +} + +interface PrepareScreenshotsOptions extends PrepareLongMemEvalV2ScreenshotsOptions { + archives: readonly PinnedDatasetFile[] +} + +function requireValue(condition: unknown, message: string): asserts condition { + if (!condition) throw new Error(message) +} + +function normalizeArchiveEntryPath(rawPath: string): string | undefined { + requireValue(rawPath.length > 0, "Archive contains an empty path") + requireValue(!rawPath.includes("\0"), "Archive path contains a null byte") + requireValue(!rawPath.includes("\\"), `Archive path uses a backslash: ${rawPath}`) + requireValue(!isAbsolute(rawPath), `Archive path is absolute: ${rawPath}`) + let candidate = rawPath + while (candidate.startsWith("./")) candidate = candidate.slice(2) + candidate = candidate.replace(/\/+$/, "") + if (!candidate) return undefined + return validateDatasetRelativePath(candidate) +} + +async function runTar(args: string[]): Promise { + const processHandle = Bun.spawn({ + cmd: ["tar", ...args], + stdout: "pipe", + stderr: "pipe", + }) + const [stdout, stderr, exitCode] = await Promise.all([ + new Response(processHandle.stdout).text(), + new Response(processHandle.stderr).text(), + processHandle.exited, + ]) + if (exitCode !== 0) { + throw new Error(`tar ${args.join(" ")} failed with exit ${exitCode}: ${stderr.trim()}`) + } + return stdout +} + +export const systemTarArchiveAdapter: ArchiveAdapter = { + async list(archivePath: string): Promise { + const [namesOutput, verboseOutput] = await Promise.all([ + runTar(["-tf", archivePath]), + runTar(["-tvf", archivePath]), + ]) + const names = namesOutput.split(/\r?\n/).filter(Boolean) + const verbose = verboseOutput.split(/\r?\n/).filter(Boolean) + requireValue(names.length === verbose.length, `Archive listing mismatch for ${archivePath}`) + return names.flatMap((path, index) => { + const typeCharacter = verbose[index][0] + requireValue( + typeCharacter === "-" || typeCharacter === "d", + `Archive contains unsupported link or special entry: ${path}` + ) + const normalized = normalizeArchiveEntryPath(path) + return normalized + ? [{ path: normalized, type: typeCharacter === "d" ? "directory" : "file" }] + : [] + }) + }, + + async extract(archivePath: string, destination: string): Promise { + await runTar([ + "--no-same-owner", + "--no-same-permissions", + "-xf", + archivePath, + "-C", + destination, + ]) + }, +} + +async function validateExtractedTree(root: string): Promise { + const rootRealPath = await realpath(root) + const pending = [rootRealPath] + while (pending.length > 0) { + const directory = pending.pop()! + for (const entry of await readdir(directory, { withFileTypes: true })) { + const path = join(directory, entry.name) + const metadata = await lstat(path) + requireValue( + !metadata.isSymbolicLink(), + `Extracted archive contains a symbolic link: ${path}` + ) + requireValue( + metadata.isDirectory() || metadata.isFile(), + `Extracted archive contains a special file: ${path}` + ) + const pathReal = await realpath(path) + const relativePath = relative(rootRealPath, pathReal) + requireValue( + relativePath !== ".." && !relativePath.startsWith(`..${sep}`) && !isAbsolute(relativePath), + `Extracted archive escaped its destination: ${path}` + ) + if (metadata.isDirectory()) pending.push(path) + } + } +} + +async function verifyArchive(dataRoot: string, archive: PinnedDatasetFile): Promise { + const relativePath = validateDatasetRelativePath(archive.relativePath) + const archivePath = resolve(dataRoot, relativePath) + const metadata = await lstat(archivePath) + requireValue( + metadata.isFile() && !metadata.isSymbolicLink(), + `Archive is not a regular file: ${relativePath}` + ) + if (archive.byteLength !== undefined) { + requireValue(metadata.size === archive.byteLength, `Archive size mismatch for ${relativePath}`) + } + requireValue( + (await sha256FileStreaming(archivePath)) === archive.sha256, + `Archive checksum mismatch for ${relativePath}` + ) + return archivePath +} + +async function extractArchiveAtomically(input: { + dataRoot: string + archive: PinnedDatasetFile + archiveAdapter: ArchiveAdapter +}): Promise { + const archivePath = await verifyArchive(input.dataRoot, input.archive) + const sourceRoot = dirname(archivePath) + const sourceName = basename(archivePath).replace(/\.tar\.gz$/i, "") + requireValue(sourceName !== basename(archivePath), `Unsupported archive name ${archivePath}`) + const destination = resolve(sourceRoot, sourceName) + + let destinationExists = false + try { + const existing = await lstat(destination) + destinationExists = true + requireValue(existing.isDirectory(), `Archive destination is not a directory: ${destination}`) + await validateExtractedTree(destination) + return destination + } catch (error) { + if (destinationExists || (error as NodeJS.ErrnoException).code !== "ENOENT") { + throw error + } + } + + const entries = await input.archiveAdapter.list(archivePath) + requireValue(entries.length > 0, `Archive is empty: ${archivePath}`) + const seen = new Set() + for (const entry of entries) { + const path = normalizeArchiveEntryPath(entry.path) + if (!path) continue + requireValue(!seen.has(path), `Archive has duplicate entry ${path}`) + seen.add(path) + requireValue( + entry.type === "file" || entry.type === "directory", + `Archive has unsupported entry ${path}` + ) + } + + const partial = resolve( + sourceRoot, + `.${sourceName}.memorybench-partial-${process.pid}-${randomUUID()}` + ) + await mkdir(partial, { mode: 0o700 }) + try { + await input.archiveAdapter.extract(archivePath, partial) + await validateExtractedTree(partial) + await rename(partial, destination) + return destination + } catch (error) { + await rm(partial, { recursive: true, force: true }) + throw error + } +} + +async function linkOrCopyDirectory( + source: string, + destination: string, + mode: ScreenshotPreparationMode +): Promise<"symlinked" | "copied"> { + if (mode === "symlink") { + try { + await symlink(relative(dirname(destination), source), destination, "dir") + return "symlinked" + } catch (error) { + const code = (error as NodeJS.ErrnoException).code + if (!["EPERM", "EACCES", "ENOTSUP", "EINVAL"].includes(code ?? "")) { + throw error + } + } + } + await cp(source, destination, { + recursive: true, + errorOnExist: true, + force: false, + }) + return "copied" +} + +async function cleanupPreparationPartials( + dataRoot: string, + archives: readonly PinnedDatasetFile[] +): Promise { + for (const entry of await readdir(dataRoot, { withFileTypes: true })) { + if (entry.isDirectory() && entry.name.startsWith(".screenshots.memorybench-partial-")) { + await rm(resolve(dataRoot, entry.name), { recursive: true, force: true }) + } + } + for (const archive of archives) { + const archivePath = resolve(dataRoot, validateDatasetRelativePath(archive.relativePath)) + const sourceRoot = dirname(archivePath) + const sourceName = basename(archivePath).replace(/\.tar\.gz$/i, "") + const prefix = `.${sourceName}.memorybench-partial-` + for (const entry of await readdir(sourceRoot, { withFileTypes: true })) { + if (entry.isDirectory() && entry.name.startsWith(prefix)) { + await rm(resolve(sourceRoot, entry.name), { + recursive: true, + force: true, + }) + } + } + } +} + +export async function validatePreparedScreenshotLayout( + dataRoot: string, + screenshotsRoot = resolve(dataRoot, "screenshots") +): Promise { + const root = await realpath(dataRoot) + const trajectoryPath = resolve(root, "trajectories.jsonl") + const stream = createReadStream(trajectoryPath, { encoding: "utf8" }) + const lines = createInterface({ input: stream, crlfDelay: Infinity }) + let count = 0 + let lineNumber = 0 + try { + for await (const line of lines) { + lineNumber += 1 + if (!line.trim()) continue + const trajectory = JSON.parse(line) as Record + requireValue(Array.isArray(trajectory.states), `Trajectory line ${lineNumber} has no states`) + for (const [stateIndex, stateValue] of trajectory.states.entries()) { + requireValue( + stateValue && typeof stateValue === "object", + `Invalid state ${stateIndex} on trajectory line ${lineNumber}` + ) + const screenshot = (stateValue as Record).screenshot + requireValue( + typeof screenshot === "string" && screenshot.length > 0, + `Missing screenshot at state ${stateIndex} on trajectory line ${lineNumber}` + ) + const relativePath = validateDatasetRelativePath(screenshot) + requireValue( + relativePath.startsWith("screenshots/"), + `Trajectory screenshot is outside screenshots/: ${relativePath}` + ) + const path = resolve(screenshotsRoot, relativePath.slice("screenshots/".length)) + const fileMetadata = await stat(path) + requireValue(fileMetadata.isFile(), `Screenshot is not a file: ${relativePath}`) + const resolvedPath = await realpath(path) + const withinDataset = relative(root, resolvedPath) + requireValue( + withinDataset !== ".." && + !withinDataset.startsWith(`..${sep}`) && + !isAbsolute(withinDataset), + `Screenshot resolves outside the dataset: ${relativePath}` + ) + count += 1 + } + } + } finally { + lines.close() + stream.destroy() + } + return count +} + +async function prepareScreenshots( + options: PrepareScreenshotsOptions +): Promise { + const dataRoot = resolve(options.dataRoot) + const mode = options.mode ?? "symlink" + requireValue(mode === "symlink" || mode === "copy", "mode must be symlink or copy") + const archiveAdapter = options.archiveAdapter ?? systemTarArchiveAdapter + const rootMetadata = await lstat(dataRoot) + requireValue(rootMetadata.isDirectory(), `Data root is not a directory: ${dataRoot}`) + const screenshotsRoot = resolve(dataRoot, "screenshots") + + const lockPath = resolve(dataRoot, ".memorybench-screenshot-preparation.lock") + const releaseLock = await acquireDatasetOperationLock({ + lockPath, + dataRoot, + operation: "screenshot preparation", + }) + + let partialRoot: string | undefined + try { + await cleanupPreparationPartials(dataRoot, options.archives) + const sourceDirectories: string[] = [] + for (const archive of options.archives) { + sourceDirectories.push( + await extractArchiveAtomically({ + dataRoot, + archive, + archiveAdapter, + }) + ) + } + + let screenshotsRootExists = false + try { + const existing = await lstat(screenshotsRoot) + screenshotsRootExists = true + requireValue( + existing.isDirectory(), + `Screenshots root is not a directory: ${screenshotsRoot}` + ) + const stateScreenshotsValidated = await validatePreparedScreenshotLayout(dataRoot) + return { + status: "already-prepared", + dataRoot, + screenshotsRoot, + sourceDirectories, + trajectoryDirectories: (await readdir(screenshotsRoot, { withFileTypes: true })).filter( + (entry) => entry.isDirectory() || entry.isSymbolicLink() + ).length, + stateScreenshotsValidated, + symlinked: 0, + copied: 0, + } + } catch (error) { + if (screenshotsRootExists || (error as NodeJS.ErrnoException).code !== "ENOENT") { + throw new Error( + `Existing screenshots root is incomplete; refusing to overwrite ${screenshotsRoot}`, + { cause: error } + ) + } + } + + partialRoot = resolve( + dataRoot, + `.screenshots.memorybench-partial-${process.pid}-${randomUUID()}` + ) + await mkdir(partialRoot, { mode: 0o700 }) + let symlinked = 0 + let copied = 0 + let trajectoryDirectories = 0 + const seen = new Set() + for (const sourceDirectory of sourceDirectories) { + for (const entry of (await readdir(sourceDirectory, { withFileTypes: true })).sort( + (left, right) => left.name.localeCompare(right.name) + )) { + if (!entry.isDirectory()) continue + requireValue( + !seen.has(entry.name), + `Duplicate trajectory screenshot directory: ${entry.name}` + ) + seen.add(entry.name) + const source = resolve(sourceDirectory, entry.name) + const destination = resolve(partialRoot, entry.name) + const result = await linkOrCopyDirectory(source, destination, mode) + if (result === "symlinked") symlinked += 1 + else copied += 1 + trajectoryDirectories += 1 + } + } + requireValue(trajectoryDirectories > 0, "No trajectory screenshot directories were extracted") + const stateScreenshotsValidated = await validatePreparedScreenshotLayout(dataRoot, partialRoot) + await rename(partialRoot, screenshotsRoot) + partialRoot = undefined + return { + status: "prepared", + dataRoot, + screenshotsRoot, + sourceDirectories, + trajectoryDirectories, + stateScreenshotsValidated, + symlinked, + copied, + } + } finally { + if (partialRoot) { + await rm(partialRoot, { recursive: true, force: true }) + } + await releaseLock() + } +} + +export async function prepareLongMemEvalV2Screenshots( + options: PrepareLongMemEvalV2ScreenshotsOptions +): Promise { + return prepareScreenshots({ ...options, archives: LONGMEMEVAL_V2_ARCHIVES }) +} + +/** A fixture seam for archive-safety and atomicity tests. */ +export async function prepareLongMemEvalV2ScreenshotsForTesting( + options: PrepareScreenshotsOptions +): Promise { + return prepareScreenshots(options) +} diff --git a/src/benchmarks/longmemeval-v2/reader.ts b/src/benchmarks/longmemeval-v2/reader.ts new file mode 100644 index 0000000..94bd237 --- /dev/null +++ b/src/benchmarks/longmemeval-v2/reader.ts @@ -0,0 +1,384 @@ +import { readFile } from "node:fs/promises" +import { getEncoding } from "js-tiktoken" +import type { + AssetRef, + QueryArtifact, + ReaderArtifact, + ReaderMessagePart, +} from "../../types/migration" +import { ArtifactStore } from "../../core/artifact-store" +import { readerFingerprint } from "../../core/fingerprints" +import { openAICompletionControls } from "./openai-model" + +export const READER_PROMPT_VERSION = "longmemeval-v2-reader-v1" +export const CONTEXT_BUDGET_VERSION = "gpt5-o200k-conservative-images-v1" + +export const DOMAIN_SYSTEM_PROMPTS = { + web: + "You are an experienced colleague in a web browsing environment that has " + + "a customized magento-based shopping website, a customized magento-based " + + "shopping admin cms website, as well as a customized forum website based " + + "on reddit/postmill. Answer based on your memory of the environment. " + + "If you do not know the answer, output exactly \\boxed{UNKNOWN}. " + + "Do not guess. Never attempt to guess an answer if you are not sure. " + + "If you believe the question's construction/premise is wrong, provide an " + + "explanation in \\boxed{} explaining why the question is flawed.", + enterprise: + "You are an experienced colleague working in a customized ServiceNow " + + "environment. Answer based on your memory of the environment. " + + "If you do not know the answer, output exactly \\boxed{UNKNOWN}. " + + "Do not guess. Never attempt to guess an answer if you are not sure. " + + "If you believe the question's construction/premise is wrong, provide an " + + "explanation in \\boxed{} explaining why the question is flawed.", +} as const + +export interface ReaderSettings { + model: string + reasoningEffort: "none" | "minimal" | "low" | "medium" | "high" | "xhigh" + maxCompletionTokens: number + maxContextTokens: number + evidenceTopK: number + maxImages: number + maxImageBytes: number + malformedResponseAttempts: number +} + +export interface ReaderModelRequest { + model: string + reasoningEffort: ReaderSettings["reasoningEffort"] + maxCompletionTokens: number + systemPrompt: string + parts: ReaderMessagePart[] +} + +export interface ReaderModelResponse { + text: string + usage?: Record + raw: unknown +} + +export interface ReaderModelClient { + generate(request: ReaderModelRequest, signal?: AbortSignal): Promise +} + +export interface ContextTokenCounter { + readonly version: string + count(systemPrompt: string, parts: ReaderMessagePart[]): number +} + +export class Gpt5ContextTokenCounter implements ContextTokenCounter { + readonly version = CONTEXT_BUDGET_VERSION + private readonly encoding = getEncoding("o200k_base") + + constructor(private readonly tokensPerImage = 1700) {} + + count(systemPrompt: string, parts: ReaderMessagePart[]): number { + let count = this.encoding.encode(systemPrompt).length + 12 + for (const part of parts) { + count += + part.type === "text" ? this.encoding.encode(part.text).length + 4 : this.tokensPerImage + } + return count + } +} + +export class OpenAIReaderClient implements ReaderModelClient { + constructor( + private readonly options: { + apiKey: string + baseUrl?: string + timeoutMs?: number + } + ) { + if (!options.apiKey) throw new Error("OPENAI_API_KEY is required") + } + + async generate(request: ReaderModelRequest, signal?: AbortSignal): Promise { + const content = [] + for (const part of request.parts) { + if (part.type === "text") { + content.push({ type: "text", text: part.text }) + } else { + if (!part.asset.absolutePath) throw new Error(`Unresolved image ${part.asset.assetId}`) + const bytes = await readFile(part.asset.absolutePath) + content.push({ + type: "image_url", + image_url: { + url: `data:${part.asset.mimeType};base64,${bytes.toString("base64")}`, + }, + }) + } + } + const controller = new AbortController() + const onAbort = () => controller.abort(signal?.reason) + signal?.addEventListener("abort", onAbort, { once: true }) + const timeout = setTimeout( + () => controller.abort(new Error("Reader request timed out")), + this.options.timeoutMs ?? 10 * 60 * 1000 + ) + try { + const response = await fetch( + `${(this.options.baseUrl ?? "https://api.openai.com").replace(/\/$/, "")}/v1/chat/completions`, + { + method: "POST", + headers: { + authorization: `Bearer ${this.options.apiKey}`, + "content-type": "application/json", + }, + body: JSON.stringify({ + model: request.model, + messages: [ + { role: "system", content: request.systemPrompt }, + { role: "user", content }, + ], + ...openAICompletionControls( + request.model, + request.maxCompletionTokens, + request.reasoningEffort + ), + }), + signal: controller.signal, + } + ) + const raw = (await response.json().catch(() => null)) as Record | null + if (!response.ok) { + throw new Error(`OpenAI reader HTTP ${response.status}`) + } + const choices = raw?.choices as Array> | undefined + const message = choices?.[0]?.message as Record | undefined + const messageContent = message?.content + let text = "" + if (typeof messageContent === "string") text = messageContent.trim() + else if (Array.isArray(messageContent)) { + text = messageContent + .map((item) => + item && typeof item === "object" && typeof item.text === "string" ? item.text : "" + ) + .filter(Boolean) + .join("\n") + .trim() + } + if (!text && typeof message?.reasoning === "string") text = message.reasoning.trim() + const usage = raw?.usage as Record | undefined + return { text, usage, raw } + } finally { + clearTimeout(timeout) + signal?.removeEventListener("abort", onAbort) + } + } +} + +interface EvidenceUnit { + parts: ReaderMessagePart[] +} + +function extractBoxedAnswer(text: string): string { + const marker = "\\boxed{" + const start = text.lastIndexOf(marker) + if (start < 0) return text.trim() + let depth = 1 + const output: string[] = [] + for (let index = start + marker.length; index < text.length && depth > 0; index += 1) { + const character = text[index] + if (character === "{") { + depth += 1 + output.push(character) + } else if (character === "}") { + depth -= 1 + if (depth > 0) output.push(character) + } else { + output.push(character) + } + } + const parsed = output.join("").trim() + return parsed || text.trim() +} + +export class LongMemEvalV2Reader { + constructor( + private readonly client: ReaderModelClient, + private readonly artifacts: ArtifactStore, + private readonly tokenCounter: ContextTokenCounter = new Gpt5ContextTokenCounter() + ) {} + + async answer(input: { + queryArtifact: QueryArtifact + domain: "web" | "enterprise" + question: string + questionImage?: AssetRef + settings: ReaderSettings + signal?: AbortSignal + }): Promise { + const systemPrompt = DOMAIN_SYSTEM_PROMPTS[input.domain] + const units: EvidenceUnit[] = [] + const seenImages = new Set() + let imageCount = 0 + for (const result of input.queryArtifact.normalizedResults.slice( + 0, + input.settings.evidenceTopK + )) { + const parts: ReaderMessagePart[] = [ + { + type: "text", + text: result.text, + provenance: { + rank: result.rank, + score: result.score, + trajectoryId: result.trajectoryId, + stateIndex: result.stateIndex, + documentIds: result.documentIds, + }, + }, + ] + for (const screenshot of result.screenshotRefs) { + if ( + seenImages.has(screenshot.sha256) || + imageCount >= input.settings.maxImages || + screenshot.byteLength > input.settings.maxImageBytes + ) { + continue + } + const materialized = await this.artifacts.materializeAsset(screenshot) + seenImages.add(screenshot.sha256) + imageCount += 1 + parts.push({ + type: "image", + asset: materialized, + caption: `Screenshot for retrieval rank ${result.rank}`, + provenance: { rank: result.rank, trajectoryId: result.trajectoryId }, + }) + } + units.push({ parts }) + } + + const intro: ReaderMessagePart = { + type: "text", + text: `### Memory context:\n${units.length === 0 ? "(empty)" : ""}`, + } + const questionPart: ReaderMessagePart = { + type: "text", + text: `\n\n### Question to answer:\n${input.question}`, + } + let materializedQuestionImage: AssetRef | undefined + if (input.questionImage) { + if (input.questionImage.byteLength > input.settings.maxImageBytes) { + throw new Error(`Question image exceeds maxImageBytes`) + } + materializedQuestionImage = await this.artifacts.materializeAsset(input.questionImage) + } + const suffix: ReaderMessagePart[] = [ + questionPart, + ...(materializedQuestionImage + ? [{ type: "image" as const, asset: materializedQuestionImage }] + : []), + ] + const fits = (count: number): boolean => { + const parts = [intro, ...units.slice(0, count).flatMap((unit) => unit.parts), ...suffix] + return this.tokenCounter.count(systemPrompt, parts) <= input.settings.maxContextTokens + } + if (!fits(0)) { + throw new Error("System prompt, question, and question image exceed the context budget") + } + let low = 0 + let high = units.length + while (low < high) { + const middle = Math.floor((low + high + 1) / 2) + if (fits(middle)) low = middle + else high = middle - 1 + } + const parts = [intro, ...units.slice(0, low).flatMap((unit) => unit.parts), ...suffix] + const imageHashes = parts + .filter( + (part): part is Extract => part.type === "image" + ) + .map((part) => part.asset.sha256) + const fingerprint = readerFingerprint({ + queryArtifact: input.queryArtifact, + model: input.settings.model, + settings: { ...input.settings }, + promptVersion: READER_PROMPT_VERSION, + imageHashes, + contextBudgetVersion: this.tokenCounter.version, + }) + const cached = await this.loadCached(input.queryArtifact.questionId, fingerprint) + if (cached) return { ...cached, cacheHit: true } + + const rawAttempts: unknown[] = [] + const started = performance.now() + let response: ReaderModelResponse | undefined + for (let attempt = 1; attempt <= input.settings.malformedResponseAttempts; attempt += 1) { + response = await this.client.generate( + { + model: input.settings.model, + reasoningEffort: input.settings.reasoningEffort, + maxCompletionTokens: input.settings.maxCompletionTokens, + systemPrompt, + parts, + }, + input.signal + ) + rawAttempts.push(response.raw) + if (response.text.trim()) break + response = undefined + } + if (!response) throw new Error("Reader returned malformed empty responses") + const artifact: ReaderArtifact = { + schemaVersion: 1, + questionId: input.queryArtifact.questionId, + readerFingerprint: fingerprint, + model: input.settings.model, + reasoningEffort: input.settings.reasoningEffort, + systemPrompt, + parts, + sentAssetIds: parts + .filter( + (part): part is Extract => part.type === "image" + ) + .map((part) => part.asset.assetId), + omittedItems: units.length - low, + responseText: response.text.trim(), + parsedAnswer: extractBoxedAnswer(response.text), + rawAttempts, + usage: response.usage, + durationMs: performance.now() - started, + cacheHit: false, + createdAt: new Date().toISOString(), + } + const portableArtifact: ReaderArtifact = { + ...artifact, + parts: artifact.parts.map((part) => + part.type === "image" + ? { ...part, asset: { ...part.asset, absolutePath: undefined } } + : part + ), + } + await this.artifacts.writeJson( + `readers/${artifact.questionId}/${fingerprint}.json`, + portableArtifact + ) + return artifact + } + + private async loadCached( + questionId: string, + fingerprint: string + ): Promise { + try { + const artifact = await this.artifacts.readJson( + `readers/${questionId}/${fingerprint}.json` + ) + for (const part of artifact.parts) { + if (part.type === "image") { + const stored = await this.artifacts.describe(part.asset.relativePath) + if (stored.sha256 !== part.asset.sha256 || stored.byteLength !== part.asset.byteLength) { + return null + } + part.asset.absolutePath = this.artifacts.resolve(part.asset.relativePath) + } + } + return artifact + } catch { + return null + } + } +} diff --git a/src/benchmarks/longmemeval-v2/source.ts b/src/benchmarks/longmemeval-v2/source.ts new file mode 100644 index 0000000..bc8b9ff --- /dev/null +++ b/src/benchmarks/longmemeval-v2/source.ts @@ -0,0 +1,290 @@ +import { createHash } from "node:crypto" +import { createReadStream } from "node:fs" +import { lstat, mkdir, readFile, rm, stat, writeFile } from "node:fs/promises" +import { isAbsolute, posix, resolve } from "node:path" + +export const LONGMEMEVAL_V2_REPOSITORY = "xiaowu0162/longmemeval-v2" +export const LONGMEMEVAL_V2_PINNED_REVISION = "f152293e235517d504809563c833d7190b8c713b" +export const LONGMEMEVAL_V2_CHECKSUM_MANIFEST = "checksums.sha256" +export const LONGMEMEVAL_V2_CHECKSUM_MANIFEST_SHA256 = + "b17a18daa52873f915808502217c3c5fab39d20638544f986401155c9e8d67a6" +export const LONGMEMEVAL_V2_QUESTION_IMAGE_COUNT = 29 + +export interface PinnedDatasetFile { + relativePath: string + sha256: string + byteLength?: number +} + +export interface LongMemEvalV2SnapshotSpec { + repository: string + revision: string + checksumManifest: PinnedDatasetFile + requiredFiles: Readonly>> + archives: readonly PinnedDatasetFile[] + questionImageCount: number +} + +export const LONGMEMEVAL_V2_REQUIRED_FILES: Readonly< + Record> +> = { + LICENSE: { + sha256: "d547f7673579465fcecc8f257fcdb410f51c82fd784a10b1587e83036f9c29e1", + byteLength: 9109, + }, + "questions.jsonl": { + sha256: "0a3ae5ebea938c24d7800e1e0b0828e08ae1646f939a53853b2b8cdc08e292b7", + byteLength: 286186, + }, + "trajectories.jsonl": { + sha256: "363cec9a8e87aa8d9101ce4e600aadbf7031d674056ebe4f969e8424abc5f3c6", + byteLength: 1195604539, + }, + "haystacks/lme_v2_small.json": { + sha256: "9b5301defb23a088a5f06e45ff8d5f35e569d78305a66d492046a9fff9b46593", + byteLength: 822632, + }, + "haystacks/lme_v2_medium.json": { + sha256: "4756d5126347f0d18f045bb6c47b08cb3b23e9db24386cc48a9b2879e7969b59", + byteLength: 4054244, + }, +} + +export const LONGMEMEVAL_V2_ARCHIVES: readonly PinnedDatasetFile[] = [ + { + relativePath: "trajectory_screenshots/web_screenshots.tar.gz", + sha256: "68699c6842412e09a6f89d3c05c5ae8813275918002b52d82dec43ab24dd01fb", + byteLength: 2562302847, + }, + { + relativePath: "trajectory_screenshots/enterprise_screenshots_base.tar.gz", + sha256: "5c4a67ae0856aa1ede9b040e7da7c7a2d0b76fdd6344ef87380bcdf9f4b6d7a3", + byteLength: 3354163660, + }, +] + +export const LONGMEMEVAL_V2_SNAPSHOT: LongMemEvalV2SnapshotSpec = { + repository: LONGMEMEVAL_V2_REPOSITORY, + revision: LONGMEMEVAL_V2_PINNED_REVISION, + checksumManifest: { + relativePath: LONGMEMEVAL_V2_CHECKSUM_MANIFEST, + sha256: LONGMEMEVAL_V2_CHECKSUM_MANIFEST_SHA256, + byteLength: 3561, + }, + requiredFiles: LONGMEMEVAL_V2_REQUIRED_FILES, + archives: LONGMEMEVAL_V2_ARCHIVES, + questionImageCount: LONGMEMEVAL_V2_QUESTION_IMAGE_COUNT, +} + +function requireValue(condition: unknown, message: string): asserts condition { + if (!condition) throw new Error(message) +} + +async function processIsAlive(pid: number): Promise { + try { + process.kill(pid, 0) + return true + } catch (error) { + return (error as NodeJS.ErrnoException).code !== "ESRCH" + } +} + +export async function acquireDatasetOperationLock(input: { + lockPath: string + dataRoot: string + operation: string +}): Promise<() => Promise> { + for (let attempt = 0; attempt < 2; attempt += 1) { + try { + await mkdir(input.lockPath) + try { + await writeFile( + resolve(input.lockPath, "owner.json"), + `${JSON.stringify({ + pid: process.pid, + dataRoot: resolve(input.dataRoot), + operation: input.operation, + createdAt: new Date().toISOString(), + })}\n`, + { encoding: "utf8", mode: 0o600 } + ) + } catch (error) { + await rm(input.lockPath, { recursive: true, force: true }) + throw error + } + return async () => { + await rm(input.lockPath, { recursive: true, force: true }) + } + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error + let owner: { pid?: unknown; dataRoot?: unknown } | undefined + try { + owner = JSON.parse(await readFile(resolve(input.lockPath, "owner.json"), "utf8")) as { + pid?: unknown + dataRoot?: unknown + } + } catch { + throw new Error(`Dataset operation lock is unreadable: ${input.lockPath}`) + } + const pid = owner.pid + const sameRoot = owner.dataRoot === resolve(input.dataRoot) + if ( + attempt === 0 && + sameRoot && + typeof pid === "number" && + Number.isInteger(pid) && + pid > 0 && + !(await processIsAlive(pid)) + ) { + await rm(input.lockPath, { recursive: true, force: true }) + continue + } + throw new Error(`Another ${input.operation} operation holds ${input.lockPath}`) + } + } + throw new Error(`Could not acquire dataset operation lock: ${input.lockPath}`) +} + +export function validateDatasetRelativePath(relativePath: string): string { + requireValue(relativePath.length > 0, "Dataset path must not be empty") + requireValue(!relativePath.includes("\0"), "Dataset path contains a null byte") + requireValue(!relativePath.includes("\\"), `Dataset path uses a backslash: ${relativePath}`) + requireValue(!isAbsolute(relativePath), `Dataset path must be relative: ${relativePath}`) + requireValue(!/^[A-Za-z]:/.test(relativePath), `Dataset path has a drive prefix: ${relativePath}`) + const normalized = posix.normalize(relativePath) + requireValue( + normalized === relativePath && + normalized !== "." && + normalized !== ".." && + !normalized.startsWith("../"), + `Unsafe dataset path: ${relativePath}` + ) + return normalized +} + +export function parseChecksumManifest( + text: string, + spec: LongMemEvalV2SnapshotSpec = LONGMEMEVAL_V2_SNAPSHOT +): PinnedDatasetFile[] { + const files: PinnedDatasetFile[] = [] + const seen = new Set() + for (const [index, rawLine] of text.split(/\r?\n/).entries()) { + if (!rawLine.trim()) continue + const match = /^([0-9a-f]{64}) {2}(.+)$/.exec(rawLine) + requireValue(match, `Invalid checksum line ${index + 1}`) + const relativePath = validateDatasetRelativePath(match[2]) + requireValue(!seen.has(relativePath), `Duplicate checksum path: ${relativePath}`) + seen.add(relativePath) + files.push({ relativePath, sha256: match[1] }) + } + requireValue(files.length > 0, "Checksum manifest is empty") + + const byPath = new Map(files.map((file) => [file.relativePath, file])) + for (const [relativePath, expected] of Object.entries(spec.requiredFiles)) { + const actual = byPath.get(relativePath) + requireValue(actual, `Checksum manifest is missing ${relativePath}`) + requireValue(actual.sha256 === expected.sha256, `Pinned checksum mismatch for ${relativePath}`) + } + const questionImages = files.filter((file) => + file.relativePath.startsWith("question_screenshots/") + ) + requireValue( + questionImages.length === spec.questionImageCount, + `Expected ${spec.questionImageCount} question images, found ${questionImages.length}` + ) + return files +} + +export function selectRuntimeSnapshotFiles( + files: PinnedDatasetFile[], + spec: LongMemEvalV2SnapshotSpec = LONGMEMEVAL_V2_SNAPSHOT +): PinnedDatasetFile[] { + const requiredPaths = new Set(Object.keys(spec.requiredFiles)) + const selected = files.filter( + (file) => + requiredPaths.has(file.relativePath) || file.relativePath.startsWith("question_screenshots/") + ) + requireValue( + selected.length === requiredPaths.size + spec.questionImageCount, + "Checksum manifest does not contain the complete runtime dataset" + ) + return selected +} + +export async function sha256FileStreaming(path: string): Promise { + const hash = createHash("sha256") + for await (const chunk of createReadStream(path)) { + hash.update(chunk as Buffer) + } + return hash.digest("hex") +} + +export interface VerifiedDatasetSnapshot { + repository: string + revision: string + files: Array> +} + +export async function verifyDatasetSnapshot( + dataRoot: string, + spec: LongMemEvalV2SnapshotSpec = LONGMEMEVAL_V2_SNAPSHOT +): Promise { + const root = resolve(dataRoot) + const checksumPath = resolve( + root, + validateDatasetRelativePath(spec.checksumManifest.relativePath) + ) + const checksumStat = await stat(checksumPath) + requireValue(checksumStat.isFile(), `Checksum manifest is not a file: ${checksumPath}`) + if (spec.checksumManifest.byteLength !== undefined) { + requireValue( + checksumStat.size === spec.checksumManifest.byteLength, + `Checksum manifest size mismatch: expected ${spec.checksumManifest.byteLength}, got ${checksumStat.size}` + ) + } + requireValue( + (await sha256FileStreaming(checksumPath)) === spec.checksumManifest.sha256, + "Checksum manifest does not match the pinned snapshot" + ) + + const manifestFiles = selectRuntimeSnapshotFiles( + parseChecksumManifest(await readFile(checksumPath, "utf8"), spec), + spec + ) + const allFiles = [{ ...spec.checksumManifest }, ...manifestFiles, ...spec.archives] + const seen = new Set() + const verified: Array> = [] + for (const expected of allFiles) { + const relativePath = validateDatasetRelativePath(expected.relativePath) + if (seen.has(relativePath)) continue + seen.add(relativePath) + const absolutePath = resolve(root, relativePath) + const fileStat = await lstat(absolutePath) + requireValue( + fileStat.isFile() && !fileStat.isSymbolicLink(), + `Snapshot entry must be a regular file: ${relativePath}` + ) + if (expected.byteLength !== undefined) { + requireValue( + fileStat.size === expected.byteLength, + `Size mismatch for ${relativePath}: expected ${expected.byteLength}, got ${fileStat.size}` + ) + } + const actualHash = await sha256FileStreaming(absolutePath) + requireValue( + actualHash === expected.sha256, + `Checksum mismatch for ${relativePath}: expected ${expected.sha256}, got ${actualHash}` + ) + verified.push({ + relativePath, + sha256: expected.sha256, + byteLength: fileStat.size, + }) + } + + return { + repository: spec.repository, + revision: spec.revision, + files: verified, + } +} diff --git a/src/benchmarks/longmemeval-v2/types.ts b/src/benchmarks/longmemeval-v2/types.ts new file mode 100644 index 0000000..cce027c --- /dev/null +++ b/src/benchmarks/longmemeval-v2/types.ts @@ -0,0 +1,72 @@ +import type { AssetRef } from "../../types/migration" + +export type LongMemEvalV2Domain = "web" | "enterprise" +export type LongMemEvalV2Tier = "small" | "medium" + +export interface LongMemEvalV2Question { + id: string + domain: LongMemEvalV2Domain + environment: string + question_type: string + question: string + image: string | null + answer: string + eval_function: string +} + +export interface LongMemEvalV2State { + state_index?: number + step?: number + url: string + action: string | null + thought?: string | null + thoughts?: string | null + accessibility_tree?: string + text?: string + screenshot: string +} + +export interface LongMemEvalV2Trajectory { + id: string + domain: LongMemEvalV2Domain + goal: string + start_url: string + outcome: string | null + states: LongMemEvalV2State[] +} + +export interface PreparedTrajectoryState { + stateIndex: number + step: number + url: string + action: string | null + thoughts: string | null + accessibilityTree: string + screenshot: AssetRef +} + +export interface PreparedTrajectory { + id: string + domain: LongMemEvalV2Domain + goal: string + startUrl: string + outcome: string | null + states: PreparedTrajectoryState[] + contentHash: string +} + +export interface LongMemEvalV2QuestionPlan { + question: LongMemEvalV2Question + questionImage?: AssetRef + orderedTrajectoryIds: string[] + haystackHash: string + buildKey: string +} + +export interface LongMemEvalV2BuildGroup { + buildKey: string + domain: LongMemEvalV2Domain + tier: LongMemEvalV2Tier + orderedTrajectoryIds: string[] + questionIds: string[] +} diff --git a/src/cli/commands/longmemeval-v2.ts b/src/cli/commands/longmemeval-v2.ts new file mode 100644 index 0000000..38d62a1 --- /dev/null +++ b/src/cli/commands/longmemeval-v2.ts @@ -0,0 +1,447 @@ +import { resolve } from "node:path" +import { atomicWriteJson } from "../../core/canonical" +import { + LongMemEvalV2Runner, + inspectLongMemEvalV2Run, + type LongMemEvalV2RunThrough, +} from "../../orchestrator/longmemeval-v2" +import { BuildAwareRunStore } from "../../orchestrator/build-aware-run-store" +import { + AdvancedSupermemoryProvider, + supermemoryPreflightGatePath, +} from "../../providers/supermemory/advanced" +import type { BuildAwareRunConfig } from "../../types/build-aware" +import { downloadLongMemEvalV2Dataset } from "../../benchmarks/longmemeval-v2/download" +import { + prepareLongMemEvalV2Screenshots, + type ScreenshotPreparationMode, +} from "../../benchmarks/longmemeval-v2/prepare" + +const PINNED_DATASET_REVISION = "f152293e235517d504809563c833d7190b8c713b" + +type Action = + | "download" + | "prepare" + | "preflight" + | "dry-run" + | "canary" + | "build" + | "query" + | "evaluate" + | "run" + | "resume" + | "inspect" + +interface ParsedArgs { + action: Action + runId?: string + datasetPath: string + revision: string + tier: "small" | "medium" + allowMedium: boolean + domain: "web" | "enterprise" | "all" + questionIds?: string[] + limit?: number + perCategory?: number + seed: string + topK: number + evidenceTopK: number + threshold: number + readerModel: string + evaluatorModel: string + reasoningEffort: BuildAwareRunConfig["reader"]["reasoningEffort"] + evaluatorReasoningEffort: string + questionConcurrency: number + buildConcurrency: number + trajectoryConcurrency: number + maxInFlightRequests: number + maxTrajectoryAttempts: number + indexingTimeoutMs: number + serviceBaseUrl: string + forceBuild: boolean + freshQuery: boolean + keepPreflightDocuments: boolean + prepareMode: ScreenshotPreparationMode + preflightReadinessTimeoutMs: number + preflightSearchTimeoutMs: number + preflightPollMs: number + preflightMaxAgeMs: number + continueOnIndexingTimeout: boolean +} + +function help(): void { + console.log(` +LongMemEval-V2 build-aware workflow + +Usage: + bun run src/index.ts lme-v2 [options] + +Actions: + download Download the exact pinned snapshot atomically and verify checksums + prepare Safely extract archives and build the common screenshots view + preflight Probe the live Supermemory V3/V4 service contract and clean probes + dry-run Validate dataset, selection, assets, conversion, and build plans + canary Build and query exactly one trajectory (not an official score) + build Create or resume reusable Memory Builds + query Build if needed, then retrieve with immutable artifacts + evaluate Build, query, read with GPT-5, and run official evaluation + run Execute the complete official pipeline and report + resume Resume an existing run from its durable checkpoints + inspect Print an existing checkpoint/report without network calls + +Important options: + -r, --run-id ID Stable run identifier + --dataset PATH Prepared dataset root + --revision SHA Exact dataset revision + --tier small|medium Medium additionally requires --allow-medium + --domain web|enterprise|all + --question-id ID[,ID...] Exact question selection + --limit N Deterministic prefix selection + --per-category N Deterministic category sample + --seed VALUE Replayable sample seed + --top-k N Authoritative retrieval top-K + --evidence-top-k N Reader evidence limit (must be <= top-K) + --reader-model MODEL Default: gpt-5 + --reasoning-effort LEVEL Default: high + --evaluator-model MODEL Default: gpt-5 + --force-build Explicitly clear the exact build before rebuilding + --fresh-query Bypass query cache, retaining immutable old artifacts + --max-trajectory-attempts N Finite non-timeout retries per trajectory (default: 4) + --indexing-timeout-ms N Hard per-trajectory readiness deadline (default: 1800000) + --preflight-readiness-ms N Live readiness deadline (default: 300000) + --preflight-search-ms N Live search deadline (default: 120000) + --preflight-max-age-hours N Maximum accepted gate age (default: 24) + --strict-ingestion Fail instead of skipping documents that exceed the indexing deadline +`) +} + +function positiveInteger(raw: string | undefined, flag: string): number { + const value = Number(raw) + if (!Number.isInteger(value) || value < 1) throw new Error(`${flag} requires an integer >= 1`) + return value +} + +function finiteNumber(raw: string | undefined, flag: string): number { + const value = Number(raw) + if (!Number.isFinite(value)) throw new Error(`${flag} requires a finite number`) + return value +} + +function generateRunId(action: Action): string { + return `lme-v2-${action}-${new Date() + .toISOString() + .replace(/[-:.TZ]/g, "") + .slice(0, 14)}` +} + +function parse(args: string[]): ParsedArgs | null { + if (args.length === 0 || ["help", "--help", "-h"].includes(args[0])) return null + const action = args[0] as Action + if ( + ![ + "preflight", + "download", + "prepare", + "dry-run", + "canary", + "build", + "query", + "evaluate", + "run", + "resume", + "inspect", + ].includes(action) + ) { + throw new Error(`Unknown lme-v2 action: ${args[0]}`) + } + const parsed: ParsedArgs = { + action, + datasetPath: "data/benchmarks/longmemeval-v2", + revision: PINNED_DATASET_REVISION, + tier: "small", + allowMedium: false, + domain: "all", + seed: "memorybench-longmemeval-v2", + topK: 20, + evidenceTopK: 20, + threshold: 0, + readerModel: "gpt-5", + evaluatorModel: "gpt-5", + reasoningEffort: "high", + evaluatorReasoningEffort: "high", + questionConcurrency: 5, + buildConcurrency: 2, + trajectoryConcurrency: 4, + maxInFlightRequests: 20, + maxTrajectoryAttempts: 4, + indexingTimeoutMs: 30 * 60_000, + serviceBaseUrl: "https://api.supermemory.ai", + forceBuild: false, + freshQuery: false, + keepPreflightDocuments: false, + prepareMode: "symlink", + preflightReadinessTimeoutMs: 300_000, + preflightSearchTimeoutMs: 120_000, + preflightPollMs: 5_000, + preflightMaxAgeMs: 24 * 60 * 60_000, + continueOnIndexingTimeout: true, + } + const questionIds: string[] = [] + for (let index = 1; index < args.length; index += 1) { + const flag = args[index] + const next = () => { + const value = args[++index] + if (!value || value.startsWith("--")) throw new Error(`${flag} requires a value`) + return value + } + if (flag === "-r" || flag === "--run-id") parsed.runId = next() + else if (flag === "--dataset") parsed.datasetPath = next() + else if (flag === "--revision") parsed.revision = next() + else if (flag === "--tier") { + const value = next() + if (value !== "small" && value !== "medium") throw new Error("Invalid --tier") + parsed.tier = value + } else if (flag === "--allow-medium") parsed.allowMedium = true + else if (flag === "--domain") { + const value = next() + if (!["web", "enterprise", "all"].includes(value)) throw new Error("Invalid --domain") + parsed.domain = value as ParsedArgs["domain"] + } else if (flag === "--question-id") { + questionIds.push( + ...next() + .split(",") + .map((value) => value.trim()) + .filter(Boolean) + ) + } else if (flag === "--limit") parsed.limit = positiveInteger(next(), flag) + else if (flag === "--per-category") parsed.perCategory = positiveInteger(next(), flag) + else if (flag === "--seed") parsed.seed = next() + else if (flag === "--top-k") parsed.topK = positiveInteger(next(), flag) + else if (flag === "--evidence-top-k") parsed.evidenceTopK = positiveInteger(next(), flag) + else if (flag === "--threshold") parsed.threshold = finiteNumber(next(), flag) + else if (flag === "--reader-model") parsed.readerModel = next() + else if (flag === "--evaluator-model") parsed.evaluatorModel = next() + else if (flag === "--reasoning-effort") { + const value = next() + if (!["none", "minimal", "low", "medium", "high", "xhigh"].includes(value)) { + throw new Error("Invalid --reasoning-effort") + } + parsed.reasoningEffort = value as ParsedArgs["reasoningEffort"] + } else if (flag === "--evaluator-reasoning-effort") { + parsed.evaluatorReasoningEffort = next() + } else if (flag === "--question-concurrency") { + parsed.questionConcurrency = positiveInteger(next(), flag) + } else if (flag === "--build-concurrency") { + parsed.buildConcurrency = positiveInteger(next(), flag) + } else if (flag === "--trajectory-concurrency") { + parsed.trajectoryConcurrency = positiveInteger(next(), flag) + } else if (flag === "--max-in-flight-requests") { + parsed.maxInFlightRequests = positiveInteger(next(), flag) + } else if (flag === "--max-trajectory-attempts") { + parsed.maxTrajectoryAttempts = positiveInteger(next(), flag) + } else if (flag === "--indexing-timeout-ms") { + parsed.indexingTimeoutMs = positiveInteger(next(), flag) + } else if (flag === "--base-url") parsed.serviceBaseUrl = next() + else if (flag === "--force-build") parsed.forceBuild = true + else if (flag === "--fresh-query") parsed.freshQuery = true + else if (flag === "--keep-preflight-documents") parsed.keepPreflightDocuments = true + else if (flag === "--prepare-mode") { + const value = next() + if (value !== "symlink" && value !== "copy") { + throw new Error("--prepare-mode must be symlink or copy") + } + parsed.prepareMode = value + } else if (flag === "--preflight-readiness-ms") { + parsed.preflightReadinessTimeoutMs = positiveInteger(next(), flag) + } else if (flag === "--preflight-search-ms") { + parsed.preflightSearchTimeoutMs = positiveInteger(next(), flag) + } else if (flag === "--preflight-poll-ms") { + parsed.preflightPollMs = positiveInteger(next(), flag) + } else if (flag === "--preflight-max-age-hours") { + const hours = finiteNumber(next(), flag) + if (hours <= 0) throw new Error(`${flag} requires a number > 0`) + parsed.preflightMaxAgeMs = hours * 60 * 60_000 + } else if (flag === "--strict-ingestion") parsed.continueOnIndexingTimeout = false + else throw new Error(`Unknown option: ${flag}`) + } + if (questionIds.length > 0) parsed.questionIds = [...new Set(questionIds)] + if (parsed.tier === "medium" && !parsed.allowMedium) { + throw new Error("Medium is an explicit high-cost tier; pass --allow-medium") + } + if (parsed.action === "canary" && parsed.questionIds?.length !== 1) { + throw new Error("Canary requires exactly one --question-id") + } + if (["resume", "inspect"].includes(parsed.action) && !parsed.runId) { + throw new Error(`${parsed.action} requires --run-id`) + } + return parsed +} + +function configFrom(parsed: ParsedArgs): BuildAwareRunConfig { + return { + provider: "supermemory", + benchmark: "longmemeval-v2", + mode: parsed.action === "canary" ? "one-trajectory-canary" : "benchmark", + datasetPath: resolve(parsed.datasetPath), + datasetRevision: parsed.revision, + tier: parsed.tier, + domain: parsed.domain, + questionIds: parsed.questionIds, + limit: parsed.limit, + perCategory: parsed.perCategory, + seed: parsed.seed, + retrieval: { + topK: parsed.topK, + threshold: parsed.threshold, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: true, + metadataFilter: {}, + }, + reader: { + model: parsed.readerModel, + reasoningEffort: parsed.reasoningEffort, + maxCompletionTokens: 20_000, + maxContextTokens: 200_000, + evidenceTopK: parsed.evidenceTopK, + maxImages: 100, + maxImageBytes: 20 * 1024 * 1024, + malformedResponseAttempts: 3, + }, + evaluator: { + model: parsed.evaluatorModel, + reasoningEffort: parsed.evaluatorReasoningEffort, + maxCompletionTokens: 4096, + }, + build: { + serviceBaseUrl: parsed.serviceBaseUrl, + dreaming: "instant", + rootFilterMode: "self", + maxDocumentChars: 200_000, + trajectoryConcurrency: parsed.trajectoryConcurrency, + maxInFlightRequests: parsed.maxInFlightRequests, + maxTrajectoryAttempts: parsed.maxTrajectoryAttempts, + indexingTimeoutMs: parsed.indexingTimeoutMs, + pollIntervalMs: 2_000, + preflightMaxAgeMs: parsed.preflightMaxAgeMs, + continueOnIndexingTimeout: parsed.continueOnIndexingTimeout, + }, + execution: { + buildConcurrency: parsed.buildConcurrency, + questionConcurrency: parsed.questionConcurrency, + }, + } +} + +function throughFor(action: Action): LongMemEvalV2RunThrough { + if (action === "dry-run") return "plan" + if (action === "build") return "build" + if (action === "query" || action === "canary") return "query" + if (action === "evaluate") return "evaluate" + return "report" +} + +export async function longMemEvalV2Command(args: string[]): Promise { + const parsed = parse(args) + if (!parsed) { + help() + return + } + if (parsed.action === "inspect") { + console.log(JSON.stringify(await inspectLongMemEvalV2Run(parsed.runId!), null, 2)) + return + } + if (parsed.action === "download") { + console.log( + JSON.stringify(await downloadLongMemEvalV2Dataset({ dataRoot: parsed.datasetPath }), null, 2) + ) + return + } + if (parsed.action === "prepare") { + console.log( + JSON.stringify( + await prepareLongMemEvalV2Screenshots({ + dataRoot: parsed.datasetPath, + mode: parsed.prepareMode, + }), + null, + 2 + ) + ) + return + } + if (parsed.action === "preflight") { + const apiKey = process.env.SUPERMEMORY_API_KEY + if (!apiKey) throw new Error("SUPERMEMORY_API_KEY is required for preflight") + const provider = new AdvancedSupermemoryProvider({ + apiKey, + baseUrl: parsed.serviceBaseUrl, + maxInFlightRequests: parsed.maxInFlightRequests, + }) + const report = await provider.preflight({ + searchTopK: parsed.topK, + keepDocuments: parsed.keepPreflightDocuments, + readinessTimeoutMs: parsed.preflightReadinessTimeoutMs, + searchVisibilityTimeoutMs: parsed.preflightSearchTimeoutMs, + searchPollMs: parsed.preflightPollMs, + onCheck: (check) => { + console.error(`[preflight] ${check.ok ? "PASS" : "FAIL"} ${check.check}`) + }, + }) + if (parsed.runId) { + const store = new BuildAwareRunStore(parsed.runId) + await atomicWriteJson(resolve(store.runRoot, "preflight.json"), report) + } + console.log( + JSON.stringify( + { + allPassed: report.allPassed, + blockers: report.blockers, + checks: report.checks, + cleaned: !parsed.keepPreflightDocuments, + }, + null, + 2 + ) + ) + if (!report.allPassed) throw new Error("Supermemory preflight failed") + const gatePath = supermemoryPreflightGatePath("data/preflights-v2", parsed.serviceBaseUrl) + await atomicWriteJson(gatePath, report) + console.error(`[preflight] gate written to ${gatePath}`) + return + } + + let config: BuildAwareRunConfig + const runId = parsed.runId ?? generateRunId(parsed.action) + if (parsed.action === "resume") { + config = (await new BuildAwareRunStore(runId).load()).config + } else { + config = configFrom(parsed) + } + const runner = new LongMemEvalV2Runner({ runId, config }) + const checkpoint = await runner.execute({ + through: parsed.action === "resume" ? "report" : throughFor(parsed.action), + forceBuild: parsed.forceBuild, + freshQuery: parsed.freshQuery, + }) + console.log( + JSON.stringify( + { + runId: checkpoint.runId, + mode: checkpoint.config.mode, + status: checkpoint.status, + currentStage: checkpoint.currentStage, + datasetFingerprint: checkpoint.datasetFingerprint, + buildIds: checkpoint.buildIds, + questionCount: checkpoint.targetQuestionIds.length, + checkpointPath: runner.runStore.checkpointPath, + }, + null, + 2 + ) + ) +} diff --git a/src/cli/index.ts b/src/cli/index.ts index b3c29d6..f969cb9 100644 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -7,6 +7,7 @@ import { statusCommand } from "./commands/status" import { listQuestionsCommand } from "./commands/list-questions" import { showFailuresCommand } from "./commands/show-failures" import { serveCommand } from "./commands/serve" +import { longMemEvalV2Command } from "./commands/longmemeval-v2" import { getAvailableProviders } from "../providers" import { getAvailableBenchmarks } from "../benchmarks" import { listModelsByProvider, MODEL_ALIASES, DEFAULT_ANSWERING_MODEL } from "../utils/models" @@ -27,6 +28,7 @@ Commands: show-failures Show failed questions from a run with full debugging data status Check run status serve Start the web UI server + lme-v2 Build-aware LongMemEval-V2 workflow help Show help (use 'help providers', 'help models', 'help benchmarks' for details) Examples: @@ -190,6 +192,10 @@ export async function cli(args: string[]): Promise { case "serve": await serveCommand(commandArgs) break + case "lme-v2": + case "longmemeval-v2": + await longMemEvalV2Command(commandArgs) + break case "help": case "--help": case "-h": diff --git a/src/core/artifact-store.ts b/src/core/artifact-store.ts new file mode 100644 index 0000000..7927891 --- /dev/null +++ b/src/core/artifact-store.ts @@ -0,0 +1,166 @@ +import { access, copyFile, mkdir, readFile, rename, stat, writeFile } from "node:fs/promises" +import { existsSync, lstatSync, mkdirSync } from "node:fs" +import { basename, dirname, extname, isAbsolute, join, relative, resolve, sep } from "node:path" +import type { AssetRef } from "../types/migration" +import { canonicalJson, sha256, sha256File } from "./canonical" + +export interface StoredArtifact { + relativePath: string + sha256: string + byteLength: number +} + +function assertRelativePath(path: string): void { + if (!path || isAbsolute(path) || path === ".." || path.startsWith(`..${sep}`)) { + throw new Error(`Artifact path must stay inside the artifact root: ${path}`) + } +} + +function isSensitiveKey(key: string): boolean { + const normalized = key.replace(/[^a-zA-Z0-9]/g, "").toLowerCase() + return ( + normalized === "authorization" || + normalized === "token" || + normalized === "secret" || + normalized.endsWith("apikey") || + normalized.endsWith("accesstoken") || + normalized.endsWith("refreshtoken") || + normalized.endsWith("clientsecret") + ) +} + +function redactSecrets(value: unknown, knownSecrets: string[]): unknown { + if (typeof value === "string") { + let output = value + for (const secret of knownSecrets) { + if (secret.length >= 8) output = output.split(secret).join("[REDACTED]") + } + output = output.replace(/\b(?:sk|sm)_[A-Za-z0-9_-]{16,}\b/g, "[REDACTED]") + return output + } + if (Array.isArray(value)) return value.map((item) => redactSecrets(item, knownSecrets)) + if (value && typeof value === "object") { + return Object.fromEntries( + Object.entries(value as Record).map(([key, item]) => [ + key, + isSensitiveKey(key) ? "[REDACTED]" : redactSecrets(item, knownSecrets), + ]) + ) + } + return value +} + +export class ArtifactStore { + readonly root: string + private readonly knownSecrets: string[] + + constructor(root: string, secretEnvironmentNames: string[] = []) { + this.root = resolve(root) + mkdirSync(this.root, { recursive: true }) + this.knownSecrets = secretEnvironmentNames + .map((name) => process.env[name]) + .filter((value): value is string => Boolean(value)) + } + + resolve(relativePath: string): string { + assertRelativePath(relativePath) + const absolutePath = resolve(this.root, relativePath) + const fromRoot = relative(this.root, absolutePath) + if (fromRoot === ".." || fromRoot.startsWith(`..${sep}`) || isAbsolute(fromRoot)) { + throw new Error(`Artifact path escapes root: ${relativePath}`) + } + let current = this.root + for (const component of fromRoot.split(sep).filter(Boolean)) { + current = join(current, component) + if (existsSync(current) && lstatSync(current).isSymbolicLink()) { + throw new Error(`Artifact path contains a symlink: ${relativePath}`) + } + } + return absolutePath + } + + async writeJson(relativePath: string, value: unknown): Promise { + const safeValue = redactSecrets(value, this.knownSecrets) + const bytes = Buffer.from(`${JSON.stringify(safeValue, null, 2)}\n`, "utf8") + return this.writeImmutable(relativePath, bytes) + } + + async writeCanonicalJson(relativePath: string, value: unknown): Promise { + const safeValue = redactSecrets(value, this.knownSecrets) + return this.writeImmutable(relativePath, Buffer.from(canonicalJson(safeValue), "utf8")) + } + + async writeImmutable(relativePath: string, bytes: Uint8Array): Promise { + const absolutePath = this.resolve(relativePath) + const expectedHash = sha256(bytes) + await mkdir(dirname(absolutePath), { recursive: true }) + try { + await access(absolutePath) + const existingHash = await sha256File(absolutePath) + if (existingHash !== expectedHash) { + throw new Error(`Immutable artifact collision at ${relativePath}`) + } + const existingStat = await stat(absolutePath) + return { relativePath, sha256: existingHash, byteLength: existingStat.size } + } catch (error) { + if (error instanceof Error && error.message.startsWith("Immutable artifact collision")) { + throw error + } + } + const temporaryPath = `${absolutePath}.tmp-${process.pid}-${crypto.randomUUID()}` + await writeFile(temporaryPath, bytes, { mode: 0o600 }) + await rename(temporaryPath, absolutePath) + return { relativePath, sha256: expectedHash, byteLength: bytes.byteLength } + } + + async materializeAsset(asset: AssetRef): Promise { + if (!asset.absolutePath) throw new Error(`Asset ${asset.assetId} is not resolved`) + const actualHash = await sha256File(asset.absolutePath) + if (actualHash !== asset.sha256) { + throw new Error(`Asset bytes changed for ${asset.relativePath}`) + } + const extension = extname(asset.relativePath).toLowerCase() + const relativePath = `assets/${asset.sha256}${extension}` + const target = this.resolve(relativePath) + await mkdir(dirname(target), { recursive: true }) + try { + await access(target) + if ((await sha256File(target)) !== asset.sha256) { + throw new Error(`Content-addressed asset collision for ${asset.sha256}`) + } + } catch (error) { + if (error instanceof Error && error.message.startsWith("Content-addressed asset collision")) { + throw error + } + const temporaryPath = `${target}.tmp-${process.pid}-${crypto.randomUUID()}` + await copyFile(asset.absolutePath, temporaryPath) + if ((await sha256File(temporaryPath)) !== asset.sha256) { + throw new Error(`Copied asset hash mismatch for ${asset.relativePath}`) + } + await rename(temporaryPath, target) + } + const targetStat = await stat(target) + if (targetStat.size !== asset.byteLength) { + throw new Error(`Asset size mismatch for ${asset.relativePath}`) + } + return { + ...asset, + absolutePath: target, + relativePath, + } + } + + async readJson(relativePath: string): Promise { + return JSON.parse(await readFile(this.resolve(relativePath), "utf8")) as T + } + + async describe(relativePath: string): Promise { + const absolutePath = this.resolve(relativePath) + const fileStat = await stat(absolutePath) + return { + relativePath, + sha256: await sha256File(absolutePath), + byteLength: fileStat.size, + } + } +} diff --git a/src/core/build-engine.ts b/src/core/build-engine.ts new file mode 100644 index 0000000..3d39611 --- /dev/null +++ b/src/core/build-engine.ts @@ -0,0 +1,373 @@ +import type { BuildProvider, RemoteDocumentState, RemoteDocumentStatus } from "../types/provider" +import type { MemoryBuildPlan, PhysicalDocument } from "../types/migration" +import { BuildStore, type StoredDocument } from "./build-store" + +export interface BuildEngineOptions { + trajectoryConcurrency: number + maxTrajectoryAttempts: number + indexingTimeoutMs: number + pollIntervalMs: number + leaseMs: number + continueOnIndexingTimeout: boolean + signal?: AbortSignal + sleep?: (milliseconds: number, signal?: AbortSignal) => Promise +} + +const DEFAULT_OPTIONS: BuildEngineOptions = { + trajectoryConcurrency: 4, + maxTrajectoryAttempts: 4, + indexingTimeoutMs: 30 * 60 * 1000, + pollIntervalMs: 2000, + leaseMs: 60_000, + continueOnIndexingTimeout: false, +} + +const SKIPPED_INDEXING_TIMEOUT_PREFIX = "INDEXING_TIMEOUT_SKIPPED:" + +class SkippedIndexingTimeoutError extends Error {} + +function defaultSleep(milliseconds: number, signal?: AbortSignal): Promise { + return new Promise((resolve, reject) => { + if (signal?.aborted) return reject(signal.reason ?? new Error("Operation aborted")) + const timer = setTimeout(resolve, milliseconds) + signal?.addEventListener( + "abort", + () => { + clearTimeout(timer) + reject(signal.reason ?? new Error("Operation aborted")) + }, + { once: true } + ) + }) +} + +function throwIfAborted(signal?: AbortSignal): void { + if (signal?.aborted) throw signal.reason ?? new Error("Operation aborted") +} + +export class BuildEngine { + private readonly options: BuildEngineOptions + private readonly documentByCustomId: Map + + constructor( + private readonly plan: MemoryBuildPlan, + private readonly provider: BuildProvider, + private readonly store: BuildStore, + options: Partial = {} + ) { + this.options = { ...DEFAULT_OPTIONS, ...options } + this.documentByCustomId = new Map( + plan.documents.map((document) => [document.customId, document]) + ) + if ( + !Number.isInteger(this.options.trajectoryConcurrency) || + this.options.trajectoryConcurrency < 1 + ) { + throw new Error("trajectoryConcurrency must be a positive integer") + } + if (this.options.leaseMs <= this.options.pollIntervalMs * 2) { + throw new Error("leaseMs must be more than two poll intervals") + } + } + + async run(): Promise<"ready" | "degraded"> { + const existing = this.store.getBuild(this.plan.buildId) + const created = this.store.registerBuild(this.plan) + if (!created && existing?.status === "degraded" && this.options.continueOnIndexingTimeout) { + return "degraded" + } + this.store.setBuildStatus(this.plan.buildId, "ingesting") + if (created) { + await this.reconcileAmbiguous() + } else { + await this.reconcileSavedRemoteState() + this.store.reopenTrajectoriesWithNonReadyDocuments(this.plan.buildId) + } + const workers = Array.from( + { length: Math.min(this.options.trajectoryConcurrency, this.plan.orderedSourceIds.length) }, + (_, index) => this.worker(`worker-${process.pid}-${index}-${crypto.randomUUID()}`) + ) + await Promise.all(workers) + const summary = this.store.buildSummary(this.plan.buildId) + if (summary.trajectories.failed > 0 || summary.documents.failed > 0) { + const failedTrajectories = this.store.getFailedTrajectories(this.plan.buildId) + const boundedFailuresMayDegrade = + this.options.continueOnIndexingTimeout && + failedTrajectories.length === summary.trajectories.failed + if (boundedFailuresMayDegrade) { + const skippedDocumentCount = Object.entries(summary.documents).reduce( + (total, [status, count]) => total + (status === "ready" ? 0 : count), + 0 + ) + const message = `Build degraded after skipping ${skippedDocumentCount} non-ready documents across ${summary.trajectories.failed} trajectories after bounded ingestion failures` + this.store.setBuildStatus(this.plan.buildId, "degraded", message) + this.store.recordEvent(this.plan.buildId, "build_degraded", "build", this.plan.buildId, { + message, + failedTrajectories, + summary, + }) + return "degraded" + } + const message = `Build failed: ${summary.trajectories.failed} trajectories and ${summary.documents.failed} documents failed` + this.store.setBuildStatus(this.plan.buildId, "failed", message) + throw new Error(message) + } + const trajectoryTotal = Object.values(summary.trajectories).reduce( + (sum, value) => sum + value, + 0 + ) + const documentTotal = Object.values(summary.documents).reduce((sum, value) => sum + value, 0) + if ( + summary.trajectories.ready !== trajectoryTotal || + summary.documents.ready !== documentTotal + ) { + const message = "Build stopped before every required document reached ready" + this.store.setBuildStatus(this.plan.buildId, "failed", message) + throw new Error(message) + } + this.store.setBuildStatus(this.plan.buildId, "ready") + this.store.recordEvent(this.plan.buildId, "build_ready", "build", this.plan.buildId, summary) + return "ready" + } + + async verifyRemoteHealth(options: { allowDegraded?: boolean } = {}): Promise { + const states = await this.provider.verifyBuildHealth(this.plan) + const byCustomId = new Map(states.map((state) => [state.customId, state])) + const expectedReady = options.allowDegraded + ? new Set(this.store.getDocumentCustomIdsByStatus(this.plan.buildId, "ready")) + : new Set(this.plan.documents.map((document) => document.customId)) + const unhealthy = this.plan.documents.filter( + (document) => + expectedReady.has(document.customId) && + byCustomId.get(document.customId)?.status !== "ready" + ) + if (unhealthy.length > 0) { + throw new Error( + `Remote build health check failed for ${unhealthy.length} documents: ${unhealthy + .slice(0, 5) + .map((document) => document.customId) + .join(", ")}` + ) + } + if (options.allowDegraded) { + const skippedStillVisible = this.store + .getDocumentCustomIdsByStatus(this.plan.buildId, "failed") + .filter((customId) => byCustomId.get(customId)?.status !== "absent") + if (skippedStillVisible.length > 0) { + throw new Error( + `Remote degraded-build health check found ${skippedStillVisible.length} skipped documents still visible: ${skippedStillVisible + .slice(0, 5) + .join(", ")}` + ) + } + } + } + + private async worker(workerId: string): Promise { + while (true) { + throwIfAborted(this.options.signal) + const trajectoryId = this.store.claimTrajectory( + this.plan.buildId, + workerId, + this.options.leaseMs + ) + if (!trajectoryId) { + const summary = this.store.buildSummary(this.plan.buildId) + const waiting = + summary.trajectories.planned + + summary.trajectories.retryable + + summary.trajectories.processing + if (waiting === 0) return + // A different process may still own an unexpired lease. Returning here + // would make run() misclassify resumable work as a terminal partial + // build. Wait until that worker finishes or its lease becomes + // claimable, then try again. + await (this.options.sleep ?? defaultSleep)(this.options.pollIntervalMs, this.options.signal) + continue + } + let heartbeatError: unknown + const heartbeat = setInterval( + () => { + try { + this.store.renewTrajectoryLease( + this.plan.buildId, + trajectoryId, + workerId, + this.options.leaseMs + ) + } catch (error) { + heartbeatError ??= error + } + }, + Math.max(1, Math.floor(this.options.leaseMs / 3)) + ) + try { + await this.processTrajectory(trajectoryId, workerId) + if (heartbeatError) throw heartbeatError + this.store.markTrajectoryReady(this.plan.buildId, trajectoryId, workerId) + } catch (error) { + const message = error instanceof Error ? error.message : String(error) + const attempt = this.store.getTrajectoryAttempt(this.plan.buildId, trajectoryId) + if ( + error instanceof SkippedIndexingTimeoutError || + (!this.options.signal?.aborted && attempt >= this.options.maxTrajectoryAttempts) + ) { + this.store.markTrajectoryFailed(this.plan.buildId, trajectoryId, workerId, message) + } else { + // A user stop is resumable. Persist the claimed trajectory as + // retryable so the next process can reconcile any accepted documents + // and continue from the durable checkpoint. + this.store.markTrajectoryRetryable(this.plan.buildId, trajectoryId, workerId, message) + } + } finally { + clearInterval(heartbeat) + } + } + } + + private async processTrajectory(trajectoryId: string, workerId: string): Promise { + let stored = this.store.getTrajectoryDocuments(this.plan.buildId, trajectoryId) + if ( + stored.some((document) => ["submitting", "accepted", "indexing"].includes(document.status)) + ) { + await this.reconcileDocuments(stored) + stored = this.store.getTrajectoryDocuments(this.plan.buildId, trajectoryId) + } + if (stored.some((document) => document.status === "failed")) { + const failed = stored.filter((document) => document.status === "failed") + await this.provider.deleteDocuments( + this.plan, + failed.map((document) => document.customId) + ) + for (const document of failed) this.store.resetDocumentToPlanned(document.customId) + stored = this.store.getTrajectoryDocuments(this.plan.buildId, trajectoryId) + } + const pending = stored.filter((document) => ["planned", "retryable"].includes(document.status)) + if (pending.length > 0) { + const physicalDocuments = pending.map((document) => { + const physical = this.documentByCustomId.get(document.customId) + if (!physical) throw new Error(`Missing physical document ${document.customId}`) + return physical + }) + for (const document of pending) this.store.markDocumentSubmitting(document.customId) + let submitted: RemoteDocumentState[] + try { + submitted = await this.provider.submitDocumentBatch({ + build: this.plan, + trajectoryId, + documents: physicalDocuments, + }) + } catch (error) { + await this.reconcileDocuments( + this.store.getTrajectoryDocuments(this.plan.buildId, trajectoryId) + ) + throw error + } + await this.applyStates(submitted, new Set(pending.map((document) => document.customId))) + const returned = new Set(submitted.map((state) => state.customId)) + const missing = pending.filter((document) => !returned.has(document.customId)) + if (missing.length > 0) await this.reconcileDocuments(missing) + } + + const deadline = Date.now() + this.options.indexingTimeoutMs + while (true) { + throwIfAborted(this.options.signal) + this.store.renewTrajectoryLease( + this.plan.buildId, + trajectoryId, + workerId, + this.options.leaseMs + ) + stored = this.store.getTrajectoryDocuments(this.plan.buildId, trajectoryId) + if (stored.every((document) => document.status === "ready")) return + if (stored.some((document) => document.status === "failed")) { + throw new Error(`A remote document failed for trajectory ${trajectoryId}`) + } + if (Date.now() >= deadline) { + if (this.options.continueOnIndexingTimeout) { + const unresolved = stored.filter((document) => document.status !== "ready") + const message = `${SKIPPED_INDEXING_TIMEOUT_PREFIX} ${unresolved.length} documents for trajectory ${trajectoryId} did not become ready within ${this.options.indexingTimeoutMs}ms` + await this.provider.deleteDocuments( + this.plan, + unresolved.map((document) => document.customId) + ) + for (const document of unresolved) { + this.store.markDocumentFailed(document.customId, message) + } + throw new SkippedIndexingTimeoutError(message) + } + throw new Error(`Indexing timed out for trajectory ${trajectoryId}`) + } + const unresolved = stored.filter((document) => document.status !== "ready") + await this.reconcileDocuments(unresolved) + await (this.options.sleep ?? defaultSleep)(this.options.pollIntervalMs, this.options.signal) + } + } + + private async reconcileAmbiguous(): Promise { + const ambiguous = this.store.getAmbiguousDocuments(this.plan.buildId) + for (let index = 0; index < ambiguous.length; index += 100) { + await this.reconcileDocuments(ambiguous.slice(index, index + 100)) + } + } + + private async reconcileSavedRemoteState(): Promise { + const stored = this.plan.orderedSourceIds.flatMap((trajectoryId) => + this.store.getTrajectoryDocuments(this.plan.buildId, trajectoryId) + ) + const reconcilable = stored.filter((document) => document.status !== "failed") + for (let index = 0; index < reconcilable.length; index += 100) { + await this.reconcileDocuments(reconcilable.slice(index, index + 100)) + } + } + + private async reconcileDocuments(documents: StoredDocument[]): Promise { + if (documents.length === 0) return + const states = await this.provider.reconcileDocuments( + this.plan, + documents.map((document) => document.customId) + ) + await this.applyStates(states, new Set(documents.map((document) => document.customId))) + const returned = new Set(states.map((state) => state.customId)) + for (const document of documents) { + if (!returned.has(document.customId)) { + this.store.resetDocumentToPlanned(document.customId) + } + } + } + + private async applyStates( + states: RemoteDocumentState[], + expectedCustomIds: Set + ): Promise { + for (const state of states) { + if (!expectedCustomIds.has(state.customId)) { + throw new Error(`Provider returned unexpected customId ${state.customId}`) + } + this.applyState(state) + } + } + + private applyState(state: RemoteDocumentState): void { + if (state.remoteId && (state.status === "pending" || state.status === "ready")) { + this.store.markDocumentAccepted( + state.customId, + state.remoteId, + state.raw ?? { status: state.status } + ) + } + const handlers: Record void> = { + absent: () => this.store.resetDocumentToPlanned(state.customId), + pending: () => this.store.markDocumentIndexing(state.customId, state.remoteId), + ready: () => this.store.markDocumentReady(state.customId, state.remoteId), + failed: () => + this.store.markDocumentFailed(state.customId, state.error ?? "Remote document failed"), + unknown: () => + this.store.markDocumentFailed( + state.customId, + state.error ?? "Unknown remote document status" + ), + } + handlers[state.status]() + } +} diff --git a/src/core/build-store.ts b/src/core/build-store.ts new file mode 100644 index 0000000..2a3ed5a --- /dev/null +++ b/src/core/build-store.ts @@ -0,0 +1,639 @@ +import { Database } from "bun:sqlite" +import { mkdirSync } from "node:fs" +import { dirname } from "node:path" +import type { MemoryBuildPlan, PhysicalDocument } from "../types/migration" +import { canonicalJson } from "./canonical" + +export type BuildStatus = "planned" | "ingesting" | "ready" | "failed" | "degraded" +export type TrajectoryStatus = "planned" | "processing" | "ready" | "retryable" | "failed" +export type BuildDocumentStatus = + | "planned" + | "submitting" + | "accepted" + | "indexing" + | "ready" + | "retryable" + | "failed" + +export interface StoredDocument { + customId: string + buildId: string + trajectoryId: string + trajectoryOrder: number + logicalDocumentId: string + documentOrdinal: number + partIndex: number + partCount: number + contentHash: string + remoteId?: string + status: BuildDocumentStatus + attempts: number + lastError?: string +} + +function now(): string { + return new Date().toISOString() +} + +export class BuildStore { + readonly path: string + private readonly db: Database + + constructor(path: string) { + this.path = path + mkdirSync(dirname(path), { recursive: true }) + this.db = new Database(path, { create: true, strict: true }) + this.db.exec("PRAGMA journal_mode = WAL") + this.db.exec("PRAGMA synchronous = FULL") + this.db.exec("PRAGMA foreign_keys = ON") + this.db.exec("PRAGMA busy_timeout = 10000") + this.migrate() + } + + private migrate(): void { + const version = this.db.query("PRAGMA user_version").get() as { user_version: number } + if (version.user_version > 1) { + throw new Error( + `Build checkpoint schema ${version.user_version} is newer than supported schema 1` + ) + } + if (version.user_version === 0) { + this.db.exec(` + CREATE TABLE builds ( + build_id TEXT PRIMARY KEY, + build_fingerprint TEXT NOT NULL UNIQUE, + container_tag TEXT NOT NULL, + provider TEXT NOT NULL, + plan_json TEXT NOT NULL, + status TEXT NOT NULL, + error TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + CREATE TABLE trajectories ( + build_id TEXT NOT NULL, + trajectory_id TEXT NOT NULL, + trajectory_order INTEGER NOT NULL, + plan_hash TEXT NOT NULL, + status TEXT NOT NULL, + attempts INTEGER NOT NULL DEFAULT 0, + lease_owner TEXT, + lease_expires_at INTEGER, + error TEXT, + updated_at TEXT NOT NULL, + PRIMARY KEY (build_id, trajectory_id), + UNIQUE (build_id, trajectory_order), + FOREIGN KEY (build_id) REFERENCES builds(build_id) + ); + CREATE TABLE documents ( + custom_id TEXT PRIMARY KEY, + build_id TEXT NOT NULL, + trajectory_id TEXT NOT NULL, + trajectory_order INTEGER NOT NULL, + logical_document_id TEXT NOT NULL, + document_ordinal INTEGER NOT NULL, + part_index INTEGER NOT NULL, + part_count INTEGER NOT NULL, + content_hash TEXT NOT NULL, + remote_id TEXT, + status TEXT NOT NULL, + attempts INTEGER NOT NULL DEFAULT 0, + last_error TEXT, + response_json TEXT, + updated_at TEXT NOT NULL, + UNIQUE (build_id, trajectory_id, document_ordinal, part_index), + FOREIGN KEY (build_id, trajectory_id) + REFERENCES trajectories(build_id, trajectory_id) + ); + CREATE INDEX documents_build_status ON documents(build_id, status); + CREATE INDEX documents_remote_id ON documents(remote_id); + CREATE TABLE events ( + event_id INTEGER PRIMARY KEY AUTOINCREMENT, + build_id TEXT NOT NULL, + event_type TEXT NOT NULL, + entity_type TEXT, + entity_id TEXT, + details_json TEXT, + created_at TEXT NOT NULL, + FOREIGN KEY (build_id) REFERENCES builds(build_id) + ); + PRAGMA user_version = 1; + `) + } + } + + registerBuild(plan: MemoryBuildPlan): boolean { + const planJson = canonicalJson({ + ...plan, + documents: plan.documents.map((document) => ({ + ...document, + screenshotRef: document.screenshotRef + ? { ...document.screenshotRef, absolutePath: undefined } + : undefined, + })), + documentPlans: plan.documentPlans.map((documentPlan) => ({ + ...documentPlan, + documents: documentPlan.documents.map((document) => ({ + ...document, + spec: { + ...document.spec, + screenshotRef: document.spec.screenshotRef + ? { ...document.spec.screenshotRef, absolutePath: undefined } + : undefined, + }, + })), + })), + }) + const existing = this.db + .query( + "SELECT build_fingerprint, container_tag, provider, plan_json FROM builds WHERE build_id = ?" + ) + .get(plan.buildId) as { + build_fingerprint: string + container_tag: string + provider: string + plan_json: string + } | null + if (existing) { + if ( + existing.build_fingerprint !== plan.buildFingerprint || + existing.container_tag !== plan.containerTag || + existing.provider !== plan.provider || + existing.plan_json !== planJson + ) { + throw new Error(`Build ${plan.buildId} checkpoint does not match the requested plan`) + } + return false + } + + const documentPlans = new Map(plan.documentPlans.map((item) => [item.trajectoryId, item])) + const trajectoryOrder = new Map( + plan.orderedSourceIds.map((trajectoryId, index) => [trajectoryId, index]) + ) + const documentsByTrajectory = new Map() + for (const document of plan.documents) { + const items = documentsByTrajectory.get(document.trajectoryId) ?? [] + items.push(document) + documentsByTrajectory.set(document.trajectoryId, items) + } + + const transaction = this.db.transaction(() => { + const timestamp = now() + this.db + .query( + `INSERT INTO builds + (build_id, build_fingerprint, container_tag, provider, plan_json, status, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, 'planned', ?, ?)` + ) + .run( + plan.buildId, + plan.buildFingerprint, + plan.containerTag, + plan.provider, + planJson, + timestamp, + timestamp + ) + for (const trajectoryId of plan.orderedSourceIds) { + const trajectoryPlan = documentPlans.get(trajectoryId) + if (!trajectoryPlan) throw new Error(`Missing document plan for ${trajectoryId}`) + const order = trajectoryOrder.get(trajectoryId)! + this.db + .query( + `INSERT INTO trajectories + (build_id, trajectory_id, trajectory_order, plan_hash, status, updated_at) + VALUES (?, ?, ?, ?, 'planned', ?)` + ) + .run(plan.buildId, trajectoryId, order, trajectoryPlan.planHash, timestamp) + for (const document of documentsByTrajectory.get(trajectoryId) ?? []) { + this.db + .query( + `INSERT INTO documents + (custom_id, build_id, trajectory_id, trajectory_order, logical_document_id, + document_ordinal, part_index, part_count, content_hash, status, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'planned', ?)` + ) + .run( + document.customId, + plan.buildId, + trajectoryId, + order, + document.logicalDocumentId, + document.documentOrdinal, + document.partIndex, + document.partCount, + document.contentHash, + timestamp + ) + } + } + this.recordEvent(plan.buildId, "build_registered", "build", plan.buildId, { + trajectories: plan.orderedSourceIds.length, + documents: plan.documents.length, + }) + }) + transaction.immediate() + return true + } + + getBuild(buildId: string): { + buildId: string + buildFingerprint: string + containerTag: string + provider: string + status: BuildStatus + error?: string + } | null { + const row = this.db + .query( + `SELECT build_id, build_fingerprint, container_tag, provider, status, error + FROM builds WHERE build_id = ?` + ) + .get(buildId) as { + build_id: string + build_fingerprint: string + container_tag: string + provider: string + status: BuildStatus + error: string | null + } | null + return row + ? { + buildId: row.build_id, + buildFingerprint: row.build_fingerprint, + containerTag: row.container_tag, + provider: row.provider, + status: row.status, + error: row.error ?? undefined, + } + : null + } + + setBuildStatus(buildId: string, status: BuildStatus, error?: string): void { + this.db + .query("UPDATE builds SET status = ?, error = ?, updated_at = ? WHERE build_id = ?") + .run(status, error ?? null, now(), buildId) + } + + claimTrajectory(buildId: string, workerId: string, leaseMs: number): string | null { + if (!Number.isInteger(leaseMs) || leaseMs < 1) throw new Error("leaseMs must be positive") + const transaction = this.db.transaction(() => { + const timestamp = Date.now() + const row = this.db + .query( + `SELECT trajectory_id FROM trajectories + WHERE build_id = ? + AND ( + status IN ('planned', 'retryable') + OR (status = 'processing' AND COALESCE(lease_expires_at, 0) <= ?) + ) + ORDER BY trajectory_order + LIMIT 1` + ) + .get(buildId, timestamp) as { trajectory_id: string } | null + if (!row) return null + const result = this.db + .query( + `UPDATE trajectories + SET status = 'processing', attempts = attempts + 1, lease_owner = ?, + lease_expires_at = ?, error = NULL, updated_at = ? + WHERE build_id = ? AND trajectory_id = ? + AND ( + status IN ('planned', 'retryable') + OR (status = 'processing' AND COALESCE(lease_expires_at, 0) <= ?) + )` + ) + .run(workerId, timestamp + leaseMs, now(), buildId, row.trajectory_id, timestamp) + return result.changes === 1 ? row.trajectory_id : null + }) + return transaction.immediate() + } + + renewTrajectoryLease( + buildId: string, + trajectoryId: string, + workerId: string, + leaseMs: number + ): void { + const result = this.db + .query( + `UPDATE trajectories + SET lease_expires_at = ?, updated_at = ? + WHERE build_id = ? AND trajectory_id = ? AND status = 'processing' AND lease_owner = ?` + ) + .run(Date.now() + leaseMs, now(), buildId, trajectoryId, workerId) + if (result.changes !== 1) { + throw new Error(`Worker ${workerId} no longer owns ${trajectoryId}`) + } + } + + getTrajectoryAttempt(buildId: string, trajectoryId: string): number { + const row = this.db + .query("SELECT attempts FROM trajectories WHERE build_id = ? AND trajectory_id = ?") + .get(buildId, trajectoryId) as { attempts: number } | null + if (!row) throw new Error(`Unknown trajectory ${trajectoryId}`) + return row.attempts + } + + getFailedTrajectories(buildId: string): Array<{ trajectoryId: string; error?: string }> { + const rows = this.db + .query( + `SELECT trajectory_id, error FROM trajectories + WHERE build_id = ? AND status = 'failed' + ORDER BY trajectory_order` + ) + .all(buildId) as Array<{ trajectory_id: string; error: string | null }> + return rows.map((row) => ({ + trajectoryId: row.trajectory_id, + error: row.error ?? undefined, + })) + } + + getDocumentCustomIdsByStatus(buildId: string, status: BuildDocumentStatus): string[] { + const rows = this.db + .query( + `SELECT custom_id FROM documents + WHERE build_id = ? AND status = ? + ORDER BY trajectory_order, document_ordinal, part_index` + ) + .all(buildId, status) as Array<{ custom_id: string }> + return rows.map((row) => row.custom_id) + } + + markTrajectoryReady(buildId: string, trajectoryId: string, workerId: string): void { + const notReady = this.db + .query( + `SELECT COUNT(*) AS count FROM documents + WHERE build_id = ? AND trajectory_id = ? AND status != 'ready'` + ) + .get(buildId, trajectoryId) as { count: number } + if (notReady.count > 0) { + throw new Error(`Cannot complete ${trajectoryId}; ${notReady.count} documents are not ready`) + } + const result = this.db + .query( + `UPDATE trajectories + SET status = 'ready', lease_owner = NULL, lease_expires_at = NULL, error = NULL, updated_at = ? + WHERE build_id = ? AND trajectory_id = ? AND lease_owner = ?` + ) + .run(now(), buildId, trajectoryId, workerId) + if (result.changes !== 1) throw new Error(`Worker ${workerId} no longer owns ${trajectoryId}`) + } + + markTrajectoryRetryable( + buildId: string, + trajectoryId: string, + workerId: string, + error: string + ): void { + this.db + .query( + `UPDATE trajectories + SET status = 'retryable', lease_owner = NULL, lease_expires_at = NULL, error = ?, updated_at = ? + WHERE build_id = ? AND trajectory_id = ? AND lease_owner = ?` + ) + .run(error.slice(0, 2000), now(), buildId, trajectoryId, workerId) + } + + markTrajectoryFailed( + buildId: string, + trajectoryId: string, + workerId: string, + error: string + ): void { + this.db + .query( + `UPDATE trajectories + SET status = 'failed', lease_owner = NULL, lease_expires_at = NULL, error = ?, updated_at = ? + WHERE build_id = ? AND trajectory_id = ? AND lease_owner = ?` + ) + .run(error.slice(0, 2000), now(), buildId, trajectoryId, workerId) + } + + getTrajectoryDocuments(buildId: string, trajectoryId: string): StoredDocument[] { + const rows = this.db + .query( + `SELECT custom_id, build_id, trajectory_id, trajectory_order, logical_document_id, + document_ordinal, part_index, part_count, content_hash, remote_id, + status, attempts, last_error + FROM documents + WHERE build_id = ? AND trajectory_id = ? + ORDER BY document_ordinal, part_index` + ) + .all(buildId, trajectoryId) as Array<{ + custom_id: string + build_id: string + trajectory_id: string + trajectory_order: number + logical_document_id: string + document_ordinal: number + part_index: number + part_count: number + content_hash: string + remote_id: string | null + status: BuildDocumentStatus + attempts: number + last_error: string | null + }> + return rows.map((row) => ({ + customId: row.custom_id, + buildId: row.build_id, + trajectoryId: row.trajectory_id, + trajectoryOrder: row.trajectory_order, + logicalDocumentId: row.logical_document_id, + documentOrdinal: row.document_ordinal, + partIndex: row.part_index, + partCount: row.part_count, + contentHash: row.content_hash, + remoteId: row.remote_id ?? undefined, + status: row.status, + attempts: row.attempts, + lastError: row.last_error ?? undefined, + })) + } + + getAmbiguousDocuments(buildId: string): StoredDocument[] { + const trajectoryIds = this.db + .query( + `SELECT DISTINCT trajectory_id FROM documents + WHERE build_id = ? AND status IN ('submitting', 'accepted', 'indexing')` + ) + .all(buildId) as Array<{ trajectory_id: string }> + return trajectoryIds.flatMap((row) => + this.getTrajectoryDocuments(buildId, row.trajectory_id).filter((document) => + ["submitting", "accepted", "indexing"].includes(document.status) + ) + ) + } + + /** + * A locally-ready trajectory must be made claimable again when a remote + * health reconciliation moves any of its documents out of `ready`. + */ + reopenTrajectoriesWithNonReadyDocuments(buildId: string): number { + const result = this.db + .query( + `UPDATE trajectories + SET status = 'planned', lease_owner = NULL, lease_expires_at = NULL, + error = NULL, updated_at = ? + WHERE build_id = ? + AND status = 'ready' + AND EXISTS ( + SELECT 1 FROM documents + WHERE documents.build_id = trajectories.build_id + AND documents.trajectory_id = trajectories.trajectory_id + AND documents.status != 'ready' + )` + ) + .run(now(), buildId) + return result.changes + } + + /** + * Used only after an explicit remote clear. It never silently discards a + * remote namespace and therefore keeps `--force` semantics auditable. + */ + resetBuildForReingestion(buildId: string): void { + const transaction = this.db.transaction(() => { + this.db + .query( + `UPDATE documents + SET status = 'planned', remote_id = NULL, response_json = NULL, + last_error = NULL, updated_at = ? + WHERE build_id = ?` + ) + .run(now(), buildId) + this.db + .query( + `UPDATE trajectories + SET status = 'planned', attempts = 0, lease_owner = NULL, + lease_expires_at = NULL, error = NULL, updated_at = ? + WHERE build_id = ?` + ) + .run(now(), buildId) + this.db + .query( + `UPDATE builds SET status = 'planned', error = NULL, updated_at = ? + WHERE build_id = ?` + ) + .run(now(), buildId) + this.recordEvent(buildId, "build_explicitly_reset", "build", buildId) + }) + transaction.immediate() + } + + markDocumentSubmitting(customId: string): void { + this.db + .query( + `UPDATE documents SET status = 'submitting', attempts = attempts + 1, + last_error = NULL, updated_at = ? WHERE custom_id = ?` + ) + .run(now(), customId) + } + + markDocumentAccepted(customId: string, remoteId: string, response: unknown): void { + this.db + .query( + `UPDATE documents SET status = 'accepted', remote_id = ?, response_json = ?, + last_error = NULL, updated_at = ? WHERE custom_id = ?` + ) + .run(remoteId, canonicalJson(response), now(), customId) + } + + markDocumentIndexing(customId: string, remoteId?: string): void { + this.db + .query( + `UPDATE documents SET status = 'indexing', remote_id = COALESCE(?, remote_id), + updated_at = ? WHERE custom_id = ?` + ) + .run(remoteId ?? null, now(), customId) + } + + markDocumentReady(customId: string, remoteId?: string): void { + this.db + .query( + `UPDATE documents SET status = 'ready', remote_id = COALESCE(?, remote_id), + last_error = NULL, updated_at = ? WHERE custom_id = ?` + ) + .run(remoteId ?? null, now(), customId) + } + + resetDocumentToPlanned(customId: string): void { + this.db + .query( + `UPDATE documents SET status = 'planned', remote_id = NULL, response_json = NULL, + last_error = NULL, updated_at = ? WHERE custom_id = ?` + ) + .run(now(), customId) + } + + markDocumentRetryable(customId: string, error: string): void { + this.db + .query( + `UPDATE documents SET status = 'retryable', last_error = ?, updated_at = ? + WHERE custom_id = ?` + ) + .run(error.slice(0, 2000), now(), customId) + } + + markDocumentFailed(customId: string, error: string): void { + this.db + .query( + `UPDATE documents SET status = 'failed', last_error = ?, updated_at = ? + WHERE custom_id = ?` + ) + .run(error.slice(0, 2000), now(), customId) + } + + buildSummary(buildId: string): { + trajectories: Record + documents: Record + } { + const trajectoryCounts = Object.fromEntries( + ["planned", "processing", "ready", "retryable", "failed"].map((status) => [status, 0]) + ) as Record + const documentCounts = Object.fromEntries( + ["planned", "submitting", "accepted", "indexing", "ready", "retryable", "failed"].map( + (status) => [status, 0] + ) + ) as Record + const trajectories = this.db + .query( + "SELECT status, COUNT(*) AS count FROM trajectories WHERE build_id = ? GROUP BY status" + ) + .all(buildId) as Array<{ status: TrajectoryStatus; count: number }> + const documents = this.db + .query("SELECT status, COUNT(*) AS count FROM documents WHERE build_id = ? GROUP BY status") + .all(buildId) as Array<{ status: BuildDocumentStatus; count: number }> + for (const row of trajectories) trajectoryCounts[row.status] = row.count + for (const row of documents) documentCounts[row.status] = row.count + return { trajectories: trajectoryCounts, documents: documentCounts } + } + + recordEvent( + buildId: string, + eventType: string, + entityType?: string, + entityId?: string, + details?: unknown + ): void { + this.db + .query( + `INSERT INTO events + (build_id, event_type, entity_type, entity_id, details_json, created_at) + VALUES (?, ?, ?, ?, ?, ?)` + ) + .run( + buildId, + eventType, + entityType ?? null, + entityId ?? null, + details === undefined ? null : canonicalJson(details), + now() + ) + } + + close(): void { + this.db.close() + } +} diff --git a/src/core/canonical.ts b/src/core/canonical.ts new file mode 100644 index 0000000..1927f09 --- /dev/null +++ b/src/core/canonical.ts @@ -0,0 +1,61 @@ +import { createHash } from "node:crypto" +import { mkdir, readFile, rename, writeFile } from "node:fs/promises" +import { dirname } from "node:path" + +export type JsonPrimitive = string | number | boolean | null +export type JsonValue = JsonPrimitive | JsonValue[] | { [key: string]: JsonValue } + +function normalizeForJson(value: unknown): JsonValue { + if (value === null || typeof value === "string" || typeof value === "boolean") { + return value + } + if (typeof value === "number") { + if (!Number.isFinite(value)) { + throw new TypeError(`Cannot canonicalize non-finite number: ${value}`) + } + return value + } + if (Array.isArray(value)) { + return value.map(normalizeForJson) + } + if (typeof value === "object") { + const record = value as Record + const output: Record = {} + for (const key of Object.keys(record).sort()) { + const item = record[key] + if (item !== undefined) output[key] = normalizeForJson(item) + } + return output + } + throw new TypeError(`Cannot canonicalize ${typeof value}`) +} + +export function canonicalJson(value: unknown): string { + return JSON.stringify(normalizeForJson(value)) +} + +export function sha256(value: string | Uint8Array): string { + return createHash("sha256").update(value).digest("hex") +} + +export function stableHash(value: unknown): string { + return sha256(canonicalJson(value)) +} + +export async function sha256File(path: string): Promise { + return sha256(await readFile(path)) +} + +export async function atomicWriteJson(path: string, value: unknown): Promise { + await mkdir(dirname(path), { recursive: true }) + const temporaryPath = `${path}.tmp-${process.pid}-${crypto.randomUUID()}` + await writeFile(temporaryPath, `${JSON.stringify(value, null, 2)}\n`, { + encoding: "utf8", + mode: 0o600, + }) + await rename(temporaryPath, path) +} + +export async function readJson(path: string): Promise { + return JSON.parse(await readFile(path, "utf8")) as T +} diff --git a/src/core/document-plan.ts b/src/core/document-plan.ts new file mode 100644 index 0000000..2c75f7a --- /dev/null +++ b/src/core/document-plan.ts @@ -0,0 +1,311 @@ +import { existsSync } from "node:fs" +import type { + DocumentPlan, + DocumentSpec, + MetadataValue, + PhysicalDocument, + ValidatedDocumentPlan, +} from "../types/migration" +import { canonicalJson, sha256, stableHash } from "./canonical" + +export const DOCUMENT_PLAN_VERSION = 1 +export const SPLITTER_VERSION = 1 + +export const RESERVED_METADATA_KEYS = new Set([ + "benchmark", + "adapterSchemaVersion", + "buildFingerprint", + "runFingerprint", + "tier", + "domain", + "haystackHash", + "trajectoryId", + "trajectoryOrder", + "documentType", + "stateIndex", + "step", + "documentOrdinal", + "partIndex", + "partCount", + "contentHash", + "screenshotPath", + "logicalDocumentId", +]) + +const METADATA_KEY = /^[a-zA-Z0-9_.-]+$/ +const LOGICAL_ID = /^[A-Za-z0-9_.:-]+$/ +const METADATA_KEY_MAX = 100 +const METADATA_VALUE_MAX = 1024 + +export interface TrajectoryConverter { + readonly name: string + readonly version: number + readonly sourceHash: string + convert(trajectory: TTrajectory, context: TContext): DocumentPlan +} + +function portableDocumentSpec(document: DocumentSpec): unknown { + return { + ...document, + screenshotRef: document.screenshotRef + ? { ...document.screenshotRef, absolutePath: undefined } + : undefined, + } +} + +function portableDocumentPlan(plan: DocumentPlan): unknown { + return { + ...plan, + documents: plan.documents.map(portableDocumentSpec), + } +} + +function fail(trajectoryId: string, message: string): never { + throw new Error(`[trajectory ${trajectoryId}] ${message}`) +} + +function validateMetadata( + trajectoryId: string, + logicalId: string, + metadata: Record +): void { + for (const [key, value] of Object.entries(metadata)) { + if (!key || key.length > METADATA_KEY_MAX || !METADATA_KEY.test(key)) { + fail( + trajectoryId, + `${logicalId}: metadata key ${JSON.stringify(key)} must match [a-zA-Z0-9_.-]+ and be <= ${METADATA_KEY_MAX} chars` + ) + } + if (RESERVED_METADATA_KEYS.has(key)) { + fail(trajectoryId, `${logicalId}: metadata key ${key} is reserved`) + } + const values = Array.isArray(value) ? value : [value] + if (Array.isArray(value) && !value.every((item) => typeof item === "string")) { + fail(trajectoryId, `${logicalId}: metadata arrays may contain only strings`) + } + if ( + values.some( + (item) => + !["string", "number", "boolean"].includes(typeof item) || + (typeof item === "string" && item.length > METADATA_VALUE_MAX) || + (typeof item === "number" && !Number.isFinite(item)) + ) + ) { + fail(trajectoryId, `${logicalId}: unsupported metadata value for ${key}`) + } + } +} + +function topologicalOrder(trajectoryId: string, documents: DocumentSpec[]): number[] { + const indexes = new Map(documents.map((document, index) => [document.logicalDocumentId, index])) + const degrees = documents.map(() => 0) + const dependents = documents.map(() => [] as number[]) + for (const [index, document] of documents.entries()) { + for (const dependency of document.dependsOn) { + const dependencyIndex = indexes.get(dependency) + if (dependencyIndex === undefined) { + fail(trajectoryId, `${document.logicalDocumentId}: dependency ${dependency} does not exist`) + } + degrees[index] += 1 + dependents[dependencyIndex].push(index) + } + } + const ready = degrees + .map((degree, index) => ({ degree, index })) + .filter(({ degree }) => degree === 0) + .map(({ index }) => index) + const order: number[] = [] + while (ready.length > 0) { + ready.sort((a, b) => a - b) + const index = ready.shift()! + order.push(index) + for (const dependent of dependents[index]) { + degrees[dependent] -= 1 + if (degrees[dependent] === 0) ready.push(dependent) + } + } + if (order.length !== documents.length) { + const cyclic = documents + .filter((_, index) => degrees[index] > 0) + .map((document) => document.logicalDocumentId) + fail(trajectoryId, `dependency graph contains a cycle involving ${cyclic.join(", ")}`) + } + return order +} + +export function validateDocumentPlan(input: { + plan: DocumentPlan + converter: TrajectoryConverter + trajectory: TTrajectory + context: TContext + checkDeterminism?: boolean +}): ValidatedDocumentPlan { + const { plan, converter, trajectory, context, checkDeterminism = true } = input + const trajectoryId = plan.trajectoryId + if (!trajectoryId) fail("", "trajectoryId is required") + if (plan.documents.length === 0) fail(trajectoryId, "plan contains no documents") + if (plan.batchUpload && plan.documents.some((document) => document.dependsOn.length > 0)) { + fail(trajectoryId, "batch-upload plans cannot declare dependencies") + } + + const ids = new Set() + const contentIds = new Map() + for (const document of plan.documents) { + const id = document.logicalDocumentId + if (!id || !LOGICAL_ID.test(id)) { + fail(trajectoryId, `invalid logicalDocumentId ${JSON.stringify(id)}`) + } + if (ids.has(id)) fail(trajectoryId, `duplicate logicalDocumentId ${id}`) + ids.add(id) + if (!document.content.trim()) fail(trajectoryId, `${id}: content must not be empty`) + if (new Set(document.dependsOn).size !== document.dependsOn.length) { + fail(trajectoryId, `${id}: duplicate dependency`) + } + if (document.dependsOn.includes(id)) fail(trajectoryId, `${id}: self dependency`) + if ( + document.stateIndex !== undefined && + (!Number.isInteger(document.stateIndex) || document.stateIndex < 0) + ) { + fail(trajectoryId, `${id}: stateIndex must be an integer >= 0`) + } + if (document.step !== undefined && (!Number.isInteger(document.step) || document.step < 0)) { + fail(trajectoryId, `${id}: step must be an integer >= 0`) + } + validateMetadata(trajectoryId, id, document.metadata) + for (const attachment of document.localAttachmentPaths) { + if (!existsSync(attachment)) fail(trajectoryId, `${id}: missing attachment ${attachment}`) + } + if ( + document.screenshotRef && + (!document.screenshotRef.absolutePath || !existsSync(document.screenshotRef.absolutePath)) + ) { + fail( + trajectoryId, + `${id}: missing screenshot ${document.screenshotRef.absolutePath ?? "(unresolved)"}` + ) + } + const contentHash = sha256(document.content) + const earlier = contentIds.get(contentHash) + if (earlier && !document.allowDuplicateContent) { + fail(trajectoryId, `${id}: content duplicates ${earlier}`) + } + if (!earlier) contentIds.set(contentHash, id) + } + + for (const document of plan.documents) { + for (const dependency of document.dependsOn) { + if (!ids.has(dependency)) { + fail(trajectoryId, `${document.logicalDocumentId}: unknown dependency ${dependency}`) + } + } + } + + if (checkDeterminism) { + const repeated = converter.convert(trajectory, context) + if ( + canonicalJson(portableDocumentPlan(repeated)) !== canonicalJson(portableDocumentPlan(plan)) + ) { + fail(trajectoryId, `${converter.name} produced non-deterministic output`) + } + } + + const order = topologicalOrder(trajectoryId, plan.documents) + const ordinalByOriginalIndex = new Map( + order.map((originalIndex, ordinal) => [originalIndex, ordinal]) + ) + const indexById = new Map( + plan.documents.map((document, index) => [document.logicalDocumentId, index]) + ) + const documents = order.map((originalIndex, documentOrdinal) => { + const spec = plan.documents[originalIndex] + return { + spec, + documentOrdinal, + contentHash: sha256(spec.content), + dependsOnOrdinals: spec.dependsOn + .map((id) => ordinalByOriginalIndex.get(indexById.get(id)!)!) + .sort((a, b) => a - b), + } + }) + return { + trajectoryId, + planHash: stableHash(portableDocumentPlan(plan)), + documents, + batchUpload: plan.batchUpload, + declaredInvariants: [...plan.declaredInvariants], + } +} + +export function splitContent(content: string, maxChars: number): string[] { + if (!Number.isInteger(maxChars) || maxChars < 1) { + throw new Error("maxChars must be an integer >= 1") + } + if (content.length <= maxChars) return [content] + const parts: string[] = [] + let remaining = content + while (remaining.length > maxChars) { + const window = remaining.slice(0, maxChars + 1) + let cut = -1 + for (const marker of ["\n\n", "\n", " "]) { + const candidate = window.lastIndexOf(marker, maxChars) + if (candidate >= Math.max(1, Math.floor(maxChars / 2))) { + cut = candidate + marker.length + break + } + } + if (cut <= 0 || cut > maxChars) cut = maxChars + parts.push(remaining.slice(0, cut)) + remaining = remaining.slice(cut) + } + if (remaining) parts.push(remaining) + if (parts.some((part) => part.length === 0 || part.length > maxChars)) { + throw new Error("splitter produced an invalid part") + } + if (parts.join("") !== content) throw new Error("splitter reassembly mismatch") + return parts +} + +export function createPhysicalDocuments(input: { + plan: ValidatedDocumentPlan + buildFingerprint: string + maxDocumentChars: number +}): PhysicalDocument[] { + const { plan, buildFingerprint, maxDocumentChars } = input + const output: PhysicalDocument[] = [] + for (const document of plan.documents) { + const parts = splitContent(document.spec.content, maxDocumentChars) + if (plan.batchUpload && parts.length > 1) { + fail( + plan.trajectoryId, + `${document.spec.logicalDocumentId}: a batch document cannot be split` + ) + } + parts.forEach((content, partIndex) => { + const contentHash = sha256(content) + const customId = `lme2-${sha256( + canonicalJson({ + buildFingerprint, + trajectoryId: plan.trajectoryId, + documentOrdinal: document.documentOrdinal, + partIndex, + }) + ).slice(0, 56)}` + output.push({ + trajectoryId: plan.trajectoryId, + logicalDocumentId: document.spec.logicalDocumentId, + documentOrdinal: document.documentOrdinal, + partIndex, + partCount: parts.length, + content, + contentHash, + customId, + documentType: document.spec.documentType, + stateIndex: document.spec.stateIndex, + step: document.spec.step, + screenshotRef: document.spec.screenshotRef, + metadata: { ...document.spec.metadata }, + }) + }) + } + return output +} diff --git a/src/core/fingerprints.ts b/src/core/fingerprints.ts new file mode 100644 index 0000000..9cbcac4 --- /dev/null +++ b/src/core/fingerprints.ts @@ -0,0 +1,93 @@ +import type { + EvaluationArtifact, + MemoryBuildPlan, + QueryArtifact, + RetrievalConfig, +} from "../types/migration" +import { sha256, stableHash } from "./canonical" + +export const FINGERPRINT_SCHEMA_VERSION = 1 + +export function buildFingerprint(input: { + benchmark: string + datasetFingerprint: string + tier: string + domain: string + orderedSourceIds: string[] + sourceContentHashes: string[] + converter: { name: string; version: number; sourceHash: string } + validatedPlanHashes: string[] + provider: string + providerBuildConfig: Record + documentPlanVersion: number + splitterVersion: number +}): string { + return stableHash({ schemaVersion: FINGERPRINT_SCHEMA_VERSION, ...input }) +} + +export function queryFingerprint(input: { + buildFingerprint: string + questionText: string + questionImageHash?: string + retrieval: RetrievalConfig + normalizerVersion: number +}): string { + return stableHash({ + schemaVersion: FINGERPRINT_SCHEMA_VERSION, + buildFingerprint: input.buildFingerprint, + questionTextHash: sha256(input.questionText), + questionImageHash: input.questionImageHash, + retrieval: input.retrieval, + normalizerVersion: input.normalizerVersion, + }) +} + +export function readerFingerprint(input: { + queryArtifact: Pick + model: string + settings: Record + promptVersion: string + imageHashes: string[] + contextBudgetVersion: string +}): string { + return stableHash({ + schemaVersion: FINGERPRINT_SCHEMA_VERSION, + queryArtifact: { + queryFingerprint: input.queryArtifact.queryFingerprint, + normalizedResults: input.queryArtifact.normalizedResults.map((result) => ({ + ...result, + screenshotRefs: result.screenshotRefs.map((asset) => ({ + ...asset, + absolutePath: undefined, + })), + })), + }, + model: input.model, + settings: input.settings, + promptVersion: input.promptVersion, + imageHashes: input.imageHashes, + contextBudgetVersion: input.contextBudgetVersion, + }) +} + +export function evaluatorFingerprint(input: { + answerArtifactHash: string + groundTruth: string + evalFunction: string + evaluatorModel?: string + settings: Record + promptVersion: string + implementationVersion: string +}): string { + return stableHash({ schemaVersion: FINGERPRINT_SCHEMA_VERSION, ...input }) +} + +export function memoryBuildId(plan: Pick): string { + return `mb-${plan.buildFingerprint.slice(0, 24)}` +} + +export function evaluationArtifactHash( + artifact: Pick +): string { + return stableHash(artifact) +} diff --git a/src/core/migration-core.test.ts b/src/core/migration-core.test.ts new file mode 100644 index 0000000..a882fe0 --- /dev/null +++ b/src/core/migration-core.test.ts @@ -0,0 +1,1065 @@ +import { afterEach, describe, expect, test } from "bun:test" +import { mkdtemp, mkdir, readFile, rm, symlink, unlink, writeFile } from "node:fs/promises" +import { join } from "node:path" +import { tmpdir } from "node:os" +import type { + BuildBatchRequest, + BuildProvider, + BuildSearchRequest, + BuildSearchResponse, + RemoteDocumentState, +} from "../types/provider" +import type { + AssetRef, + DocumentPlan, + DocumentSpec, + MemoryBuildPlan, + NormalizedRetrievalResult, + ProviderCapabilities, + RetrievalConfig, + ValidatedDocumentPlan, +} from "../types/migration" +import { ArtifactStore } from "./artifact-store" +import { BuildEngine } from "./build-engine" +import { BuildStore } from "./build-store" +import { canonicalJson, sha256, stableHash } from "./canonical" +import { + createPhysicalDocuments, + splitContent, + type TrajectoryConverter, + validateDocumentPlan, +} from "./document-plan" +import { + buildFingerprint, + evaluatorFingerprint, + memoryBuildId, + queryFingerprint, + readerFingerprint, +} from "./fingerprints" +import { requireProviderCapabilities } from "./provider-capabilities" +import { QueryRunner } from "./query-runner" + +const temporaryRoots: string[] = [] + +async function temporaryRoot(prefix: string): Promise { + const root = await mkdtemp(join(tmpdir(), prefix)) + temporaryRoots.push(root) + return root +} + +afterEach(async () => { + await Promise.all( + temporaryRoots.splice(0).map((root) => rm(root, { recursive: true, force: true })) + ) +}) + +const CAPABILITIES: ProviderCapabilities = { + deterministicExternalIds: true, + batchUpload: true, + documentDependencies: false, + ingestionMetadataFilters: true, + searchMetadataFilters: true, + searchModes: ["hybrid", "memories"], + reranking: true, + queryRewriting: true, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, +} + +const RETRIEVAL: RetrievalConfig = { + topK: 2, + threshold: 0, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: false, + metadataFilter: { runFingerprint: "build-fingerprint" }, +} + +function documentSpec( + logicalDocumentId: string, + content = `content-${logicalDocumentId}`, + overrides: Partial = {} +): DocumentSpec { + return { + logicalDocumentId, + content, + metadata: {}, + sourceStateIndices: [], + localAttachmentPaths: [], + dependsOn: [], + allowParallelUpload: true, + documentType: "state", + allowDuplicateContent: false, + ...overrides, + } +} + +class StaticConverter implements TrajectoryConverter<{ id: string }, { suffix?: string }> { + readonly name = "test-converter" + readonly version = 1 + readonly sourceHash = "source-hash" + + constructor(private readonly factory: () => DocumentPlan) {} + + convert(): DocumentPlan { + return this.factory() + } +} + +function validatedPlan( + trajectoryId: string, + documents = [documentSpec("state-0000")], + batchUpload = true +): ValidatedDocumentPlan { + const plan: DocumentPlan = { + trajectoryId, + documents, + batchUpload, + declaredInvariants: ["test invariant"], + } + return validateDocumentPlan({ + plan, + converter: new StaticConverter(() => plan), + trajectory: { id: trajectoryId }, + context: {}, + }) +} + +function buildPlan( + trajectoryIds = ["trajectory-1"], + options: { documentCount?: number; fingerprint?: string } = {} +): MemoryBuildPlan { + const fingerprint = options.fingerprint ?? "b".repeat(64) + const plans = trajectoryIds.map((trajectoryId) => + validatedPlan( + trajectoryId, + Array.from({ length: options.documentCount ?? 1 }, (_, index) => + documentSpec(`state-${index.toString().padStart(4, "0")}`, `${trajectoryId}-${index}`) + ) + ) + ) + const documents = plans.flatMap((plan) => + createPhysicalDocuments({ + plan, + buildFingerprint: fingerprint, + maxDocumentChars: 10_000, + }) + ) + return { + schemaVersion: 1, + buildId: `build-${fingerprint.slice(0, 8)}`, + benchmark: "longmemeval-v2", + provider: "fake", + datasetFingerprint: "dataset-fingerprint", + tier: "small", + domain: "web", + orderedSourceIds: [...trajectoryIds], + sourceContentHashes: trajectoryIds.map((id) => sha256(id)), + converter: { + name: "structured-accessibility", + version: 1, + sourceHash: "converter-source", + }, + providerBuildConfig: { dreaming: "instant" }, + buildFingerprint: fingerprint, + containerTag: `lme2-test-${fingerprint.slice(0, 12)}`, + documentPlans: plans, + documents, + } +} + +function result( + rank: number, + overrides: Partial = {} +): NormalizedRetrievalResult { + return { + rank, + score: 1 - rank / 10, + kind: "memory", + text: `result-${rank}`, + chunks: [], + documentIds: [`remote-${rank}`], + screenshotRefs: [], + provenanceValid: true, + ...overrides, + } +} + +class FakeBuildProvider implements BuildProvider { + readonly name = "fake" + readonly capabilities = CAPABILITIES + readonly remote = new Map() + readonly submittedCustomIds: string[] = [] + submitCalls = 0 + reconcileCalls = 0 + searchCalls = 0 + deleteCalls = 0 + deleteFailure = false + ambiguousSubmitOnce = false + failuresBeforeStore = 0 + permanentFailure = false + reconcileDelayMs = 0 + pollsBeforeReady = 0 + searchResults: NormalizedRetrievalResult[] = [result(0)] + remoteDurationMs = 321 + + async submitDocumentBatch(request: BuildBatchRequest): Promise { + this.submitCalls += 1 + if (this.failuresBeforeStore > 0) { + this.failuresBeforeStore -= 1 + throw new Error("transport failed before remote store") + } + const states = request.documents.map((document) => { + this.submittedCustomIds.push(document.customId) + const state: RemoteDocumentState = this.permanentFailure + ? { + customId: document.customId, + remoteId: `remote-${document.customId}`, + status: "failed", + error: "permanent remote failure", + } + : { + customId: document.customId, + remoteId: `remote-${document.customId}`, + status: this.pollsBeforeReady > 0 ? "pending" : "ready", + } + this.remote.set(document.customId, state) + return { ...state } + }) + if (this.ambiguousSubmitOnce) { + this.ambiguousSubmitOnce = false + throw new Error("response lost after remote success") + } + return states + } + + async reconcileDocuments( + _build: MemoryBuildPlan, + customIds: string[] + ): Promise { + this.reconcileCalls += 1 + if (this.reconcileDelayMs > 0) await Bun.sleep(this.reconcileDelayMs) + return customIds.map((customId) => { + const current = this.remote.get(customId) + if (!current) return { customId, status: "absent" } + if (current.status === "pending" && this.reconcileCalls > this.pollsBeforeReady) { + const ready = { ...current, status: "ready" as const } + this.remote.set(customId, ready) + return ready + } + return { ...current } + }) + } + + async searchBuild(request: BuildSearchRequest): Promise { + this.searchCalls += 1 + return { + request: { + containerTag: request.build.containerTag, + filters: { runFingerprint: request.build.buildFingerprint }, + limit: request.config.topK, + }, + rawResponse: { results: this.searchResults }, + normalizedResults: this.searchResults.map((item) => ({ ...item })), + remoteDurationMs: this.remoteDurationMs, + } + } + + async verifyBuildHealth(build: MemoryBuildPlan): Promise { + return build.documents.map( + (document) => + this.remote.get(document.customId) ?? { + customId: document.customId, + status: "absent", + } + ) + } + + async deleteDocuments(_build: MemoryBuildPlan, customIds: string[]): Promise { + this.deleteCalls += 1 + if (this.deleteFailure) throw new Error("HTTP 409: document is still processing") + for (const customId of customIds) this.remote.delete(customId) + } + + async clearBuild(): Promise { + this.remote.clear() + } +} + +describe("canonical values and four-level fingerprints", () => { + test("canonical JSON is key-order stable and rejects unsafe values", () => { + expect(canonicalJson({ z: 1, a: { d: 4, b: 2 }, omitted: undefined })).toBe( + '{"a":{"b":2,"d":4},"z":1}' + ) + expect(stableHash({ a: 1, b: 2 })).toBe(stableHash({ b: 2, a: 1 })) + expect(() => canonicalJson({ bad: Number.NaN })).toThrow("non-finite") + expect(() => canonicalJson({ bad: BigInt(1) })).toThrow("bigint") + expect(() => canonicalJson({ bad: () => true })).toThrow("function") + }) + + test("build identity is deterministic, order-sensitive, and independent from query settings", () => { + const input = { + benchmark: "longmemeval-v2", + datasetFingerprint: "dataset", + tier: "small", + domain: "web", + orderedSourceIds: ["a", "b"], + sourceContentHashes: ["ha", "hb"], + converter: { name: "structured", version: 1, sourceHash: "source" }, + validatedPlanHashes: ["pa", "pb"], + provider: "supermemory", + providerBuildConfig: { dreaming: "instant", metadata: { b: 2, a: 1 } }, + documentPlanVersion: 1, + splitterVersion: 1, + } + const first = buildFingerprint(input) + expect(buildFingerprint({ ...input })).toBe(first) + expect( + buildFingerprint({ + ...input, + providerBuildConfig: { metadata: { a: 1, b: 2 }, dreaming: "instant" }, + }) + ).toBe(first) + expect(buildFingerprint({ ...input, orderedSourceIds: ["b", "a"] })).not.toBe(first) + expect(buildFingerprint({ ...input, sourceContentHashes: ["changed", "hb"] })).not.toBe(first) + expect(memoryBuildId({ buildFingerprint: first })).toBe(`mb-${first.slice(0, 24)}`) + }) + + test("query, reader, and evaluator fingerprints invalidate only their dependencies", () => { + const queryA = queryFingerprint({ + buildFingerprint: "build", + questionText: "question", + retrieval: RETRIEVAL, + normalizerVersion: 1, + }) + const queryB = queryFingerprint({ + buildFingerprint: "build", + questionText: "question", + retrieval: { ...RETRIEVAL, topK: 1 }, + normalizerVersion: 1, + }) + expect(queryB).not.toBe(queryA) + + const queryArtifact = { + queryFingerprint: queryA, + normalizedResults: [result(0)], + } + const readerA = readerFingerprint({ + queryArtifact, + model: "gpt-5", + settings: { reasoningEffort: "high" }, + promptVersion: "p1", + imageHashes: ["image-a"], + contextBudgetVersion: "budget-1", + }) + expect( + readerFingerprint({ + queryArtifact, + model: "gpt-5", + settings: { reasoningEffort: "high" }, + promptVersion: "p1", + imageHashes: ["image-b"], + contextBudgetVersion: "budget-1", + }) + ).not.toBe(readerA) + + const evaluatorA = evaluatorFingerprint({ + answerArtifactHash: "answer", + groundTruth: "gold", + evalFunction: "mc_choice_match|require_non_empty=true", + settings: {}, + promptVersion: "prompt-1", + implementationVersion: "implementation-1", + }) + expect( + evaluatorFingerprint({ + answerArtifactHash: "answer", + groundTruth: "gold", + evalFunction: "mc_choice_match|require_non_empty=true", + settings: {}, + promptVersion: "prompt-2", + implementationVersion: "implementation-1", + }) + ).not.toBe(evaluatorA) + }) +}) + +describe("document-plan validation and lossless physical documents", () => { + test("validates deterministic plans and creates stable custom IDs", () => { + const plan: DocumentPlan = { + trajectoryId: "trajectory", + documents: [ + documentSpec("overview", "goal"), + documentSpec("state", "observed state", { + dependsOn: ["overview"], + allowParallelUpload: false, + }), + ], + batchUpload: false, + declaredInvariants: ["no question leakage"], + } + const converter = new StaticConverter(() => plan) + const validatedA = validateDocumentPlan({ + plan, + converter, + trajectory: { id: "trajectory" }, + context: {}, + }) + const validatedB = validateDocumentPlan({ + plan, + converter, + trajectory: { id: "trajectory" }, + context: {}, + }) + expect(validatedA.planHash).toBe(validatedB.planHash) + expect(validatedA.documents[1].dependsOnOrdinals).toEqual([0]) + const physicalA = createPhysicalDocuments({ + plan: validatedA, + buildFingerprint: "f".repeat(64), + maxDocumentChars: 100, + }) + const physicalB = createPhysicalDocuments({ + plan: validatedB, + buildFingerprint: "f".repeat(64), + maxDocumentChars: 100, + }) + expect(physicalA).toEqual(physicalB) + expect(physicalA.every((item) => /^lme2-[a-f0-9]{56}$/.test(item.customId))).toBeTrue() + }) + + test("rejects malformed identities, metadata, dependencies, duplicate content, and drift", () => { + const cases: Array<{ plan: DocumentPlan; message: string }> = [ + { + plan: { + trajectoryId: "t", + documents: [documentSpec("bad id")], + batchUpload: false, + declaredInvariants: [], + }, + message: "invalid logicalDocumentId", + }, + { + plan: { + trajectoryId: "t", + documents: [documentSpec("a", "a", { metadata: { runFingerprint: "x" } })], + batchUpload: false, + declaredInvariants: [], + }, + message: "reserved", + }, + { + plan: { + trajectoryId: "t", + documents: [ + documentSpec("a", "a", { dependsOn: ["b"] }), + documentSpec("b", "b", { dependsOn: ["a"] }), + ], + batchUpload: false, + declaredInvariants: [], + }, + message: "cycle", + }, + { + plan: { + trajectoryId: "t", + documents: [documentSpec("a", "same"), documentSpec("b", "same")], + batchUpload: false, + declaredInvariants: [], + }, + message: "duplicates", + }, + { + plan: { + trajectoryId: "t", + documents: [documentSpec("a", "a"), documentSpec("b", "b", { dependsOn: ["a"] })], + batchUpload: true, + declaredInvariants: [], + }, + message: "batch-upload", + }, + ] + for (const item of cases) { + expect(() => + validateDocumentPlan({ + plan: item.plan, + converter: new StaticConverter(() => item.plan), + trajectory: { id: "t" }, + context: {}, + }) + ).toThrow(item.message) + } + + let call = 0 + const nondeterministic = new StaticConverter(() => ({ + trajectoryId: "t", + documents: [documentSpec("a", `content-${call++}`)], + batchUpload: false, + declaredInvariants: [], + })) + const first = nondeterministic.convert() + expect(() => + validateDocumentPlan({ + plan: first, + converter: nondeterministic, + trajectory: { id: "t" }, + context: {}, + }) + ).toThrow("non-deterministic") + }) + + test("splits losslessly but rejects oversized documents in a batch plan", () => { + const content = `${"paragraph words ".repeat(80)}\n\n${"尾".repeat(80)}` + const parts = splitContent(content, 100) + expect(parts.join("")).toBe(content) + expect(parts.every((part) => part.length > 0 && part.length <= 100)).toBeTrue() + expect(splitContent(content, 100)).toEqual(parts) + expect(() => splitContent("x", 0)).toThrow("integer >= 1") + + const plan = validatedPlan("trajectory", [documentSpec("state", "x".repeat(101))]) + expect(() => + createPhysicalDocuments({ + plan, + buildFingerprint: "f".repeat(64), + maxDocumentChars: 100, + }) + ).toThrow("batch document cannot be split") + }) +}) + +describe("provider capability gate", () => { + test("fails before work when required capabilities are absent", () => { + expect(() => + requireProviderCapabilities("fake", CAPABILITIES, [ + "deterministicExternalIds", + "batchUpload", + "searchMetadataFilters", + ]) + ).not.toThrow() + expect(() => + requireProviderCapabilities( + "weak", + { ...CAPABILITIES, batchUpload: false, searchModes: [] }, + ["batchUpload", "searchModes"] + ) + ).toThrow("batchUpload, searchModes") + }) +}) + +describe("SQLite build checkpoint and leases", () => { + test("registers idempotently, survives reopen, and rejects plan drift", async () => { + const root = await temporaryRoot("memorybench-build-store-") + const path = join(root, "checkpoint.sqlite3") + const plan = buildPlan(["t1", "t2"], { documentCount: 2 }) + let store = new BuildStore(path) + store.registerBuild(plan) + store.registerBuild(plan) + expect(store.buildSummary(plan.buildId)).toEqual({ + trajectories: { + planned: 2, + processing: 0, + ready: 0, + retryable: 0, + failed: 0, + }, + documents: { + planned: 4, + submitting: 0, + accepted: 0, + indexing: 0, + ready: 0, + retryable: 0, + failed: 0, + }, + }) + store.close() + + store = new BuildStore(path) + expect(store.getBuild(plan.buildId)?.buildFingerprint).toBe(plan.buildFingerprint) + expect(() => store.registerBuild({ ...plan, containerTag: "different-container" })).toThrow( + "does not match" + ) + store.close() + }) + + test("enforces lease ownership, renewal, expiry, and the all-ready barrier", async () => { + const root = await temporaryRoot("memorybench-build-lease-") + const plan = buildPlan() + const storeA = new BuildStore(join(root, "checkpoint.sqlite3")) + const storeB = new BuildStore(join(root, "checkpoint.sqlite3")) + storeA.registerBuild(plan) + expect(storeA.claimTrajectory(plan.buildId, "worker-a", 20)).toBe("trajectory-1") + expect(storeB.claimTrajectory(plan.buildId, "worker-b", 20)).toBeNull() + expect(() => storeB.renewTrajectoryLease(plan.buildId, "trajectory-1", "worker-b", 20)).toThrow( + "no longer owns" + ) + expect(() => storeA.markTrajectoryReady(plan.buildId, "trajectory-1", "worker-a")).toThrow( + "documents are not ready" + ) + storeA.renewTrajectoryLease(plan.buildId, "trajectory-1", "worker-a", 20) + await Bun.sleep(5) + expect(storeB.claimTrajectory(plan.buildId, "worker-b", 20)).toBeNull() + await Bun.sleep(25) + expect(storeB.claimTrajectory(plan.buildId, "worker-b", 20)).toBe("trajectory-1") + storeA.close() + storeB.close() + }) + + test("persists the ambiguous submitting state across a simulated crash", async () => { + const root = await temporaryRoot("memorybench-build-crash-") + const path = join(root, "checkpoint.sqlite3") + const plan = buildPlan() + let store = new BuildStore(path) + store.registerBuild(plan) + expect(store.claimTrajectory(plan.buildId, "crashed-worker", 1)).toBe("trajectory-1") + store.markDocumentSubmitting(plan.documents[0].customId) + store.close() + + store = new BuildStore(path) + expect(store.getAmbiguousDocuments(plan.buildId)).toEqual([ + expect.objectContaining({ + customId: plan.documents[0].customId, + status: "submitting", + attempts: 1, + }), + ]) + store.close() + }) +}) + +describe("durable build engine", () => { + const options = { + trajectoryConcurrency: 2, + maxTrajectoryAttempts: 3, + indexingTimeoutMs: 2_000, + pollIntervalMs: 1, + leaseMs: 100, + sleep: async () => {}, + } + + test("builds concurrently and an identical rerun performs zero uploads", async () => { + const root = await temporaryRoot("memorybench-engine-idempotent-") + const plan = buildPlan(["t1", "t2", "t3"], { documentCount: 2 }) + const provider = new FakeBuildProvider() + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + const engine = new BuildEngine(plan, provider, store, options) + await engine.run() + expect(store.getBuild(plan.buildId)?.status).toBe("ready") + expect(provider.submitCalls).toBe(3) + expect(new Set(provider.submittedCustomIds).size).toBe(plan.documents.length) + await engine.run() + expect(provider.submitCalls).toBe(3) + await engine.verifyRemoteHealth() + store.close() + }) + + test("reconciles a response lost after remote success without duplication", async () => { + const root = await temporaryRoot("memorybench-engine-ambiguous-") + const plan = buildPlan() + const provider = new FakeBuildProvider() + provider.ambiguousSubmitOnce = true + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + await new BuildEngine(plan, provider, store, options).run() + expect(provider.submitCalls).toBe(1) + expect(provider.submittedCustomIds).toEqual([plan.documents[0].customId]) + expect(provider.reconcileCalls).toBeGreaterThan(0) + expect(store.buildSummary(plan.buildId).documents.ready).toBe(1) + store.close() + }) + + test("resumes a crash before remote submission and retries a transport failure", async () => { + const root = await temporaryRoot("memorybench-engine-resume-") + const plan = buildPlan() + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + store.registerBuild(plan) + expect(store.claimTrajectory(plan.buildId, "dead-worker", 1)).toBe("trajectory-1") + store.markDocumentSubmitting(plan.documents[0].customId) + await Bun.sleep(3) + + const provider = new FakeBuildProvider() + provider.failuresBeforeStore = 1 + await new BuildEngine(plan, provider, store, options).run() + expect(provider.submitCalls).toBe(2) + expect(provider.submittedCustomIds).toEqual([plan.documents[0].customId]) + expect(store.getBuild(plan.buildId)?.status).toBe("ready") + store.close() + }) + + test("waits for an unexpired crashed-worker lease instead of failing the build", async () => { + const root = await temporaryRoot("memorybench-engine-live-lease-resume-") + const plan = buildPlan() + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + store.registerBuild(plan) + expect(store.claimTrajectory(plan.buildId, "dead-worker", 40)).toBe("trajectory-1") + store.markDocumentSubmitting(plan.documents[0].customId) + + const provider = new FakeBuildProvider() + await new BuildEngine(plan, provider, store, { + ...options, + pollIntervalMs: 5, + leaseMs: 100, + sleep: (milliseconds) => Bun.sleep(milliseconds), + }).run() + + expect(provider.submitCalls).toBe(1) + expect(store.getTrajectoryAttempt(plan.buildId, "trajectory-1")).toBe(2) + expect(store.getBuild(plan.buildId)?.status).toBe("ready") + store.close() + }) + + test("never marks a partial or permanently failed build ready", async () => { + const root = await temporaryRoot("memorybench-engine-failure-") + const plan = buildPlan() + const provider = new FakeBuildProvider() + provider.permanentFailure = true + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + await expect( + new BuildEngine(plan, provider, store, { + ...options, + maxTrajectoryAttempts: 2, + }).run() + ).rejects.toThrow("Build failed") + expect(store.getBuild(plan.buildId)?.status).toBe("failed") + expect(store.buildSummary(plan.buildId).documents.failed).toBe(1) + expect(provider.submitCalls).toBe(2) + store.close() + }) + + test("degrades after bounded cleanup conflicts leave documents indexing in non-strict mode", async () => { + const root = await temporaryRoot("memorybench-engine-degraded-bounded-failure-") + const plan = buildPlan() + const provider = new FakeBuildProvider() + provider.pollsBeforeReady = Number.MAX_SAFE_INTEGER + provider.deleteFailure = true + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + const engine = new BuildEngine(plan, provider, store, { + ...options, + maxTrajectoryAttempts: 2, + indexingTimeoutMs: 10, + pollIntervalMs: 1, + continueOnIndexingTimeout: true, + sleep: (milliseconds) => Bun.sleep(milliseconds), + }) + + expect(await engine.run()).toBe("degraded") + expect(provider.submitCalls).toBe(1) + expect(provider.deleteCalls).toBe(2) + expect(store.getBuild(plan.buildId)).toMatchObject({ + status: "degraded", + error: expect.stringContaining("bounded ingestion failures"), + }) + expect(store.buildSummary(plan.buildId)).toEqual({ + trajectories: { + planned: 0, + processing: 0, + ready: 0, + retryable: 0, + failed: 1, + }, + documents: { + planned: 0, + submitting: 0, + accepted: 0, + indexing: 1, + ready: 0, + retryable: 0, + failed: 0, + }, + }) + await engine.verifyRemoteHealth({ allowDegraded: true }) + store.close() + }) + + test("bounds indexing waits and records an explicit degraded build", async () => { + const root = await temporaryRoot("memorybench-engine-degraded-timeout-") + const plan = buildPlan() + const provider = new FakeBuildProvider() + provider.pollsBeforeReady = Number.MAX_SAFE_INTEGER + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + const engine = new BuildEngine(plan, provider, store, { + ...options, + indexingTimeoutMs: 10, + pollIntervalMs: 1, + continueOnIndexingTimeout: true, + sleep: (milliseconds) => Bun.sleep(milliseconds), + }) + + expect(await engine.run()).toBe("degraded") + expect(provider.submitCalls).toBe(1) + expect(provider.deleteCalls).toBe(1) + expect(store.getTrajectoryAttempt(plan.buildId, "trajectory-1")).toBe(1) + expect(store.getBuild(plan.buildId)?.status).toBe("degraded") + expect(store.buildSummary(plan.buildId)).toEqual({ + trajectories: { + planned: 0, + processing: 0, + ready: 0, + retryable: 0, + failed: 1, + }, + documents: { + planned: 0, + submitting: 0, + accepted: 0, + indexing: 0, + ready: 0, + retryable: 0, + failed: 1, + }, + }) + await engine.verifyRemoteHealth({ allowDegraded: true }) + + expect(await engine.run()).toBe("degraded") + expect(provider.submitCalls).toBe(1) + expect(provider.deleteCalls).toBe(1) + store.close() + }) + + test("renews the trajectory lease while a provider poll is slow", async () => { + const root = await temporaryRoot("memorybench-engine-heartbeat-") + const plan = buildPlan() + const provider = new FakeBuildProvider() + provider.pollsBeforeReady = 1 + provider.reconcileDelayMs = 180 + const path = join(root, "checkpoint.sqlite3") + const store = new BuildStore(path) + const observer = new BuildStore(path) + const run = new BuildEngine(plan, provider, store, { + ...options, + leaseMs: 90, + pollIntervalMs: 10, + sleep: async () => {}, + }).run() + await Bun.sleep(120) + expect(observer.claimTrajectory(plan.buildId, "lease-thief", 90)).toBeNull() + await run + store.close() + observer.close() + }) + + test("a user stop leaves a claimed trajectory retryable and the next run resumes it", async () => { + const root = await temporaryRoot("memorybench-engine-user-stop-") + const plan = buildPlan() + const provider = new FakeBuildProvider() + provider.pollsBeforeReady = Number.MAX_SAFE_INTEGER + provider.reconcileDelayMs = 30 + const store = new BuildStore(join(root, "checkpoint.sqlite3")) + const controller = new AbortController() + const run = new BuildEngine(plan, provider, store, { + ...options, + signal: controller.signal, + sleep: (milliseconds, signal) => + new Promise((resolve, reject) => { + const timer = setTimeout(resolve, milliseconds) + signal?.addEventListener( + "abort", + () => { + clearTimeout(timer) + reject(signal.reason ?? new Error("stopped")) + }, + { once: true } + ) + }), + }).run() + await Bun.sleep(5) + controller.abort(new Error("Stopped from UI")) + await expect(run).rejects.toThrow("Stopped from UI") + expect(store.buildSummary(plan.buildId).trajectories.retryable).toBe(1) + expect(store.buildSummary(plan.buildId).trajectories.failed).toBe(0) + + provider.pollsBeforeReady = 0 + provider.reconcileDelayMs = 0 + await new BuildEngine(plan, provider, store, options).run() + expect(store.getBuild(plan.buildId)?.status).toBe("ready") + store.close() + }) +}) + +describe("artifact safety and immutable query execution", () => { + test("redacts credentials without destroying non-secret token metrics", async () => { + const root = await temporaryRoot("memorybench-artifacts-redaction-") + const envName = "MEMORYBENCH_TEST_SECRET" + const previous = process.env[envName] + process.env[envName] = "known-secret-value-12345" + try { + const store = new ArtifactStore(root, [envName]) + await store.writeJson("safe/result.json", { + apiKey: "key-by-name", + authorization: "Bearer hidden", + nested: { + value: process.env[envName], + generated: "sm_abcdefghijklmnopqrstuvwxyz", + prompt_tokens: 42, + }, + }) + const text = await readFile(join(root, "safe/result.json"), "utf8") + expect(text).not.toContain("known-secret-value-12345") + expect(text).not.toContain("sm_abcdefghijklmnopqrstuvwxyz") + expect(text).not.toContain("key-by-name") + expect(text).toContain('"prompt_tokens": 42') + } finally { + if (previous === undefined) delete process.env[envName] + else process.env[envName] = previous + } + }) + + test("rejects traversal and symlink escapes from the artifact root", async () => { + const root = await temporaryRoot("memorybench-artifacts-root-") + const outside = await temporaryRoot("memorybench-artifacts-outside-") + const store = new ArtifactStore(root) + expect(() => store.resolve("../outside.json")).toThrow("inside") + await symlink(outside, join(root, "escape")) + await expect(store.writeJson("escape/leak.json", { leaked: true })).rejects.toThrow("symlink") + }) + + test("immutable writes are idempotent and reject collisions", async () => { + const root = await temporaryRoot("memorybench-artifacts-immutable-") + const store = new ArtifactStore(root) + const first = await store.writeImmutable("immutable/value.bin", Buffer.from("first")) + const second = await store.writeImmutable("immutable/value.bin", Buffer.from("first")) + expect(second).toEqual(first) + await expect( + store.writeImmutable("immutable/value.bin", Buffer.from("different")) + ).rejects.toThrow("collision") + }) + + test("query cache preserves remote timing and invalidates by retrieval config", async () => { + const root = await temporaryRoot("memorybench-query-cache-") + const artifacts = new ArtifactStore(root) + const provider = new FakeBuildProvider() + const build = buildPlan() + const runner = new QueryRunner(provider, artifacts) + const first = await runner.run({ + build, + questionId: "question-1", + query: "Where is the setting?", + config: RETRIEVAL, + }) + const second = await runner.run({ + build, + questionId: "question-1", + query: "Where is the setting?", + config: RETRIEVAL, + }) + expect(first.cacheHit).toBeFalse() + expect(second.cacheHit).toBeTrue() + expect(second.remoteDurationMs).toBe(321) + expect(second.wallDurationMs).toBeGreaterThanOrEqual(0) + expect(provider.searchCalls).toBe(1) + + await runner.run({ + build, + questionId: "question-1", + query: "Where is the setting?", + config: { ...RETRIEVAL, topK: 1 }, + }) + expect(provider.searchCalls).toBe(2) + }) + + test("rejects provider topK and provenance violations before persisting a record", async () => { + const root = await temporaryRoot("memorybench-query-contract-") + const artifacts = new ArtifactStore(root) + const provider = new FakeBuildProvider() + const runner = new QueryRunner(provider, artifacts) + provider.searchResults = [result(0), result(1), result(2)] + await expect( + runner.run({ + build: buildPlan(), + questionId: "too-many", + query: "query", + config: RETRIEVAL, + }) + ).rejects.toThrow("violated topK") + + provider.searchResults = [result(0, { provenanceValid: false })] + await expect( + runner.run({ + build: buildPlan(), + questionId: "wrong-provenance", + query: "query", + config: RETRIEVAL, + }) + ).rejects.toThrow("wrong build provenance") + }) + + test("does not accept a cache record whose identity was tampered", async () => { + const root = await temporaryRoot("memorybench-query-tamper-") + const artifacts = new ArtifactStore(root) + const provider = new FakeBuildProvider() + const build = buildPlan() + const runner = new QueryRunner(provider, artifacts) + const first = await runner.run({ + build, + questionId: "question-tamper", + query: "query", + config: RETRIEVAL, + }) + const directory = join(root, "queries", "question-tamper", first.queryFingerprint) + const recordName = (await Array.fromAsync(new Bun.Glob("*.record.json").scan(directory)))[0] + const recordPath = join(directory, recordName) + const record = JSON.parse(await readFile(recordPath, "utf8")) + record.questionId = "different-question" + record.queryFingerprint = "wrong-fingerprint" + await writeFile(recordPath, `${JSON.stringify(record, null, 2)}\n`) + + const second = await runner.run({ + build, + questionId: "question-tamper", + query: "query", + config: RETRIEVAL, + }) + expect(second.cacheHit).toBeFalse() + expect(provider.searchCalls).toBe(2) + }) + + test("a missing normalized artifact forces a fresh provider query", async () => { + const root = await temporaryRoot("memorybench-query-partial-") + const artifacts = new ArtifactStore(root) + const provider = new FakeBuildProvider() + const build = buildPlan() + const runner = new QueryRunner(provider, artifacts) + const first = await runner.run({ + build, + questionId: "question-partial", + query: "query", + config: RETRIEVAL, + }) + await unlink(artifacts.resolve(first.normalizedArtifact.relativePath)) + const second = await runner.run({ + build, + questionId: "question-partial", + query: "query", + config: RETRIEVAL, + }) + expect(second.cacheHit).toBeFalse() + expect(provider.searchCalls).toBe(2) + }) + + test("materialized assets are content-addressed and reject changed bytes", async () => { + const root = await temporaryRoot("memorybench-assets-") + const sourceRoot = await temporaryRoot("memorybench-assets-source-") + const source = join(sourceRoot, "image.png") + await writeFile(source, "image-bytes") + const bytes = Buffer.from("image-bytes") + const asset: AssetRef = { + assetId: "asset-1", + kind: "trajectory-screenshot", + absolutePath: source, + relativePath: "screenshots/image.png", + mimeType: "image/png", + sha256: sha256(bytes), + byteLength: bytes.byteLength, + } + const stored = await new ArtifactStore(root).materializeAsset(asset) + expect(stored.relativePath).toBe(`assets/${asset.sha256}.png`) + expect(await readFile(stored.absolutePath!, "utf8")).toBe("image-bytes") + await writeFile(source, "changed") + await expect(new ArtifactStore(root).materializeAsset(asset)).rejects.toThrow("bytes changed") + }) +}) diff --git a/src/core/provider-capabilities.ts b/src/core/provider-capabilities.ts new file mode 100644 index 0000000..7aca53a --- /dev/null +++ b/src/core/provider-capabilities.ts @@ -0,0 +1,19 @@ +import type { ProviderCapabilities } from "../types/migration" + +export type RequiredCapability = keyof ProviderCapabilities + +export function requireProviderCapabilities( + providerName: string, + available: ProviderCapabilities, + required: RequiredCapability[] +): void { + const missing = required.filter((capability) => { + const value = available[capability] + return Array.isArray(value) ? value.length === 0 : value !== true + }) + if (missing.length > 0) { + throw new Error( + `Provider ${providerName} cannot run this workflow; missing capabilities: ${missing.join(", ")}` + ) + } +} diff --git a/src/core/query-runner.ts b/src/core/query-runner.ts new file mode 100644 index 0000000..e5237a2 --- /dev/null +++ b/src/core/query-runner.ts @@ -0,0 +1,251 @@ +import { readdir } from "node:fs/promises" +import type { BuildProvider } from "../types/provider" +import type { AssetRef, MemoryBuildPlan, QueryArtifact, RetrievalConfig } from "../types/migration" +import { canonicalJson } from "./canonical" +import { queryFingerprint } from "./fingerprints" +import { ArtifactStore } from "./artifact-store" + +export const RETRIEVAL_NORMALIZER_VERSION = 1 + +export interface QueryRunnerInput { + build: MemoryBuildPlan + questionId: string + query: string + questionImage?: AssetRef + config: RetrievalConfig + fresh?: boolean +} + +export class QueryRunner { + constructor( + private readonly provider: BuildProvider, + private readonly artifacts: ArtifactStore + ) {} + + async run(input: QueryRunnerInput): Promise { + const fingerprint = queryFingerprint({ + buildFingerprint: input.build.buildFingerprint, + questionText: input.query, + questionImageHash: input.questionImage?.sha256, + retrieval: input.config, + normalizerVersion: RETRIEVAL_NORMALIZER_VERSION, + }) + const directory = `queries/${input.questionId}/${fingerprint}` + const started = performance.now() + if (!input.fresh) { + const cached = await this.loadCached(directory, { + fingerprint, + questionId: input.questionId, + buildId: input.build.buildId, + buildFingerprint: input.build.buildFingerprint, + query: input.query, + questionImageHash: input.questionImage?.sha256, + config: input.config, + }) + if (cached) { + const assets = new Map( + input.build.documents + .flatMap((document) => (document.screenshotRef ? [document.screenshotRef] : [])) + .flatMap((asset) => [ + [asset.assetId, asset] as const, + [asset.sha256, asset] as const, + [asset.relativePath, asset] as const, + ]) + ) + return { + ...cached, + questionImage: input.questionImage, + normalizedResults: cached.normalizedResults.map((result) => ({ + ...result, + screenshotRefs: result.screenshotRefs.map( + (asset) => + assets.get(asset.assetId) ?? + assets.get(asset.sha256) ?? + assets.get(asset.relativePath) ?? + asset + ), + })), + cacheHit: true, + wallDurationMs: performance.now() - started, + } + } + } + + const response = await this.provider.searchBuild({ + build: input.build, + questionId: input.questionId, + query: input.query, + config: input.config, + }) + if (response.normalizedResults.length > input.config.topK) { + throw new Error( + `Provider ${this.provider.name} violated topK=${input.config.topK}; returned ${response.normalizedResults.length}` + ) + } + const invalid = response.normalizedResults.filter((result) => !result.provenanceValid) + if (invalid.length > 0) { + throw new Error( + `Rejected ${invalid.length} retrieval results with missing or wrong build provenance` + ) + } + + const attempt = `${new Date().toISOString().replace(/[:.]/g, "-")}-${crypto.randomUUID()}` + const rawArtifact = await this.artifacts.writeJson(`${directory}/${attempt}.raw.json`, { + schemaVersion: 1, + questionId: input.questionId, + buildId: input.build.buildId, + buildFingerprint: input.build.buildFingerprint, + request: response.request, + response: response.rawResponse, + remoteDurationMs: response.remoteDurationMs, + createdAt: new Date().toISOString(), + }) + const normalizedArtifact = await this.artifacts.writeJson( + `${directory}/${attempt}.normalized.json`, + { + schemaVersion: 1, + questionId: input.questionId, + queryFingerprint: fingerprint, + results: response.normalizedResults.map((result) => ({ + ...result, + screenshotRefs: result.screenshotRefs.map((asset) => ({ + ...asset, + absolutePath: undefined, + })), + })), + } + ) + const artifact: QueryArtifact = { + schemaVersion: 1, + questionId: input.questionId, + buildId: input.build.buildId, + buildFingerprint: input.build.buildFingerprint, + queryFingerprint: fingerprint, + query: input.query, + questionImage: input.questionImage, + config: input.config, + request: response.request, + rawArtifact, + normalizedArtifact, + normalizedResults: response.normalizedResults, + remoteDurationMs: response.remoteDurationMs, + wallDurationMs: performance.now() - started, + cacheHit: false, + createdAt: new Date().toISOString(), + } + await this.artifacts.writeJson(`${directory}/${attempt}.record.json`, { + ...artifact, + questionImage: artifact.questionImage + ? { ...artifact.questionImage, absolutePath: undefined } + : undefined, + normalizedResults: artifact.normalizedResults.map((result) => ({ + ...result, + screenshotRefs: result.screenshotRefs.map((asset) => ({ + ...asset, + absolutePath: undefined, + })), + })), + }) + return artifact + } + + private async loadCached( + directory: string, + expected: { + fingerprint: string + questionId: string + buildId: string + buildFingerprint: string + query: string + questionImageHash?: string + config: RetrievalConfig + } + ): Promise { + let names: string[] + try { + names = await readdir(this.artifacts.resolve(directory)) + } catch { + return null + } + const recordName = names + .filter((name) => name.endsWith(".record.json")) + .sort() + .at(-1) + if (!recordName) return null + try { + const artifact = await this.artifacts.readJson(`${directory}/${recordName}`) + const recomputedFingerprint = queryFingerprint({ + buildFingerprint: artifact.buildFingerprint, + questionText: artifact.query, + questionImageHash: artifact.questionImage?.sha256, + retrieval: artifact.config, + normalizerVersion: RETRIEVAL_NORMALIZER_VERSION, + }) + if ( + artifact.schemaVersion !== 1 || + artifact.questionId !== expected.questionId || + artifact.buildId !== expected.buildId || + artifact.buildFingerprint !== expected.buildFingerprint || + artifact.queryFingerprint !== expected.fingerprint || + artifact.query !== expected.query || + artifact.questionImage?.sha256 !== expected.questionImageHash || + recomputedFingerprint !== expected.fingerprint || + canonicalJson(artifact.config) !== canonicalJson(expected.config) || + !Number.isFinite(artifact.remoteDurationMs) || + artifact.remoteDurationMs < 0 || + !Number.isFinite(artifact.wallDurationMs) || + artifact.wallDurationMs < 0 || + artifact.normalizedResults.length > expected.config.topK + ) { + return null + } + const prefix = `${directory}/` + if ( + !artifact.rawArtifact.relativePath.startsWith(prefix) || + !artifact.normalizedArtifact.relativePath.startsWith(prefix) + ) { + return null + } + const raw = await this.artifacts.describe(artifact.rawArtifact.relativePath) + const normalized = await this.artifacts.describe(artifact.normalizedArtifact.relativePath) + if ( + raw.sha256 !== artifact.rawArtifact.sha256 || + normalized.sha256 !== artifact.normalizedArtifact.sha256 + ) { + return null + } + const rawPayload = await this.artifacts.readJson<{ + schemaVersion: number + questionId: string + buildId: string + buildFingerprint: string + request: Record + remoteDurationMs: number + }>(artifact.rawArtifact.relativePath) + const normalizedPayload = await this.artifacts.readJson<{ + schemaVersion: number + questionId: string + queryFingerprint: string + results: QueryArtifact["normalizedResults"] + }>(artifact.normalizedArtifact.relativePath) + if ( + rawPayload.schemaVersion !== 1 || + rawPayload.questionId !== expected.questionId || + rawPayload.buildId !== expected.buildId || + rawPayload.buildFingerprint !== expected.buildFingerprint || + rawPayload.remoteDurationMs !== artifact.remoteDurationMs || + canonicalJson(rawPayload.request) !== canonicalJson(artifact.request) || + normalizedPayload.schemaVersion !== 1 || + normalizedPayload.questionId !== expected.questionId || + normalizedPayload.queryFingerprint !== expected.fingerprint || + canonicalJson(normalizedPayload.results) !== canonicalJson(artifact.normalizedResults) + ) { + return null + } + if (artifact.normalizedResults.some((result) => !result.provenanceValid)) return null + return artifact + } catch { + return null + } + } +} diff --git a/src/index.ts b/src/index.ts index df0930c..9e1c5a4 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,4 +1,7 @@ import { cli } from "./cli" const args = process.argv.slice(2) -cli(args).catch(console.error) +cli(args).catch((error) => { + console.error(error instanceof Error ? error.message : error) + process.exitCode = 1 +}) diff --git a/src/orchestrator/build-aware-run-store.test.ts b/src/orchestrator/build-aware-run-store.test.ts new file mode 100644 index 0000000..6335681 --- /dev/null +++ b/src/orchestrator/build-aware-run-store.test.ts @@ -0,0 +1,291 @@ +import { afterEach, describe, expect, test } from "bun:test" +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises" +import { join } from "node:path" +import { tmpdir } from "node:os" +import type { BuildAwareQuestionCheckpoint, BuildAwareRunConfig } from "../types/build-aware" +import { BuildAwareRunStore } from "./build-aware-run-store" + +const temporaryRoots: string[] = [] + +async function temporaryRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "memorybench-build-aware-")) + temporaryRoots.push(root) + return root +} + +afterEach(async () => { + await Promise.all( + temporaryRoots.splice(0).map((root) => rm(root, { recursive: true, force: true })) + ) +}) + +function runConfig(overrides: Partial = {}): BuildAwareRunConfig { + return { + provider: "supermemory", + benchmark: "longmemeval-v2", + mode: "benchmark", + datasetPath: "/machine-a/datasets/longmemeval-v2", + datasetRevision: "f152293e235517d504809563c833d7190b8c713b", + tier: "small", + domain: "web", + questionIds: ["q1", "q2"], + seed: "fixed-seed", + retrieval: { + topK: 20, + threshold: 0, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: false, + metadataFilter: { runFingerprint: "build-fingerprint" }, + }, + reader: { + model: "gpt-5", + reasoningEffort: "high", + maxCompletionTokens: 20_000, + maxContextTokens: 100_000, + evidenceTopK: 20, + maxImages: 20, + maxImageBytes: 20_000_000, + malformedResponseAttempts: 5, + }, + evaluator: { + model: "gpt-5.2", + reasoningEffort: "medium", + maxCompletionTokens: 2_048, + }, + build: { + serviceBaseUrl: "https://api.supermemory.ai", + dreaming: "instant", + rootFilterMode: "self", + maxDocumentChars: 100_000, + trajectoryConcurrency: 20, + maxInFlightRequests: 40, + maxTrajectoryAttempts: 4, + indexingTimeoutMs: 1_800_000, + pollIntervalMs: 2_000, + preflightMaxAgeMs: 24 * 60 * 60_000, + }, + execution: { + buildConcurrency: 1, + questionConcurrency: 4, + }, + ...overrides, + } +} + +function question( + overrides: Partial> = {} +): Omit { + return { + questionId: "q1", + questionType: "static-environment", + question: "Where is the setting?", + groundTruth: "Settings, General", + evalFunction: + "norm_phrase_set_match|lower=true|normalize_hyphen=true|strip_punct=true|separators=,;|require_non_empty=true", + buildId: "shared-web-small-build", + questionImageHash: "image-hash", + ...overrides, + } +} + +describe("BuildAwareRunStore", () => { + test("validates run IDs before resolving filesystem paths", async () => { + const root = await temporaryRoot() + expect(() => new BuildAwareRunStore("../escape", root)).toThrow("runId") + expect(() => new BuildAwareRunStore("contains/slash", root)).toThrow("runId") + expect(() => new BuildAwareRunStore("", root)).toThrow("runId") + expect(() => new BuildAwareRunStore("a".repeat(101), root)).toThrow("runId") + expect(() => new BuildAwareRunStore("safe_Run-01", root)).not.toThrow() + }) + + test("creates a versioned build-aware checkpoint and loads it durably", async () => { + const root = await temporaryRoot() + const store = new BuildAwareRunStore("run-create", root) + const checkpoint = await store.createOrLoad(runConfig()) + expect(checkpoint).toEqual( + expect.objectContaining({ + schemaVersion: 1, + executionModel: "shared-memory-build-v1", + runId: "run-create", + status: "running", + currentStage: "plan", + targetQuestionIds: [], + buildIds: [], + buildLinks: {}, + questions: {}, + }) + ) + expect(checkpoint.configFingerprint).toHaveLength(64) + await store.flush() + + const reopened = new BuildAwareRunStore("run-create", root) + expect(await reopened.exists()).toBeTrue() + expect(await reopened.load()).toEqual(checkpoint) + }) + + test("excludes machine-local dataset path but rejects semantic config drift", async () => { + const root = await temporaryRoot() + const store = new BuildAwareRunStore("run-config", root) + const original = await store.createOrLoad(runConfig()) + const moved = await store.createOrLoad( + runConfig({ datasetPath: "/machine-b/moved/longmemeval-v2" }) + ) + expect(moved.configFingerprint).toBe(original.configFingerprint) + + await expect( + store.createOrLoad( + runConfig({ + retrieval: { ...runConfig().retrieval, topK: 10 }, + }) + ) + ).rejects.toThrow("different configuration") + await expect( + store.createOrLoad( + runConfig({ + datasetRevision: "different-revision", + }) + ) + ).rejects.toThrow("different configuration") + }) + + test("initializes questions idempotently and rejects every identity drift", async () => { + const root = await temporaryRoot() + const store = new BuildAwareRunStore("run-question", root) + const checkpoint = await store.createOrLoad(runConfig()) + await store.initializeQuestion(checkpoint, question()) + await store.initializeQuestion(checkpoint, question()) + expect(checkpoint.questions.q1.stages).toEqual({ + query: { status: "pending" }, + read: { status: "pending" }, + evaluate: { status: "pending" }, + }) + + for (const changed of [ + question({ question: "changed question" }), + question({ groundTruth: "changed ground truth" }), + question({ evalFunction: "mc_choice_match|require_non_empty=true" }), + question({ buildId: "different-build" }), + question({ questionImageHash: "different-image" }), + ]) { + await expect(store.initializeQuestion(checkpoint, changed)).rejects.toThrow( + "changed since checkpoint" + ) + } + }) + + test("persists shared build links and concurrent stage boundaries", async () => { + const root = await temporaryRoot() + const store = new BuildAwareRunStore("run-stages", root) + const checkpoint = await store.createOrLoad(runConfig()) + checkpoint.targetQuestionIds = ["q1", "q2"] + checkpoint.buildIds = ["shared-web-small-build"] + checkpoint.buildLinks = { + q1: "shared-web-small-build", + q2: "shared-web-small-build", + } + await store.initializeQuestion(checkpoint, question()) + await store.initializeQuestion( + checkpoint, + question({ + questionId: "q2", + question: "Which option?", + groundTruth: "B", + evalFunction: "mc_choice_match|require_non_empty=true", + questionImageHash: undefined, + }) + ) + + await Promise.all([ + store.updateQuestionStage(checkpoint, "q1", "query", { + status: "completed", + fingerprint: "query-fingerprint", + artifactPath: "queries/q1/record.json", + durationMs: 125, + cacheHit: false, + }), + store.updateQuestionStage(checkpoint, "q2", "query", { + status: "failed", + error: "provider unavailable", + durationMs: 40, + }), + ]) + await store.setStage(checkpoint, "read") + await store.flush() + + const reloaded = await new BuildAwareRunStore("run-stages", root).load() + expect(reloaded.buildIds).toEqual(["shared-web-small-build"]) + expect(reloaded.buildLinks).toEqual({ + q1: "shared-web-small-build", + q2: "shared-web-small-build", + }) + expect(reloaded.questions.q1.stages.query).toEqual( + expect.objectContaining({ + status: "completed", + fingerprint: "query-fingerprint", + cacheHit: false, + }) + ) + expect(reloaded.questions.q2.stages.query).toEqual( + expect.objectContaining({ + status: "failed", + error: "provider unavailable", + }) + ) + expect(reloaded.currentStage).toBe("read") + }) + + test("records run failure durably without losing question state", async () => { + const root = await temporaryRoot() + const store = new BuildAwareRunStore("run-failure", root) + const checkpoint = await store.createOrLoad(runConfig()) + await store.initializeQuestion(checkpoint, question()) + await store.updateQuestionStage(checkpoint, "q1", "query", { + status: "completed", + artifactPath: "queries/q1/record.json", + }) + await store.fail(checkpoint, new Error("reader failed")) + await store.flush() + + const reloaded = await new BuildAwareRunStore("run-failure", root).load() + expect(reloaded.status).toBe("failed") + expect(reloaded.error).toBe("reader failed") + expect(reloaded.questions.q1.stages.query.status).toBe("completed") + + await store.setStage(checkpoint, "read", "running") + const resumed = await new BuildAwareRunStore("run-failure", root).load() + expect(resumed.status).toBe("running") + expect(resumed.currentStage).toBe("read") + expect(resumed.error).toBeUndefined() + }) + + test("rejects corrupted schema, execution model, and run identity", async () => { + const root = await temporaryRoot() + const store = new BuildAwareRunStore("run-corrupt", root) + const checkpoint = await store.createOrLoad(runConfig()) + for (const corruption of [ + { ...checkpoint, schemaVersion: 2 }, + { ...checkpoint, executionModel: "legacy-question-containers" }, + { ...checkpoint, runId: "another-run" }, + ]) { + await writeFile(store.checkpointPath, `${JSON.stringify(corruption)}\n`) + await expect(store.load()).rejects.toThrow("Invalid build-aware checkpoint") + } + }) + + test("atomic checkpoint persistence leaves parseable JSON", async () => { + const root = await temporaryRoot() + const store = new BuildAwareRunStore("run-json", root) + const checkpoint = await store.createOrLoad(runConfig()) + for (let index = 0; index < 10; index += 1) { + await store.setStage(checkpoint, index % 2 === 0 ? "build" : "query") + JSON.parse(await readFile(store.checkpointPath, "utf8")) + } + await store.flush() + expect(JSON.parse(await readFile(store.checkpointPath, "utf8")).runId).toBe("run-json") + }) +}) diff --git a/src/orchestrator/build-aware-run-store.ts b/src/orchestrator/build-aware-run-store.ts new file mode 100644 index 0000000..d194777 --- /dev/null +++ b/src/orchestrator/build-aware-run-store.ts @@ -0,0 +1,156 @@ +import { access, mkdir, readFile } from "node:fs/promises" +import { resolve } from "node:path" +import type { + BuildAwareQuestionCheckpoint, + BuildAwareRunCheckpoint, + BuildAwareRunConfig, + BuildAwareStage, + StageState, +} from "../types/build-aware" +import { atomicWriteJson, stableHash } from "../core/canonical" + +function validateRunId(runId: string): void { + if (!/^[A-Za-z0-9_-]{1,100}$/.test(runId)) { + throw new Error("runId must match [A-Za-z0-9_-]+ and be <= 100 characters") + } +} + +export class BuildAwareRunStore { + readonly runRoot: string + readonly checkpointPath: string + private saveQueue: Promise = Promise.resolve() + + constructor( + readonly runId: string, + root = "data/runs-v2" + ) { + validateRunId(runId) + this.runRoot = resolve(root, runId) + this.checkpointPath = resolve(this.runRoot, "checkpoint.json") + } + + async exists(): Promise { + try { + await access(this.checkpointPath) + return true + } catch { + return false + } + } + + async createOrLoad(config: BuildAwareRunConfig): Promise { + const configFingerprint = stableHash({ + ...config, + datasetPath: undefined, + }) + if (await this.exists()) { + const checkpoint = await this.load() + if (checkpoint.configFingerprint !== configFingerprint) { + throw new Error( + `Run ${this.runId} already exists with a different configuration; use a new run ID` + ) + } + return checkpoint + } + await mkdir(this.runRoot, { recursive: true }) + const timestamp = new Date().toISOString() + const checkpoint: BuildAwareRunCheckpoint = { + schemaVersion: 1, + executionModel: "shared-memory-build-v1", + runId: this.runId, + configFingerprint, + status: "running", + currentStage: "plan", + config, + targetQuestionIds: [], + buildIds: [], + buildLinks: {}, + questions: {}, + createdAt: timestamp, + updatedAt: timestamp, + } + await this.save(checkpoint) + return checkpoint + } + + async load(): Promise { + const checkpoint = JSON.parse( + await readFile(this.checkpointPath, "utf8") + ) as BuildAwareRunCheckpoint + if ( + checkpoint.schemaVersion !== 1 || + checkpoint.executionModel !== "shared-memory-build-v1" || + checkpoint.runId !== this.runId + ) { + throw new Error(`Invalid build-aware checkpoint at ${this.checkpointPath}`) + } + return checkpoint + } + + async save(checkpoint: BuildAwareRunCheckpoint): Promise { + checkpoint.updatedAt = new Date().toISOString() + this.saveQueue = this.saveQueue.then(() => atomicWriteJson(this.checkpointPath, checkpoint)) + await this.saveQueue + } + + async initializeQuestion( + checkpoint: BuildAwareRunCheckpoint, + input: Omit + ): Promise { + const existing = checkpoint.questions[input.questionId] + if (existing) { + if ( + existing.question !== input.question || + existing.groundTruth !== input.groundTruth || + existing.evalFunction !== input.evalFunction || + existing.buildId !== input.buildId || + existing.questionImageHash !== input.questionImageHash + ) { + throw new Error(`Question ${input.questionId} changed since checkpoint creation`) + } + return + } + checkpoint.questions[input.questionId] = { + ...input, + stages: { + query: { status: "pending" }, + read: { status: "pending" }, + evaluate: { status: "pending" }, + }, + } + await this.save(checkpoint) + } + + async updateQuestionStage( + checkpoint: BuildAwareRunCheckpoint, + questionId: string, + stage: "query" | "read" | "evaluate", + update: Partial + ): Promise { + const question = checkpoint.questions[questionId] + if (!question) throw new Error(`Unknown checkpoint question ${questionId}`) + question.stages[stage] = { ...question.stages[stage], ...update } + await this.save(checkpoint) + } + + async setStage( + checkpoint: BuildAwareRunCheckpoint, + stage: BuildAwareStage, + status: BuildAwareRunCheckpoint["status"] = "running" + ): Promise { + checkpoint.currentStage = stage + checkpoint.status = status + if (status === "running") checkpoint.error = undefined + await this.save(checkpoint) + } + + async fail(checkpoint: BuildAwareRunCheckpoint, error: unknown): Promise { + checkpoint.status = "failed" + checkpoint.error = error instanceof Error ? error.message : String(error) + await this.save(checkpoint) + } + + async flush(): Promise { + await this.saveQueue + } +} diff --git a/src/orchestrator/index.ts b/src/orchestrator/index.ts index 64578bb..2c6c4b4 100644 --- a/src/orchestrator/index.ts +++ b/src/orchestrator/index.ts @@ -365,3 +365,5 @@ export class Orchestrator { export const orchestrator = new Orchestrator() export { CheckpointManager } from "./checkpoint" +export { BuildAwareRunStore } from "./build-aware-run-store" +export { LongMemEvalV2Runner, inspectLongMemEvalV2Run } from "./longmemeval-v2" diff --git a/src/orchestrator/longmemeval-v2.test.ts b/src/orchestrator/longmemeval-v2.test.ts new file mode 100644 index 0000000..da1c74b --- /dev/null +++ b/src/orchestrator/longmemeval-v2.test.ts @@ -0,0 +1,799 @@ +import { afterEach, describe, expect, test } from "bun:test" +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises" +import { tmpdir } from "node:os" +import { dirname, join, resolve } from "node:path" +import { + AUDITED_LONGMEMEVAL_V2_DATASET_VALIDATION, + type LongMemEvalV2DatasetValidationProfile, +} from "../benchmarks/longmemeval-v2/dataset" +import type { + ReaderModelClient, + ReaderModelRequest, + ReaderModelResponse, +} from "../benchmarks/longmemeval-v2/reader" +import { + type StrictJudgeCallback, + type StrictJudgeRequest, +} from "../benchmarks/longmemeval-v2/evaluation" +import { LONGMEMEVAL_V2_PINNED_REVISION } from "../benchmarks/longmemeval-v2/source" +import { longMemEvalV2Command } from "../cli/commands/longmemeval-v2" +import { atomicWriteJson, sha256 } from "../core/canonical" +import { + supermemoryPreflightGatePath, + type SupermemoryPreflightReport, +} from "../providers/supermemory/advanced" +import type { BuildAwareReport, BuildAwareRunConfig } from "../types/build-aware" +import type { + BuildBatchRequest, + BuildProvider, + BuildSearchRequest, + BuildSearchResponse, + RemoteDocumentState, +} from "../types/provider" +import type { MemoryBuildPlan, ProviderCapabilities } from "../types/migration" +import { LongMemEvalV2Runner, limitLongMemEvalV2Haystacks } from "./longmemeval-v2" + +const temporaryRoots: string[] = [] +const PNG_BYTES = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]) + +test("haystack limiting keeps complete deterministic builds and their linked questions", () => { + const question = (id: string, buildKey: string) => + ({ question: { id }, buildKey }) as unknown as Parameters< + typeof limitLongMemEvalV2Haystacks + >[0]["questions"][number] + const build = (buildKey: string) => + ({ buildKey }) as Parameters[0]["builds"][number] + const planned = { + questions: [question("q-1", "build-a"), question("q-2", "build-b"), question("q-3", "build-a")], + builds: [build("build-a"), build("build-b")], + } + + const limited = limitLongMemEvalV2Haystacks(planned, 1) + expect(limited.builds.map((item) => item.buildKey)).toEqual(["build-a"]) + expect(limited.questions.map((item) => item.question.id)).toEqual(["q-1", "q-3"]) + expect(() => limitLongMemEvalV2Haystacks(planned, 3)).toThrow("exceeds 2 available") +}) + +afterEach(async () => { + await Promise.all( + temporaryRoots.splice(0).map((root) => rm(root, { recursive: true, force: true })) + ) +}) + +async function temporaryRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "memorybench-lme2-runner-")) + temporaryRoots.push(root) + return root +} + +interface MiniDatasetFixture { + root: string + validationProfile: LongMemEvalV2DatasetValidationProfile +} + +async function writeMiniDataset(root: string): Promise { + const datasetRoot = resolve(root, "dataset") + await mkdir(resolve(datasetRoot, "haystacks"), { recursive: true }) + await mkdir(resolve(datasetRoot, "screenshots/trajectory-1"), { recursive: true }) + await mkdir(resolve(datasetRoot, "question_screenshots"), { recursive: true }) + + const trajectoryScreenshot = Buffer.concat([PNG_BYTES, Buffer.from("trajectory")]) + const questionScreenshot = Buffer.concat([PNG_BYTES, Buffer.from("question")]) + await writeFile(resolve(datasetRoot, "screenshots/trajectory-1/0.png"), trajectoryScreenshot) + await writeFile(resolve(datasetRoot, "question_screenshots/gotcha.png"), questionScreenshot) + + const questions = [ + { + id: "q-gotcha", + domain: "web", + environment: "browser", + question_type: "errors-gotchas", + question: "What is the gotcha?", + image: "question_screenshots/gotcha.png", + answer: "The visible toggle is read-only", + eval_function: "llm_gotchas_checker|require_non_empty=true", + }, + { + id: "q-choice", + domain: "web", + environment: "browser", + question_type: "static-environment", + question: "Which option was selected?", + image: null, + answer: "B", + eval_function: "mc_choice_match|require_non_empty=true", + }, + ] + const trajectories = [ + { + id: "trajectory-1", + domain: "web", + goal: "Inspect the visible setting", + start_url: "https://example.test/settings", + outcome: "The toggle was read-only and option B was selected.", + states: [ + { + state_index: 0, + step: 0, + url: "https://example.test/settings", + action: null, + thought: "Inspect the setting", + accessibility_tree: + "heading 'Settings'\nswitch 'Visible toggle', disabled=true\nradio 'B', checked=true", + screenshot: "screenshots/trajectory-1/0.png", + }, + ], + }, + ] + const haystacks = { + "q-gotcha": ["trajectory-1"], + "q-choice": ["trajectory-1"], + } + const files = new Map([ + [ + "questions.jsonl", + Buffer.from(questions.map((value) => `${JSON.stringify(value)}\n`).join("")), + ], + [ + "trajectories.jsonl", + Buffer.from(trajectories.map((value) => `${JSON.stringify(value)}\n`).join("")), + ], + ["haystacks/lme_v2_small.json", Buffer.from(`${JSON.stringify(haystacks)}\n`)], + ]) + for (const [relativePath, bytes] of files) { + await writeFile(resolve(datasetRoot, relativePath), bytes) + } + + return { + root: datasetRoot, + validationProfile: { + expectedCounts: { + questions: 2, + trajectories: 1, + states: 1, + assets: 2, + uniqueBuilds: { small: 1, medium: 1 }, + }, + requiredFiles: Object.fromEntries( + [...files].map(([relativePath, bytes]) => [ + relativePath, + { sha256: sha256(bytes), byteLength: bytes.byteLength }, + ]) + ), + }, + } +} + +const CAPABILITIES: ProviderCapabilities = { + deterministicExternalIds: true, + batchUpload: true, + documentDependencies: false, + ingestionMetadataFilters: true, + searchMetadataFilters: true, + searchModes: ["hybrid"], + reranking: true, + queryRewriting: false, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, +} + +class FakeBuildProvider implements BuildProvider { + readonly name = "fake-build-provider" + readonly capabilities = CAPABILITIES + readonly remote = new Map() + submitCalls = 0 + reconcileCalls = 0 + searchCalls = 0 + + async submitDocumentBatch(request: BuildBatchRequest): Promise { + this.submitCalls += 1 + return request.documents.map((document) => { + const state: RemoteDocumentState = { + customId: document.customId, + remoteId: `remote-${document.customId}`, + status: "ready", + } + this.remote.set(document.customId, state) + return { ...state } + }) + } + + async reconcileDocuments( + _build: BuildBatchRequest["build"], + customIds: string[] + ): Promise { + this.reconcileCalls += 1 + return customIds.map((customId) => this.remote.get(customId) ?? { customId, status: "absent" }) + } + + async searchBuild(request: BuildSearchRequest): Promise { + this.searchCalls += 1 + const screenshot = request.build.documents.find( + (document) => document.screenshotRef + )?.screenshotRef + if (!screenshot) throw new Error("Fixture build is missing its screenshot") + return { + request: { + containerTag: request.build.containerTag, + filters: { runFingerprint: request.build.buildFingerprint }, + limit: request.config.topK, + }, + rawResponse: { fixture: true, questionId: request.questionId }, + normalizedResults: [ + { + rank: 0, + score: 0.99, + kind: "memory", + text: "The visible toggle is read-only. Option B is selected.", + chunks: [], + documentIds: [request.build.documents[0].customId], + trajectoryId: "trajectory-1", + stateIndex: 0, + screenshotRefs: [screenshot], + provenanceValid: true, + }, + ], + remoteDurationMs: 7, + } + } + + async verifyBuildHealth(build: BuildBatchRequest["build"]): Promise { + return build.documents.map( + (document) => + this.remote.get(document.customId) ?? { + customId: document.customId, + status: "absent", + } + ) + } + + async deleteDocuments(_build: BuildBatchRequest["build"], customIds: string[]): Promise { + for (const customId of customIds) this.remote.delete(customId) + } + + async clearBuild(): Promise { + this.remote.clear() + } +} + +class BoundedFailureBuildProvider extends FakeBuildProvider { + override async submitDocumentBatch(_request: BuildBatchRequest): Promise { + this.submitCalls += 1 + throw new Error("bounded fixture ingestion failure") + } +} + +class FakeReaderClient implements ReaderModelClient { + readonly requests: ReaderModelRequest[] = [] + + constructor(private readonly failures = new Set()) {} + + async generate(request: ReaderModelRequest): Promise { + this.requests.push(request) + const text = request.parts + .filter( + (part): part is Extract<(typeof request.parts)[number], { type: "text" }> => + part.type === "text" + ) + .map((part) => part.text) + .join("\n") + const questionId = text.includes("Which option was selected?") ? "q-choice" : "q-gotcha" + if (this.failures.has(questionId)) throw new Error(`reader failure for ${questionId}`) + return { + text: + questionId === "q-choice" + ? "The selected option is \\boxed{B}" + : "The key issue is \\boxed{The visible toggle is read-only}", + raw: { fixture: true, questionId }, + usage: { input_tokens: 100, output_tokens: 10 }, + } + } +} + +class AbortAwareReaderClient implements ReaderModelClient { + private markStarted!: () => void + readonly started = new Promise((resolve) => { + this.markStarted = resolve + }) + + async generate(_request: ReaderModelRequest, signal?: AbortSignal): Promise { + this.markStarted() + return new Promise((resolve, reject) => { + if (signal?.aborted) { + reject(signal.reason ?? new Error("Run aborted")) + return + } + signal?.addEventListener("abort", () => reject(signal.reason ?? new Error("Run aborted")), { + once: true, + }) + }) + } +} + +class FakeStrictJudge { + readonly requests: StrictJudgeRequest[] = [] + readonly callback: StrictJudgeCallback = async (request) => { + this.requests.push(request) + return { + text: '{"label":1,"reason":"fixture gotcha matches"}', + rawResponse: { fixtureJudge: true }, + } + } +} + +function config( + datasetPath: string, + options: { + mode?: BuildAwareRunConfig["mode"] + questionIds?: string[] + provider?: BuildAwareRunConfig["provider"] + } = {} +): BuildAwareRunConfig { + return { + provider: options.provider ?? "supermemory", + benchmark: "longmemeval-v2", + mode: options.mode ?? "benchmark", + datasetPath, + datasetRevision: LONGMEMEVAL_V2_PINNED_REVISION, + tier: "small", + domain: "web", + questionIds: options.questionIds, + seed: "deterministic-runner-fixture", + retrieval: { + topK: 2, + threshold: 0, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: false, + metadataFilter: {}, + }, + reader: { + model: "fake-reader", + reasoningEffort: "high", + maxCompletionTokens: 100, + maxContextTokens: 10_000, + evidenceTopK: 2, + maxImages: 10, + maxImageBytes: 1_000_000, + malformedResponseAttempts: 1, + }, + evaluator: { + model: "fake-strict-judge", + reasoningEffort: "high", + maxCompletionTokens: 100, + }, + build: { + serviceBaseUrl: "https://fixture.invalid", + dreaming: "instant", + rootFilterMode: "self", + maxDocumentChars: 100_000, + trajectoryConcurrency: 2, + maxInFlightRequests: 2, + maxTrajectoryAttempts: 2, + indexingTimeoutMs: 1_000, + pollIntervalMs: 1, + preflightMaxAgeMs: 24 * 60 * 60_000, + }, + execution: { + buildConcurrency: 2, + questionConcurrency: 2, + }, + } +} + +function runnerOptions(root: string): { + runRoot: string + buildRoot: string + cacheRoot: string +} { + return { + runRoot: resolve(root, "runs"), + buildRoot: resolve(root, "builds"), + cacheRoot: resolve(root, "artifacts"), + } +} + +async function writePassingPreflightGate( + root: string, + runConfig: BuildAwareRunConfig +): Promise { + const path = supermemoryPreflightGatePath(root, runConfig.build.serviceBaseUrl) + await mkdir(dirname(path), { recursive: true }) + const report: SupermemoryPreflightReport = { + schemaVersion: 1, + generatedAt: new Date().toISOString(), + baseUrl: runConfig.build.serviceBaseUrl, + identity: { + buildId: "preflight-fixture", + containerTag: "preflight-fixture", + runFingerprint: "preflight-fixture", + }, + searchContract: { + searchMode: "hybrid", + standaloneChunksExpected: true, + deprecatedIncludeChunks: false, + requestedTopK: runConfig.retrieval.topK, + }, + checks: [], + allPassed: true, + blockers: [], + requestBudget: { + configuredCap: 2, + effectiveCap: 2, + inFlight: 0, + peakInFlight: 1, + throttleEvents: 0, + successStreak: 1, + notBeforeMs: 0, + }, + } + await atomicWriteJson(path, report) + return path +} + +async function loadReport(runner: LongMemEvalV2Runner): Promise { + return JSON.parse( + await readFile(resolve(runner.runStore.runRoot, "report.json"), "utf8") + ) as BuildAwareReport +} + +describe("LongMemEvalV2Runner end-to-end", () => { + test("continues through report with an explicit degraded warning in non-strict mode", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const provider = new BoundedFailureBuildProvider() + const runConfig = config(fixture.root, { questionIds: ["q-choice"] }) + runConfig.build.continueOnIndexingTimeout = true + const runner = new LongMemEvalV2Runner({ + runId: "fixture-degraded-non-strict", + config: runConfig, + provider, + readerClient: new FakeReaderClient(), + strictJudge: new FakeStrictJudge().callback, + datasetValidationProfile: fixture.validationProfile, + ...runnerOptions(root), + }) + + const checkpoint = await runner.execute() + const report = await loadReport(runner) + + expect(checkpoint.status).toBe("completed") + expect(checkpoint.currentStage).toBe("report") + expect(provider.submitCalls).toBe(runConfig.build.maxTrajectoryAttempts) + expect(provider.searchCalls).toBe(1) + expect(report.officiallyComparable).toBeFalse() + expect(report.builds).toEqual([ + expect.objectContaining({ + status: "degraded", + skippedTrajectoryCount: 1, + skippedDocumentCount: expect.any(Number), + }), + ]) + expect(report.builds[0].skippedDocumentCount).toBeGreaterThan(0) + expect(report.ineligibilityReasons).toEqual([ + expect.stringContaining("after bounded ingestion failures"), + ]) + }) + + test("keeps exhausted bounded ingestion failures fatal in strict mode", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const provider = new BoundedFailureBuildProvider() + const runConfig = config(fixture.root, { questionIds: ["q-choice"] }) + runConfig.build.continueOnIndexingTimeout = false + const runner = new LongMemEvalV2Runner({ + runId: "fixture-failed-strict", + config: runConfig, + provider, + readerClient: new FakeReaderClient(), + strictJudge: new FakeStrictJudge().callback, + datasetValidationProfile: fixture.validationProfile, + ...runnerOptions(root), + }) + + await expect(runner.execute()).rejects.toThrow("Build failed") + const checkpoint = await runner.runStore.load() + expect(checkpoint.status).toBe("failed") + expect(checkpoint.currentStage).toBe("build") + expect(provider.submitCalls).toBe(runConfig.build.maxTrajectoryAttempts) + expect(provider.searchCalls).toBe(0) + expect(await Bun.file(resolve(runner.runStore.runRoot, "report.json")).exists()).toBeFalse() + }) + + test("fingerprints and reports the persisted non-Supermemory provider identity", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const provider = new FakeBuildProvider() + const runner = new LongMemEvalV2Runner({ + runId: "fixture-rag-provider", + config: config(fixture.root, { questionIds: ["q-choice"], provider: "rag" }), + provider, + readerClient: new FakeReaderClient(), + strictJudge: new FakeStrictJudge().callback, + datasetValidationProfile: fixture.validationProfile, + ...runnerOptions(root), + }) + await runner.execute() + const report = await loadReport(runner) + const checkpoint = await runner.runStore.load() + const buildPlan = JSON.parse( + await readFile( + resolve(runner.runStore.runRoot, "builds", `${checkpoint.buildIds[0]}.plan.json`), + "utf8" + ) + ) as MemoryBuildPlan + expect(report.provider).toBe("rag") + expect(buildPlan.provider).toBe("rag") + expect(buildPlan.providerBuildConfig).toMatchObject({ + adapter: "memorybench-build-aware-v1", + extractionModel: "gpt-4o-mini", + embeddingModel: "text-embedding-3-small", + }) + }) + + test("persists a resumable failed checkpoint when the UI aborts during reading", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const provider = new FakeBuildProvider() + const reader = new AbortAwareReaderClient() + const controller = new AbortController() + const runner = new LongMemEvalV2Runner({ + runId: "fixture-ui-stop", + config: config(fixture.root, { questionIds: ["q-choice"] }), + provider, + readerClient: reader, + datasetValidationProfile: fixture.validationProfile, + signal: controller.signal, + ...runnerOptions(root), + }) + + const execution = runner.execute() + await reader.started + controller.abort(new Error("Stopped from the MemoryBench UI")) + + await expect(execution).rejects.toThrow("Stopped from the MemoryBench UI") + const checkpoint = await runner.runStore.load() + expect(checkpoint.status).toBe("failed") + expect(checkpoint.error).toBe("Stopped from the MemoryBench UI") + expect(checkpoint.questions["q-choice"].stages.read.status).toBe("failed") + expect(await Bun.file(resolve(runner.runStore.runRoot, "report.json")).exists()).toBeFalse() + }) + + test("executes plan through report with one shared multimodal build and reuses it on a second run", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const provider = new FakeBuildProvider() + const reader = new FakeReaderClient() + const judge = new FakeStrictJudge() + const shared = runnerOptions(root) + const runConfig = config(fixture.root) + + const firstRunner = new LongMemEvalV2Runner({ + runId: "fixture-first", + config: runConfig, + provider, + readerClient: reader, + strictJudge: judge.callback, + datasetValidationProfile: fixture.validationProfile, + ...shared, + }) + const first = await firstRunner.execute() + const firstReport = await loadReport(firstRunner) + const selection = JSON.parse( + await readFile(resolve(firstRunner.runStore.runRoot, "selection.json"), "utf8") + ) as { questionIds: string[]; buildLinks: Record } + + expect(first.currentStage).toBe("report") + expect(first.status).toBe("completed") + expect(selection.questionIds).toEqual(["q-gotcha", "q-choice"]) + expect(first.buildIds).toHaveLength(1) + expect(new Set(Object.values(first.buildLinks)).size).toBe(1) + expect(provider.submitCalls).toBe(1) + expect(provider.searchCalls).toBe(2) + expect(first.questions["q-gotcha"].stages.evaluate.status).toBe("completed") + expect(first.questions["q-choice"].stages.evaluate.status).toBe("completed") + expect(first.questions["q-gotcha"].evaluationArtifact?.request?.kind).toBe("gotcha") + expect(first.questions["q-choice"].evaluationArtifact?.request).toBeUndefined() + expect(judge.requests).toHaveLength(1) + expect(firstReport.targetQuestionCount).toBe(2) + expect(firstReport.completedQuestionCount).toBe(2) + expect(firstReport.official.overall.overall_full_set).toBe(1) + expect(firstReport.builds).toEqual([ + expect.objectContaining({ + trajectoryCount: 1, + linkedQuestionIds: ["q-gotcha", "q-choice"], + reused: false, + }), + ]) + + const gotchaRequest = reader.requests.find((request) => + request.parts.some( + (part) => part.type === "text" && part.text.includes("What is the gotcha?") + ) + ) + expect(gotchaRequest).toBeDefined() + expect( + gotchaRequest!.parts.filter((part) => part.type === "image").map((part) => part.asset.kind) + ).toEqual(["trajectory-screenshot", "question-image"]) + expect(firstReport.diagnostics.contextImagesSent).toBe(3) + + const secondRunner = new LongMemEvalV2Runner({ + runId: "fixture-second", + config: runConfig, + provider, + readerClient: reader, + strictJudge: judge.callback, + datasetValidationProfile: fixture.validationProfile, + ...shared, + }) + const second = await secondRunner.execute() + const secondReport = await loadReport(secondRunner) + + expect(second.status).toBe("completed") + expect(provider.submitCalls).toBe(1) + expect(provider.reconcileCalls).toBeGreaterThan(0) + expect(provider.searchCalls).toBe(2) + expect(reader.requests).toHaveLength(2) + expect(judge.requests).toHaveLength(1) + expect(secondReport.builds[0].reused).toBeTrue() + expect(secondReport.diagnostics.queryCacheHits).toBe(2) + expect(secondReport.diagnostics.readerCacheHits).toBe(2) + expect(second.questions["q-gotcha"].stages.evaluate.cacheHit).toBeTrue() + expect(second.questions["q-choice"].stages.evaluate.cacheHit).toBeTrue() + }) + + test("keeps a failed reader question in the official denominator", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const reader = new FakeReaderClient(new Set(["q-choice"])) + const judge = new FakeStrictJudge() + const runner = new LongMemEvalV2Runner({ + runId: "fixture-reader-failure", + config: config(fixture.root), + provider: new FakeBuildProvider(), + readerClient: reader, + strictJudge: judge.callback, + datasetValidationProfile: fixture.validationProfile, + ...runnerOptions(root), + }) + + const checkpoint = await runner.execute() + const report = await loadReport(runner) + + expect(checkpoint.status).toBe("completed") + expect(checkpoint.questions["q-choice"].stages.read.status).toBe("failed") + expect(checkpoint.questions["q-choice"].stages.evaluate.status).toBe("blocked") + expect(report.targetQuestionCount).toBe(2) + expect(report.completedQuestionCount).toBe(1) + expect(report.failedQuestionCount).toBe(1) + expect(report.official.overall.count_all_questions).toBe(2) + expect(report.official.overall.overall_full_set).toBe(0.5) + expect(report.official.execution).toEqual({ + completed: 1, + failed: 0, + pending: 0, + blocked: 1, + }) + expect(report.diagnostics.failedQuestions).toEqual([ + { + questionId: "q-choice", + stage: "read", + error: "reader failure for q-choice", + }, + ]) + }) + + test("allows a one-trajectory canary through query but refuses scoring", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const provider = new FakeBuildProvider() + const runner = new LongMemEvalV2Runner({ + runId: "fixture-canary", + config: config(fixture.root, { + mode: "one-trajectory-canary", + questionIds: ["q-gotcha"], + }), + provider, + datasetValidationProfile: fixture.validationProfile, + ...runnerOptions(root), + }) + + const checkpoint = await runner.execute({ through: "query" }) + expect(checkpoint.currentStage).toBe("query") + expect(checkpoint.status).toBe("completed") + expect(checkpoint.questions["q-gotcha"].stages.query.status).toBe("completed") + expect(provider.searchCalls).toBe(1) + await expect(runner.execute({ through: "evaluate" })).rejects.toThrow( + "not an official benchmark run" + ) + }) + + test("blocks live-style builds until a fresh passing preflight gate covers topK", async () => { + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const provider = new FakeBuildProvider() + const runConfig = config(fixture.root) + const preflightRoot = resolve(root, "preflights") + const shared = runnerOptions(root) + + const missingGateRunner = new LongMemEvalV2Runner({ + runId: "fixture-missing-preflight", + config: runConfig, + provider, + requirePreflight: true, + preflightRoot, + datasetValidationProfile: fixture.validationProfile, + ...shared, + }) + await expect(missingGateRunner.execute({ through: "build" })).rejects.toThrow( + "No readable passing Supermemory preflight gate" + ) + expect(provider.submitCalls).toBe(0) + + await writePassingPreflightGate(preflightRoot, runConfig) + const gatedRunner = new LongMemEvalV2Runner({ + runId: "fixture-passing-preflight", + config: runConfig, + provider, + requirePreflight: true, + preflightRoot, + datasetValidationProfile: fixture.validationProfile, + ...shared, + }) + const checkpoint = await gatedRunner.execute({ through: "build" }) + expect(checkpoint.status).toBe("completed") + expect(checkpoint.preflightGate).toMatchObject({ + schemaVersion: 1, + baseUrl: runConfig.build.serviceBaseUrl, + testedTopK: runConfig.retrieval.topK, + }) + expect(checkpoint.preflightGate?.reportFingerprint).toMatch(/^[a-f0-9]{64}$/) + expect(provider.submitCalls).toBe(1) + }) + + test("keeps the full audited snapshot profile as the production default", async () => { + expect(AUDITED_LONGMEMEVAL_V2_DATASET_VALIDATION.expectedCounts).toEqual({ + questions: 451, + trajectories: 1870, + states: 48609, + assets: 48638, + uniqueBuilds: { small: 2, medium: 447 }, + }) + expect(AUDITED_LONGMEMEVAL_V2_DATASET_VALIDATION.requiredFiles.LICENSE).toEqual({ + sha256: "d547f7673579465fcecc8f257fcdb410f51c82fd784a10b1587e83036f9c29e1", + byteLength: 9109, + }) + + const root = await temporaryRoot() + const fixture = await writeMiniDataset(root) + const runner = new LongMemEvalV2Runner({ + runId: "fixture-without-profile", + config: config(fixture.root), + ...runnerOptions(root), + }) + await expect(runner.execute({ through: "plan" })).rejects.toThrow( + "Expected 451 questions, found 2" + ) + }) +}) + +describe("LongMemEval-V2 CLI safety gates", () => { + test("rejects medium without explicit authorization and rejects unknown flags", async () => { + await expect(longMemEvalV2Command(["dry-run", "--tier", "medium"])).rejects.toThrow( + "pass --allow-medium" + ) + await expect(longMemEvalV2Command(["dry-run", "--unknown-runner-option"])).rejects.toThrow( + "Unknown option: --unknown-runner-option" + ) + await expect( + longMemEvalV2Command(["dry-run", "--preflight-max-age-hours", "0"]) + ).rejects.toThrow("requires a number > 0") + }) +}) diff --git a/src/orchestrator/longmemeval-v2.ts b/src/orchestrator/longmemeval-v2.ts new file mode 100644 index 0000000..7d85cc8 --- /dev/null +++ b/src/orchestrator/longmemeval-v2.ts @@ -0,0 +1,959 @@ +import { mkdir } from "node:fs/promises" +import { resolve } from "node:path" +import { ArtifactStore } from "../core/artifact-store" +import { BuildEngine } from "../core/build-engine" +import { BuildStore } from "../core/build-store" +import { atomicWriteJson, stableHash } from "../core/canonical" +import { requireProviderCapabilities } from "../core/provider-capabilities" +import { QueryRunner } from "../core/query-runner" +import { + LongMemEvalV2Dataset, + type LongMemEvalV2DatasetValidationProfile, +} from "../benchmarks/longmemeval-v2/dataset" +import { planLongMemEvalV2Build } from "../benchmarks/longmemeval-v2/planner" +import { + LongMemEvalV2Reader, + OpenAIReaderClient, + type ReaderModelClient, +} from "../benchmarks/longmemeval-v2/reader" +import { + StrictJudgeError, + aggregateLongMemEvalV2, + createOpenAIStrictJudge, + evaluateLongMemEvalV2, + isUnknownAnswer, + type LongMemEvalV2AggregateRecord, + type StrictJudgeCallback, +} from "../benchmarks/longmemeval-v2/evaluation" +import { + AdvancedSupermemoryProvider, + supermemoryPreflightGatePath, + validateSupermemoryPreflightReport, + type SupermemoryPreflightReport, +} from "../providers/supermemory/advanced" +import type { BuildProvider } from "../types/provider" +import type { ProviderName } from "../types/provider" +import type { + BuildAwareReport, + BuildAwareRunCheckpoint, + BuildAwareRunConfig, +} from "../types/build-aware" +import type { + AssetRef, + DatasetManifest, + EvaluationArtifact, + MemoryBuildPlan, +} from "../types/migration" +import type { + LongMemEvalV2BuildGroup, + LongMemEvalV2QuestionPlan, + PreparedTrajectory, +} from "../benchmarks/longmemeval-v2/types" +import { BuildAwareRunStore } from "./build-aware-run-store" + +export type LongMemEvalV2RunThrough = "plan" | "build" | "query" | "read" | "evaluate" | "report" + +const STAGE_ORDER: LongMemEvalV2RunThrough[] = [ + "plan", + "build", + "query", + "read", + "evaluate", + "report", +] + +export interface LongMemEvalV2RunnerOptions { + runId: string + config: BuildAwareRunConfig + runRoot?: string + buildRoot?: string + cacheRoot?: string + supermemoryApiKey?: string + openAIApiKey?: string + provider?: BuildProvider + readerClient?: ReaderModelClient + strictJudge?: StrictJudgeCallback + datasetValidationProfile?: LongMemEvalV2DatasetValidationProfile + preflightRoot?: string + requirePreflight?: boolean + signal?: AbortSignal +} + +export interface LongMemEvalV2ExecuteOptions { + through?: LongMemEvalV2RunThrough + forceBuild?: boolean + freshQuery?: boolean +} + +interface PreparedRun { + manifest: DatasetManifest + questions: LongMemEvalV2QuestionPlan[] + trajectories: Map + builds: MemoryBuildPlan[] + buildByQuestionId: Map +} + +interface BuildExecution { + plan: MemoryBuildPlan + reused: boolean + status: "ready" | "degraded" + skippedTrajectoryCount: number + skippedDocumentCount: number +} + +export function limitLongMemEvalV2Haystacks( + planned: { + questions: LongMemEvalV2QuestionPlan[] + builds: LongMemEvalV2BuildGroup[] + }, + limit: number | undefined +): { + questions: LongMemEvalV2QuestionPlan[] + builds: LongMemEvalV2BuildGroup[] +} { + if (limit === undefined) return planned + assertPositiveInteger(limit, "haystackLimit") + if (limit > planned.builds.length) { + throw new Error( + `haystackLimit ${limit} exceeds ${planned.builds.length} available exact haystacks` + ) + } + const builds = planned.builds.slice(0, limit) + const buildKeys = new Set(builds.map((build) => build.buildKey)) + return { + builds, + questions: planned.questions.filter((question) => buildKeys.has(question.buildKey)), + } +} + +function portableAsset(asset: AssetRef): AssetRef { + return { ...asset, absolutePath: undefined } +} + +function portableBuild(plan: MemoryBuildPlan): MemoryBuildPlan { + return { + ...plan, + documents: plan.documents.map((document) => ({ + ...document, + screenshotRef: document.screenshotRef ? portableAsset(document.screenshotRef) : undefined, + })), + documentPlans: plan.documentPlans.map((documentPlan) => ({ + ...documentPlan, + documents: documentPlan.documents.map((document) => ({ + ...document, + spec: { + ...document.spec, + screenshotRef: document.spec.screenshotRef + ? portableAsset(document.spec.screenshotRef) + : undefined, + }, + })), + })), + } +} + +function collectAssets( + questions: LongMemEvalV2QuestionPlan[], + trajectories: Map +): AssetRef[] { + const assets = new Map() + for (const question of questions) { + if (question.questionImage) assets.set(question.questionImage.sha256, question.questionImage) + } + for (const trajectory of trajectories.values()) { + for (const state of trajectory.states) { + const existing = assets.get(state.screenshot.sha256) + if ( + existing && + (existing.byteLength !== state.screenshot.byteLength || + existing.mimeType !== state.screenshot.mimeType) + ) { + throw new Error(`Asset hash collision for ${state.screenshot.sha256}`) + } + assets.set(state.screenshot.sha256, state.screenshot) + } + } + return [...assets.values()] + .sort((left, right) => + `${left.kind}:${left.relativePath}:${left.sha256}`.localeCompare( + `${right.kind}:${right.relativePath}:${right.sha256}` + ) + ) + .map(portableAsset) +} + +function assertPositiveInteger(value: number, field: string): void { + if (!Number.isInteger(value) || value < 1) { + throw new Error(`${field} must be a positive integer`) + } +} + +function validateConfig(config: BuildAwareRunConfig): void { + if (config.benchmark !== "longmemeval-v2") { + throw new Error("The build-aware runner only supports longmemeval-v2") + } + const providers: ProviderName[] = ["supermemory", "filesystem", "rag", "mem0", "zep"] + if (!providers.includes(config.provider)) { + throw new Error(`Unsupported LongMemEval-V2 provider: ${config.provider}`) + } + if (!["benchmark", "one-trajectory-canary"].includes(config.mode)) { + throw new Error("Invalid run mode") + } + if (!["small", "medium"].includes(config.tier)) throw new Error("Invalid tier") + if (!["web", "enterprise", "all"].includes(config.domain)) throw new Error("Invalid domain") + if (!config.datasetRevision.trim()) throw new Error("An exact dataset revision is required") + if (config.retrieval.topK < 1 || !Number.isInteger(config.retrieval.topK)) { + throw new Error("retrieval.topK must be a positive integer") + } + if (config.reader.evidenceTopK < 1 || !Number.isInteger(config.reader.evidenceTopK)) { + throw new Error("reader.evidenceTopK must be a positive integer") + } + if (config.reader.evidenceTopK > config.retrieval.topK) { + throw new Error("reader.evidenceTopK cannot exceed retrieval.topK") + } + if (config.reader.maxContextTokens <= config.reader.maxCompletionTokens) { + throw new Error("Reader context budget must exceed max completion tokens") + } + assertPositiveInteger(config.build.trajectoryConcurrency, "build.trajectoryConcurrency") + assertPositiveInteger(config.build.maxInFlightRequests, "build.maxInFlightRequests") + assertPositiveInteger(config.build.preflightMaxAgeMs, "build.preflightMaxAgeMs") + assertPositiveInteger(config.execution.buildConcurrency, "execution.buildConcurrency") + assertPositiveInteger(config.execution.questionConcurrency, "execution.questionConcurrency") + if (config.haystackLimit !== undefined) { + assertPositiveInteger(config.haystackLimit, "haystackLimit") + } +} + +async function mapConcurrent( + values: T[], + concurrency: number, + task: (value: T) => Promise +): Promise { + let cursor = 0 + const workers = Array.from({ length: Math.min(concurrency, values.length) }, async () => { + while (cursor < values.length) { + const index = cursor + cursor += 1 + await task(values[index]) + } + }) + await Promise.all(workers) +} + +function stageReached(through: LongMemEvalV2RunThrough, stage: LongMemEvalV2RunThrough): boolean { + return STAGE_ORDER.indexOf(through) >= STAGE_ORDER.indexOf(stage) +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? error.message : String(error) +} + +function throwIfAborted(signal?: AbortSignal): void { + if (signal?.aborted) throw signal.reason ?? new Error("Run aborted") +} + +export class LongMemEvalV2Runner { + readonly runStore: BuildAwareRunStore + readonly buildRoot: string + readonly cacheRoot: string + private provider?: BuildProvider + private readonly cacheArtifacts: ArtifactStore + + constructor(private readonly options: LongMemEvalV2RunnerOptions) { + validateConfig(options.config) + this.runStore = new BuildAwareRunStore(options.runId, options.runRoot ?? "data/runs-v2") + this.buildRoot = resolve(options.buildRoot ?? "data/memory-builds-v2") + this.cacheRoot = resolve(options.cacheRoot ?? "data/artifacts-v2") + this.provider = options.provider + this.cacheArtifacts = new ArtifactStore(this.cacheRoot, [ + "SUPERMEMORY_API_KEY", + "OPENAI_API_KEY", + "MEM0_API_KEY", + "ZEP_API_KEY", + ]) + } + + async execute( + executeOptions: LongMemEvalV2ExecuteOptions = {} + ): Promise { + throwIfAborted(this.options.signal) + const through = executeOptions.through ?? "report" + if (this.options.config.mode === "one-trajectory-canary" && stageReached(through, "read")) { + throw new Error( + "A one-trajectory canary may only plan, build, or query; it is not an official benchmark run" + ) + } + const checkpoint = await this.runStore.createOrLoad(this.options.config) + checkpoint.artifactRoot = this.cacheRoot + checkpoint.buildRoot = this.buildRoot + try { + const prepared = await this.prepare(checkpoint) + throwIfAborted(this.options.signal) + if (!stageReached(through, "build")) { + await this.runStore.setStage(checkpoint, "plan", "completed") + return checkpoint + } + + await this.requirePassingPreflight(checkpoint) + throwIfAborted(this.options.signal) + await this.runStore.setStage(checkpoint, "build") + const builds = await this.build(prepared, executeOptions.forceBuild ?? false) + throwIfAborted(this.options.signal) + await this.runStore.setStage( + checkpoint, + stageReached(through, "query") ? "query" : "build", + stageReached(through, "query") ? "running" : "completed" + ) + if (!stageReached(through, "query")) return checkpoint + + await this.query(checkpoint, prepared, executeOptions.freshQuery ?? false) + throwIfAborted(this.options.signal) + if (!stageReached(through, "read")) { + await this.runStore.setStage(checkpoint, "query", "completed") + return checkpoint + } + + await this.runStore.setStage(checkpoint, "read") + await this.read(checkpoint, prepared) + throwIfAborted(this.options.signal) + if (!stageReached(through, "evaluate")) { + await this.runStore.setStage(checkpoint, "read", "completed") + return checkpoint + } + + await this.runStore.setStage(checkpoint, "evaluate") + await this.evaluate(checkpoint, prepared) + throwIfAborted(this.options.signal) + if (!stageReached(through, "report")) { + await this.runStore.setStage(checkpoint, "evaluate", "completed") + return checkpoint + } + + await this.runStore.setStage(checkpoint, "report") + await this.report(checkpoint, prepared, builds) + await this.runStore.setStage(checkpoint, "report", "completed") + return checkpoint + } catch (error) { + await this.runStore.fail(checkpoint, error) + throw error + } finally { + await this.runStore.flush() + } + } + + private async requirePassingPreflight(checkpoint: BuildAwareRunCheckpoint): Promise { + const required = + this.options.requirePreflight ?? + (this.options.config.provider === "supermemory" && + (this.options.provider === undefined || + this.options.provider instanceof AdvancedSupermemoryProvider)) + if (!required) return + + const gatePath = supermemoryPreflightGatePath( + this.options.preflightRoot ?? "data/preflights-v2", + this.options.config.build.serviceBaseUrl + ) + let report: SupermemoryPreflightReport + try { + report = (await Bun.file(gatePath).json()) as SupermemoryPreflightReport + } catch { + throw new Error( + `No readable passing Supermemory preflight gate exists for ${this.options.config.build.serviceBaseUrl}; run lme-v2 preflight before build` + ) + } + validateSupermemoryPreflightReport(report, { + baseUrl: this.options.config.build.serviceBaseUrl, + requiredTopK: this.options.config.retrieval.topK, + maxAgeMs: this.options.config.build.preflightMaxAgeMs, + }) + checkpoint.preflightGate = { + schemaVersion: 1, + reportFingerprint: stableHash(report), + generatedAt: report.generatedAt, + baseUrl: report.baseUrl, + testedTopK: report.searchContract.requestedTopK, + } + await this.runStore.save(checkpoint) + } + + private async prepare(checkpoint: BuildAwareRunCheckpoint): Promise { + throwIfAborted(this.options.signal) + await this.runStore.setStage(checkpoint, "plan") + const dataset = new LongMemEvalV2Dataset({ + dataRoot: this.options.config.datasetPath, + tier: this.options.config.tier, + revision: this.options.config.datasetRevision, + validationProfile: this.options.datasetValidationProfile, + }) + await dataset.load() + throwIfAborted(this.options.signal) + const selected = dataset.selectQuestions({ + domain: this.options.config.domain === "all" ? undefined : this.options.config.domain, + ids: this.options.config.questionIds, + limit: this.options.config.limit, + perCategory: this.options.config.perCategory, + seed: this.options.config.seed, + }) + let planned = limitLongMemEvalV2Haystacks( + dataset.planQuestions(selected), + this.options.config.haystackLimit + ) + if (this.options.config.mode === "one-trajectory-canary") { + if (planned.questions.length !== 1 || planned.builds.length !== 1) { + throw new Error("A one-trajectory canary requires exactly one selected question") + } + planned.builds[0] = { + ...planned.builds[0], + orderedTrajectoryIds: planned.builds[0].orderedTrajectoryIds.slice(0, 1), + } + } + await dataset.resolveQuestionImages(planned.questions) + throwIfAborted(this.options.signal) + const trajectoryIds = [ + ...new Set(planned.builds.flatMap((build) => build.orderedTrajectoryIds)), + ] + const trajectories = await dataset.loadTrajectories(trajectoryIds) + throwIfAborted(this.options.signal) + const baseManifest = await dataset.createManifest() + const assets = collectAssets(planned.questions, trajectories) + const manifest: DatasetManifest = { + ...baseManifest, + assets, + assetScope: "selected-run", + assetsFingerprint: stableHash(assets), + } + const builds = planned.builds.map((group) => + planLongMemEvalV2Build({ + manifest, + group, + trajectories, + options: { + provider: this.options.config.provider, + providerBuildConfig: { + adapter: "memorybench-build-aware-v1", + ...(this.options.config.provider === "filesystem" || + this.options.config.provider === "rag" + ? { + extractionModel: "gpt-4o-mini", + extractionPromptVersion: 1, + } + : {}), + ...(this.options.config.provider === "rag" + ? { + embeddingModel: "text-embedding-3-small", + chunkSizeCharacters: 1600, + chunkOverlapCharacters: 320, + hybridWeights: { vector: 0.7, bm25: 0.3 }, + } + : {}), + ...(this.options.config.provider === "supermemory" + ? { serviceBaseUrl: this.options.config.build.serviceBaseUrl } + : {}), + dreaming: this.options.config.build.dreaming, + rootFilterMode: this.options.config.build.rootFilterMode, + maxDocumentChars: this.options.config.build.maxDocumentChars, + }, + }, + }) + ) + const buildByKey = new Map( + planned.builds.map((group, index) => [group.buildKey, builds[index]]) + ) + const buildByQuestionId = new Map() + checkpoint.targetQuestionIds = planned.questions.map((plan) => plan.question.id) + checkpoint.buildIds = builds.map((build) => build.buildId) + checkpoint.buildLinks = {} + checkpoint.datasetFingerprint = manifest.fingerprint + checkpoint.datasetManifestPath = "dataset-manifest.json" + for (const questionPlan of planned.questions) { + const build = buildByKey.get(questionPlan.buildKey) + if (!build) throw new Error(`No build for question ${questionPlan.question.id}`) + buildByQuestionId.set(questionPlan.question.id, build) + checkpoint.buildLinks[questionPlan.question.id] = build.buildId + await this.runStore.initializeQuestion(checkpoint, { + questionId: questionPlan.question.id, + questionType: questionPlan.question.question_type, + question: questionPlan.question.question, + groundTruth: questionPlan.question.answer, + evalFunction: questionPlan.question.eval_function, + buildId: build.buildId, + questionImageHash: questionPlan.questionImage?.sha256, + }) + } + await atomicWriteJson(resolve(this.runStore.runRoot, "dataset-manifest.json"), { + ...manifest, + dataRoot: undefined, + assets: manifest.assets.map(portableAsset), + }) + await atomicWriteJson(resolve(this.runStore.runRoot, "selection.json"), { + schemaVersion: 1, + mode: this.options.config.mode, + seed: this.options.config.seed, + questionIds: checkpoint.targetQuestionIds, + buildLinks: checkpoint.buildLinks, + assetsFingerprint: manifest.assetsFingerprint, + }) + await mkdir(resolve(this.runStore.runRoot, "builds"), { recursive: true }) + for (const build of builds) { + await atomicWriteJson( + resolve(this.runStore.runRoot, "builds", `${build.buildId}.plan.json`), + portableBuild(build) + ) + } + await this.runStore.save(checkpoint) + return { + manifest, + questions: planned.questions, + trajectories, + builds, + buildByQuestionId, + } + } + + private getProvider(): BuildProvider { + if (this.provider) return this.provider + if (this.options.config.provider !== "supermemory") { + throw new Error( + `No safe LongMemEval-V2 build adapter was injected for ${this.options.config.provider}` + ) + } + const apiKey = this.options.supermemoryApiKey ?? process.env.SUPERMEMORY_API_KEY + if (!apiKey) throw new Error("SUPERMEMORY_API_KEY is required for build/query stages") + this.provider = new AdvancedSupermemoryProvider({ + apiKey, + baseUrl: this.options.config.build.serviceBaseUrl, + maxInFlightRequests: this.options.config.build.maxInFlightRequests, + }) + return this.provider + } + + private validateProvider(provider: BuildProvider): void { + requireProviderCapabilities(provider.name, provider.capabilities, [ + "deterministicExternalIds", + "batchUpload", + "remoteClear", + "readinessStates", + "durableLocalPersistence", + "splitPhaseSafe", + ]) + if (!provider.capabilities.searchModes.includes(this.options.config.retrieval.searchMode)) { + throw new Error( + `Provider ${provider.name} does not support ${this.options.config.retrieval.searchMode} search` + ) + } + if (this.options.config.retrieval.rerank && !provider.capabilities.reranking) { + throw new Error(`Provider ${provider.name} does not support reranking`) + } + if (this.options.config.retrieval.rewriteQuery && !provider.capabilities.queryRewriting) { + throw new Error(`Provider ${provider.name} does not support query rewriting`) + } + } + + private async build(prepared: PreparedRun, forceBuild: boolean): Promise { + const provider = this.getProvider() + this.validateProvider(provider) + const results: BuildExecution[] = [] + await mapConcurrent( + prepared.builds, + this.options.config.execution.buildConcurrency, + async (plan) => { + if (this.options.signal?.aborted) { + throw this.options.signal.reason ?? new Error("Run aborted") + } + const directory = resolve(this.buildRoot, provider.name, plan.buildFingerprint) + await mkdir(directory, { recursive: true }) + await atomicWriteJson(resolve(directory, "plan.json"), portableBuild(plan)) + const store = new BuildStore(resolve(directory, "checkpoint.sqlite")) + try { + const existing = store.getBuild(plan.buildId) + const reused = existing?.status === "ready" && !forceBuild + if (forceBuild && existing) { + await provider.clearBuild(plan) + store.resetBuildForReingestion(plan.buildId) + } + const engine = new BuildEngine(plan, provider, store, { + trajectoryConcurrency: this.options.config.build.trajectoryConcurrency, + maxTrajectoryAttempts: this.options.config.build.maxTrajectoryAttempts, + indexingTimeoutMs: this.options.config.build.indexingTimeoutMs, + pollIntervalMs: this.options.config.build.pollIntervalMs, + leaseMs: Math.max(this.options.config.build.pollIntervalMs * 3 + 1, 60_000), + continueOnIndexingTimeout: this.options.config.build.continueOnIndexingTimeout ?? false, + signal: this.options.signal, + }) + const status = await engine.run() + await engine.verifyRemoteHealth({ allowDegraded: status === "degraded" }) + const buildSummary = store.buildSummary(plan.buildId) + const skippedDocumentCount = Object.entries(buildSummary.documents).reduce( + (total, [documentStatus, count]) => total + (documentStatus === "ready" ? 0 : count), + 0 + ) + await atomicWriteJson(resolve(directory, "summary.json"), { + schemaVersion: 1, + buildId: plan.buildId, + buildFingerprint: plan.buildFingerprint, + containerTag: plan.containerTag, + provider: provider.name, + status, + serviceBaseUrl: this.options.config.build.serviceBaseUrl, + providerCapabilities: provider.capabilities, + requestBudget: + provider instanceof AdvancedSupermemoryProvider + ? provider.client.budgetSnapshot + : undefined, + summary: buildSummary, + verifiedAt: new Date().toISOString(), + }) + results.push({ + plan, + reused, + status, + skippedTrajectoryCount: buildSummary.trajectories.failed, + skippedDocumentCount, + }) + } finally { + store.close() + } + } + ) + return results.sort( + (left, right) => prepared.builds.indexOf(left.plan) - prepared.builds.indexOf(right.plan) + ) + } + + private async query( + checkpoint: BuildAwareRunCheckpoint, + prepared: PreparedRun, + fresh: boolean + ): Promise { + const provider = this.getProvider() + this.validateProvider(provider) + const runner = new QueryRunner(provider, this.cacheArtifacts) + await mapConcurrent( + prepared.questions, + this.options.config.execution.questionConcurrency, + async (questionPlan) => { + throwIfAborted(this.options.signal) + const id = questionPlan.question.id + const question = checkpoint.questions[id] + const build = prepared.buildByQuestionId.get(id)! + const startedAt = new Date().toISOString() + await this.runStore.updateQuestionStage(checkpoint, id, "query", { + status: "running", + startedAt, + error: undefined, + }) + try { + const artifact = await runner.run({ + build, + questionId: id, + query: questionPlan.question.question, + questionImage: questionPlan.questionImage, + config: this.options.config.retrieval, + fresh, + }) + throwIfAborted(this.options.signal) + question.queryArtifact = artifact + await this.runStore.updateQuestionStage(checkpoint, id, "query", { + status: "completed", + fingerprint: artifact.queryFingerprint, + artifactPath: artifact.normalizedArtifact.relativePath, + completedAt: new Date().toISOString(), + durationMs: artifact.wallDurationMs, + cacheHit: artifact.cacheHit, + }) + } catch (error) { + await this.runStore.updateQuestionStage(checkpoint, id, "query", { + status: "failed", + error: errorMessage(error), + completedAt: new Date().toISOString(), + }) + await this.runStore.updateQuestionStage(checkpoint, id, "read", { + status: "blocked", + error: "Blocked by query failure", + }) + await this.runStore.updateQuestionStage(checkpoint, id, "evaluate", { + status: "blocked", + error: "Blocked by query failure", + }) + throwIfAborted(this.options.signal) + } + } + ) + } + + private async read(checkpoint: BuildAwareRunCheckpoint, prepared: PreparedRun): Promise { + const apiKey = this.options.openAIApiKey ?? process.env.OPENAI_API_KEY + const readerClient = + this.options.readerClient ?? (apiKey ? new OpenAIReaderClient({ apiKey }) : undefined) + if (!readerClient) throw new Error("OPENAI_API_KEY is required for the read stage") + const reader = new LongMemEvalV2Reader(readerClient, this.cacheArtifacts) + await mapConcurrent( + prepared.questions, + this.options.config.execution.questionConcurrency, + async (questionPlan) => { + throwIfAborted(this.options.signal) + const id = questionPlan.question.id + const question = checkpoint.questions[id] + if (question.stages.query.status !== "completed" || !question.queryArtifact) { + return + } + await this.runStore.updateQuestionStage(checkpoint, id, "read", { + status: "running", + startedAt: new Date().toISOString(), + error: undefined, + }) + try { + const artifact = await reader.answer({ + queryArtifact: question.queryArtifact, + domain: questionPlan.question.domain, + question: questionPlan.question.question, + questionImage: questionPlan.questionImage, + settings: this.options.config.reader, + signal: this.options.signal, + }) + throwIfAborted(this.options.signal) + question.readerArtifact = artifact + await this.runStore.updateQuestionStage(checkpoint, id, "read", { + status: "completed", + fingerprint: artifact.readerFingerprint, + artifactPath: `readers/${id}/${artifact.readerFingerprint}.json`, + completedAt: new Date().toISOString(), + durationMs: artifact.durationMs, + cacheHit: artifact.cacheHit, + }) + } catch (error) { + await this.runStore.updateQuestionStage(checkpoint, id, "read", { + status: "failed", + error: errorMessage(error), + completedAt: new Date().toISOString(), + }) + await this.runStore.updateQuestionStage(checkpoint, id, "evaluate", { + status: "blocked", + error: "Blocked by reader failure", + }) + throwIfAborted(this.options.signal) + } + } + ) + } + + private async evaluate( + checkpoint: BuildAwareRunCheckpoint, + prepared: PreparedRun + ): Promise { + const apiKey = this.options.openAIApiKey ?? process.env.OPENAI_API_KEY + const judge = + this.options.strictJudge ?? (apiKey ? createOpenAIStrictJudge({ apiKey }) : undefined) + await mapConcurrent( + prepared.questions, + this.options.config.execution.questionConcurrency, + async (questionPlan) => { + throwIfAborted(this.options.signal) + const id = questionPlan.question.id + const question = checkpoint.questions[id] + if (question.stages.read.status !== "completed" || !question.readerArtifact) { + return + } + const cacheKey = stableHash({ + schemaVersion: 1, + protocol: "longmemeval-v2-official", + implementationVersion: "longmemeval-v2-official-evaluator-v1", + questionId: id, + questionType: question.questionType, + question: question.question, + responseText: question.readerArtifact.responseText, + groundTruth: question.groundTruth, + evalFunction: question.evalFunction, + evaluator: this.options.config.evaluator, + }) + const path = `evaluations/${id}/${cacheKey}.json` + await this.runStore.updateQuestionStage(checkpoint, id, "evaluate", { + status: "running", + startedAt: new Date().toISOString(), + error: undefined, + }) + try { + let artifact: EvaluationArtifact + let cacheHit = false + try { + artifact = await this.cacheArtifacts.readJson(path) + cacheHit = true + } catch { + artifact = await evaluateLongMemEvalV2({ + questionId: id, + questionType: question.questionType, + question: question.question, + responseText: question.readerArtifact.responseText, + groundTruth: question.groundTruth, + evalFunction: question.evalFunction, + evaluatorModel: this.options.config.evaluator.model, + evaluatorSettings: { + reasoningEffort: this.options.config.evaluator.reasoningEffort, + maxCompletionTokens: this.options.config.evaluator.maxCompletionTokens, + }, + judge, + }) + throwIfAborted(this.options.signal) + await this.cacheArtifacts.writeJson(path, artifact) + } + question.evaluationArtifact = artifact + await this.runStore.updateQuestionStage(checkpoint, id, "evaluate", { + status: "completed", + fingerprint: artifact.evaluatorFingerprint, + artifactPath: path, + completedAt: new Date().toISOString(), + durationMs: artifact.durationMs, + cacheHit, + }) + } catch (error) { + const attemptPath = `evaluations/${id}/${cacheKey}-${crypto.randomUUID()}.failure.json` + const failure = { + schemaVersion: 1, + questionId: id, + evaluatorCacheKey: cacheKey, + error: errorMessage(error), + request: error instanceof StrictJudgeError ? error.request : undefined, + rawResponse: error instanceof StrictJudgeError ? error.rawResponse : undefined, + createdAt: new Date().toISOString(), + } + await this.cacheArtifacts.writeJson(attemptPath, failure) + await this.runStore.updateQuestionStage(checkpoint, id, "evaluate", { + status: "failed", + artifactPath: attemptPath, + error: errorMessage(error), + completedAt: new Date().toISOString(), + }) + throwIfAborted(this.options.signal) + } + } + ) + } + + private async report( + checkpoint: BuildAwareRunCheckpoint, + prepared: PreparedRun, + buildExecutions: BuildExecution[] + ): Promise { + throwIfAborted(this.options.signal) + const records: LongMemEvalV2AggregateRecord[] = prepared.questions.map((questionPlan) => { + const question = checkpoint.questions[questionPlan.question.id] + const status = + question.stages.evaluate.status === "completed" + ? "completed" + : question.stages.evaluate.status === "blocked" + ? "blocked" + : question.stages.evaluate.status === "failed" + ? "failed" + : "pending" + return { + questionId: question.questionId, + questionType: question.questionType, + evalFunction: question.evalFunction, + status, + score: question.evaluationArtifact?.score, + isUnknown: question.evaluationArtifact + ? isUnknownAnswer(question.evaluationArtifact.answer) + : false, + } + }) + const official = aggregateLongMemEvalV2(records) + const degradedBuilds = buildExecutions.filter((execution) => execution.status === "degraded") + const failedQuestions = prepared.questions.flatMap((questionPlan) => { + const question = checkpoint.questions[questionPlan.question.id] + for (const stage of ["query", "read", "evaluate"] as const) { + const state = question.stages[stage] + if (state.status === "failed") { + return [{ questionId: question.questionId, stage, error: state.error ?? "failed" }] + } + } + return [] + }) + const report: BuildAwareReport = { + schemaVersion: 1, + protocol: "longmemeval-v2-official", + runId: checkpoint.runId, + benchmark: "longmemeval-v2", + provider: this.options.config.provider, + converter: "Structured Accessibility Converter", + targetQuestionCount: prepared.questions.length, + completedQuestionCount: records.filter((record) => record.status === "completed").length, + failedQuestionCount: records.filter((record) => record.status !== "completed").length, + officiallyComparable: degradedBuilds.length === 0, + ineligibilityReasons: degradedBuilds.map( + (execution) => + `${execution.plan.buildId} skipped ${execution.skippedDocumentCount} non-ready documents across ${execution.skippedTrajectoryCount} trajectories after bounded ingestion failures` + ), + buildIds: prepared.builds.map((build) => build.buildId), + builds: prepared.builds.map((build) => ({ + buildId: build.buildId, + buildFingerprint: build.buildFingerprint, + containerTag: build.containerTag, + domain: build.domain, + trajectoryCount: build.orderedSourceIds.length, + documentCount: build.documents.length, + linkedQuestionIds: prepared.questions + .filter( + (question) => + prepared.buildByQuestionId.get(question.question.id)?.buildId === build.buildId + ) + .map((question) => question.question.id), + reused: + buildExecutions.find((execution) => execution.plan.buildId === build.buildId)?.reused ?? + true, + status: + buildExecutions.find((execution) => execution.plan.buildId === build.buildId)?.status ?? + "ready", + skippedTrajectoryCount: + buildExecutions.find((execution) => execution.plan.buildId === build.buildId) + ?.skippedTrajectoryCount ?? 0, + skippedDocumentCount: + buildExecutions.find((execution) => execution.plan.buildId === build.buildId) + ?.skippedDocumentCount ?? 0, + })), + official, + diagnostics: { + queryCacheHits: records.filter( + (record) => checkpoint.questions[record.questionId].stages.query.cacheHit + ).length, + readerCacheHits: records.filter( + (record) => checkpoint.questions[record.questionId].stages.read.cacheHit + ).length, + remoteSearchLatencyMs: records.flatMap((record) => { + const artifact = checkpoint.questions[record.questionId].queryArtifact + return artifact && !artifact.cacheHit ? [artifact.remoteDurationMs] : [] + }), + queryWallLatencyMs: records.flatMap((record) => { + const artifact = checkpoint.questions[record.questionId].queryArtifact + return artifact ? [artifact.wallDurationMs] : [] + }), + contextImagesSent: records.reduce( + (total, record) => + total + + (checkpoint.questions[record.questionId].readerArtifact?.sentAssetIds.length ?? 0), + 0 + ), + failedQuestions, + }, + createdAt: new Date().toISOString(), + } + await atomicWriteJson(resolve(this.runStore.runRoot, "report.json"), report) + return report + } +} + +export async function inspectLongMemEvalV2Run( + runId: string, + runRoot = "data/runs-v2" +): Promise<{ + checkpoint: BuildAwareRunCheckpoint + report?: BuildAwareReport +}> { + const store = new BuildAwareRunStore(runId, runRoot) + const checkpoint = await store.load() + try { + const report = await Bun.file(resolve(store.runRoot, "report.json")).json() + return { checkpoint, report: report as BuildAwareReport } + } catch { + return { checkpoint } + } +} diff --git a/src/prompts/extraction.ts b/src/prompts/extraction.ts index daca2b9..295fcdd 100644 --- a/src/prompts/extraction.ts +++ b/src/prompts/extraction.ts @@ -70,7 +70,8 @@ Rules: */ export async function extractMemories( openai: ReturnType, - session: UnifiedSession + session: UnifiedSession, + signal?: AbortSignal ): Promise { const prompt = buildExtractionPrompt(session) @@ -79,6 +80,7 @@ export async function extractMemories( prompt, maxTokens: 2000, temperature: 0, + abortSignal: signal, } const { text } = await generateText(params as Parameters[0]) diff --git a/src/providers/build-aware/index.ts b/src/providers/build-aware/index.ts new file mode 100644 index 0000000..7fe662b --- /dev/null +++ b/src/providers/build-aware/index.ts @@ -0,0 +1,48 @@ +import { getProviderConfig } from "../../utils/config" +import type { BuildProvider, ProviderName } from "../../types/provider" +import { AdvancedSupermemoryProvider } from "../supermemory/advanced" +import { FilesystemProvider } from "../filesystem" +import { RAGProvider } from "../rag" +import { LegacyBuildProviderAdapter } from "./legacy-adapter" + +export const LONGMEMEVAL_V2_BUILD_PROVIDERS = ["supermemory", "filesystem", "rag"] as const +export type LongMemEvalV2BuildProviderName = (typeof LONGMEMEVAL_V2_BUILD_PROVIDERS)[number] + +export function isLongMemEvalV2BuildProviderName( + provider: ProviderName +): provider is LongMemEvalV2BuildProviderName { + return (LONGMEMEVAL_V2_BUILD_PROVIDERS as readonly string[]).includes(provider) +} + +export async function createLongMemEvalV2BuildProvider(input: { + provider: LongMemEvalV2BuildProviderName + serviceBaseUrl: string + maxInFlightRequests: number + operationTimeoutMs: number + signal?: AbortSignal +}): Promise { + if (input.provider === "supermemory") { + const config = getProviderConfig("supermemory") + if (!config.apiKey) throw new Error("SUPERMEMORY_API_KEY is required for Supermemory") + return new AdvancedSupermemoryProvider( + { + apiKey: config.apiKey, + baseUrl: input.serviceBaseUrl, + maxInFlightRequests: input.maxInFlightRequests, + }, + { + cleanupTimeoutMs: input.operationTimeoutMs, + signal: input.signal, + } + ) + } + + const provider = input.provider === "filesystem" ? new FilesystemProvider() : new RAGProvider() + await provider.initialize(getProviderConfig(input.provider)) + return new LegacyBuildProviderAdapter(provider, { + operationTimeoutMs: input.operationTimeoutMs, + signal: input.signal, + }) +} + +export * from "./legacy-adapter" diff --git a/src/providers/build-aware/legacy-adapter.test.ts b/src/providers/build-aware/legacy-adapter.test.ts new file mode 100644 index 0000000..1e7dd37 --- /dev/null +++ b/src/providers/build-aware/legacy-adapter.test.ts @@ -0,0 +1,199 @@ +import { describe, expect, test } from "bun:test" +import { LegacyBuildProviderAdapter } from "./legacy-adapter" +import type { + BuildAwareSessionBridge, + IngestOptions, + IngestResult, + Provider, + SearchOptions, +} from "../../types/provider" +import type { MemoryBuildPlan } from "../../types/migration" +import type { UnifiedSession } from "../../types/unified" + +const capabilities = { + deterministicExternalIds: true, + batchUpload: false, + documentDependencies: false, + ingestionMetadataFilters: false, + searchMetadataFilters: false, + searchModes: ["hybrid"] as const, + reranking: false, + queryRewriting: false, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, +} + +class FakeLegacyProvider implements Provider, BuildAwareSessionBridge { + readonly name = "rag" + readonly capabilities = capabilities + readonly containers = new Map>() + + async initialize(): Promise {} + + async ingest(sessions: UnifiedSession[], options: IngestOptions): Promise { + if (options.signal?.aborted) throw options.signal.reason + const container = this.containers.get(options.containerTag) ?? new Map() + for (const session of sessions) container.set(session.sessionId, structuredClone(session)) + this.containers.set(options.containerTag, container) + return { documentIds: sessions.map((session) => session.sessionId) } + } + + async awaitIndexing(): Promise {} + + async search(_query: string, options: SearchOptions): Promise { + return [...(this.containers.get(options.containerTag)?.values() ?? [])].map((session) => ({ + id: `result-${session.sessionId}`, + sessionId: session.sessionId, + content: session.messages[0].content, + score: 0.9, + metadata: session.metadata, + })) + } + + async clear(containerTag: string): Promise { + this.containers.delete(containerTag) + } + + async inspectSessions(containerTag: string, sessionIds: string[]) { + const container = this.containers.get(containerTag) + return sessionIds.map((sessionId) => { + const session = container?.get(sessionId) + return session + ? { sessionId, status: "ready" as const, metadata: session.metadata } + : { sessionId, status: "absent" as const } + }) + } + + async deleteSessions(containerTag: string, sessionIds: string[]): Promise { + const container = this.containers.get(containerTag) + for (const sessionId of sessionIds) container?.delete(sessionId) + } +} + +function plan(): MemoryBuildPlan { + return { + schemaVersion: 1, + buildId: "mb-test", + benchmark: "longmemeval-v2", + provider: "rag", + datasetFingerprint: "dataset", + tier: "small", + domain: "web", + orderedSourceIds: ["trajectory-1"], + sourceContentHashes: ["source"], + converter: { name: "test", version: 1, sourceHash: "source" }, + providerBuildConfig: {}, + buildFingerprint: "build-fingerprint", + containerTag: "container-test", + documentPlans: [], + documents: [ + { + trajectoryId: "trajectory-1", + logicalDocumentId: "state-0", + documentOrdinal: 0, + partIndex: 0, + partCount: 1, + content: "A screenshot-backed memory", + contentHash: "content-hash", + customId: "lme2-document-1", + documentType: "state", + stateIndex: 0, + screenshotRef: { + assetId: "asset-1", + kind: "trajectory-screenshot", + relativePath: "screenshots/trajectory-1/0.png", + absolutePath: "/tmp/screenshot.png", + mimeType: "image/png", + sha256: "a".repeat(64), + byteLength: 123, + }, + metadata: {}, + }, + ], + } +} + +describe("legacy build-aware provider adapter", () => { + test("reconciles exact metadata, restores only the contributing screenshot, and cleans up", async () => { + const legacy = new FakeLegacyProvider() + const adapter = new LegacyBuildProviderAdapter(legacy, { operationTimeoutMs: 1_000 }) + const build = plan() + const states = await adapter.submitDocumentBatch({ + build, + trajectoryId: "trajectory-1", + documents: build.documents, + }) + expect(states).toEqual([ + expect.objectContaining({ customId: "lme2-document-1", status: "ready" }), + ]) + + const search = await adapter.searchBuild({ + build, + questionId: "question-1", + query: "memory", + config: { + topK: 20, + threshold: 0, + searchMode: "hybrid", + rerank: false, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: false, + metadataFilter: {}, + }, + }) + expect(search.normalizedResults).toHaveLength(1) + expect(search.normalizedResults[0]).toEqual( + expect.objectContaining({ + documentIds: ["lme2-document-1"], + trajectoryId: "trajectory-1", + screenshotRefs: [build.documents[0].screenshotRef], + provenanceValid: true, + }) + ) + + await adapter.deleteDocuments(build, ["lme2-document-1"]) + expect((await adapter.verifyBuildHealth(build))[0].status).toBe("absent") + }) + + test("rejects stale sidecar metadata and drops search results without an exact custom ID", async () => { + const legacy = new FakeLegacyProvider() + const adapter = new LegacyBuildProviderAdapter(legacy, { operationTimeoutMs: 1_000 }) + const build = plan() + await adapter.submitDocumentBatch({ + build, + trajectoryId: "trajectory-1", + documents: build.documents, + }) + const session = legacy.containers.get(build.containerTag)!.get("lme2-document-1")! + session.metadata = { ...session.metadata, buildFingerprint: "wrong-build" } + expect((await adapter.reconcileDocuments(build, ["lme2-document-1"]))[0].status).toBe("absent") + + const originalSearch = legacy.search.bind(legacy) + legacy.search = async () => [{ content: "unattributed", score: 1 }] + const result = await adapter.searchBuild({ + build, + questionId: "question-1", + query: "memory", + config: { + topK: 20, + threshold: 0, + searchMode: "hybrid", + rerank: false, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: false, + metadataFilter: {}, + }, + }) + expect(result.normalizedResults).toEqual([]) + legacy.search = originalSearch + }) +}) diff --git a/src/providers/build-aware/legacy-adapter.ts b/src/providers/build-aware/legacy-adapter.ts new file mode 100644 index 0000000..ab9274a --- /dev/null +++ b/src/providers/build-aware/legacy-adapter.ts @@ -0,0 +1,370 @@ +import type { + BuildAwareSessionBridge, + BuildBatchRequest, + BuildProvider, + BuildSearchRequest, + BuildSearchResponse, + Provider, + RemoteDocumentState, +} from "../../types/provider" +import type { + AssetRef, + MemoryBuildPlan, + NormalizedRetrievalResult, + PhysicalDocument, +} from "../../types/migration" +import type { UnifiedSession } from "../../types/unified" + +export interface LegacyBuildProviderAdapterOptions { + operationTimeoutMs: number + signal?: AbortSignal +} + +/** + * Adapts local legacy providers that expose an exact, durable session bridge to + * the shared MemoryBuild contract. This intentionally excludes Mem0 and Zep: + * their current SDK paths cannot prove exact per-document reconciliation and + * cleanup after an interrupted async ingestion. + */ +export class LegacyBuildProviderAdapter implements BuildProvider { + readonly name: string + readonly capabilities + private readonly bridge: BuildAwareSessionBridge + + constructor( + private readonly provider: Provider, + private readonly options: LegacyBuildProviderAdapterOptions + ) { + if (!isSessionBridge(provider)) { + throw new Error(`Provider ${provider.name} has no exact build-aware session bridge`) + } + if (!Number.isInteger(options.operationTimeoutMs) || options.operationTimeoutMs < 1) { + throw new Error("operationTimeoutMs must be a positive integer") + } + this.name = provider.name + this.bridge = provider + this.capabilities = { + deterministicExternalIds: true, + batchUpload: true, + documentDependencies: false, + // Isolation is provided by a content-addressed container tag, not by + // provider metadata filters. Keep these flags honest. + ingestionMetadataFilters: false, + searchMetadataFilters: false, + searchModes: provider.capabilities.searchModes, + reranking: provider.capabilities.reranking, + queryRewriting: provider.capabilities.queryRewriting, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, + } + } + + async submitDocumentBatch(request: BuildBatchRequest): Promise { + validateBatch(request) + const sessions = request.documents.map((document) => toSession(request.build, document)) + const ingestion = await withTimeout( + (signal) => + this.provider.ingest(sessions, { + containerTag: request.build.containerTag, + metadata: buildMetadata(request.build), + signal, + }), + this.options.operationTimeoutMs, + `${this.name} ingestion`, + this.options.signal + ) + await withTimeout( + () => this.provider.awaitIndexing(ingestion, request.build.containerTag), + this.options.operationTimeoutMs, + `${this.name} indexing`, + this.options.signal + ) + return this.reconcileDocuments( + request.build, + request.documents.map((document) => document.customId) + ) + } + + async reconcileDocuments( + build: MemoryBuildPlan, + customIds: string[] + ): Promise { + const expected = expectedDocuments(build, customIds) + const states = await withTimeout( + () => this.bridge.inspectSessions(build.containerTag, customIds), + this.options.operationTimeoutMs, + `${this.name} reconciliation`, + this.options.signal + ) + const returned = new Map(states.map((state) => [state.sessionId, state])) + return customIds.map((customId) => { + const document = expected.get(customId)! + const state = returned.get(customId) + const ready = + state?.status === "ready" && + state.metadata?.buildFingerprint === build.buildFingerprint && + state.metadata?.buildId === build.buildId && + state.metadata?.contentHash === document.contentHash && + state.metadata?.customId === document.customId + return { + customId, + remoteId: ready ? `${this.name}:${customId}` : undefined, + status: ready ? "ready" : "absent", + raw: state, + } + }) + } + + async searchBuild(request: BuildSearchRequest): Promise { + const started = performance.now() + const raw = await withTimeout( + (signal) => + this.provider.search(request.query, { + containerTag: request.build.containerTag, + limit: request.config.topK, + threshold: request.config.threshold, + signal, + }), + this.options.operationTimeoutMs, + `${this.name} search`, + this.options.signal + ) + const values = Array.isArray(raw) ? raw : [] + const normalizedResults = values + .map((value, rank) => normalizeResult(request.build, value, rank)) + .filter( + (result) => + result.provenanceValid && + (result.score === undefined || result.score >= request.config.threshold) + ) + .slice(0, request.config.topK) + .map((result, rank) => ({ ...result, rank })) + return { + request: { + provider: this.name, + containerTag: request.build.containerTag, + limit: request.config.topK, + threshold: request.config.threshold, + searchMode: request.config.searchMode, + }, + rawResponse: raw, + normalizedResults, + remoteDurationMs: performance.now() - started, + } + } + + async verifyBuildHealth(build: MemoryBuildPlan): Promise { + return this.reconcileDocuments( + build, + build.documents.map((document) => document.customId) + ) + } + + async deleteDocuments(build: MemoryBuildPlan, customIds: string[]): Promise { + expectedDocuments(build, customIds) + await withTimeout( + () => this.bridge.deleteSessions(build.containerTag, customIds), + this.options.operationTimeoutMs, + `${this.name} exact cleanup`, + this.options.signal + ) + const remaining = await this.reconcileDocuments(build, customIds) + if (remaining.some((state) => state.status !== "absent")) { + throw new Error(`${this.name} exact cleanup did not remove every requested document`) + } + } + + async clearBuild(build: MemoryBuildPlan): Promise { + await withTimeout( + () => this.provider.clear(build.containerTag), + this.options.operationTimeoutMs, + `${this.name} build cleanup`, + this.options.signal + ) + } +} + +function isSessionBridge(provider: Provider): provider is Provider & BuildAwareSessionBridge { + const candidate = provider as Partial + return ( + typeof candidate.inspectSessions === "function" && + typeof candidate.deleteSessions === "function" + ) +} + +function validateBatch(request: BuildBatchRequest): void { + for (const document of request.documents) { + if (document.trajectoryId !== request.trajectoryId) { + throw new Error( + `Document ${document.customId} belongs to ${document.trajectoryId}, not ${request.trajectoryId}` + ) + } + } +} + +function expectedDocuments( + build: MemoryBuildPlan, + customIds: string[] +): Map { + const documents = new Map(build.documents.map((document) => [document.customId, document])) + for (const customId of customIds) { + if (!documents.has(customId)) { + throw new Error(`Document ${customId} does not belong to build ${build.buildId}`) + } + } + return documents +} + +function buildMetadata(build: MemoryBuildPlan): Record { + return { + benchmark: build.benchmark, + buildId: build.buildId, + buildFingerprint: build.buildFingerprint, + } +} + +function toSession(build: MemoryBuildPlan, document: PhysicalDocument): UnifiedSession { + return { + sessionId: document.customId, + messages: [{ role: "user", speaker: "user", content: document.content }], + metadata: { + ...document.metadata, + ...buildMetadata(build), + customId: document.customId, + contentHash: document.contentHash, + trajectoryId: document.trajectoryId, + documentType: document.documentType, + documentOrdinal: document.documentOrdinal, + partIndex: document.partIndex, + partCount: document.partCount, + ...(document.stateIndex !== undefined ? { stateIndex: document.stateIndex } : {}), + ...(document.step !== undefined ? { step: document.step } : {}), + ...(document.screenshotRef + ? { + screenshotAssetId: document.screenshotRef.assetId, + screenshotSha256: document.screenshotRef.sha256, + } + : {}), + }, + } +} + +function normalizeResult( + build: MemoryBuildPlan, + value: unknown, + rank: number +): NormalizedRetrievalResult { + const record = asRecord(value) + const metadata = asRecord(record.metadata) + const candidateIds = [ + ...new Set( + [ + stringValue(record.sessionId), + stringValue(record.session_id), + stringValue(record.customId), + stringValue(metadata.sessionId), + stringValue(metadata.session_id), + stringValue(metadata.customId), + ].filter((item): item is string => Boolean(item)) + ), + ] + const matchedDocuments = build.documents.filter((item) => candidateIds.includes(item.customId)) + const document = matchedDocuments.length === 1 ? matchedDocuments[0] : undefined + const trajectoryId = document?.trajectoryId + const screenshotRefs = uniqueAssets( + matchedDocuments.flatMap((item) => (item.screenshotRef ? [item.screenshotRef] : [])) + ) + const declaredFingerprint = + stringValue(metadata.buildFingerprint) ?? stringValue(record.buildFingerprint) + const declaredBuildId = stringValue(metadata.buildId) ?? stringValue(record.buildId) + const provenanceValid = + (!declaredFingerprint || declaredFingerprint === build.buildFingerprint) && + (!declaredBuildId || declaredBuildId === build.buildId) && + candidateIds.length > 0 && + matchedDocuments.length === 1 && + candidateIds.every((id) => id === document?.customId) + const text = resultText(value) + return { + rank, + score: numberValue(record.score), + kind: `${build.provider}-memory`, + text, + chunks: text ? [text] : [], + providerResultId: + stringValue(record.id) ?? stringValue(record.memory_id) ?? `${build.provider}-${rank}`, + documentIds: matchedDocuments.map((item) => item.customId), + trajectoryId, + stateIndex: integerValue(metadata.stateIndex) ?? document?.stateIndex, + screenshotRefs, + provenanceValid, + } +} + +function uniqueAssets(assets: AssetRef[]): AssetRef[] { + const seen = new Set() + return assets.filter((asset) => { + if (seen.has(asset.assetId)) return false + seen.add(asset.assetId) + return true + }) +} + +function resultText(value: unknown): string { + if (typeof value === "string") return value + const record = asRecord(value) + for (const key of ["content", "memory", "text", "fact", "summary", "name"]) { + const found = stringValue(record[key]) + if (found) return found + } + return JSON.stringify(value) +} + +function asRecord(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : {} +} + +function stringValue(value: unknown): string | undefined { + return typeof value === "string" && value.trim() ? value : undefined +} + +function numberValue(value: unknown): number | undefined { + return typeof value === "number" && Number.isFinite(value) ? value : undefined +} + +function integerValue(value: unknown): number | undefined { + return typeof value === "number" && Number.isInteger(value) ? value : undefined +} + +async function withTimeout( + operation: (signal: AbortSignal) => Promise, + milliseconds: number, + label: string, + parentSignal?: AbortSignal +): Promise { + const controller = new AbortController() + let timer: ReturnType | undefined + const onParentAbort = () => + controller.abort(parentSignal?.reason ?? new Error(`${label} aborted`)) + parentSignal?.addEventListener("abort", onParentAbort, { once: true }) + if (parentSignal?.aborted) onParentAbort() + try { + return await Promise.race([ + operation(controller.signal), + new Promise((_, reject) => { + timer = setTimeout(() => { + const error = new Error(`${label} timed out after ${milliseconds}ms`) + controller.abort(error) + reject(error) + }, milliseconds) + }), + ]) + } finally { + if (timer) clearTimeout(timer) + parentSignal?.removeEventListener("abort", onParentAbort) + } +} diff --git a/src/providers/filesystem/index.ts b/src/providers/filesystem/index.ts index 0c51f7a..ccf0acf 100644 --- a/src/providers/filesystem/index.ts +++ b/src/providers/filesystem/index.ts @@ -1,4 +1,4 @@ -import { mkdir, readdir, readFile, writeFile, rm } from "node:fs/promises" +import { mkdir, readdir, readFile, writeFile, rename, rm, unlink } from "node:fs/promises" import { join } from "node:path" import { createOpenAI } from "@ai-sdk/openai" import type { @@ -8,6 +8,7 @@ import type { IngestResult, SearchOptions, IndexingProgressCallback, + BuildAwareSessionBridge, } from "../../types/provider" import type { UnifiedSession } from "../../types/unified" import { logger } from "../../utils/logger" @@ -33,7 +34,10 @@ function tokenize(text: string): string[] { * Returns a score between 0 and 1 representing the fraction of query terms found, * with a small frequency bonus for repeated matches. */ -function scoreDocument(queryTerms: string[], docText: string): { score: number; matchCount: number } { +function scoreDocument( + queryTerms: string[], + docText: string +): { score: number; matchCount: number } { if (queryTerms.length === 0) return { score: 0, matchCount: 0 } const docLower = docText.toLowerCase() @@ -75,8 +79,23 @@ function scoreDocument(queryTerms: string[], docText: string): { score: number; * This represents the MEMORY.md approach: use an LLM to extract key facts, preferences, * events, and relationships from conversations, then store them as searchable markdown. */ -export class FilesystemProvider implements Provider { +export class FilesystemProvider implements Provider, BuildAwareSessionBridge { name = "filesystem" + capabilities = { + deterministicExternalIds: true, + batchUpload: false, + documentDependencies: false, + ingestionMetadataFilters: false, + searchMetadataFilters: false, + searchModes: ["memories"] as const, + reranking: false, + queryRewriting: false, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, + } prompts = FILESYSTEM_PROMPTS concurrency = { default: 50, @@ -104,7 +123,8 @@ export class FilesystemProvider implements Provider { const documentIds: string[] = [] for (const session of sessions) { - const extractedMemories = await extractMemories(this.openai, session) + const extractedMemories = await extractMemories(this.openai, session, options.signal) + if (options.signal?.aborted) throw options.signal.reason ?? new Error("Ingestion aborted") // Build a memory file with date header + extracted content const date = @@ -116,7 +136,16 @@ export class FilesystemProvider implements Provider { const safeId = sanitizePath(session.sessionId) const filePath = join(memoriesDir, `${safeId}.md`) - await writeFile(filePath, content, "utf-8") + const sidecarPath = join(memoriesDir, `${safeId}.json`) + await atomicWrite(filePath, content) + await atomicWrite( + sidecarPath, + JSON.stringify({ + schemaVersion: 1, + sessionId: session.sessionId, + metadata: session.metadata ?? {}, + }) + ) documentIds.push(safeId) logger.debug(`Extracted and stored memories for session ${session.sessionId}`) } @@ -197,6 +226,65 @@ export class FilesystemProvider implements Provider { logger.warn(`Failed to clear filesystem data: ${e}`) } } + + async inspectSessions( + containerTag: string, + sessionIds: string[] + ): Promise< + Array<{ + sessionId: string + status: "ready" | "absent" + metadata?: Record + }> + > { + const memoriesDir = join(BASE_DIR, sanitizePath(containerTag), "memories") + return Promise.all( + sessionIds.map(async (sessionId) => { + try { + const safeId = sanitizePath(sessionId) + const [content, rawSidecar] = await Promise.all([ + readFile(join(memoriesDir, `${safeId}.md`), "utf8"), + readFile(join(memoriesDir, `${safeId}.json`), "utf8"), + ]) + const sidecar = JSON.parse(rawSidecar) as { + schemaVersion?: number + sessionId?: string + metadata?: Record + } + if ( + !content.trim() || + sidecar.schemaVersion !== 1 || + sidecar.sessionId !== sessionId || + !sidecar.metadata + ) { + return { sessionId, status: "absent" as const } + } + return { sessionId, status: "ready" as const, metadata: sidecar.metadata } + } catch { + return { sessionId, status: "absent" as const } + } + }) + ) + } + + async deleteSessions(containerTag: string, sessionIds: string[]): Promise { + const memoriesDir = join(BASE_DIR, sanitizePath(containerTag), "memories") + await Promise.all( + sessionIds.flatMap((sessionId) => { + const safeId = sanitizePath(sessionId) + return [ + unlink(join(memoriesDir, `${safeId}.md`)).catch(() => undefined), + unlink(join(memoriesDir, `${safeId}.json`)).catch(() => undefined), + ] + }) + ) + } +} + +async function atomicWrite(path: string, content: string): Promise { + const temporary = `${path}.${process.pid}.${crypto.randomUUID()}.tmp` + await writeFile(temporary, content, "utf8") + await rename(temporary, path) } /** Sanitize a string for safe use as a filesystem path component */ diff --git a/src/providers/index.ts b/src/providers/index.ts index 5f71566..6787fe2 100644 --- a/src/providers/index.ts +++ b/src/providers/index.ts @@ -40,3 +40,5 @@ export function getProviderInfo(name: ProviderName): { } export { SupermemoryProvider, Mem0Provider, ZepProvider, FilesystemProvider, RAGProvider } +export * from "./supermemory/advanced" +export * from "./build-aware" diff --git a/src/providers/mem0/index.ts b/src/providers/mem0/index.ts index e01d343..2ffcfa3 100644 --- a/src/providers/mem0/index.ts +++ b/src/providers/mem0/index.ts @@ -53,6 +53,21 @@ const CUSTOM_INSTRUCTIONS = `Generate personal memories that follow these guidel export class Mem0Provider implements Provider { name = "mem0" + capabilities = { + deterministicExternalIds: false, + batchUpload: false, + documentDependencies: false, + ingestionMetadataFilters: false, + searchMetadataFilters: false, + searchModes: ["memories"] as const, + reranking: false, + queryRewriting: false, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, + } prompts = MEM0_PROMPTS concurrency = { default: 50, diff --git a/src/providers/rag/index.ts b/src/providers/rag/index.ts index c90b723..c5d0fd7 100644 --- a/src/providers/rag/index.ts +++ b/src/providers/rag/index.ts @@ -1,7 +1,11 @@ import { embedMany, embed } from "ai" import { createOpenAI } from "@ai-sdk/openai" +import { Database } from "bun:sqlite" +import { mkdir, rm } from "node:fs/promises" +import { join } from "node:path" import type { Provider, + BuildAwareSessionBridge, ProviderConfig, IngestOptions, IngestResult, @@ -23,6 +27,7 @@ const CHUNK_OVERLAP = 320 const EMBEDDING_BATCH_SIZE = 100 /** Embedding model to use */ const EMBEDDING_MODEL = "text-embedding-3-small" +const BASE_DIR = join(process.cwd(), "data", "providers", "rag") // ─── Chunking ──────────────────────────────────────────────────────────────── @@ -30,7 +35,11 @@ const EMBEDDING_MODEL = "text-embedding-3-small" * Split text into overlapping chunks, attempting to break on sentence boundaries. * Follows the chunking approach from OpenClaw/QMD: ~400 tokens with overlap. */ -function chunkText(text: string, chunkSize: number = CHUNK_SIZE, overlap: number = CHUNK_OVERLAP): string[] { +function chunkText( + text: string, + chunkSize: number = CHUNK_SIZE, + overlap: number = CHUNK_OVERLAP +): string[] { if (text.length <= chunkSize) { return [text.trim()] } @@ -84,8 +93,23 @@ function chunkText(text: string, chunkSize: number = CHUNK_SIZE, overlap: number * memory/YYYY-MM-DD.md daily logs) * - No external memory service required - all local except for LLM + embedding API */ -export class RAGProvider implements Provider { +export class RAGProvider implements Provider, BuildAwareSessionBridge { name = "rag" + capabilities = { + deterministicExternalIds: true, + batchUpload: false, + documentDependencies: false, + ingestionMetadataFilters: false, + searchMetadataFilters: false, + searchModes: ["hybrid"] as const, + reranking: false, + queryRewriting: false, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, + } prompts = RAG_PROMPTS concurrency = { default: 20, @@ -96,6 +120,8 @@ export class RAGProvider implements Provider { private searchEngine = new HybridSearchEngine() private openai: ReturnType | null = null private apiKey: string = "" + private loadedContainers = new Set() + private databases = new Map() async initialize(config: ProviderConfig): Promise { this.apiKey = config.apiKey @@ -103,11 +129,15 @@ export class RAGProvider implements Provider { throw new Error("RAG provider requires OPENAI_API_KEY for memory extraction and embeddings") } this.openai = createOpenAI({ apiKey: this.apiKey }) - logger.info("Initialized RAG memory provider (OpenClaw/QMD-style with LLM extraction + hybrid search)") + await mkdir(BASE_DIR, { recursive: true }) + logger.info( + "Initialized RAG memory provider (OpenClaw/QMD-style with LLM extraction + hybrid search)" + ) } async ingest(sessions: UnifiedSession[], options: IngestOptions): Promise { if (!this.openai) throw new Error("Provider not initialized") + await this.ensureLoaded(options.containerTag) const allChunks: Array<{ text: string @@ -119,7 +149,8 @@ export class RAGProvider implements Provider { // Step 1: Extract memories from each session via LLM, then chunk for (const session of sessions) { - const extracted = await extractMemories(this.openai, session) + const extracted = await extractMemories(this.openai, session, options.signal) + if (options.signal?.aborted) throw options.signal.reason ?? new Error("Ingestion aborted") // Extract ISO date for OpenClaw-style date organization const isoDate = (session.metadata?.date as string) || "unknown" @@ -160,7 +191,9 @@ export class RAGProvider implements Provider { const { embeddings } = await embedMany({ model: embeddingModel, values: texts, + abortSignal: options.signal, }) + if (options.signal?.aborted) throw options.signal.reason ?? new Error("Ingestion aborted") for (let j = 0; j < batch.length; j++) { const chunk = batch[j] @@ -182,7 +215,10 @@ export class RAGProvider implements Provider { } // Step 3: Add to search engine + const sessionIds = sessions.map((session) => session.sessionId) + this.searchEngine.removeSessions(options.containerTag, sessionIds) this.searchEngine.addChunks(options.containerTag, embeddedChunks) + this.persistSessions(options.containerTag, sessionIds) const documentIds = embeddedChunks.map((c) => c.id) logger.debug( @@ -207,12 +243,14 @@ export class RAGProvider implements Provider { async search(query: string, options: SearchOptions): Promise { if (!this.openai) throw new Error("Provider not initialized") + await this.ensureLoaded(options.containerTag) // Generate query embedding const embeddingModel = this.openai.embedding(EMBEDDING_MODEL) const { embedding: queryEmbedding } = await embed({ model: embeddingModel, value: query, + abortSignal: options.signal, }) const limit = options.limit || 10 @@ -230,8 +268,147 @@ export class RAGProvider implements Provider { async clear(containerTag: string): Promise { this.searchEngine.clear(containerTag) + this.loadedContainers.delete(containerTag) + this.databases.get(containerTag)?.close() + this.databases.delete(containerTag) + const path = this.containerPath(containerTag) + await Promise.all([ + rm(path, { force: true }), + rm(`${path}-wal`, { force: true }), + rm(`${path}-shm`, { force: true }), + ]) logger.info(`Cleared RAG data for: ${containerTag}`) } + + async inspectSessions( + containerTag: string, + sessionIds: string[] + ): Promise< + Array<{ + sessionId: string + status: "ready" | "absent" + metadata?: Record + }> + > { + await this.ensureLoaded(containerTag) + const chunks = this.searchEngine.getChunks(containerTag) + return sessionIds.map((sessionId) => { + const chunk = chunks.find((candidate) => candidate.sessionId === sessionId) + return chunk + ? { sessionId, status: "ready" as const, metadata: chunk.metadata } + : { sessionId, status: "absent" as const } + }) + } + + async deleteSessions(containerTag: string, sessionIds: string[]): Promise { + await this.ensureLoaded(containerTag) + this.searchEngine.removeSessions(containerTag, sessionIds) + this.persistSessions(containerTag, sessionIds) + } + + private containerPath(containerTag: string): string { + return join(BASE_DIR, `${sanitizePath(containerTag)}.sqlite`) + } + + private async ensureLoaded(containerTag: string): Promise { + if (this.loadedContainers.has(containerTag)) return + const db = new Database(this.containerPath(containerTag), { create: true, strict: true }) + db.exec("PRAGMA journal_mode = WAL") + db.exec("PRAGMA synchronous = FULL") + db.exec("PRAGMA busy_timeout = 10000") + db.exec(` + CREATE TABLE IF NOT EXISTS metadata ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS chunks ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + chunk_index INTEGER NOT NULL, + content TEXT NOT NULL, + embedding_json TEXT NOT NULL, + date TEXT, + metadata_json TEXT, + UNIQUE(session_id, chunk_index) + ); + CREATE INDEX IF NOT EXISTS chunks_session_id ON chunks(session_id); + `) + const existingTag = db.query("SELECT value FROM metadata WHERE key = 'containerTag'").get() as { + value: string + } | null + if (existingTag && existingTag.value !== containerTag) { + db.close() + throw new Error("Persisted RAG container identity does not match") + } + db.query("INSERT OR REPLACE INTO metadata (key, value) VALUES ('schemaVersion', '1')").run() + db.query("INSERT OR REPLACE INTO metadata (key, value) VALUES ('containerTag', ?)").run( + containerTag + ) + const rows = db + .query( + `SELECT id, session_id, chunk_index, content, embedding_json, date, metadata_json + FROM chunks ORDER BY session_id, chunk_index, id` + ) + .all() as Array<{ + id: string + session_id: string + chunk_index: number + content: string + embedding_json: string + date: string | null + metadata_json: string | null + }> + this.searchEngine.replaceChunks( + containerTag, + rows.map((row) => ({ + id: row.id, + sessionId: row.session_id, + chunkIndex: row.chunk_index, + content: row.content, + embedding: JSON.parse(row.embedding_json) as number[], + date: row.date ?? undefined, + metadata: row.metadata_json + ? (JSON.parse(row.metadata_json) as Record) + : undefined, + })) + ) + this.databases.set(containerTag, db) + this.loadedContainers.add(containerTag) + } + + private persistSessions(containerTag: string, sessionIds: string[]): void { + const db = this.databases.get(containerTag) + if (!db) throw new Error(`RAG container ${containerTag} is not initialized`) + const selected = new Set(sessionIds) + const chunks = this.searchEngine + .getChunks(containerTag) + .filter((chunk) => selected.has(chunk.sessionId)) + const transaction = db.transaction(() => { + const remove = db.query("DELETE FROM chunks WHERE session_id = ?") + for (const sessionId of sessionIds) remove.run(sessionId) + const insert = db.query( + `INSERT INTO chunks + (id, session_id, chunk_index, content, embedding_json, date, metadata_json) + VALUES (?, ?, ?, ?, ?, ?, ?)` + ) + for (const chunk of chunks) { + insert.run( + chunk.id, + chunk.sessionId, + chunk.chunkIndex, + chunk.content, + JSON.stringify(chunk.embedding), + chunk.date ?? null, + chunk.metadata ? JSON.stringify(chunk.metadata) : null + ) + } + }) + transaction.immediate() + } +} + +function sanitizePath(input: string): string { + return input.replace(/[^a-zA-Z0-9_.-]/g, "_") } export default RAGProvider diff --git a/src/providers/rag/search.test.ts b/src/providers/rag/search.test.ts new file mode 100644 index 0000000..20e5b14 --- /dev/null +++ b/src/providers/rag/search.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, test } from "bun:test" +import { HybridSearchEngine, type Chunk } from "./search" + +function chunk(id: string, sessionId: string, content = "same memory"): Chunk { + return { + id, + sessionId, + chunkIndex: 0, + content, + embedding: [1, 0], + } +} + +describe("durable RAG search index primitives", () => { + test("deterministic upserts do not duplicate BM25 documents", () => { + const engine = new HybridSearchEngine() + engine.addChunks("build", [chunk("b", "session-b"), chunk("a", "session-a")]) + engine.addChunks("build", [chunk("a", "session-a", "same memory updated")]) + expect(engine.getChunkCount("build")).toBe(2) + expect(engine.getChunks("build").find((item) => item.id === "a")?.content).toBe( + "same memory updated" + ) + }) + + test("session replacement removes stale chunk ordinals and ties break by chunk ID", () => { + const engine = new HybridSearchEngine() + engine.addChunks("build", [ + { ...chunk("b", "session-1"), chunkIndex: 1 }, + chunk("a", "session-1"), + ]) + expect(engine.search("build", [1, 0], "same", 2).map((item) => item.sessionId)).toEqual([ + "session-1", + "session-1", + ]) + engine.removeSessions("build", ["session-1"]) + engine.addChunks("build", [chunk("a", "session-1", "replacement")]) + expect(engine.getChunks("build").map((item) => item.id)).toEqual(["a"]) + }) +}) diff --git a/src/providers/rag/search.ts b/src/providers/rag/search.ts index df2a365..b792d60 100644 --- a/src/providers/rag/search.ts +++ b/src/providers/rag/search.ts @@ -207,6 +207,12 @@ function addToBM25Index(index: BM25Index, chunkId: string, text: string): void { } } +function buildBM25Index(chunks: Iterable): BM25Index { + const index = createBM25Index() + for (const chunk of chunks) addToBM25Index(index, chunk.id, chunk.content) + return index +} + function searchBM25(index: BM25Index, query: string): Map { const queryTerms = tokenize(query) const scores = new Map() @@ -283,11 +289,40 @@ export class HybridSearchEngine { for (const chunk of chunks) { container.chunks.set(chunk.id, chunk) - addToBM25Index(container.bm25Index, chunk.id, chunk.content) } + // Rebuild instead of incrementally appending so deterministic upserts do + // not inflate BM25 document counts after a retry or resume. + container.bm25Index = buildBM25Index(container.chunks.values()) + } + + replaceChunks(containerTag: string, chunks: Chunk[]): void { + const byId = new Map(chunks.map((chunk) => [chunk.id, chunk])) + this.containers.set(containerTag, { + chunks: byId, + bm25Index: buildBM25Index(byId.values()), + }) + } + + getChunks(containerTag: string): Chunk[] { + return [...(this.containers.get(containerTag)?.chunks.values() ?? [])] + } + + hasSession(containerTag: string, sessionId: string): boolean { + return this.getChunks(containerTag).some((chunk) => chunk.sessionId === sessionId) + } + + removeSessions(containerTag: string, sessionIds: string[]): void { + const removed = new Set(sessionIds) + const kept = this.getChunks(containerTag).filter((chunk) => !removed.has(chunk.sessionId)) + this.replaceChunks(containerTag, kept) } - search(containerTag: string, queryEmbedding: number[], query: string, limit: number): SearchResult[] { + search( + containerTag: string, + queryEmbedding: number[], + query: string, + limit: number + ): SearchResult[] { const container = this.containers.get(containerTag) if (!container || container.chunks.size === 0) return [] @@ -334,7 +369,7 @@ export class HybridSearchEngine { } // Sort by hybrid score descending - hybridScores.sort((a, b) => b.score - a.score) + hybridScores.sort((a, b) => b.score - a.score || a.chunkId.localeCompare(b.chunkId)) // Return top results return hybridScores.slice(0, limit).map((result) => { diff --git a/src/providers/supermemory/advanced/advanced.test.ts b/src/providers/supermemory/advanced/advanced.test.ts new file mode 100644 index 0000000..f54f84d --- /dev/null +++ b/src/providers/supermemory/advanced/advanced.test.ts @@ -0,0 +1,891 @@ +import { describe, expect, test } from "bun:test" +import type { MemoryBuildPlan, PhysicalDocument, RetrievalConfig } from "../../../types/migration" +import { + AdaptiveRequestBudget, + AdvancedSupermemoryClient, + SupermemoryHttpError, + type AdvancedSupermemoryApi, + type V3DocumentInput, + type V4SearchRequest, +} from "./client" +import { + AdvancedSupermemoryBuild, + SupermemoryCleanupTimeoutError, + UnsafeSupermemoryCleanupError, +} from "./build" +import { AdvancedSupermemoryProvider } from "./provider" +import { + AdvancedSupermemoryRetrieval, + SupermemoryProvenanceError, + buildSupermemorySearchFilter, +} from "./retrieval" +import { + AdvancedSupermemoryPreflight, + supermemoryPreflightGatePath, + validateSupermemoryPreflightReport, +} from "./preflight" + +function response(body: unknown, status = 200, headers?: HeadersInit): Response { + return new Response(body === null ? null : JSON.stringify(body), { + status, + headers: { "Content-Type": "application/json", ...headers }, + }) +} + +function fakeFetch( + handler: (url: string, init: RequestInit) => Promise | Response +): typeof fetch { + return (async (input: RequestInfo | URL, init?: RequestInit) => + handler(String(input), init ?? {})) as typeof fetch +} + +function emptyBudgetSnapshot() { + return { + configuredCap: 4, + effectiveCap: 4, + inFlight: 0, + peakInFlight: 1, + throttleEvents: 0, + successStreak: 0, + notBeforeMs: 0, + } +} + +function fakeApi(overrides: Partial = {}): AdvancedSupermemoryApi { + return { + baseUrl: "https://fake.supermemory.test", + requestCount: 0, + budgetSnapshot: emptyBudgetSnapshot(), + async addDocument() { + return { id: "document-id", status: "queued" } + }, + async addDocumentsBatch(input) { + return { + results: input.documents.map((document, index) => ({ + id: `remote-${index}`, + customId: document.customId, + status: "queued", + })), + } + }, + async getDocument() { + return null + }, + async listDocumentsByCustomIds() { + return [] + }, + async searchV4() { + return { results: [] } + }, + async deleteDocument() {}, + ...overrides, + } +} + +function physicalDocument(customId = "lme2-document-1"): PhysicalDocument { + return { + trajectoryId: "trajectory-1", + logicalDocumentId: "state-0000", + documentOrdinal: 0, + partIndex: 0, + partCount: 1, + content: "A structured accessibility observation.", + contentHash: "a".repeat(64), + customId, + documentType: "state", + stateIndex: 0, + step: 1, + screenshotRef: { + assetId: "screenshot-1", + kind: "trajectory-screenshot", + relativePath: "screenshots/state-0.png", + mimeType: "image/png", + sha256: "b".repeat(64), + byteLength: 123, + }, + metadata: { + evidenceFormat: "structured-accessibility-v1", + }, + } +} + +function buildPlan(document = physicalDocument()): MemoryBuildPlan { + return { + schemaVersion: 1, + buildId: "build-1", + benchmark: "longmemeval-v2", + provider: "supermemory", + datasetFingerprint: "dataset-fingerprint", + tier: "small", + domain: "web", + orderedSourceIds: ["trajectory-1"], + sourceContentHashes: ["c".repeat(64)], + converter: { + name: "structured-accessibility", + version: 1, + sourceHash: "d".repeat(64), + }, + providerBuildConfig: {}, + buildFingerprint: "e".repeat(64), + containerTag: "lme2-small-web-build", + documentPlans: [], + documents: [document], + } +} + +describe("AdvancedSupermemoryClient", () => { + test("honors base URL, Retry-After, shared pressure, and secret redaction", async () => { + const apiKey = "super-secret-api-key" + const requests: Array<{ url: string; init: RequestInit }> = [] + const sleeps: number[] = [] + const events: Array> = [] + let now = 0 + let attempt = 0 + const sleep = async (milliseconds: number) => { + sleeps.push(milliseconds) + now += milliseconds + } + const budget = new AdaptiveRequestBudget({ + maxInFlight: 4, + clock: () => now, + sleep, + }) + const client = new AdvancedSupermemoryClient({ + apiKey, + baseUrl: "https://custom.supermemory.test/root/", + maxAttempts: 2, + fetch: fakeFetch((url, init) => { + requests.push({ url, init }) + attempt += 1 + if (attempt === 1) { + return response({ error: `Bearer ${apiKey} ${apiKey}` }, 429, { "Retry-After": "2" }) + } + return response({ results: [{ id: "remote-1", status: "queued" }] }) + }), + sleep, + clock: () => now, + random: () => 0, + budget, + eventLogger: (_event, details) => events.push(details), + }) + + const result = await client.addDocumentsBatch({ + containerTag: "container", + dreaming: "instant", + documents: [ + { + content: "content", + customId: "custom-1", + metadata: { runFingerprint: "run-1" }, + }, + ], + }) + + expect(result.results).toHaveLength(1) + expect(requests.map((request) => request.url)).toEqual([ + "https://custom.supermemory.test/root/v3/documents/batch", + "https://custom.supermemory.test/root/v3/documents/batch", + ]) + expect(new Headers(requests[0].init.headers).get("authorization")).toBe(`Bearer ${apiKey}`) + expect(sleeps).toEqual([2_000]) + expect(client.budgetSnapshot.effectiveCap).toBe(2) + expect(client.budgetSnapshot.throttleEvents).toBe(1) + expect(JSON.stringify(events)).not.toContain(apiKey) + }) + + test("does not expose secrets from permanent response bodies", async () => { + const apiKey = "do-not-leak-me" + const client = new AdvancedSupermemoryClient({ + apiKey, + maxAttempts: 1, + fetch: fakeFetch(() => response({ authorization: `Bearer ${apiKey}`, token: apiKey }, 400)), + budget: new AdaptiveRequestBudget({ maxInFlight: 1 }), + }) + + let thrown: unknown + try { + await client.addDocument({ + containerTag: "container", + document: { content: "content", customId: "custom", metadata: {} }, + }) + } catch (error) { + thrown = error + } + expect(thrown).toBeInstanceOf(SupermemoryHttpError) + expect(String(thrown)).not.toContain(apiKey) + expect(String(thrown)).toContain("") + }) + + test("cancels an in-flight cleanup request without retrying it", async () => { + const controller = new AbortController() + const stopped = new Error("run stopped") + let requests = 0 + const client = new AdvancedSupermemoryClient({ + apiKey: "test-key", + maxAttempts: 3, + fetch: fakeFetch((_url, init) => { + requests += 1 + return new Promise((_, reject) => { + const signal = init.signal as AbortSignal + signal.addEventListener("abort", () => reject(signal.reason), { once: true }) + }) + }), + budget: new AdaptiveRequestBudget({ maxInFlight: 1 }), + }) + + const deletion = client.deleteDocument("remote-1", controller.signal) + await Bun.sleep(0) + controller.abort(stopped) + + await expect(deletion).rejects.toBe(stopped) + expect(requests).toBe(1) + }) +}) + +describe("AdvancedSupermemoryBuild", () => { + test("reconciles an ambiguous 409 by custom ID and polls to ready", async () => { + let listCalls = 0 + let now = 0 + const api = fakeApi({ + async addDocumentsBatch() { + throw new SupermemoryHttpError("conflict", { + statusCode: 409, + retryable: false, + }) + }, + async listDocumentsByCustomIds(customIds) { + listCalls += 1 + return customIds.map((customId) => ({ + id: `remote-${customId}`, + customId, + status: listCalls >= 2 ? "done" : "processing", + metadata: { + buildId: "build-1", + runFingerprint: "run-1", + }, + memories: [], + })) + }, + }) + const build = new AdvancedSupermemoryBuild(api, { + clock: () => now, + sleep: async (milliseconds) => { + now += milliseconds + }, + }) + const input = { + trajectoryId: "trajectory-1", + identity: { + buildId: "build-1", + containerTag: "container-1", + runFingerprint: "run-1", + }, + documents: [ + { + customId: "custom-1", + content: "content", + metadata: {}, + filterByMetadata: { + runFingerprint: "run-1", + causalKey: "custom-1", + }, + }, + ], + } + + const submission = await build.submitTrajectoryBatch(input) + expect(submission.reconciled).toBe(true) + expect(submission.documents[0].status).toBe("indexing") + + const ready = await build.awaitReady({ + customIds: ["custom-1"], + containerTag: "container-1", + timeoutMs: 100, + initialPollMs: 10, + maxPollMs: 10, + }) + expect(ready[0].status).toBe("ready") + expect(ready[0].memoryCount).toBe(0) + }) + + test("refuses cleanup when exact build metadata does not match", async () => { + let deleted = false + const api = fakeApi({ + async listDocumentsByCustomIds() { + return [ + { + id: "remote-1", + customId: "custom-1", + status: "done", + metadata: { buildId: "another-build", runFingerprint: "run-1" }, + }, + ] + }, + async deleteDocument() { + deleted = true + }, + }) + const build = new AdvancedSupermemoryBuild(api) + await expect( + build.cleanupExactBuild({ + identity: { + buildId: "build-1", + containerTag: "container", + runFingerprint: "run-1", + }, + customIds: ["custom-1"], + }) + ).rejects.toBeInstanceOf(UnsafeSupermemoryCleanupError) + expect(deleted).toBe(false) + }) + + test("reconciles a still-processing delete conflict before retrying exact cleanup", async () => { + let now = 0 + let deleteCalls = 0 + const sleeps: number[] = [] + const remote = new Map>([ + [ + "custom-1", + { + id: "remote-1", + customId: "custom-1", + status: "processing", + metadata: { buildId: "build-1", runFingerprint: "run-1" }, + }, + ], + ]) + const api = fakeApi({ + async listDocumentsByCustomIds(customIds) { + return customIds + .map((customId) => remote.get(customId)) + .filter((document): document is Record => document !== undefined) + }, + async deleteDocument() { + deleteCalls += 1 + if (deleteCalls === 1) { + throw new SupermemoryHttpError( + 'HTTP 409 during delete_document: {"error":"Document is still processing"}', + { statusCode: 409, retryable: false } + ) + } + remote.delete("custom-1") + }, + }) + const build = new AdvancedSupermemoryBuild(api, { + clock: () => now, + sleep: async (milliseconds) => { + sleeps.push(milliseconds) + now += milliseconds + }, + cleanupTimeoutMs: 100, + cleanupInitialPollMs: 10, + cleanupMaxPollMs: 10, + }) + + const result = await build.cleanupExactBuild({ + identity: { + buildId: "build-1", + containerTag: "container", + runFingerprint: "run-1", + }, + customIds: ["custom-1"], + }) + + expect(result).toEqual({ deleted: ["custom-1"], absent: [] }) + expect(deleteCalls).toBe(2) + expect(sleeps).toEqual([10]) + expect(remote.size).toBe(0) + }) + + test("bounds a persistent processing conflict without treating the document as absent", async () => { + let now = 0 + let deleteCalls = 0 + const api = fakeApi({ + async listDocumentsByCustomIds() { + return [ + { + id: "remote-1", + customId: "custom-1", + status: "processing", + metadata: { buildId: "build-1", runFingerprint: "run-1" }, + }, + ] + }, + async deleteDocument() { + deleteCalls += 1 + throw new SupermemoryHttpError("Document is still processing", { + statusCode: 409, + retryable: false, + }) + }, + }) + const build = new AdvancedSupermemoryBuild(api, { + clock: () => now, + sleep: async (milliseconds) => { + now += milliseconds + }, + cleanupTimeoutMs: 20, + cleanupInitialPollMs: 10, + cleanupMaxPollMs: 10, + }) + + await expect( + build.cleanupExactBuild({ + identity: { + buildId: "build-1", + containerTag: "container", + runFingerprint: "run-1", + }, + customIds: ["custom-1"], + }) + ).rejects.toBeInstanceOf(SupermemoryCleanupTimeoutError) + expect(deleteCalls).toBe(3) + }) + + test("cancels processing-conflict reconciliation from the run signal", async () => { + const controller = new AbortController() + let deleteCalls = 0 + const api = fakeApi({ + async listDocumentsByCustomIds() { + return [ + { + id: "remote-1", + customId: "custom-1", + status: "processing", + metadata: { buildId: "build-1", runFingerprint: "run-1" }, + }, + ] + }, + async deleteDocument() { + deleteCalls += 1 + throw new SupermemoryHttpError("Document is still processing", { + statusCode: 409, + retryable: false, + }) + }, + }) + const stopped = new Error("run stopped") + const build = new AdvancedSupermemoryBuild(api, { + signal: controller.signal, + sleep: async () => { + controller.abort(stopped) + }, + cleanupTimeoutMs: 100, + cleanupInitialPollMs: 10, + cleanupMaxPollMs: 10, + }) + + await expect( + build.cleanupExactBuild({ + identity: { + buildId: "build-1", + containerTag: "container", + runFingerprint: "run-1", + }, + customIds: ["custom-1"], + }) + ).rejects.toBe(stopped) + expect(deleteCalls).toBe(1) + }) +}) + +describe("AdvancedSupermemoryProvider", () => { + test("implements BuildProvider with self-scoped batches, verified screenshots, and deletion", async () => { + const remote = new Map>() + let batch: + | { + documents: V3DocumentInput[] + containerTag: string + dreaming?: string + } + | undefined + let searchRequest: V4SearchRequest | undefined + const api = fakeApi({ + async addDocumentsBatch(input) { + batch = input + for (const [index, document] of input.documents.entries()) { + remote.set(document.customId, { + id: `remote-${index}`, + customId: document.customId, + status: "processing", + metadata: document.metadata, + memories: [], + }) + } + return { + results: input.documents.map((document, index) => ({ + id: `remote-${index}`, + customId: document.customId, + status: "queued", + })), + } + }, + async listDocumentsByCustomIds(customIds) { + return customIds + .map((customId) => remote.get(customId)) + .filter((document): document is Record => document !== undefined) + .map((document) => ({ ...document, status: "done" })) + }, + async deleteDocument(idOrCustomId) { + for (const [customId, document] of remote) { + if (customId === idOrCustomId || document.id === idOrCustomId) remote.delete(customId) + } + }, + async searchV4(request) { + searchRequest = request + const metadata = batch!.documents[0].metadata + return { + results: [ + { + id: "result-1", + memory: "Timezone is UTC.", + similarity: 0.9, + chunks: [{ content: "same chunk" }, { content: "same chunk" }], + documents: [{ id: "remote-0", summary: "Settings", metadata }], + }, + ], + } + }, + }) + const provider = new AdvancedSupermemoryProvider(api) + const document = physicalDocument() + const build = buildPlan(document) + + const submitted = await provider.submitDocumentBatch({ + build, + trajectoryId: "trajectory-1", + documents: [document], + }) + expect(submitted[0].status).toBe("pending") + expect(batch?.dreaming).toBe("instant") + expect(batch?.containerTag).toBe(build.containerTag) + expect(batch?.documents[0].metadata).toMatchObject({ + runFingerprint: build.buildFingerprint, + buildFingerprint: build.buildFingerprint, + buildId: build.buildId, + trajectoryId: "trajectory-1", + causalKey: document.customId, + screenshotAssetId: "screenshot-1", + screenshotSha256: "b".repeat(64), + screenshotMimeType: "image/png", + screenshotByteLength: 123, + }) + expect(batch?.documents[0].filterByMetadata).toEqual({ + runFingerprint: build.buildFingerprint, + buildFingerprint: build.buildFingerprint, + trajectoryId: "trajectory-1", + causalKey: document.customId, + }) + + const reconciled = await provider.reconcileDocuments(build, [document.customId]) + expect(reconciled[0].status).toBe("ready") + + const config: RetrievalConfig = { + topK: 17, + threshold: 0.1, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: true, + metadataFilter: {}, + } + const search = await provider.searchBuild({ + build, + questionId: "question-1", + query: "What is the timezone?", + config, + }) + expect(searchRequest).toMatchObject({ + limit: 17, + containerTag: build.containerTag, + filters: { + AND: [ + { + key: "runFingerprint", + value: build.buildFingerprint, + filterType: "metadata", + }, + ], + }, + }) + expect(search.normalizedResults[0].chunks).toEqual(["same chunk"]) + expect(search.normalizedResults[0].screenshotRefs).toEqual([document.screenshotRef!]) + expect(search.normalizedResults[0].provenanceValid).toBe(true) + + await provider.deleteDocuments(build, [document.customId]) + expect(remote.size).toBe(0) + }) +}) + +describe("AdvancedSupermemoryRetrieval", () => { + test("builds the current V4 logical filter contract and keeps the build boundary mandatory", () => { + expect( + buildSupermemorySearchFilter( + { + domain: ["web", "enterprise"], + stateIndex: 4, + verified: true, + }, + "expected-run" + ) + ).toEqual({ + AND: [ + { + key: "runFingerprint", + value: "expected-run", + filterType: "metadata", + }, + { + OR: [ + { + key: "domain", + value: "web", + filterType: "array_contains", + }, + { + key: "domain", + value: "enterprise", + filterType: "array_contains", + }, + ], + }, + { + key: "stateIndex", + value: "4", + filterType: "numeric", + numericOperator: "=", + }, + { + key: "verified", + value: "true", + filterType: "metadata", + }, + ], + }) + + expect( + buildSupermemorySearchFilter( + { + OR: [ + { key: "trajectoryId", value: "one" }, + { + AND: [ + { + key: "stateIndex", + value: "2", + filterType: "numeric", + numericOperator: ">=", + }, + { key: "domain", value: "web", negate: false }, + ], + }, + ], + }, + "expected-run" + ) + ).toEqual({ + AND: [ + { + key: "runFingerprint", + value: "expected-run", + filterType: "metadata", + }, + { + OR: [ + { key: "trajectoryId", value: "one" }, + { + AND: [ + { + key: "stateIndex", + value: "2", + filterType: "numeric", + numericOperator: ">=", + }, + { key: "domain", value: "web", negate: false }, + ], + }, + ], + }, + ], + }) + }) + + test("rejects filters that weaken provenance or violate the V4 contract", () => { + expect(() => + buildSupermemorySearchFilter({ runFingerprint: "another-run" }, "expected-run") + ).toThrow("cannot override") + expect(() => + buildSupermemorySearchFilter( + { + OR: [{ key: "runFingerprint", value: "another-run" }], + }, + "expected-run" + ) + ).toThrow("cannot override") + expect(() => + buildSupermemorySearchFilter( + { + AND: [ + { + key: "stateIndex", + value: "not-a-number", + filterType: "numeric", + }, + ], + }, + "expected-run" + ) + ).toThrow("requires a numeric value") + expect(() => buildSupermemorySearchFilter({ "bad key": "value" }, "expected-run")).toThrow( + "Invalid search metadata key" + ) + }) + + test("rejects wrong run fingerprint even when the container matches", async () => { + const api = fakeApi({ + async searchV4() { + return { + results: [ + { + id: "wrong-run", + memory: "contaminated", + metadata: { runFingerprint: "another-run" }, + }, + ], + } + }, + }) + const retrieval = new AdvancedSupermemoryRetrieval(api) + await expect( + retrieval.search({ + identity: { + buildId: "build-1", + containerTag: "shared-container", + runFingerprint: "expected-run", + }, + query: "question", + config: { topK: 10 }, + }) + ).rejects.toBeInstanceOf(SupermemoryProvenanceError) + }) +}) + +describe("AdvancedSupermemoryPreflight", () => { + test("exposes explicit hooks, validates the gate, and cleans exact probe documents", async () => { + const remote = new Map>() + const observedChecks: string[] = [] + const api = fakeApi({ + async addDocument(input) { + const document = input.document + const existing = remote.get(document.customId) + if (existing) return existing as { id: string } + const created = { + id: `remote-${remote.size}`, + customId: document.customId, + status: "done", + metadata: document.metadata, + memories: [], + } + remote.set(document.customId, created) + return created + }, + async addDocumentsBatch(input) { + const results = input.documents.map((document) => { + const created = { + id: `remote-${remote.size}`, + customId: document.customId, + status: "done", + metadata: document.metadata, + memories: [], + } + remote.set(document.customId, created) + return created + }) + return { results } + }, + async listDocumentsByCustomIds(customIds) { + return customIds + .map((customId) => remote.get(customId)) + .filter((document): document is Record => document !== undefined) + }, + async searchV4(request) { + const first = remote.values().next().value as Record | undefined + return { + results: first + ? [ + { + id: "probe-result", + memory: "The unique marker is visible.", + metadata: first.metadata, + }, + ] + : [], + requestLimit: request.limit, + } + }, + async deleteDocument(idOrCustomId) { + for (const [customId, document] of remote) { + if (customId === idOrCustomId || document.id === idOrCustomId) remote.delete(customId) + } + }, + }) + const now = Date.parse("2026-07-27T00:00:00.000Z") + const report = await new AdvancedSupermemoryPreflight(api, { + sessionId: "test-session", + searchTopK: 100, + readinessTimeoutMs: 100, + searchVisibilityTimeoutMs: 100, + searchPollMs: 1, + clock: () => now, + sleep: async () => {}, + onCheck: (check) => observedChecks.push(check.check), + }).run() + + expect(report.allPassed).toBe(true) + expect(observedChecks).toContain("v3_trajectory_batch") + expect(observedChecks).toContain("search_run_fingerprint_filter") + expect(observedChecks).toContain("cleanup") + expect(remote.size).toBe(0) + expect(() => + validateSupermemoryPreflightReport(report, { + baseUrl: api.baseUrl, + requiredTopK: 100, + maxAgeMs: 1_000, + now, + }) + ).not.toThrow() + expect(() => + validateSupermemoryPreflightReport(report, { + baseUrl: "https://another.supermemory.test", + requiredTopK: 100, + maxAgeMs: 1_000, + now, + }) + ).toThrow("base URL") + expect(() => + validateSupermemoryPreflightReport(report, { + baseUrl: api.baseUrl, + requiredTopK: 101, + maxAgeMs: 1_000, + now, + }) + ).toThrow("search contract") + expect(() => + validateSupermemoryPreflightReport(report, { + baseUrl: api.baseUrl, + requiredTopK: 100, + maxAgeMs: 1_000, + now: now + 1_001, + }) + ).toThrow("stale") + expect(supermemoryPreflightGatePath("/tmp/preflight-gates", `${api.baseUrl}/`)).toBe( + supermemoryPreflightGatePath("/tmp/preflight-gates", api.baseUrl) + ) + expect(supermemoryPreflightGatePath("/tmp/preflight-gates", api.baseUrl)).not.toBe( + supermemoryPreflightGatePath("/tmp/preflight-gates", "https://another.supermemory.test") + ) + }) +}) diff --git a/src/providers/supermemory/advanced/build.ts b/src/providers/supermemory/advanced/build.ts new file mode 100644 index 0000000..ce49f1d --- /dev/null +++ b/src/providers/supermemory/advanced/build.ts @@ -0,0 +1,575 @@ +import type { AdvancedSupermemoryApi, SupermemoryMetadata, V3DocumentInput } from "./client" +import { + SupermemoryContractError, + SupermemoryHttpError, + SupermemoryRetryExhaustedError, + isRecord, +} from "./client" + +const RECONCILIATION_BATCH_SIZE = 100 +const DEFAULT_CLEANUP_TIMEOUT_MS = 5 * 60_000 +const DEFAULT_CLEANUP_INITIAL_POLL_MS = 1_000 +const DEFAULT_CLEANUP_MAX_POLL_MS = 5_000 + +export interface SupermemoryBuildIdentity { + buildId: string + containerTag: string + runFingerprint: string +} + +export interface SupermemoryBuildDocument { + customId: string + content: string + metadata: SupermemoryMetadata + filterByMetadata?: SupermemoryMetadata +} + +export interface SupermemoryTrajectoryBatch { + trajectoryId: string + identity: SupermemoryBuildIdentity + documents: SupermemoryBuildDocument[] +} + +export type RemoteDocumentStatus = "absent" | "accepted" | "indexing" | "ready" | "failed" + +export interface ReconciledDocument { + customId: string + remoteId?: string + status: RemoteDocumentStatus + remoteStatus?: string + metadata?: Record + memoryCount?: number + raw?: Record +} + +export interface TrajectoryBatchSubmission { + trajectoryId: string + documents: ReconciledDocument[] + reconciled: boolean + rawResponse?: Record +} + +export interface ReadinessProgress { + ready: ReconciledDocument[] + pending: ReconciledDocument[] + failed: ReconciledDocument[] +} + +export class SupermemoryBatchSubmissionError extends Error { + readonly states: ReconciledDocument[] + + constructor(message: string, states: ReconciledDocument[], cause?: unknown) { + super(message, cause === undefined ? undefined : { cause }) + this.name = "SupermemoryBatchSubmissionError" + this.states = states + } +} + +export class SupermemoryReadinessTimeoutError extends Error { + readonly states: ReconciledDocument[] + + constructor(message: string, states: ReconciledDocument[]) { + super(message) + this.name = "SupermemoryReadinessTimeoutError" + this.states = states + } +} + +export class SupermemoryRemoteDocumentFailedError extends Error { + readonly documents: ReconciledDocument[] + + constructor(documents: ReconciledDocument[]) { + super( + `${documents.length} Supermemory document${documents.length === 1 ? "" : "s"} failed: ${documents + .slice(0, 3) + .map((document) => document.customId) + .join(", ")}` + ) + this.name = "SupermemoryRemoteDocumentFailedError" + this.documents = documents + } +} + +export class UnsafeSupermemoryCleanupError extends Error { + constructor(message: string) { + super(message) + this.name = "UnsafeSupermemoryCleanupError" + } +} + +export class SupermemoryCleanupTimeoutError extends Error { + readonly states: ReconciledDocument[] + + constructor(message: string, states: ReconciledDocument[]) { + super(message) + this.name = "SupermemoryCleanupTimeoutError" + this.states = states + } +} + +export interface AdvancedSupermemoryBuildOptions { + sleep?: (milliseconds: number) => Promise + clock?: () => number + cleanupTimeoutMs?: number + cleanupInitialPollMs?: number + cleanupMaxPollMs?: number + signal?: AbortSignal +} + +export class AdvancedSupermemoryBuild { + private readonly sleep: (milliseconds: number) => Promise + private readonly clock: () => number + private readonly cleanupTimeoutMs: number + private readonly cleanupInitialPollMs: number + private readonly cleanupMaxPollMs: number + private readonly signal?: AbortSignal + + constructor( + private readonly client: AdvancedSupermemoryApi, + options: AdvancedSupermemoryBuildOptions = {} + ) { + this.sleep = options.sleep ?? ((milliseconds) => Bun.sleep(milliseconds)) + this.clock = options.clock ?? Date.now + this.cleanupTimeoutMs = options.cleanupTimeoutMs ?? DEFAULT_CLEANUP_TIMEOUT_MS + this.cleanupInitialPollMs = options.cleanupInitialPollMs ?? DEFAULT_CLEANUP_INITIAL_POLL_MS + this.cleanupMaxPollMs = options.cleanupMaxPollMs ?? DEFAULT_CLEANUP_MAX_POLL_MS + this.signal = options.signal + if ( + this.cleanupTimeoutMs < 1 || + this.cleanupInitialPollMs < 0 || + this.cleanupMaxPollMs < this.cleanupInitialPollMs + ) { + throw new Error("Invalid cleanup timeout or polling configuration") + } + } + + async submitTrajectoryBatch( + input: SupermemoryTrajectoryBatch + ): Promise { + validateIdentity(input.identity) + if (!input.trajectoryId.trim()) throw new Error("trajectoryId must not be empty") + if (input.documents.length < 1 || input.documents.length > 600) { + throw new Error("A trajectory batch must contain between 1 and 600 documents") + } + const customIds = input.documents.map((document) => document.customId) + if (new Set(customIds).size !== customIds.length) { + throw new Error(`Trajectory ${input.trajectoryId} has duplicate custom IDs`) + } + + const documents: V3DocumentInput[] = input.documents.map((document) => ({ + content: document.content, + customId: document.customId, + metadata: withBuildMetadata(document.metadata, input.identity, input.trajectoryId), + ...(document.filterByMetadata + ? { filterByMetadata: validateFilter(document.filterByMetadata, input.identity) } + : {}), + })) + + let rawResponse: Record + try { + rawResponse = await this.client.addDocumentsBatch({ + documents, + containerTag: input.identity.containerTag, + dreaming: "instant", + }) + } catch (error) { + if (!isAmbiguousSubmissionError(error)) throw error + return this.reconcileAmbiguousSubmission(input, error) + } + + const results = rawResponse.results + if (!Array.isArray(results) || results.length !== documents.length) { + return this.reconcileAmbiguousSubmission( + input, + new SupermemoryContractError( + `V3 batch returned ${Array.isArray(results) ? results.length : "no"} results for ${ + documents.length + } documents` + ) + ) + } + + const accepted = results.map((result, index): ReconciledDocument => { + const record = isRecord(result) ? result : {} + const remoteId = typeof record.id === "string" && record.id ? record.id : undefined + const remoteStatus = typeof record.status === "string" ? record.status : undefined + return { + customId: documents[index].customId, + remoteId, + remoteStatus, + status: classifyRemoteStatus(remoteStatus, remoteId !== undefined), + raw: record, + } + }) + + if (accepted.some((document) => !document.remoteId || document.status === "failed")) { + return this.reconcileAmbiguousSubmission( + input, + new SupermemoryContractError("V3 batch contained missing IDs or failed results") + ) + } + + return { + trajectoryId: input.trajectoryId, + documents: accepted, + reconciled: false, + rawResponse, + } + } + + async reconcileByCustomId( + customIds: string[], + containerTag: string, + signal?: AbortSignal + ): Promise { + if (!containerTag.trim()) throw new Error("containerTag must not be empty") + if (new Set(customIds).size !== customIds.length) { + throw new Error("Custom IDs for reconciliation must be unique") + } + const found = new Map() + for (let offset = 0; offset < customIds.length; offset += RECONCILIATION_BATCH_SIZE) { + const batch = customIds.slice(offset, offset + RECONCILIATION_BATCH_SIZE) + throwIfAborted(signal) + const remoteDocuments = await this.client.listDocumentsByCustomIds( + batch, + containerTag, + signal + ) + for (const remote of remoteDocuments) { + const customId = stringField(remote, "customId") + if (!customId || !batch.includes(customId)) continue + if (found.has(customId)) { + throw new SupermemoryContractError( + `Reconciliation returned duplicate documents for customId ${customId}` + ) + } + const remoteId = stringField(remote, "id") + const remoteStatus = stringField(remote, "status") + found.set(customId, { + customId, + remoteId, + remoteStatus, + status: classifyRemoteStatus(remoteStatus, remoteId !== undefined), + metadata: extractMetadata(remote), + memoryCount: extractMemoryCount(remote), + raw: remote, + }) + } + } + + return customIds.map( + (customId) => + found.get(customId) ?? { + customId, + status: "absent", + } + ) + } + + async awaitReady(input: { + customIds: string[] + containerTag: string + timeoutMs?: number + initialPollMs?: number + maxPollMs?: number + signal?: AbortSignal + onProgress?: (progress: ReadinessProgress) => void + }): Promise { + const timeoutMs = input.timeoutMs ?? 30 * 60_000 + const maxPollMs = input.maxPollMs ?? 5_000 + let pollMs = input.initialPollMs ?? 1_000 + if (timeoutMs < 1 || pollMs < 0 || maxPollMs < pollMs) { + throw new Error("Invalid readiness timeout or polling configuration") + } + const deadline = this.clock() + timeoutMs + + while (true) { + if (input.signal?.aborted) throw input.signal.reason ?? new Error("Readiness wait aborted") + const states = await this.reconcileByCustomId( + input.customIds, + input.containerTag, + input.signal + ) + const progress = { + ready: states.filter((document) => document.status === "ready"), + failed: states.filter((document) => document.status === "failed"), + pending: states.filter( + (document) => document.status !== "ready" && document.status !== "failed" + ), + } + input.onProgress?.(progress) + if (progress.failed.length > 0) { + throw new SupermemoryRemoteDocumentFailedError(progress.failed) + } + if (progress.ready.length === states.length) return states + if (this.clock() >= deadline) { + throw new SupermemoryReadinessTimeoutError( + `${progress.pending.length} documents were not ready within ${timeoutMs}ms`, + states + ) + } + await abortableSleep( + this.sleep, + Math.min(pollMs, Math.max(0, deadline - this.clock())), + input.signal + ) + pollMs = Math.min(maxPollMs, Math.max(1, Math.ceil(pollMs * 1.5))) + } + } + + /** + * Delete only explicitly enumerated documents whose persisted metadata proves + * they belong to the exact build. Container-wide deletion is intentionally + * unsupported. + */ + async cleanupExactBuild(input: { + identity: SupermemoryBuildIdentity + customIds: string[] + }): Promise<{ deleted: string[]; absent: string[] }> { + validateIdentity(input.identity) + if (input.customIds.length === 0) { + throw new UnsafeSupermemoryCleanupError("Exact-build cleanup requires explicit custom IDs") + } + throwIfAborted(this.signal) + const deadline = this.clock() + this.cleanupTimeoutMs + const states = await this.reconcileByCustomId( + input.customIds, + input.identity.containerTag, + this.signal + ) + const deletable: ReconciledDocument[] = [] + const absent: string[] = [] + + for (const state of states) { + if (state.status === "absent") { + absent.push(state.customId) + continue + } + assertExactBuildOwnership(state, input.identity) + deletable.push(state) + } + + for (const state of deletable) { + await this.deleteWithReconciliation(state, input.identity, deadline) + } + + const remaining = await this.reconcileByCustomId( + deletable.map((document) => document.customId), + input.identity.containerTag, + this.signal + ) + const notDeleted = remaining.filter((document) => document.status !== "absent") + if (notDeleted.length > 0) { + throw new UnsafeSupermemoryCleanupError( + `Cleanup verification found ${notDeleted.length} remaining documents` + ) + } + return { + deleted: deletable.map((document) => document.customId), + absent, + } + } + + private async deleteWithReconciliation( + initialState: ReconciledDocument, + identity: SupermemoryBuildIdentity, + deadline: number + ): Promise { + let state = initialState + let pollMs = this.cleanupInitialPollMs + + while (true) { + throwIfAborted(this.signal) + assertExactBuildOwnership(state, identity) + + try { + await this.client.deleteDocument(state.remoteId ?? state.customId, this.signal) + } catch (error) { + if (!isDeleteConflict(error)) throw error + } + + const [reconciled] = await this.reconcileByCustomId( + [state.customId], + identity.containerTag, + this.signal + ) + if (reconciled.status === "absent") return + assertExactBuildOwnership(reconciled, identity) + state = reconciled + + if (this.clock() >= deadline) { + throw new SupermemoryCleanupTimeoutError( + `Supermemory cleanup could not delete ${state.customId} within ${this.cleanupTimeoutMs}ms`, + [state] + ) + } + await abortableSleep( + this.sleep, + Math.min(pollMs, Math.max(0, deadline - this.clock())), + this.signal + ) + pollMs = Math.min(this.cleanupMaxPollMs, Math.max(1, Math.ceil(Math.max(1, pollMs) * 1.5))) + } + } + + private async reconcileAmbiguousSubmission( + input: SupermemoryTrajectoryBatch, + cause: unknown + ): Promise { + const states = await this.reconcileByCustomId( + input.documents.map((document) => document.customId), + input.identity.containerTag + ) + if (states.every((document) => document.status !== "absent")) { + return { + trajectoryId: input.trajectoryId, + documents: states, + reconciled: true, + } + } + throw new SupermemoryBatchSubmissionError( + `Trajectory ${input.trajectoryId} has an ambiguous V3 batch submission; resume must not re-upload ${ + states.filter((document) => document.status === "absent").length + } absent custom IDs without a durable decision`, + states, + cause + ) + } +} + +function withBuildMetadata( + metadata: SupermemoryMetadata, + identity: SupermemoryBuildIdentity, + trajectoryId: string +): SupermemoryMetadata { + assertCompatibleMetadata(metadata, "runFingerprint", identity.runFingerprint) + assertCompatibleMetadata(metadata, "buildId", identity.buildId) + assertCompatibleMetadata(metadata, "trajectoryId", trajectoryId) + return { + ...metadata, + runFingerprint: identity.runFingerprint, + buildId: identity.buildId, + trajectoryId, + } +} + +function validateFilter( + filter: SupermemoryMetadata, + identity: SupermemoryBuildIdentity +): SupermemoryMetadata { + if (filter.runFingerprint !== undefined && filter.runFingerprint !== identity.runFingerprint) { + throw new Error("filterByMetadata cannot reference another run fingerprint") + } + return { ...filter } +} + +function assertCompatibleMetadata( + metadata: SupermemoryMetadata, + field: string, + expected: string +): void { + if (metadata[field] !== undefined && metadata[field] !== expected) { + throw new Error(`Document metadata ${field} conflicts with the build identity`) + } +} + +function validateIdentity(identity: SupermemoryBuildIdentity): void { + for (const [field, value] of Object.entries(identity)) { + if (!value.trim()) throw new Error(`${field} must not be empty`) + } +} + +function isAmbiguousSubmissionError(error: unknown): boolean { + return ( + error instanceof SupermemoryRetryExhaustedError || + error instanceof SupermemoryContractError || + (error instanceof SupermemoryHttpError && error.statusCode === 409) + ) +} + +function isDeleteConflict(error: unknown): boolean { + const classified = error instanceof SupermemoryRetryExhaustedError ? error.lastError : error + return classified instanceof SupermemoryHttpError && classified.statusCode === 409 +} + +function assertExactBuildOwnership( + state: ReconciledDocument, + identity: SupermemoryBuildIdentity +): void { + const metadata = state.metadata ?? {} + if ( + metadata.runFingerprint !== identity.runFingerprint || + metadata.buildId !== identity.buildId + ) { + throw new UnsafeSupermemoryCleanupError( + `Refusing to delete ${state.customId}: remote metadata does not match exact build` + ) + } +} + +function throwIfAborted(signal?: AbortSignal): void { + if (signal?.aborted) throw signal.reason ?? new Error("Supermemory operation aborted") +} + +async function abortableSleep( + sleep: (milliseconds: number) => Promise, + milliseconds: number, + signal?: AbortSignal +): Promise { + throwIfAborted(signal) + if (!signal) { + await sleep(milliseconds) + return + } + let onAbort: (() => void) | undefined + try { + await Promise.race([ + sleep(milliseconds), + new Promise((_, reject) => { + onAbort = () => reject(signal.reason ?? new Error("Supermemory operation aborted")) + signal.addEventListener("abort", onAbort, { once: true }) + if (signal.aborted) onAbort() + }), + ]) + } finally { + if (onAbort) signal.removeEventListener("abort", onAbort) + } + throwIfAborted(signal) +} + +function classifyRemoteStatus(status: string | undefined, exists: boolean): RemoteDocumentStatus { + const normalized = status?.toLowerCase() + if (normalized === "done" || normalized === "completed" || normalized === "ready") return "ready" + if (normalized === "failed" || normalized === "error" || normalized === "rejected") + return "failed" + if ( + normalized === "processing" || + normalized === "indexing" || + normalized === "pending" || + normalized === "queued" + ) { + return "indexing" + } + return exists ? "accepted" : "absent" +} + +function extractMetadata(remote: Record): Record { + if (isRecord(remote.metadata)) return remote.metadata + if (isRecord(remote.document) && isRecord(remote.document.metadata)) { + return remote.document.metadata + } + return {} +} + +function extractMemoryCount(remote: Record): number | undefined { + for (const field of ["memoryEntries", "memory_entries", "memories"]) { + const value = remote[field] + if (Array.isArray(value)) return value.length + } + return undefined +} + +function stringField(record: Record, field: string): string | undefined { + const value = record[field] + return typeof value === "string" && value ? value : undefined +} diff --git a/src/providers/supermemory/advanced/client.ts b/src/providers/supermemory/advanced/client.ts new file mode 100644 index 0000000..d050429 --- /dev/null +++ b/src/providers/supermemory/advanced/client.ts @@ -0,0 +1,712 @@ +import { createHash } from "node:crypto" + +const RETRYABLE_HTTP_STATUSES = new Set([408, 429, 500, 502, 503, 504]) +const BODY_SNIPPET_LIMIT = 500 +const DEFAULT_BASE_URL = "https://api.supermemory.ai" + +export type SupermemoryMetadataValue = string | number | boolean | string[] +export type SupermemoryMetadata = Record + +export interface V3DocumentInput { + content: string + customId: string + metadata: SupermemoryMetadata + filterByMetadata?: SupermemoryMetadata +} + +export interface V3DocumentResponse { + id: string + customId?: string + status?: string + metadata?: Record + [key: string]: unknown +} + +export interface V3BatchResponse { + results: Array> + [key: string]: unknown +} + +export interface V4SearchRequest { + q: string + containerTag: string + limit: number + threshold: number + searchMode: "hybrid" | "memories" + rerank: boolean + rewriteQuery: boolean + include: { + summaries: boolean + documents: boolean + relatedMemories: boolean + } + filters?: Record +} + +export interface RequestBudgetSnapshot { + configuredCap: number + effectiveCap: number + inFlight: number + peakInFlight: number + throttleEvents: number + successStreak: number + notBeforeMs: number +} + +export type AdvancedSupermemoryEventLogger = ( + event: string, + details: Record +) => void + +export class SupermemoryHttpError extends Error { + readonly statusCode?: number + readonly retryable: boolean + readonly retryAfterMs?: number + + constructor( + message: string, + options: { + statusCode?: number + retryable: boolean + retryAfterMs?: number + cause?: unknown + } + ) { + super(message, options.cause === undefined ? undefined : { cause: options.cause }) + this.name = "SupermemoryHttpError" + this.statusCode = options.statusCode + this.retryable = options.retryable + this.retryAfterMs = options.retryAfterMs + } +} + +export class SupermemoryRetryExhaustedError extends Error { + readonly attempts: number + readonly lastError: SupermemoryHttpError + + constructor(operation: string, attempts: number, lastError: SupermemoryHttpError) { + super(`${operation} failed after ${attempts} attempts: ${lastError.message}`, { + cause: lastError, + }) + this.name = "SupermemoryRetryExhaustedError" + this.attempts = attempts + this.lastError = lastError + } +} + +export class SupermemoryContractError extends Error { + readonly statusCode?: number + + constructor(message: string, statusCode?: number) { + super(message) + this.name = "SupermemoryContractError" + this.statusCode = statusCode + } +} + +export interface AdaptiveRequestBudgetOptions { + maxInFlight: number + recoverySuccesses?: number + clock?: () => number + sleep?: (milliseconds: number) => Promise + eventLogger?: AdvancedSupermemoryEventLogger +} + +/** + * One async request budget shared by upload, polling, reconciliation, search, + * cleanup, and every client connected to the same account/base URL. + */ +export class AdaptiveRequestBudget { + private configuredCap: number + private effectiveCap: number + private inFlight = 0 + private peakInFlight = 0 + private throttleEvents = 0 + private successStreak = 0 + private notBeforeMs = 0 + private readonly recoverySuccesses: number + private readonly clock: () => number + private readonly sleep: (milliseconds: number) => Promise + private readonly eventLogger?: AdvancedSupermemoryEventLogger + private readonly waiters: Array<() => void> = [] + + constructor(options: AdaptiveRequestBudgetOptions) { + if (!Number.isInteger(options.maxInFlight) || options.maxInFlight < 1) { + throw new Error("maxInFlight must be an integer >= 1") + } + this.configuredCap = options.maxInFlight + this.effectiveCap = options.maxInFlight + this.recoverySuccesses = options.recoverySuccesses ?? 20 + this.clock = options.clock ?? Date.now + this.sleep = options.sleep ?? ((milliseconds) => Bun.sleep(milliseconds)) + this.eventLogger = options.eventLogger + } + + restrictTo(maxInFlight: number): void { + if (!Number.isInteger(maxInFlight) || maxInFlight < 1) { + throw new Error("maxInFlight must be an integer >= 1") + } + this.configuredCap = Math.min(this.configuredCap, maxInFlight) + this.effectiveCap = Math.min(this.effectiveCap, this.configuredCap) + this.wakeWaiters() + } + + async run(operation: () => Promise): Promise { + await this.acquire() + try { + return await operation() + } finally { + this.inFlight -= 1 + this.wakeWaiters() + } + } + + recordSuccess(): void { + this.successStreak += 1 + if (this.successStreak >= this.recoverySuccesses && this.effectiveCap < this.configuredCap) { + this.effectiveCap += 1 + this.successStreak = 0 + this.emit("request_budget_recovered", { effectiveCap: this.effectiveCap }) + this.wakeWaiters() + } + } + + recordPressure(reason: string, retryAfterMs?: number): void { + this.successStreak = 0 + this.effectiveCap = Math.max(1, Math.floor(this.effectiveCap / 2)) + if (retryAfterMs !== undefined && retryAfterMs > 0) { + this.notBeforeMs = Math.max(this.notBeforeMs, this.clock() + retryAfterMs) + } + this.throttleEvents += 1 + this.emit("request_budget_pressure", { + reason, + effectiveCap: this.effectiveCap, + retryAfterMs, + }) + } + + snapshot(): RequestBudgetSnapshot { + return { + configuredCap: this.configuredCap, + effectiveCap: this.effectiveCap, + inFlight: this.inFlight, + peakInFlight: this.peakInFlight, + throttleEvents: this.throttleEvents, + successStreak: this.successStreak, + notBeforeMs: this.notBeforeMs, + } + } + + private async acquire(): Promise { + while (true) { + const waitForThrottle = this.notBeforeMs - this.clock() + if (waitForThrottle > 0) { + await this.sleep(waitForThrottle) + continue + } + if (this.inFlight < this.effectiveCap) { + this.inFlight += 1 + this.peakInFlight = Math.max(this.peakInFlight, this.inFlight) + return + } + await new Promise((resolve) => this.waiters.push(resolve)) + } + } + + private wakeWaiters(): void { + for (const resolve of this.waiters.splice(0)) resolve() + } + + private emit(event: string, details: Record): void { + try { + this.eventLogger?.(event, details) + } catch { + // Observability must never change request behavior. + } + } +} + +const sharedBudgets = new Map() + +function accountBudgetKey(baseUrl: string, apiKey: string): string { + const keyHash = createHash("sha256").update(apiKey).digest("hex").slice(0, 16) + return `${baseUrl}:${keyHash}` +} + +export function getSharedSupermemoryRequestBudget(options: { + baseUrl: string + apiKey: string + maxInFlight: number + eventLogger?: AdvancedSupermemoryEventLogger +}): AdaptiveRequestBudget { + const key = accountBudgetKey(options.baseUrl, options.apiKey) + const existing = sharedBudgets.get(key) + if (existing) { + existing.restrictTo(options.maxInFlight) + return existing + } + const created = new AdaptiveRequestBudget({ + maxInFlight: options.maxInFlight, + eventLogger: options.eventLogger, + }) + sharedBudgets.set(key, created) + return created +} + +export interface AdvancedSupermemoryClientOptions { + apiKey: string + baseUrl?: string + maxInFlightRequests?: number + maxAttempts?: number + requestTimeoutMs?: number + backoffBaseMs?: number + backoffMaxMs?: number + fetch?: typeof fetch + sleep?: (milliseconds: number) => Promise + clock?: () => number + random?: () => number + budget?: AdaptiveRequestBudget + eventLogger?: AdvancedSupermemoryEventLogger + userAgent?: string +} + +export interface AdvancedSupermemoryApi { + readonly baseUrl: string + readonly requestCount: number + readonly budgetSnapshot: RequestBudgetSnapshot + addDocument(input: { + document: V3DocumentInput + containerTag: string + dreaming?: "instant" | string + maxAttempts?: number + }): Promise + addDocumentsBatch(input: { + documents: V3DocumentInput[] + containerTag: string + dreaming?: "instant" | string + maxAttempts?: number + }): Promise + getDocument(idOrCustomId: string): Promise | null> + listDocumentsByCustomIds( + customIds: string[], + containerTag?: string, + signal?: AbortSignal + ): Promise>> + searchV4(request: V4SearchRequest, maxAttempts?: number): Promise> + deleteDocument(idOrCustomId: string, signal?: AbortSignal): Promise +} + +interface RequestOptions { + body?: Record + maxAttempts?: number + operation: string + timeoutMs?: number + allowNotFound?: boolean + signal?: AbortSignal +} + +export class AdvancedSupermemoryClient implements AdvancedSupermemoryApi { + readonly baseUrl: string + private readonly apiKey: string + private readonly maxAttempts: number + private readonly requestTimeoutMs: number + private readonly backoffBaseMs: number + private readonly backoffMaxMs: number + private readonly fetchImpl: typeof fetch + private readonly sleep: (milliseconds: number) => Promise + private readonly clock: () => number + private readonly random: () => number + private readonly budget: AdaptiveRequestBudget + private readonly eventLogger?: AdvancedSupermemoryEventLogger + private readonly userAgent: string + private requests = 0 + + constructor(options: AdvancedSupermemoryClientOptions) { + if (!options.apiKey.trim()) throw new Error("Supermemory API key is required") + this.baseUrl = normalizeBaseUrl(options.baseUrl ?? DEFAULT_BASE_URL) + this.apiKey = options.apiKey + this.maxAttempts = options.maxAttempts ?? 8 + this.requestTimeoutMs = options.requestTimeoutMs ?? 120_000 + this.backoffBaseMs = options.backoffBaseMs ?? 1_000 + this.backoffMaxMs = options.backoffMaxMs ?? 60_000 + this.fetchImpl = options.fetch ?? fetch + this.sleep = options.sleep ?? ((milliseconds) => Bun.sleep(milliseconds)) + this.clock = options.clock ?? Date.now + this.random = options.random ?? Math.random + this.eventLogger = options.eventLogger + this.userAgent = options.userAgent ?? "memorybench-supermemory-advanced/1" + this.budget = + options.budget ?? + getSharedSupermemoryRequestBudget({ + baseUrl: this.baseUrl, + apiKey: this.apiKey, + maxInFlight: options.maxInFlightRequests ?? 20, + eventLogger: options.eventLogger, + }) + + if (!Number.isInteger(this.maxAttempts) || this.maxAttempts < 1) { + throw new Error("maxAttempts must be an integer >= 1") + } + } + + get requestCount(): number { + return this.requests + } + + get budgetSnapshot(): RequestBudgetSnapshot { + return this.budget.snapshot() + } + + async addDocument(input: { + document: V3DocumentInput + containerTag: string + dreaming?: "instant" | string + maxAttempts?: number + }): Promise { + validateDocument(input.document) + const payload = await this.request("POST", "/v3/documents", { + operation: "add_document", + maxAttempts: input.maxAttempts, + body: { + ...input.document, + containerTag: requireNonEmpty(input.containerTag, "containerTag"), + ...(input.dreaming ? { dreaming: input.dreaming } : {}), + }, + }) + if (!isRecord(payload) || typeof payload.id !== "string" || !payload.id) { + throw new SupermemoryContractError("add_document returned an unexpected response") + } + return payload as V3DocumentResponse + } + + async addDocumentsBatch(input: { + documents: V3DocumentInput[] + containerTag: string + dreaming?: "instant" | string + maxAttempts?: number + }): Promise { + if (input.documents.length < 1 || input.documents.length > 600) { + throw new Error("A V3 document batch must contain between 1 and 600 documents") + } + input.documents.forEach(validateDocument) + const payload = await this.request("POST", "/v3/documents/batch", { + operation: "add_documents_batch", + maxAttempts: input.maxAttempts, + body: { + documents: input.documents, + containerTag: requireNonEmpty(input.containerTag, "containerTag"), + ...(input.dreaming ? { dreaming: input.dreaming } : {}), + }, + }) + if (!isRecord(payload) || !Array.isArray(payload.results)) { + throw new SupermemoryContractError("add_documents_batch returned an unexpected response") + } + return payload as unknown as V3BatchResponse + } + + async getDocument(idOrCustomId: string): Promise | null> { + const payload = await this.request( + "GET", + `/v3/documents/${encodeURIComponent(requireNonEmpty(idOrCustomId, "document ID"))}`, + { + operation: "get_document", + allowNotFound: true, + } + ) + if (payload === null) return null + if (!isRecord(payload)) { + throw new SupermemoryContractError("get_document returned a non-object response") + } + return payload + } + + async listDocumentsByCustomIds( + customIds: string[], + containerTag?: string, + signal?: AbortSignal + ): Promise>> { + if (customIds.length === 0) return [] + const ids = customIds.map((value) => requireNonEmpty(value, "customId")) + const payload = await this.request("POST", "/v3/documents/documents/by-ids", { + operation: "list_documents_by_custom_ids", + body: { + ids, + by: "customId", + ...(containerTag ? { containerTags: [containerTag] } : {}), + }, + signal, + }) + if (!isRecord(payload) || !Array.isArray(payload.documents)) return [] + return payload.documents.filter(isRecord) + } + + async searchV4( + searchRequest: V4SearchRequest, + maxAttempts?: number + ): Promise> { + if (!Number.isInteger(searchRequest.limit) || searchRequest.limit < 1) { + throw new Error("V4 search limit must be an integer >= 1") + } + const payload = await this.request("POST", "/v4/search", { + operation: "search_v4", + maxAttempts, + body: { ...searchRequest }, + }) + if (!isRecord(payload) || !Array.isArray(payload.results)) { + throw new SupermemoryContractError("search_v4 returned an unexpected response") + } + return payload + } + + async deleteDocument(idOrCustomId: string, signal?: AbortSignal): Promise { + await this.request( + "DELETE", + `/v3/documents/${encodeURIComponent(requireNonEmpty(idOrCustomId, "document ID"))}`, + { + operation: "delete_document", + allowNotFound: true, + signal, + } + ) + } + + private async request( + method: string, + path: string, + options: RequestOptions + ): Promise { + const attemptsAllowed = options.maxAttempts ?? this.maxAttempts + if (!Number.isInteger(attemptsAllowed) || attemptsAllowed < 1) { + throw new Error("maxAttempts must be an integer >= 1") + } + let lastError: SupermemoryHttpError | undefined + + for (let attempt = 1; attempt <= attemptsAllowed; attempt += 1) { + throwIfAborted(options.signal, options.operation) + try { + const result = await this.attempt(method, path, options) + this.budget.recordSuccess() + return result + } catch (error) { + throwIfAborted(options.signal, options.operation) + if (!(error instanceof SupermemoryHttpError)) throw error + if (options.allowNotFound && error.statusCode === 404) return null + if (!error.retryable) throw error + lastError = error + this.budget.recordPressure( + `${options.operation}:${error.statusCode ?? "transport"}`, + error.retryAfterMs + ) + this.emit("supermemory_request_retry", { + operation: options.operation, + path, + attempt, + maxAttempts: attemptsAllowed, + statusCode: error.statusCode, + error: error.message, + }) + if (attempt >= attemptsAllowed) break + await abortableSleep( + this.sleep, + this.retryDelay(attempt, error.retryAfterMs), + options.signal, + options.operation + ) + } + } + + throw new SupermemoryRetryExhaustedError( + options.operation, + attemptsAllowed, + lastError ?? + new SupermemoryHttpError("request failed without a classified error", { + retryable: true, + }) + ) + } + + private async attempt( + method: string, + path: string, + options: RequestOptions + ): Promise { + const controller = new AbortController() + const timeoutMs = options.timeoutMs ?? this.requestTimeoutMs + const timeout = setTimeout(() => controller.abort(), timeoutMs) + const onParentAbort = () => controller.abort(options.signal?.reason) + options.signal?.addEventListener("abort", onParentAbort, { once: true }) + if (options.signal?.aborted) onParentAbort() + + let response: Response + let text: string + try { + const completed = await this.budget.run(async () => { + this.requests += 1 + const fetched = await this.fetchImpl(`${this.baseUrl}${path}`, { + method, + headers: { + Authorization: `Bearer ${this.apiKey}`, + "Content-Type": "application/json", + "User-Agent": this.userAgent, + }, + body: options.body === undefined ? undefined : JSON.stringify(options.body), + signal: controller.signal, + }) + return { response: fetched, text: await fetched.text() } + }) + response = completed.response + text = completed.text + } catch (error) { + throw new SupermemoryHttpError( + `transport error during ${options.operation}: ${ + error instanceof Error ? error.name : "unknown" + }`, + { retryable: true, cause: error } + ) + } finally { + clearTimeout(timeout) + options.signal?.removeEventListener("abort", onParentAbort) + } + + const payload = parseResponseBody(text) + if (response.ok) return payload + + const retryAfterMs = parseRetryAfter(response.headers.get("retry-after"), this.clock()) + const retryable = + RETRYABLE_HTTP_STATUSES.has(response.status) || + (response.status >= 500 && response.status <= 599) + const snippet = safeBodySnippet(text, [this.apiKey]) + throw new SupermemoryHttpError( + `HTTP ${response.status} during ${options.operation}${snippet ? `: ${snippet}` : ""}`, + { + statusCode: response.status, + retryable, + retryAfterMs, + } + ) + } + + private retryDelay(attempt: number, retryAfterMs?: number): number { + if (retryAfterMs !== undefined) return Math.min(retryAfterMs, this.backoffMaxMs) + const exponential = Math.min(this.backoffMaxMs, this.backoffBaseMs * 2 ** (attempt - 1)) + return exponential * (0.5 + this.random() / 2) + } + + private emit(event: string, details: Record): void { + try { + this.eventLogger?.(event, redact(details, [this.apiKey]) as Record) + } catch { + // Logging is best effort. + } + } +} + +export function redact(value: T, secrets: string[] = []): T { + return redactUnknown(value, secrets.filter(Boolean)) as T +} + +function redactUnknown(value: unknown, secrets: string[]): unknown { + if (typeof value === "string") return sanitizeText(value, secrets) + if (Array.isArray(value)) return value.map((item) => redactUnknown(item, secrets)) + if (!isRecord(value)) return value + const output: Record = {} + for (const [key, item] of Object.entries(value)) { + if (/(authorization|api[-_]?key|secret|token|password)/i.test(key)) { + output[key] = "" + } else { + output[key] = redactUnknown(item, secrets) + } + } + return output +} + +function sanitizeText(value: string, secrets: string[]): string { + let sanitized = value.replace(/Bearer\s+[A-Za-z0-9._~+/=-]+/gi, "Bearer ") + for (const secret of secrets) { + sanitized = sanitized.split(secret).join("") + } + return sanitized +} + +function safeBodySnippet(body: string, secrets: string[]): string { + const sanitized = sanitizeText(body.trim(), secrets).replace(/[\u0000-\u001f\u007f]/g, " ") + return sanitized.length > BODY_SNIPPET_LIMIT + ? `${sanitized.slice(0, BODY_SNIPPET_LIMIT)}...` + : sanitized +} + +function throwIfAborted(signal: AbortSignal | undefined, operation: string): void { + if (signal?.aborted) throw signal.reason ?? new Error(`${operation} aborted`) +} + +async function abortableSleep( + sleep: (milliseconds: number) => Promise, + milliseconds: number, + signal: AbortSignal | undefined, + operation: string +): Promise { + throwIfAborted(signal, operation) + if (!signal) { + await sleep(milliseconds) + return + } + let onAbort: (() => void) | undefined + try { + await Promise.race([ + sleep(milliseconds), + new Promise((_, reject) => { + onAbort = () => reject(signal.reason ?? new Error(`${operation} aborted`)) + signal.addEventListener("abort", onAbort, { once: true }) + if (signal.aborted) onAbort() + }), + ]) + } finally { + if (onAbort) signal.removeEventListener("abort", onAbort) + } + throwIfAborted(signal, operation) +} + +function parseRetryAfter(value: string | null, nowMs: number): number | undefined { + if (!value) return undefined + const seconds = Number(value.trim()) + if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000) + const dateMs = Date.parse(value) + if (Number.isNaN(dateMs)) return undefined + return Math.max(0, dateMs - nowMs) +} + +function parseResponseBody(text: string): unknown | null { + if (!text) return null + try { + return JSON.parse(text) + } catch { + return null + } +} + +function normalizeBaseUrl(input: string): string { + const trimmed = input.trim().replace(/\/+$/, "") + const url = new URL(trimmed) + if (url.protocol !== "https:" && url.protocol !== "http:") { + throw new Error("Supermemory base URL must use HTTP or HTTPS") + } + if (url.username || url.password) { + throw new Error("Supermemory base URL must not contain credentials") + } + return trimmed +} + +function requireNonEmpty(value: string, field: string): string { + if (!value.trim()) throw new Error(`${field} must not be empty`) + return value +} + +function validateDocument(document: V3DocumentInput): void { + requireNonEmpty(document.content, "document content") + requireNonEmpty(document.customId, "customId") + if (!isRecord(document.metadata)) throw new Error("document metadata must be an object") +} + +export function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value) +} diff --git a/src/providers/supermemory/advanced/index.ts b/src/providers/supermemory/advanced/index.ts new file mode 100644 index 0000000..6dad136 --- /dev/null +++ b/src/providers/supermemory/advanced/index.ts @@ -0,0 +1,5 @@ +export * from "./client" +export * from "./build" +export * from "./retrieval" +export * from "./preflight" +export * from "./provider" diff --git a/src/providers/supermemory/advanced/preflight.ts b/src/providers/supermemory/advanced/preflight.ts new file mode 100644 index 0000000..4dd4766 --- /dev/null +++ b/src/providers/supermemory/advanced/preflight.ts @@ -0,0 +1,414 @@ +import { createHash, randomUUID } from "node:crypto" +import { resolve } from "node:path" +import type { + AdvancedSupermemoryApi, + RequestBudgetSnapshot, + SupermemoryMetadata, + V3DocumentInput, +} from "./client" +import { redact } from "./client" +import { AdvancedSupermemoryBuild, type SupermemoryBuildIdentity } from "./build" +import { AdvancedSupermemoryRetrieval, type AdvancedRetrievalConfig } from "./retrieval" + +export const SUPERMEMORY_PREFLIGHT_SCHEMA_VERSION = 1 + +function normalizeBaseUrl(value: string): string { + return value.trim().replace(/\/+$/, "") +} + +export function supermemoryPreflightGatePath(root: string, baseUrl: string): string { + const normalized = normalizeBaseUrl(baseUrl) + if (!normalized) throw new Error("Supermemory preflight base URL must not be empty") + const serviceId = createHash("sha256").update(normalized).digest("hex").slice(0, 24) + return resolve(root, "supermemory", serviceId, "latest-passed.json") +} + +export interface SupermemoryPreflightCheck { + check: string + ok: boolean + details?: Record +} + +export interface SupermemoryPreflightReport { + schemaVersion: number + generatedAt: string + baseUrl: string + identity: SupermemoryBuildIdentity + searchContract: { + searchMode: "hybrid" + standaloneChunksExpected: true + deprecatedIncludeChunks: false + requestedTopK: number + } + checks: SupermemoryPreflightCheck[] + allPassed: boolean + blockers: string[] + requestBudget: RequestBudgetSnapshot +} + +export interface SupermemoryPreflightOptions { + searchTopK?: number + readinessTimeoutMs?: number + searchVisibilityTimeoutMs?: number + searchPollMs?: number + keepDocuments?: boolean + sessionId?: string + containerPrefix?: string + clock?: () => number + sleep?: (milliseconds: number) => Promise + onCheck?: (check: SupermemoryPreflightCheck) => void +} + +/** + * Live-account contract probe. Constructing this class is side-effect free; + * network work occurs only when the caller explicitly invokes run(). + */ +export class AdvancedSupermemoryPreflight { + private readonly build: AdvancedSupermemoryBuild + private readonly retrieval: AdvancedSupermemoryRetrieval + private readonly clock: () => number + private readonly sleep: (milliseconds: number) => Promise + private readonly options: Required< + Pick< + SupermemoryPreflightOptions, + | "searchTopK" + | "readinessTimeoutMs" + | "searchVisibilityTimeoutMs" + | "searchPollMs" + | "keepDocuments" + | "containerPrefix" + > + > & + SupermemoryPreflightOptions + + constructor( + private readonly client: AdvancedSupermemoryApi, + options: SupermemoryPreflightOptions = {} + ) { + this.clock = options.clock ?? Date.now + this.sleep = options.sleep ?? ((milliseconds) => Bun.sleep(milliseconds)) + this.options = { + ...options, + searchTopK: options.searchTopK ?? 100, + readinessTimeoutMs: options.readinessTimeoutMs ?? 5 * 60_000, + searchVisibilityTimeoutMs: options.searchVisibilityTimeoutMs ?? 2 * 60_000, + searchPollMs: options.searchPollMs ?? 5_000, + keepDocuments: options.keepDocuments ?? false, + containerPrefix: options.containerPrefix ?? "memorybench-preflight", + } + this.build = new AdvancedSupermemoryBuild(client, { + clock: this.clock, + sleep: this.sleep, + }) + this.retrieval = new AdvancedSupermemoryRetrieval(client, { clock: this.clock }) + } + + async run(): Promise { + const session = this.options.sessionId ?? randomUUID().replaceAll("-", "").slice(0, 12) + const identity: SupermemoryBuildIdentity = { + buildId: `preflight-${session}`, + containerTag: `${this.options.containerPrefix}-${session}`, + runFingerprint: `preflight-${session}`, + } + const marker = `zebra-fjord-${session}` + const checks: SupermemoryPreflightCheck[] = [] + const createdCustomIds: string[] = [] + const customId = (suffix: string) => `preflight-${session}-${suffix}` + const metadata = (extra: SupermemoryMetadata = {}): SupermemoryMetadata => ({ + benchmark: "longmemeval-v2-preflight", + buildId: identity.buildId, + runFingerprint: identity.runFingerprint, + ...extra, + }) + const record = (check: string, ok: boolean, details?: Record): void => { + const result = redact({ check, ok, ...(details ? { details } : {}) }) + checks.push(result) + try { + this.options.onCheck?.(result) + } catch { + // A display hook cannot affect the gate. + } + } + const add = async ( + suffix: string, + content: string, + extraMetadata: SupermemoryMetadata = {}, + filterByMetadata?: SupermemoryMetadata + ): Promise> => { + const document: V3DocumentInput = { + content, + customId: customId(suffix), + metadata: metadata(extraMetadata), + ...(filterByMetadata ? { filterByMetadata } : {}), + } + if (!createdCustomIds.includes(document.customId)) createdCustomIds.push(document.customId) + const response = await this.client.addDocument({ + document, + containerTag: identity.containerTag, + dreaming: "instant", + }) + return response + } + + try { + const baselineCustomId = customId("doc1") + const zeroMemoryCustomId = customId("doc4") + try { + createdCustomIds.push(baselineCustomId, zeroMemoryCustomId) + const startedAt = this.clock() + const submission = await this.build.submitTrajectoryBatch({ + trajectoryId: `preflight-trajectory-${session}`, + identity, + documents: [ + { + customId: baselineCustomId, + content: + `Preflight document one. Unique marker: ${marker}. ` + + "The admin settings page contains a Timezone selector.", + metadata: metadata({ + causalKey: `pf:${session}:0:0`, + stateIndex: 0, + }), + }, + { + customId: zeroMemoryCustomId, + content: "ok.", + metadata: metadata({ causalKey: `pf:${session}:3:0` }), + }, + ], + }) + record( + "v3_trajectory_batch", + submission.documents.length === 2 && + submission.documents.every((document) => document.status !== "absent"), + { + documentCount: submission.documents.length, + reconciled: submission.reconciled, + uploadMs: this.clock() - startedAt, + } + ) + } catch (error) { + record("v3_trajectory_batch", false, { error: safeError(error) }) + } + + try { + await add( + "doc1", + `Preflight document one. Unique marker: ${marker}. ` + + "The admin settings page contains a Timezone selector.", + { causalKey: `pf:${session}:0:0`, stateIndex: 0 } + ) + const found = await this.client.listDocumentsByCustomIds( + [baselineCustomId], + identity.containerTag + ) + record("custom_id_idempotency", found.length === 1, { + documentsWithCustomId: found.length, + }) + } catch (error) { + record("custom_id_idempotency", false, { error: safeError(error) }) + } + + try { + const startedAt = this.clock() + const ready = await this.build.awaitReady({ + customIds: [baselineCustomId], + containerTag: identity.containerTag, + timeoutMs: this.options.readinessTimeoutMs, + initialPollMs: Math.min(1_000, this.options.searchPollMs), + maxPollMs: this.options.searchPollMs, + }) + record("document_readiness", ready.length === 1 && ready[0].status === "ready", { + terminalStatus: ready[0]?.remoteStatus, + millisecondsToReady: this.clock() - startedAt, + }) + record("memory_entries_visible", ready[0]?.memoryCount !== undefined, { + memoryEntryCount: ready[0]?.memoryCount, + }) + } catch (error) { + record("document_readiness", false, { error: safeError(error) }) + record("memory_entries_visible", false, { error: safeError(error) }) + } + + const filterProbes: Array<{ + suffix: string + filter: SupermemoryMetadata + check: string + }> = [ + { + suffix: "doc2", + filter: { causalKey: `pf:${session}:0:0` }, + check: "filter_by_metadata_single", + }, + { + suffix: "doc3", + filter: { + causalKey: [`pf:${session}:0:0`, `pf:${session}:1:0`], + }, + check: "filter_by_metadata_array_acceptance", + }, + ] + for (const [index, probe] of filterProbes.entries()) { + try { + await add( + probe.suffix, + `Preflight filtered document ${index + 2} referencing marker ${marker}.`, + { causalKey: `pf:${session}:${index + 1}:0`, stateIndex: index + 1 }, + probe.filter + ) + const ready = await this.build.awaitReady({ + customIds: [customId(probe.suffix)], + containerTag: identity.containerTag, + timeoutMs: this.options.readinessTimeoutMs, + initialPollMs: Math.min(1_000, this.options.searchPollMs), + maxPollMs: this.options.searchPollMs, + }) + record(probe.check, ready[0]?.status === "ready", { + terminalStatus: ready[0]?.remoteStatus, + ...(probe.check.includes("array") + ? { note: "acceptance only; array OR semantics require a separate semantic probe" } + : {}), + }) + } catch (error) { + record(probe.check, false, { error: safeError(error) }) + } + } + + try { + const searchConfig: AdvancedRetrievalConfig = { + topK: this.options.searchTopK, + threshold: 0, + searchMode: "hybrid", + rerank: false, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: false, + } + const searchDeadline = this.clock() + this.options.searchVisibilityTimeoutMs + let outcome + do { + outcome = await this.retrieval.search({ + identity, + query: `What is the unique marker ${marker}?`, + config: searchConfig, + }) + if (outcome.normalizedResults.length > 0) break + if (this.clock() >= searchDeadline) break + await this.sleep( + Math.min(this.options.searchPollMs, Math.max(0, searchDeadline - this.clock())) + ) + } while (this.clock() <= searchDeadline) + + record("search_visibility", outcome.normalizedResults.length > 0, { + resultCount: outcome.normalizedResults.length, + }) + record("search_limit_accepted", true, { + requestedTopK: this.options.searchTopK, + returned: outcome.diagnostics.resultCount, + }) + record( + "search_run_fingerprint_filter", + outcome.diagnostics.invalidProvenanceRanks.length === 0, + { invalidRanks: outcome.diagnostics.invalidProvenanceRanks } + ) + } catch (error) { + record("search_visibility", false, { error: safeError(error) }) + record("search_limit_accepted", false, { + requestedTopK: this.options.searchTopK, + error: safeError(error), + }) + record("search_run_fingerprint_filter", false, { error: safeError(error) }) + } + + try { + const states = await this.build.awaitReady({ + customIds: [zeroMemoryCustomId], + containerTag: identity.containerTag, + timeoutMs: this.options.readinessTimeoutMs, + initialPollMs: Math.min(1_000, this.options.searchPollMs), + maxPollMs: this.options.searchPollMs, + }) + record("zero_memory_document", states[0]?.status === "ready", { + memoryEntryCount: states[0]?.memoryCount, + terminalStatus: states[0]?.remoteStatus, + }) + } catch (error) { + record("zero_memory_document", false, { error: safeError(error) }) + } + } finally { + if (!this.options.keepDocuments && createdCustomIds.length > 0) { + try { + const cleanup = await this.build.cleanupExactBuild({ + identity, + customIds: [...new Set(createdCustomIds)], + }) + record("cleanup", true, { + deleted: cleanup.deleted.length, + alreadyAbsent: cleanup.absent.length, + }) + } catch (error) { + record("cleanup", false, { error: safeError(error) }) + } + } else if (this.options.keepDocuments) { + record("cleanup", true, { skipped: true }) + } + } + + const allPassed = checks.every((check) => check.ok) + return { + schemaVersion: SUPERMEMORY_PREFLIGHT_SCHEMA_VERSION, + generatedAt: new Date(this.clock()).toISOString(), + baseUrl: this.client.baseUrl, + identity, + searchContract: { + searchMode: "hybrid", + standaloneChunksExpected: true, + deprecatedIncludeChunks: false, + requestedTopK: this.options.searchTopK, + }, + checks, + allPassed, + blockers: checks.filter((check) => !check.ok).map((check) => check.check), + requestBudget: this.client.budgetSnapshot, + } + } +} + +export function validateSupermemoryPreflightReport( + report: SupermemoryPreflightReport, + expected: { + baseUrl: string + requiredTopK: number + maxAgeMs: number + now?: number + } +): void { + if (report.schemaVersion !== SUPERMEMORY_PREFLIGHT_SCHEMA_VERSION) { + throw new Error("Supermemory preflight report schema is obsolete") + } + if (!report.allPassed || report.blockers.length > 0) { + throw new Error(`Supermemory preflight is not passing: ${report.blockers.join(", ")}`) + } + if (normalizeBaseUrl(report.baseUrl) !== normalizeBaseUrl(expected.baseUrl)) { + throw new Error("Supermemory preflight base URL does not match the configured base URL") + } + if ( + report.searchContract.searchMode !== "hybrid" || + report.searchContract.deprecatedIncludeChunks !== false || + report.searchContract.standaloneChunksExpected !== true || + report.searchContract.requestedTopK < expected.requiredTopK + ) { + throw new Error("Supermemory preflight did not validate the required V4 search contract") + } + const generatedAt = Date.parse(report.generatedAt) + const age = (expected.now ?? Date.now()) - generatedAt + if (!Number.isFinite(generatedAt) || age < 0 || age > expected.maxAgeMs) { + throw new Error("Supermemory preflight report is missing, future-dated, or stale") + } +} + +function safeError(error: unknown): string { + return redact(error instanceof Error ? error.message : String(error)).slice(0, 500) +} diff --git a/src/providers/supermemory/advanced/provider.ts b/src/providers/supermemory/advanced/provider.ts new file mode 100644 index 0000000..5fec55b --- /dev/null +++ b/src/providers/supermemory/advanced/provider.ts @@ -0,0 +1,303 @@ +import type { + BuildBatchRequest, + BuildProvider, + BuildSearchRequest, + BuildSearchResponse, + RemoteDocumentState, +} from "../../../types/provider" +import type { + AssetRef, + MemoryBuildPlan, + NormalizedRetrievalResult, + PhysicalDocument, +} from "../../../types/migration" +import { + AdvancedSupermemoryClient, + type AdvancedSupermemoryApi, + type AdvancedSupermemoryClientOptions, + type SupermemoryMetadata, +} from "./client" +import { + AdvancedSupermemoryBuild, + type AdvancedSupermemoryBuildOptions, + type ReconciledDocument, + type SupermemoryBuildIdentity, +} from "./build" +import { AdvancedSupermemoryRetrieval } from "./retrieval" +import { + AdvancedSupermemoryPreflight, + type SupermemoryPreflightOptions, + type SupermemoryPreflightReport, +} from "./preflight" + +export type AdvancedSupermemoryProviderOptions = Pick< + AdvancedSupermemoryBuildOptions, + "cleanupTimeoutMs" | "signal" +> + +export class AdvancedSupermemoryProvider implements BuildProvider { + readonly name = "supermemory" + readonly capabilities = { + deterministicExternalIds: true, + batchUpload: true, + documentDependencies: false, + ingestionMetadataFilters: true, + searchMetadataFilters: true, + searchModes: ["hybrid", "memories"] as const, + reranking: true, + queryRewriting: true, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, + } + + readonly client: AdvancedSupermemoryApi + private readonly buildDriver: AdvancedSupermemoryBuild + private readonly retrievalDriver: AdvancedSupermemoryRetrieval + + constructor( + clientOrOptions: AdvancedSupermemoryApi | AdvancedSupermemoryClientOptions, + options: AdvancedSupermemoryProviderOptions = {} + ) { + this.client = isAdvancedApi(clientOrOptions) + ? clientOrOptions + : new AdvancedSupermemoryClient(clientOrOptions) + this.buildDriver = new AdvancedSupermemoryBuild(this.client, options) + this.retrievalDriver = new AdvancedSupermemoryRetrieval(this.client) + } + + async submitDocumentBatch(request: BuildBatchRequest): Promise { + const submission = await this.buildDriver.submitTrajectoryBatch({ + trajectoryId: request.trajectoryId, + identity: buildIdentity(request.build), + documents: request.documents.map((document) => { + if (document.trajectoryId !== request.trajectoryId) { + throw new Error( + `Document ${document.customId} belongs to ${document.trajectoryId}, not ${request.trajectoryId}` + ) + } + return { + customId: document.customId, + content: document.content, + metadata: documentMetadata(request.build, document), + filterByMetadata: rootSelfFilter(request.build, request.trajectoryId, document), + } + }), + }) + return submission.documents.map(toProviderState) + } + + async reconcileDocuments( + build: MemoryBuildPlan, + customIds: string[] + ): Promise { + const states = await this.buildDriver.reconcileByCustomId(customIds, build.containerTag) + for (const state of states) { + if (state.status === "absent") continue + if ( + state.metadata?.runFingerprint !== build.buildFingerprint || + state.metadata?.buildFingerprint !== build.buildFingerprint || + state.metadata?.buildId !== build.buildId + ) { + throw new Error( + `Remote document ${state.customId} does not belong to build ${build.buildId}` + ) + } + } + return states.map(toProviderState) + } + + async searchBuild(request: BuildSearchRequest): Promise { + const outcome = await this.retrievalDriver.search({ + identity: buildIdentity(request.build), + query: request.query, + config: { + topK: request.config.topK, + threshold: request.config.threshold, + searchMode: request.config.searchMode, + rerank: request.config.rerank, + rewriteQuery: request.config.rewriteQuery, + includeSummaries: request.config.includeSummaries, + includeChunks: request.config.includeChunks, + includeDocuments: request.config.includeDocuments, + includeRelatedMemories: request.config.includeRelatedMemories, + metadataFilter: request.config.metadataFilter, + strictProvenance: true, + }, + }) + const assets = screenshotAssets(request.build) + const normalizedResults: NormalizedRetrievalResult[] = outcome.normalizedResults.map( + (result) => { + const screenshotRefs: AssetRef[] = [] + let assetsValid = true + for (const remoteAsset of result.screenshotAssets) { + const asset = assets.get(remoteAsset.assetId) ?? assets.get(remoteAsset.path) + if ( + !asset || + asset.sha256 !== remoteAsset.sha256 || + asset.mimeType !== remoteAsset.mimeType || + asset.byteLength !== remoteAsset.byteLength + ) { + assetsValid = false + continue + } + if (!screenshotRefs.some((existing) => existing.assetId === asset.assetId)) { + screenshotRefs.push(asset) + } + } + return { + rank: result.rank, + score: result.score, + kind: result.kind, + text: result.text, + summary: result.summaries.length > 0 ? result.summaries.join("\n\n") : undefined, + chunks: result.chunks, + providerResultId: result.providerResultId, + documentIds: result.documentIds, + trajectoryId: result.trajectoryId, + stateIndex: result.stateIndex, + screenshotRefs, + provenanceValid: result.provenanceValid && assetsValid, + } + } + ) + return { + request: { ...outcome.request }, + rawResponse: outcome.rawResponse, + normalizedResults, + remoteDurationMs: outcome.remoteDurationMs, + } + } + + async verifyBuildHealth(build: MemoryBuildPlan): Promise { + return this.reconcileDocuments( + build, + build.documents.map((document) => document.customId) + ) + } + + async deleteDocuments(build: MemoryBuildPlan, customIds: string[]): Promise { + if (customIds.length === 0) return + await this.buildDriver.cleanupExactBuild({ + identity: buildIdentity(build), + customIds, + }) + } + + async clearBuild(build: MemoryBuildPlan): Promise { + await this.deleteDocuments( + build, + build.documents.map((document) => document.customId) + ) + } + + async preflight(options?: SupermemoryPreflightOptions): Promise { + return new AdvancedSupermemoryPreflight(this.client, options).run() + } +} + +function buildIdentity(build: MemoryBuildPlan): SupermemoryBuildIdentity { + return { + buildId: build.buildId, + containerTag: build.containerTag, + runFingerprint: build.buildFingerprint, + } +} + +function documentMetadata(build: MemoryBuildPlan, document: PhysicalDocument): SupermemoryMetadata { + if ( + document.metadata.causalKey !== undefined && + document.metadata.causalKey !== document.customId + ) { + throw new Error(`Document ${document.customId} has a conflicting causalKey`) + } + return { + ...document.metadata, + benchmark: build.benchmark, + adapterSchemaVersion: build.schemaVersion, + buildId: build.buildId, + runFingerprint: build.buildFingerprint, + buildFingerprint: build.buildFingerprint, + tier: build.tier, + domain: build.domain, + trajectoryId: document.trajectoryId, + documentType: document.documentType, + documentOrdinal: document.documentOrdinal, + partIndex: document.partIndex, + partCount: document.partCount, + contentHash: document.contentHash, + logicalDocumentId: document.logicalDocumentId, + causalKey: document.customId, + ...(document.stateIndex !== undefined ? { stateIndex: document.stateIndex } : {}), + ...(document.step !== undefined ? { step: document.step } : {}), + ...(document.screenshotRef + ? { + screenshotPath: document.screenshotRef.relativePath, + screenshotAssetId: document.screenshotRef.assetId, + screenshotSha256: document.screenshotRef.sha256, + screenshotMimeType: document.screenshotRef.mimeType, + screenshotByteLength: document.screenshotRef.byteLength, + } + : {}), + } +} + +function rootSelfFilter( + build: MemoryBuildPlan, + trajectoryId: string, + document: PhysicalDocument +): SupermemoryMetadata { + return { + runFingerprint: build.buildFingerprint, + buildFingerprint: build.buildFingerprint, + trajectoryId, + causalKey: document.customId, + } +} + +function toProviderState(document: ReconciledDocument): RemoteDocumentState { + const status: RemoteDocumentState["status"] = + document.status === "ready" + ? "ready" + : document.status === "failed" + ? "failed" + : document.status === "absent" + ? "absent" + : document.status === "accepted" || document.status === "indexing" + ? "pending" + : "unknown" + return { + customId: document.customId, + remoteId: document.remoteId, + status, + raw: document.raw, + ...(status === "failed" + ? { error: `Remote Supermemory status ${document.remoteStatus ?? "failed"}` } + : {}), + } +} + +function screenshotAssets(build: MemoryBuildPlan): Map { + const assets = new Map() + for (const document of build.documents) { + const asset = document.screenshotRef + if (!asset) continue + assets.set(asset.relativePath, asset) + if (asset.absolutePath) assets.set(asset.absolutePath, asset) + assets.set(asset.assetId, asset) + } + return assets +} + +function isAdvancedApi( + value: AdvancedSupermemoryApi | AdvancedSupermemoryClientOptions +): value is AdvancedSupermemoryApi { + return ( + "addDocumentsBatch" in value && + typeof value.addDocumentsBatch === "function" && + "searchV4" in value && + typeof value.searchV4 === "function" + ) +} diff --git a/src/providers/supermemory/advanced/retrieval.ts b/src/providers/supermemory/advanced/retrieval.ts new file mode 100644 index 0000000..3f72dd5 --- /dev/null +++ b/src/providers/supermemory/advanced/retrieval.ts @@ -0,0 +1,489 @@ +import { createHash } from "node:crypto" +import type { AdvancedSupermemoryApi, V4SearchRequest } from "./client" +import { SupermemoryContractError, isRecord, redact } from "./client" +import type { SupermemoryBuildIdentity } from "./build" + +export interface AdvancedRetrievalConfig { + topK: number + threshold?: number + searchMode?: "hybrid" | "memories" + rerank?: boolean + rewriteQuery?: boolean + includeSummaries?: boolean + includeChunks?: boolean + includeDocuments?: boolean + includeRelatedMemories?: boolean + metadataFilter?: Record + maxAttempts?: number + strictProvenance?: boolean +} + +export interface AdvancedNormalizedRetrievalResult { + rank: number + score?: number + kind: "memory" | "chunk" + text: string + memory?: string + summaries: string[] + chunks: string[] + providerResultId: string + documentIds: string[] + trajectoryId?: string + stateIndex?: number + screenshotPaths: string[] + screenshotAssets: Array<{ + path: string + assetId: string + sha256: string + mimeType: string + byteLength: number + }> + provenanceValid: boolean + provenanceErrors: string[] + rawResult: Record +} + +export interface AdvancedRetrievalOutcome { + request: V4SearchRequest + rawResponse: Record + normalizedResults: AdvancedNormalizedRetrievalResult[] + remoteDurationMs: number + diagnostics: { + resultCount: number + evidenceCount: number + chunksTotal: number + chunksDeduplicated: number + invalidProvenanceRanks: number[] + } +} + +export class SupermemoryProvenanceError extends Error { + readonly invalidResults: AdvancedNormalizedRetrievalResult[] + + constructor(invalidResults: AdvancedNormalizedRetrievalResult[]) { + super( + `Supermemory returned ${invalidResults.length} result${ + invalidResults.length === 1 ? "" : "s" + } outside the expected run fingerprint at ranks ${invalidResults + .map((result) => result.rank) + .join(", ")}` + ) + this.name = "SupermemoryProvenanceError" + this.invalidResults = invalidResults + } +} + +export interface AdvancedSupermemoryRetrievalOptions { + clock?: () => number +} + +export class AdvancedSupermemoryRetrieval { + private readonly clock: () => number + + constructor( + private readonly client: AdvancedSupermemoryApi, + options: AdvancedSupermemoryRetrievalOptions = {} + ) { + this.clock = options.clock ?? Date.now + } + + async search(input: { + identity: SupermemoryBuildIdentity + query: string + config: AdvancedRetrievalConfig + }): Promise { + validateSearchInput(input) + const metadataFilter = buildSupermemorySearchFilter( + input.config.metadataFilter ?? {}, + input.identity.runFingerprint + ) + + const request: V4SearchRequest = { + q: input.query, + containerTag: input.identity.containerTag, + limit: input.config.topK, + threshold: input.config.threshold ?? 0.3, + searchMode: input.config.searchMode ?? "hybrid", + rerank: input.config.rerank ?? true, + rewriteQuery: input.config.rewriteQuery ?? false, + include: { + summaries: input.config.includeSummaries ?? true, + documents: input.config.includeDocuments ?? true, + relatedMemories: input.config.includeRelatedMemories ?? true, + }, + filters: metadataFilter, + } + + const startedAt = this.clock() + const rawResponse = await this.client.searchV4(request, input.config.maxAttempts) + const remoteDurationMs = Math.max(0, this.clock() - startedAt) + const rawResults = rawResponse.results + if (!Array.isArray(rawResults)) { + throw new SupermemoryContractError("V4 search response has no results array") + } + if (rawResults.length > input.config.topK) { + throw new SupermemoryContractError( + `V4 search returned ${rawResults.length} results for topK=${input.config.topK}` + ) + } + + const normalized = normalizeV4Results(rawResponse, { + runFingerprint: input.identity.runFingerprint, + includeChunks: input.config.includeChunks ?? true, + includeSummaries: input.config.includeSummaries ?? true, + }) + const invalidResults = normalized.results.filter((result) => !result.provenanceValid) + if ((input.config.strictProvenance ?? true) && invalidResults.length > 0) { + throw new SupermemoryProvenanceError(invalidResults) + } + + return { + request: redact(request), + rawResponse: redact(rawResponse), + normalizedResults: normalized.results, + remoteDurationMs, + diagnostics: { + resultCount: rawResults.length, + evidenceCount: normalized.results.length, + chunksTotal: normalized.chunksTotal, + chunksDeduplicated: normalized.chunksDeduplicated, + invalidProvenanceRanks: invalidResults.map((result) => result.rank), + }, + } + } +} + +type SearchFilterCondition = { + key: string + value: string + filterType?: "metadata" | "numeric" | "array_contains" | "string_contains" + numericOperator?: ">" | "<" | ">=" | "<=" | "=" + negate?: boolean | "true" | "false" + ignoreCase?: boolean | "true" | "false" +} + +type SearchFilterExpression = + | SearchFilterCondition + | { AND: SearchFilterExpression[] } + | { OR: SearchFilterExpression[] } + +function validateLogicalExpression( + value: unknown, + expectedRunFingerprint: string, + depth = 0 +): SearchFilterExpression { + if (depth >= 5 || !isRecord(value)) { + throw new Error("Invalid or over-nested Supermemory search filter") + } + if ("key" in value || "value" in value) { + if (typeof value.key !== "string" || typeof value.value !== "string") { + throw new Error("Search filter conditions require string key and value") + } + if (value.key === "runFingerprint" && value.value !== expectedRunFingerprint) { + throw new Error("Retrieval metadata filter cannot override the build runFingerprint") + } + const filterType = value.filterType + if ( + filterType !== undefined && + !["metadata", "numeric", "array_contains", "string_contains"].includes(String(filterType)) + ) { + throw new Error(`Invalid search filter type: ${String(filterType)}`) + } + const numericOperator = value.numericOperator + if ( + numericOperator !== undefined && + ![">", "<", ">=", "<=", "="].includes(String(numericOperator)) + ) { + throw new Error(`Invalid numeric search operator: ${String(numericOperator)}`) + } + if ( + filterType === "numeric" && + (value.value.trim() === "" || Number.isNaN(Number(value.value))) + ) { + throw new Error(`Numeric search filter ${value.key} requires a numeric value`) + } + for (const field of ["negate", "ignoreCase"] as const) { + const candidate = value[field] + if ( + candidate !== undefined && + typeof candidate !== "boolean" && + candidate !== "true" && + candidate !== "false" + ) { + throw new Error(`Search filter ${field} must be a boolean`) + } + } + return { + key: value.key, + value: value.value, + ...(filterType === undefined + ? {} + : { filterType: filterType as SearchFilterCondition["filterType"] }), + ...(numericOperator === undefined + ? {} + : { + numericOperator: numericOperator as SearchFilterCondition["numericOperator"], + }), + ...(value.negate === undefined + ? {} + : { negate: value.negate as SearchFilterCondition["negate"] }), + ...(value.ignoreCase === undefined + ? {} + : { ignoreCase: value.ignoreCase as SearchFilterCondition["ignoreCase"] }), + } + } + const operator = "AND" in value ? "AND" : "OR" in value ? "OR" : undefined + if (!operator || !Array.isArray(value[operator]) || value[operator].length === 0) { + throw new Error("Search filters must contain a non-empty AND or OR array") + } + return { + [operator]: value[operator].map((child) => + validateLogicalExpression(child, expectedRunFingerprint, depth + 1) + ), + } as { AND: SearchFilterExpression[] } | { OR: SearchFilterExpression[] } +} + +function flatCondition(key: string, value: unknown): SearchFilterExpression { + if (!/^[A-Za-z0-9_.-]+$/.test(key)) { + throw new Error(`Invalid search metadata key: ${key}`) + } + if (Array.isArray(value)) { + if (value.length === 0 || !value.every((item) => typeof item === "string")) { + throw new Error(`Search metadata array ${key} must contain strings`) + } + return { + OR: value.map((item) => ({ + key, + value: item, + filterType: "array_contains", + })), + } + } + if (!["string", "number", "boolean"].includes(typeof value)) { + throw new Error(`Unsupported search metadata value for ${key}`) + } + return typeof value === "number" + ? { + key, + value: String(value), + filterType: "numeric", + numericOperator: "=", + } + : { key, value: String(value), filterType: "metadata" } +} + +export function buildSupermemorySearchFilter( + configured: Record, + runFingerprint: string +): { AND: SearchFilterExpression[] } { + if (!runFingerprint.trim()) throw new Error("runFingerprint must not be empty") + const runCondition: SearchFilterCondition = { + key: "runFingerprint", + value: runFingerprint, + filterType: "metadata", + } + if ("AND" in configured || "OR" in configured) { + return { + AND: [runCondition, validateLogicalExpression(configured, runFingerprint)], + } + } + if (configured.runFingerprint !== undefined && configured.runFingerprint !== runFingerprint) { + throw new Error("Retrieval metadata filter cannot override the build runFingerprint") + } + return { + AND: [ + runCondition, + ...Object.entries(configured) + .filter(([key]) => key !== "runFingerprint") + .map(([key, value]) => flatCondition(key, value)), + ], + } +} + +export function normalizeV4Results( + rawResponse: Record, + options: { + runFingerprint: string + includeChunks: boolean + includeSummaries: boolean + } +): { + results: AdvancedNormalizedRetrievalResult[] + chunksTotal: number + chunksDeduplicated: number +} { + const rawResults = Array.isArray(rawResponse.results) ? rawResponse.results : [] + const seenChunkHashes = new Set() + const results: AdvancedNormalizedRetrievalResult[] = [] + let chunksTotal = 0 + let chunksDeduplicated = 0 + + for (const [rank, value] of rawResults.entries()) { + if (!isRecord(value)) continue + const resultMetadata = isRecord(value.metadata) ? value.metadata : {} + const documents = Array.isArray(value.documents) ? value.documents.filter(isRecord) : [] + const documentIds: string[] = [] + const summaries: string[] = [] + const screenshotPaths: string[] = [] + const screenshotAssets: AdvancedNormalizedRetrievalResult["screenshotAssets"] = [] + + for (const document of documents) { + const documentId = stringValue(document.id) + if (documentId) documentIds.push(documentId) + const summary = stringValue(document.summary) + if (options.includeSummaries && summary) summaries.push(summary) + const metadata = isRecord(document.metadata) ? document.metadata : {} + const screenshotPath = stringValue(metadata.screenshotPath) + if (screenshotPath && !screenshotPaths.includes(screenshotPath)) { + screenshotPaths.push(screenshotPath) + } + appendScreenshotAsset(screenshotAssets, metadata) + } + + const chunks: string[] = [] + if (options.includeChunks) { + const candidates: string[] = [] + const singular = stringValue(value.chunk) + if (singular) candidates.push(singular) + if (Array.isArray(value.chunks)) { + for (const chunk of value.chunks) { + if (typeof chunk === "string" && chunk.trim()) candidates.push(chunk) + if (isRecord(chunk)) { + const content = stringValue(chunk.content) + if (content) candidates.push(content) + } + } + } + chunksTotal += candidates.length + for (const candidate of candidates) { + const hash = createHash("sha256").update(candidate).digest("hex") + if (seenChunkHashes.has(hash)) { + chunksDeduplicated += 1 + } else { + seenChunkHashes.add(hash) + chunks.push(candidate) + } + } + } + + const memory = stringValue(value.memory) + const textParts = [...(memory ? [memory] : []), ...summaries, ...chunks] + if (textParts.length === 0) continue + + const metadataCandidates = [ + resultMetadata, + ...documents.map((document) => (isRecord(document.metadata) ? document.metadata : {})), + ].filter((metadata) => Object.keys(metadata).length > 0) + const provenanceErrors: string[] = [] + if (metadataCandidates.length === 0) { + provenanceErrors.push("result has no provenance metadata") + } else { + for (const [index, metadata] of metadataCandidates.entries()) { + if (metadata.runFingerprint !== options.runFingerprint) { + provenanceErrors.push( + `${index === 0 ? "result" : `document ${index}`} runFingerprint is missing or mismatched` + ) + } + } + } + + const primaryMetadata = + documents.length > 0 && isRecord(documents[0].metadata) + ? documents[0].metadata + : resultMetadata + const resultScreenshot = stringValue(resultMetadata.screenshotPath) + if (resultScreenshot && !screenshotPaths.includes(resultScreenshot)) { + screenshotPaths.unshift(resultScreenshot) + } + appendScreenshotAsset(screenshotAssets, resultMetadata) + for (const screenshotPath of screenshotPaths) { + if (!screenshotAssets.some((asset) => asset.path === screenshotPath)) { + provenanceErrors.push( + `screenshot ${screenshotPath} is missing hash, MIME type, byte length, or asset ID` + ) + } + } + + results.push({ + rank, + score: numberValue(value.similarity) ?? numberValue(value.score), + kind: memory ? "memory" : "chunk", + text: textParts.join("\n\n"), + memory, + summaries, + chunks, + providerResultId: stringValue(value.id) ?? `result-${rank}`, + documentIds, + trajectoryId: + stringValue(primaryMetadata.trajectoryId) ?? stringValue(resultMetadata.trajectoryId), + stateIndex: + integerValue(primaryMetadata.stateIndex) ?? integerValue(resultMetadata.stateIndex), + screenshotPaths, + screenshotAssets, + provenanceValid: provenanceErrors.length === 0, + provenanceErrors, + rawResult: redact(value), + }) + } + + return { results, chunksTotal, chunksDeduplicated } +} + +function appendScreenshotAsset( + output: AdvancedNormalizedRetrievalResult["screenshotAssets"], + metadata: Record +): void { + const path = stringValue(metadata.screenshotPath) + if (!path) return + const assetId = stringValue(metadata.screenshotAssetId) + const sha256 = stringValue(metadata.screenshotSha256) + const mimeType = stringValue(metadata.screenshotMimeType) + const byteLength = integerValue(metadata.screenshotByteLength) + if ( + !assetId || + !sha256 || + !/^[a-f0-9]{64}$/i.test(sha256) || + !mimeType?.startsWith("image/") || + byteLength === undefined || + byteLength < 0 + ) { + return + } + if (!output.some((asset) => asset.path === path && asset.assetId === assetId)) { + output.push({ path, assetId, sha256, mimeType, byteLength }) + } +} + +function validateSearchInput(input: { + identity: SupermemoryBuildIdentity + query: string + config: AdvancedRetrievalConfig +}): void { + if (!input.query.trim()) throw new Error("Retrieval query must not be empty") + if (!input.identity.containerTag.trim() || !input.identity.runFingerprint.trim()) { + throw new Error("Retrieval requires containerTag and runFingerprint") + } + if (!Number.isInteger(input.config.topK) || input.config.topK < 1) { + throw new Error("Retrieval topK must be an integer >= 1") + } + if ( + input.config.threshold !== undefined && + (!Number.isFinite(input.config.threshold) || input.config.threshold < 0) + ) { + throw new Error("Retrieval threshold must be a finite number >= 0") + } +} + +function stringValue(value: unknown): string | undefined { + return typeof value === "string" && value.trim() ? value : undefined +} + +function numberValue(value: unknown): number | undefined { + return typeof value === "number" && Number.isFinite(value) ? value : undefined +} + +function integerValue(value: unknown): number | undefined { + if (typeof value === "number" && Number.isInteger(value)) return value + if (typeof value === "string" && /^-?\d+$/.test(value)) return Number(value) + return undefined +} diff --git a/src/providers/supermemory/index.ts b/src/providers/supermemory/index.ts index 027bc32..cc7ae3c 100644 --- a/src/providers/supermemory/index.ts +++ b/src/providers/supermemory/index.ts @@ -13,6 +13,21 @@ import { SUPERMEMORY_PROMPTS } from "./prompts" export class SupermemoryProvider implements Provider { name = "supermemory" + capabilities = { + deterministicExternalIds: false, + batchUpload: false, + documentDependencies: false, + ingestionMetadataFilters: false, + searchMetadataFilters: false, + searchModes: ["hybrid"] as const, + reranking: false, + queryRewriting: false, + remoteClear: false, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, + } prompts = SUPERMEMORY_PROMPTS concurrency = { default: 50, @@ -125,11 +140,11 @@ export class SupermemoryProvider implements Provider { containerTag: options.containerTag, limit: 30, threshold: options.threshold || 0.3, - searchMode: "hybrid", - include: { - summaries: true, - chunks: true - } + searchMode: "hybrid", + include: { + summaries: true, + chunks: true, + }, }) return response.results || [] diff --git a/src/providers/zep/index.ts b/src/providers/zep/index.ts index 083b3db..dd45f6e 100644 --- a/src/providers/zep/index.ts +++ b/src/providers/zep/index.ts @@ -81,6 +81,21 @@ const ZEP_ENTITY_TYPES = { export class ZepProvider implements Provider { name = "zep" + capabilities = { + deterministicExternalIds: false, + batchUpload: true, + documentDependencies: false, + ingestionMetadataFilters: false, + searchMetadataFilters: false, + searchModes: ["memories"] as const, + reranking: true, + queryRewriting: false, + remoteClear: true, + readinessStates: true, + mediaIngestion: false, + durableLocalPersistence: true, + splitPhaseSafe: true, + } prompts = ZEP_PROMPTS concurrency = { default: 10, diff --git a/src/server/index.ts b/src/server/index.ts index 3213783..938226b 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -1,4 +1,5 @@ import { handleRunsRoutes } from "./routes/runs" +import { createLongMemEvalV2ControlHandler } from "./routes/longmemeval-v2-control" import { handleBenchmarksRoutes } from "./routes/benchmarks" import { handleLeaderboardRoutes } from "./routes/leaderboard" import { handleCompareRoutes } from "./routes/compare" @@ -21,6 +22,9 @@ const CORS_HEADERS = { } export const wsManager = new WebSocketManager() +const handleLongMemEvalV2ControlRoutes = createLongMemEvalV2ControlHandler({ + broadcast: (message) => wsManager.broadcast(message), +}) export async function startServer(options: ServerOptions): Promise { const { port, open = true } = options @@ -47,7 +51,9 @@ export async function startServer(options: ServerOptions): Promise { try { let response: Response | null = null - if (url.pathname.startsWith("/api/runs")) { + if (url.pathname.startsWith("/api/runs-v2")) { + response = await handleLongMemEvalV2ControlRoutes(req, url) + } else if (url.pathname.startsWith("/api/runs")) { response = await handleRunsRoutes(req, url) } else if (url.pathname.startsWith("/api/compare")) { response = await handleCompareRoutes(req, url) diff --git a/src/server/routes/build-aware-inspection.test.ts b/src/server/routes/build-aware-inspection.test.ts new file mode 100644 index 0000000..c5fa0b2 --- /dev/null +++ b/src/server/routes/build-aware-inspection.test.ts @@ -0,0 +1,591 @@ +import { afterEach, describe, expect, test } from "bun:test" +import { Database } from "bun:sqlite" +import { createHash } from "node:crypto" +import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises" +import { join } from "node:path" +import { tmpdir } from "node:os" +import type { BuildAwareRunCheckpoint } from "../../types/build-aware" +import { + createBuildAwareInspectionHandler, + listBuildAwareRunSummaries, +} from "./build-aware-inspection" + +const temporaryDirectories: string[] = [] + +afterEach(async () => { + await Promise.all( + temporaryDirectories + .splice(0) + .map((directory) => rm(directory, { recursive: true, force: true })) + ) +}) + +async function fixture(options?: { + unsafeArtifact?: boolean + unsafeArtifactRoot?: boolean + symlinkArtifact?: boolean + unsafeBuildRoot?: boolean + buildState?: boolean + imageAssetMode?: "valid" | "tampered" | "symlink" +}) { + const root = await mkdtemp(join(tmpdir(), "memorybench-inspection-")) + temporaryDirectories.push(root) + const runsRoot = join(root, "runs-v2") + const buildsRoot = join(root, "builds") + const artifactsRoot = join(root, "artifacts") + const datasetRoot = join(root, "dataset") + const runId = options?.unsafeArtifact ? "unsafe-run" : "inspection-run" + const runRoot = join(runsRoot, runId) + await mkdir(join(runRoot, "builds"), { recursive: true }) + await mkdir(join(artifactsRoot, "queries"), { recursive: true }) + await mkdir(join(artifactsRoot, "assets"), { recursive: true }) + await mkdir(join(datasetRoot, "screenshots/trajectory-1"), { recursive: true }) + await mkdir(buildsRoot, { recursive: true }) + + const imageBytes = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 1, 2, 3, 4]) + const imageHash = createHash("sha256").update(imageBytes).digest("hex") + const assetId = `trajectory-screenshot:${imageHash}` + const datasetImageRelativePath = "screenshots/trajectory-1/0.png" + const materializedImageRelativePath = `assets/${imageHash}.png` + if (options?.imageAssetMode === "symlink") { + const outsideImage = join(root, "outside-image.png") + await writeFile(outsideImage, imageBytes) + await symlink(outsideImage, join(datasetRoot, datasetImageRelativePath)) + } else { + const bytes = + options?.imageAssetMode === "tampered" + ? Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 4, 3, 2, 1]) + : imageBytes + await writeFile(join(datasetRoot, datasetImageRelativePath), bytes) + await writeFile(join(artifactsRoot, materializedImageRelativePath), bytes) + } + const datasetAsset = { + assetId, + kind: "trajectory-screenshot" as const, + relativePath: datasetImageRelativePath, + mimeType: "image/png", + sha256: imageHash, + byteLength: imageBytes.byteLength, + } + const materializedAsset = { + ...datasetAsset, + relativePath: materializedImageRelativePath, + } + + if (options?.buildState) { + const checkpointDirectory = join(buildsRoot, "supermemory", "build-fingerprint") + await mkdir(checkpointDirectory, { recursive: true }) + const database = new Database(join(checkpointDirectory, "checkpoint.sqlite")) + database.exec(` + CREATE TABLE builds ( + build_id TEXT PRIMARY KEY, + build_fingerprint TEXT NOT NULL, + container_tag TEXT NOT NULL, + provider TEXT NOT NULL, + status TEXT NOT NULL, + error TEXT + ); + CREATE TABLE trajectories (build_id TEXT NOT NULL, status TEXT NOT NULL); + CREATE TABLE documents (build_id TEXT NOT NULL, status TEXT NOT NULL); + INSERT INTO builds VALUES ( + 'build-1', 'sqlite-build-fingerprint', 'container-1', 'supermemory', 'ready', NULL + ); + INSERT INTO trajectories VALUES ('build-1', 'ready'); + INSERT INTO documents VALUES ('build-1', 'ready'); + INSERT INTO documents VALUES ('build-1', 'ready'); + `) + database.close() + } + + const rawBytes = Buffer.from( + JSON.stringify({ + results: [{ id: "result-1", text: "evidence" }], + authorization: "Bearer private", + nested: { + absolutePath: "/private/evidence.png", + tokenValue: "sm_12345678901234567890", + }, + }) + ) + const artifactRelativePath = options?.symlinkArtifact + ? "queries/raw-link.json" + : "queries/raw.json" + if (options?.symlinkArtifact) { + const outsidePath = join(root, "outside.json") + await writeFile(outsidePath, rawBytes) + await symlink(outsidePath, join(artifactsRoot, artifactRelativePath)) + } else { + await writeFile(join(artifactsRoot, artifactRelativePath), rawBytes) + } + const rawDescriptor = { + relativePath: options?.unsafeArtifact ? "../../outside.json" : artifactRelativePath, + sha256: createHash("sha256").update(rawBytes).digest("hex"), + byteLength: rawBytes.byteLength, + } + + const checkpoint = { + schemaVersion: 1, + executionModel: "shared-memory-build-v1", + runId, + configFingerprint: "config-fingerprint", + status: "completed", + currentStage: "report", + config: { + provider: "supermemory", + benchmark: "longmemeval-v2", + datasetPath: datasetRoot, + datasetRevision: "revision-1", + tier: "small", + domain: "all", + seed: "seed", + retrieval: { + topK: 10, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeRelatedMemories: true, + metadataFilter: {}, + }, + reader: { + model: "reader-model", + reasoningEffort: "high", + maxCompletionTokens: 1000, + maxContextTokens: 2000, + evidenceTopK: 10, + maxImages: 4, + maxImageBytes: 1000000, + malformedResponseAttempts: 2, + }, + evaluator: { + model: "evaluator-model", + reasoningEffort: "high", + maxCompletionTokens: 1000, + }, + build: { + dreaming: "instant", + rootFilterMode: "self", + maxDocumentChars: 1000, + trajectoryConcurrency: 2, + maxInFlightRequests: 4, + maxTrajectoryAttempts: 3, + indexingTimeoutMs: 1000, + pollIntervalMs: 100, + preflightMaxAgeMs: 24 * 60 * 60_000, + }, + }, + targetQuestionIds: ["q1", "q2"], + buildIds: ["build-1"], + artifactRoot: options?.unsafeArtifactRoot ? "/private/outside-artifact-root" : artifactsRoot, + buildRoot: options?.unsafeBuildRoot ? "/private/outside-build-root" : buildsRoot, + preflightGate: { + schemaVersion: 1, + reportFingerprint: "f".repeat(64), + generatedAt: "2026-01-01T00:00:00.000Z", + baseUrl: "https://api.supermemory.ai", + testedTopK: 10, + }, + buildLinks: { + q1: "build-1", + q2: "build-1", + }, + questions: { + q1: { + questionId: "q1", + questionType: "static-environment", + question: "What happened?", + groundTruth: "The event", + evalFunction: "qa", + buildId: "build-1", + stages: { + query: { status: "completed", cacheHit: true }, + read: { status: "completed", cacheHit: false }, + evaluate: { status: "completed" }, + }, + queryArtifact: { + schemaVersion: 1, + questionId: "q1", + buildId: "build-1", + buildFingerprint: "build-fingerprint", + queryFingerprint: "query-fingerprint", + query: "What happened?", + config: { + topK: 10, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeRelatedMemories: true, + metadataFilter: {}, + }, + request: {}, + rawArtifact: rawDescriptor, + normalizedArtifact: rawDescriptor, + normalizedResults: [ + { + rank: 1, + kind: "memory", + text: "evidence", + chunks: [], + documentIds: ["document-1"], + screenshotRefs: [datasetAsset], + provenanceValid: true, + }, + ], + remoteDurationMs: 10, + wallDurationMs: 12, + cacheHit: true, + createdAt: "2026-01-01T00:00:00.000Z", + }, + readerArtifact: { + schemaVersion: 1, + questionId: "q1", + readerFingerprint: "reader-fingerprint", + model: "reader-model", + systemPrompt: "prompt", + parts: + options?.imageAssetMode === "symlink" + ? [] + : [{ type: "image", asset: materializedAsset }], + sentAssetIds: options?.imageAssetMode === "symlink" ? [] : [assetId], + omittedItems: 0, + responseText: "The event", + parsedAnswer: "The event", + durationMs: 20, + createdAt: "2026-01-01T00:00:00.000Z", + }, + evaluationArtifact: { + schemaVersion: 1, + questionId: "q1", + evaluatorFingerprint: "evaluator-fingerprint", + evalFunction: "qa", + answer: "The event", + groundTruth: "The event", + score: 1, + label: "correct", + promptVersion: "v1", + implementationVersion: "v1", + durationMs: 30, + createdAt: "2026-01-01T00:00:00.000Z", + }, + }, + q2: { + questionId: "q2", + questionType: "static-environment", + question: "When?", + groundTruth: "Then", + evalFunction: "qa", + buildId: "build-1", + stages: { + query: { status: "pending" }, + read: { status: "pending" }, + evaluate: { status: "pending" }, + }, + }, + }, + createdAt: "2026-01-01T00:00:00.000Z", + updatedAt: "2026-01-01T00:01:00.000Z", + } as unknown as BuildAwareRunCheckpoint + await writeFile(join(runRoot, "checkpoint.json"), JSON.stringify(checkpoint)) + await writeFile( + join(runRoot, "control.json"), + JSON.stringify({ + schemaVersion: 1, + runId, + events: [ + { action: "start", at: "2026-01-01T00:00:00.000Z", through: "report" }, + { action: "completed", at: "2026-01-01T00:01:00.000Z", through: "report" }, + ], + }) + ) + await writeFile( + join(runRoot, "builds", "build-1.plan.json"), + JSON.stringify({ + buildId: "build-1", + buildFingerprint: "build-fingerprint", + containerTag: "container-1", + provider: "supermemory", + domain: "web", + orderedSourceIds: ["trajectory-1"], + documents: [{ customId: "document-1" }, { customId: "document-2" }], + }) + ) + + const report = { + schemaVersion: 1, + protocol: "longmemeval-v2-official", + runId, + assetId, + imageBytes, + benchmark: "longmemeval-v2", + provider: "supermemory", + converter: "Structured Accessibility Converter", + targetQuestionCount: 2, + completedQuestionCount: 1, + failedQuestionCount: 0, + buildIds: ["build-1"], + builds: [ + { + buildId: "build-1", + buildFingerprint: "build-fingerprint", + containerTag: "container-1", + domain: "web", + trajectoryCount: 1, + documentCount: 2, + linkedQuestionIds: ["q1", "q2"], + reused: false, + }, + ], + official: { + overall: { + overall_full_set: 0.5, + overall_non_abstention_only: 0.5, + overall_abstention_only: null, + count_all_questions: 2, + count_non_abstention: 2, + count_abstention: 0, + }, + non_abstention_by_category: {}, + abstention_by_category: {}, + combined_abstention_by_category: {}, + abstention_overall: {}, + execution: { completed: 1, failed: 0, pending: 1, blocked: 0 }, + }, + diagnostics: { + queryCacheHits: 1, + readerCacheHits: 0, + remoteSearchLatencyMs: [10], + queryWallLatencyMs: [12], + contextImagesSent: 0, + failedQuestions: [], + }, + createdAt: "2026-01-01T00:01:00.000Z", + } + await writeFile(join(runRoot, "report.json"), JSON.stringify(report)) + + return { + runId, + assetId, + imageBytes, + runsRoot, + handler: createBuildAwareInspectionHandler({ + runsRoot, + buildsRoot, + artifactsRoot, + }), + } +} + +async function request( + handler: ReturnType, + path: string +) { + return handler(new Request(`http://localhost${path}`), new URL(`http://localhost${path}`)) +} + +describe("build-aware inspection routes", () => { + test("lists build-aware runs as read-only summaries for the shared runs page", async () => { + const { runsRoot, runId } = await fixture() + const summaries = await listBuildAwareRunSummaries(runsRoot) + expect(summaries).toHaveLength(1) + expect(summaries[0]).toMatchObject({ + runId, + provider: "supermemory", + benchmark: "longmemeval-v2", + judge: "evaluator-model", + answeringModel: "reader-model", + status: "completed", + readOnlyInspection: true, + accuracy: 0.5, + summary: { + total: 2, + ingested: 2, + indexed: 2, + searched: 1, + answered: 1, + evaluated: 1, + }, + }) + }) + + test("surfaces an orphaned running checkpoint as failed and resumable", async () => { + const { handler, runsRoot, runId } = await fixture() + const checkpointPath = join(runsRoot, runId, "checkpoint.json") + const checkpoint = JSON.parse(await readFile(checkpointPath, "utf8")) + checkpoint.status = "running" + checkpoint.currentStage = "query" + delete checkpoint.error + await writeFile(checkpointPath, JSON.stringify(checkpoint)) + + const summaries = await listBuildAwareRunSummaries(runsRoot) + expect(summaries[0].status).toBe("failed") + + const response = await request(handler, `/api/runs/${runId}`) + expect(await response!.json()).toMatchObject({ + status: "failed", + currentStage: "query", + error: "Run process is no longer active; resume from the durable checkpoint", + }) + }) + + test("does not misclassify an independently managed CLI checkpoint as stale", async () => { + const { handler, runsRoot, runId } = await fixture() + const checkpointPath = join(runsRoot, runId, "checkpoint.json") + const checkpoint = JSON.parse(await readFile(checkpointPath, "utf8")) + checkpoint.status = "running" + checkpoint.currentStage = "query" + delete checkpoint.error + await writeFile(checkpointPath, JSON.stringify(checkpoint)) + await rm(join(runsRoot, runId, "control.json")) + + const summaries = await listBuildAwareRunSummaries(runsRoot) + expect(summaries[0].status).toBe("running") + + const response = await request(handler, `/api/runs/${runId}`) + expect(await response!.json()).toMatchObject({ status: "running", currentStage: "query" }) + }) + + test("separates official metrics, diagnostics, and build reuse", async () => { + const { handler, runId } = await fixture({ unsafeBuildRoot: true }) + const response = await request(handler, `/api/runs/${runId}`) + expect(response?.status).toBe(200) + const body = await response!.json() + expect(body.executionModel).toBe("shared-memory-build-v1") + expect(body.inspection.metricNamespaces.official.overall.overall_full_set).toBe(0.5) + expect(body.inspection.metricNamespaces.diagnostics.queryCacheHits).toBe(1) + expect(body.inspection.builds[0].reuseCount).toBe(2) + expect(body.inspection.builds[0].reused).toBe(true) + expect(body.inspection.builds[0].priorBuildReuse).toBe(false) + expect(body.inspection.builds[0].checkpointLink.status).toBe("rejected") + expect(body.inspection.control.events.map((event: { action: string }) => event.action)).toEqual( + ["start", "completed"] + ) + expect(body.questionBuildLinks).toEqual({ q1: "build-1", q2: "build-1" }) + expect(body.preflightGate).toEqual({ + schemaVersion: 1, + reportFingerprint: "f".repeat(64), + generatedAt: "2026-01-01T00:00:00.000Z", + baseUrl: "https://api.supermemory.ai", + testedTopK: 10, + }) + expect(JSON.stringify(body)).not.toContain("/private/outside-build-root") + + const compactResponse = await request(handler, `/api/runs/${runId}?compact=true`) + const compactBody = await compactResponse!.json() + expect(compactBody.questions).toBeUndefined() + expect(compactBody.summary.total).toBe(2) + + const questionsResponse = await request(handler, `/api/runs/${runId}/questions?limit=1`) + const questionsBody = await questionsResponse!.json() + expect(questionsBody.pagination).toMatchObject({ page: 1, limit: 1, total: 2, totalPages: 2 }) + expect(questionsBody.questions).toHaveLength(1) + expect(questionsBody.questions[0]).toMatchObject({ + questionId: "q1", + evaluationArtifact: { score: 1, label: "correct" }, + }) + expect(questionsBody.questions[0].queryArtifact).toBeUndefined() + expect(questionsBody.questions[0].readerArtifact).toBeUndefined() + }) + + test("serves named artifacts with integrity checks and response redaction", async () => { + const { handler, runId } = await fixture() + const detail = await request(handler, `/api/runs/${runId}/questions/q1`) + const detailBody = await detail!.json() + expect(detailBody.artifactLinks["query-raw"].available).toBe(true) + expect(detailBody.buildReuseCount).toBe(2) + + const response = await request(handler, `/api/runs/${runId}/questions/q1/artifacts/query-raw`) + expect(response?.status).toBe(200) + const body = await response!.json() + expect(body.provenance.integrity).toBe("verified") + expect(body.data.authorization).toBe("[REDACTED]") + expect(body.data.nested.absolutePath).toBeUndefined() + expect(body.data.nested.tokenValue).toBe("[REDACTED]") + }) + + test("serves only referenced image assets after root, hash, size, and MIME verification", async () => { + const { handler, runId, assetId, imageBytes } = await fixture({ imageAssetMode: "valid" }) + const response = await request( + handler, + `/api/runs/${runId}/questions/q1/assets/${encodeURIComponent(assetId)}` + ) + expect(response?.status).toBe(200) + expect(response?.headers.get("content-type")).toBe("image/png") + expect(response?.headers.get("x-content-type-options")).toBe("nosniff") + expect(Buffer.from(await response!.arrayBuffer())).toEqual(imageBytes) + + const unreferenced = await request( + handler, + `/api/runs/${runId}/questions/q1/assets/${encodeURIComponent("unreferenced-asset")}` + ) + expect(unreferenced?.status).toBe(404) + }) + + test("rejects tampered and symlink-escaped referenced images", async () => { + const tampered = await fixture({ imageAssetMode: "tampered" }) + const tamperedResponse = await request( + tampered.handler, + `/api/runs/${tampered.runId}/questions/q1/assets/${encodeURIComponent(tampered.assetId)}` + ) + expect(tamperedResponse?.status).toBe(422) + expect((await tamperedResponse!.json()).error).toContain("hash does not match") + + const escaped = await fixture({ imageAssetMode: "symlink" }) + const escapedResponse = await request( + escaped.handler, + `/api/runs/${escaped.runId}/questions/q1/assets/${encodeURIComponent(escaped.assetId)}` + ) + expect(escapedResponse?.status).toBe(422) + expect((await escapedResponse!.json()).error).toContain("outside its recorded root") + }) + + test("summarizes an allowlisted build checkpoint through a read-only database", async () => { + const { handler, runId } = await fixture({ buildState: true }) + const response = await request(handler, `/api/runs/${runId}/builds/build-1`) + expect(response?.status).toBe(200) + const body = await response!.json() + expect(body.checkpointLink).toEqual({ + status: "available", + scope: "builds", + relativePath: "supermemory/build-fingerprint/checkpoint.sqlite", + }) + expect(body.stateStore.status).toBe("ready") + expect(body.stateStore.documents).toEqual({ ready: 2 }) + expect(body.stateStore.trajectories).toEqual({ ready: 1 }) + }) + + test("rejects checkpoint artifact traversal instead of reading it", async () => { + const { handler, runId } = await fixture({ unsafeArtifact: true }) + const response = await request(handler, `/api/runs/${runId}/questions/q1/artifacts/query-raw`) + expect(response?.status).toBe(422) + expect(await response!.json()).toEqual({ + error: "Artifact path is outside the artifact root", + }) + }) + + test("rejects a recorded artifact root outside the server allowlist", async () => { + const { handler, runId } = await fixture({ unsafeArtifactRoot: true }) + const detail = await request(handler, `/api/runs/${runId}/questions/q1`) + const detailBody = await detail!.json() + expect(detailBody.artifactLinks["query-raw"].available).toBe(false) + + const response = await request(handler, `/api/runs/${runId}/questions/q1/artifacts/query-raw`) + expect(response?.status).toBe(422) + expect(await response!.json()).toEqual({ + error: "Recorded artifact root is outside the allowed artifact root", + }) + }) + + test("rejects an artifact symlink that escapes the allowlisted root", async () => { + const { handler, runId } = await fixture({ symlinkArtifact: true }) + const response = await request(handler, `/api/runs/${runId}/questions/q1/artifacts/query-raw`) + expect(response?.status).toBe(422) + expect(await response!.json()).toEqual({ + error: "Artifact is missing or outside the artifact root", + }) + }) + + test("returns null for legacy or missing runs", async () => { + const { handler } = await fixture() + const response = await request(handler, "/api/runs/legacy-run") + expect(response).toBeNull() + }) +}) diff --git a/src/server/routes/build-aware-inspection.ts b/src/server/routes/build-aware-inspection.ts new file mode 100644 index 0000000..8aa0fa5 --- /dev/null +++ b/src/server/routes/build-aware-inspection.ts @@ -0,0 +1,999 @@ +import { Database } from "bun:sqlite" +import { createHash } from "node:crypto" +import { readFile, readdir, realpath, stat } from "node:fs/promises" +import { basename, isAbsolute, parse, relative, resolve, sep } from "node:path" +import type { + BuildAwareQuestionCheckpoint, + BuildAwareReport, + BuildAwareRunCheckpoint, +} from "../../types/build-aware" +import type { AssetRef } from "../../types/migration" +import { isRunActive } from "../runState" + +const MAX_JSON_ARTIFACT_BYTES = 8 * 1024 * 1024 +const MAX_IMAGE_ASSET_BYTES = 25 * 1024 * 1024 +const SAFE_RUN_ID = /^[A-Za-z0-9_-]{1,100}$/ +const SAFE_ENTITY_ID = /^[A-Za-z0-9._:-]{1,200}$/ +const ARTIFACT_KINDS = new Set(["query-raw", "query-normalized", "reader", "evaluation"]) +const IMAGE_MIME_TYPES = new Set(["image/png", "image/jpeg", "image/gif", "image/webp"]) + +export interface BuildAwareInspectionRouteOptions { + runsRoot?: string + buildsRoot?: string + artifactsRoot?: string +} + +interface ArtifactDescriptor { + relativePath: string + sha256?: string + byteLength?: number +} + +interface ResolvedBuildLink { + status: "available" | "missing" | "rejected" + scope?: "run" | "builds" + relativePath?: string + absolutePath?: string + reason?: string +} + +interface BuildPlanSummary { + buildId: string + buildFingerprint: string + containerTag?: string + provider?: string + domain?: string + trajectoryCount?: number + documentCount?: number +} + +interface ReferencedAsset { + asset: AssetRef + scope: "dataset" | "artifacts" +} + +interface PublicControlHistory { + schemaVersion: 1 + runId: string + events: Array<{ + action: string + at?: string + through?: string + message?: string + }> +} + +function json(data: unknown, status = 200): Response { + return new Response(JSON.stringify(data), { + status, + headers: { "Content-Type": "application/json" }, + }) +} + +function isInside(root: string, target: string): boolean { + const fromRoot = relative(root, target) + return ( + fromRoot === "" || + (fromRoot !== ".." && !fromRoot.startsWith(`..${sep}`) && !isAbsolute(fromRoot)) + ) +} + +async function resolveExistingFileWithin(root: string, candidate: string): Promise { + const rootAbsolute = resolve(root) + const targetAbsolute = resolve(rootAbsolute, candidate) + if (!isInside(rootAbsolute, targetAbsolute)) return null + + try { + const [rootReal, targetReal] = await Promise.all([ + realpath(rootAbsolute), + realpath(targetAbsolute), + ]) + if (!isInside(rootReal, targetReal)) return null + const metadata = await stat(targetReal) + return metadata.isFile() ? targetReal : null + } catch { + return null + } +} + +function imageSignatureMatches(bytes: Buffer, mimeType: string): boolean { + if (mimeType === "image/png") { + return bytes.subarray(0, 8).equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])) + } + if (mimeType === "image/jpeg") return bytes[0] === 0xff && bytes[1] === 0xd8 + if (mimeType === "image/gif") { + const signature = bytes.subarray(0, 6).toString("ascii") + return signature === "GIF87a" || signature === "GIF89a" + } + if (mimeType === "image/webp") { + return ( + bytes.subarray(0, 4).toString("ascii") === "RIFF" && + bytes.subarray(8, 12).toString("ascii") === "WEBP" + ) + } + return false +} + +function referencedAssets(question: BuildAwareQuestionCheckpoint): ReferencedAsset[] { + const assets: ReferencedAsset[] = [] + const add = (asset: AssetRef | undefined, scope: ReferencedAsset["scope"]): void => { + if (!asset || typeof asset.assetId !== "string") return + assets.push({ asset, scope }) + } + add(question.queryArtifact?.questionImage, "dataset") + for (const result of question.queryArtifact?.normalizedResults ?? []) { + for (const asset of result.screenshotRefs) add(asset, "dataset") + } + for (const part of question.readerArtifact?.parts ?? []) { + if (part.type === "image") add(part.asset, "artifacts") + } + return assets +} + +async function verifiedImageAsset(root: string, asset: AssetRef): Promise { + if ( + !asset.relativePath || + isAbsolute(asset.relativePath) || + asset.relativePath.includes("\0") || + asset.relativePath.includes("\\") || + !IMAGE_MIME_TYPES.has(asset.mimeType) || + !Number.isInteger(asset.byteLength) || + asset.byteLength < 1 || + asset.byteLength > MAX_IMAGE_ASSET_BYTES || + !/^[a-f0-9]{64}$/.test(asset.sha256) + ) { + throw new Error("Referenced image metadata is invalid") + } + const path = await resolveExistingFileWithin(root, asset.relativePath) + if (!path) throw new Error("Referenced image is missing or outside its recorded root") + const metadata = await stat(path) + if (metadata.size !== asset.byteLength) throw new Error("Referenced image size does not match") + const bytes = await readFile(path) + if (createHash("sha256").update(bytes).digest("hex") !== asset.sha256) { + throw new Error("Referenced image hash does not match") + } + if (!imageSignatureMatches(bytes, asset.mimeType)) { + throw new Error("Referenced image bytes do not match the recorded MIME type") + } + return bytes +} + +async function loadControlHistory(runRoot: string, runId: string): Promise { + const empty: PublicControlHistory = { schemaVersion: 1, runId, events: [] } + const path = await resolveExistingFileWithin(runRoot, "control.json") + if (!path) return empty + try { + const history = JSON.parse(await readFile(path, "utf8")) as Record + if (history.schemaVersion !== 1 || history.runId !== runId || !Array.isArray(history.events)) { + return empty + } + return sanitizeForResponse({ + ...history, + events: history.events.slice(-100), + }) as PublicControlHistory + } catch { + return empty + } +} + +function validateRunId(runId: string): boolean { + return SAFE_RUN_ID.test(runId) +} + +function validateEntityId(id: string): boolean { + return SAFE_ENTITY_ID.test(id) +} + +function sanitizeForResponse(value: unknown): unknown { + if (typeof value === "string") { + return value.replace(/\b(?:sk|sm)_[A-Za-z0-9_-]{16,}\b/g, "[REDACTED]") + } + if (Array.isArray(value)) return value.map(sanitizeForResponse) + if (value && typeof value === "object") { + return Object.fromEntries( + Object.entries(value as Record) + .filter(([key]) => key !== "absolutePath") + .map(([key, item]) => [ + key, + /^(?:api[-_]?key|authorization|token|secret|password|credential|tokenValue|accessToken|refreshToken|.*[-_]token)$/i.test( + key + ) + ? "[REDACTED]" + : sanitizeForResponse(item), + ]) + ) + } + return value +} + +function publicCheckpoint( + checkpoint: BuildAwareRunCheckpoint, + storageRoots: { + artifacts: "available" | "missing" | "rejected" + builds: "available" | "missing" | "rejected" + }, + uiManaged = false +) { + const { + buildLinks: _privateBuildLinks, + datasetManifestPath: _privateManifestPath, + artifactRoot: _privateArtifactRoot, + buildRoot: _privateBuildRoot, + ...safeCheckpoint + } = checkpoint + return { + ...safeCheckpoint, + ...(safeCheckpoint.status === "running" && uiManaged && !isRunActive(safeCheckpoint.runId) + ? { + status: "failed" as const, + error: "Run process is no longer active; resume from the durable checkpoint", + } + : {}), + config: { + ...safeCheckpoint.config, + datasetPath: isAbsolute(safeCheckpoint.config.datasetPath) + ? basename(safeCheckpoint.config.datasetPath) + : safeCheckpoint.config.datasetPath, + }, + questionBuildLinks: Object.fromEntries( + Object.entries(checkpoint.buildLinks).filter( + ([questionId, buildId]) => validateEntityId(questionId) && validateEntityId(buildId) + ) + ), + buildLinkCount: Object.keys(checkpoint.buildLinks).length, + storageRoots, + } +} + +async function loadCheckpoint( + runsRoot: string, + runId: string +): Promise<{ checkpoint: BuildAwareRunCheckpoint; runRoot: string } | null> { + if (!validateRunId(runId)) return null + const runRoot = resolve(runsRoot, runId) + try { + const [runsRootReal, runRootReal] = await Promise.all([realpath(runsRoot), realpath(runRoot)]) + if (!isInside(runsRootReal, runRootReal)) return null + } catch { + return null + } + const checkpointPath = await resolveExistingFileWithin(runRoot, "checkpoint.json") + if (!checkpointPath) return null + + const checkpoint = JSON.parse(await readFile(checkpointPath, "utf8")) as BuildAwareRunCheckpoint + if ( + checkpoint.schemaVersion !== 1 || + checkpoint.executionModel !== "shared-memory-build-v1" || + checkpoint.runId !== runId || + !checkpoint.questions || + !checkpoint.config + ) { + throw new Error(`Invalid build-aware checkpoint for run ${runId}`) + } + return { checkpoint, runRoot } +} + +async function loadReport(runRoot: string, runId: string): Promise { + for (const relativePath of ["report.json", "reports/report.json"]) { + const reportPath = await resolveExistingFileWithin(runRoot, relativePath) + if (!reportPath) continue + const report = JSON.parse(await readFile(reportPath, "utf8")) as BuildAwareReport + if ( + report.schemaVersion !== 1 || + report.protocol !== "longmemeval-v2-official" || + report.runId !== runId + ) { + throw new Error(`Invalid build-aware report for run ${runId}`) + } + return report + } + return null +} + +export async function listBuildAwareRunSummaries( + runsRoot = "data/runs-v2" +): Promise>> { + let entries + try { + entries = await readdir(resolve(runsRoot), { withFileTypes: true }) + } catch { + return [] + } + const summaries = await Promise.all( + entries + .filter((entry) => entry.isDirectory() && validateRunId(entry.name)) + .map(async (entry) => { + try { + const loaded = await loadCheckpoint(runsRoot, entry.name) + if (!loaded) return null + const { checkpoint, runRoot } = loaded + const report = await loadReport(runRoot, checkpoint.runId) + const control = await loadControlHistory(runRoot, checkpoint.runId) + const uiManaged = control.events.some( + (event) => event.action === "start" || event.action === "resume" + ) + const questions = Object.values(checkpoint.questions) + const queryCompleted = questions.filter( + (question) => question.stages.query.status === "completed" + ).length + const readCompleted = questions.filter( + (question) => question.stages.read.status === "completed" + ).length + const evaluateCompleted = questions.filter( + (question) => question.stages.evaluate.status === "completed" + ).length + const buildFinished = + checkpoint.status === "completed" || + ["query", "read", "evaluate", "report"].includes(checkpoint.currentStage) + return { + runId: checkpoint.runId, + provider: checkpoint.config.provider, + benchmark: checkpoint.config.benchmark, + judge: checkpoint.config.evaluator.model, + answeringModel: checkpoint.config.reader.model, + createdAt: checkpoint.createdAt, + updatedAt: checkpoint.updatedAt, + status: + checkpoint.status === "running" && uiManaged && !isRunActive(checkpoint.runId) + ? "failed" + : checkpoint.status, + summary: { + total: questions.length, + ingested: buildFinished ? questions.length : 0, + indexed: buildFinished ? questions.length : 0, + searched: queryCompleted, + answered: readCompleted, + evaluated: evaluateCompleted, + }, + accuracy: report?.official.overall.overall_full_set ?? null, + readOnlyInspection: true, + } + } catch { + return null + } + }) + ) + return summaries.filter((summary) => summary !== null) +} + +function questionStageSummary(checkpoint: BuildAwareRunCheckpoint) { + const questions = Object.values(checkpoint.questions) + const count = (stage: "query" | "read" | "evaluate", status: string) => + questions.filter((question) => question.stages[stage].status === status).length + return { + total: questions.length, + query: { + completed: count("query", "completed"), + failed: count("query", "failed"), + cacheHits: questions.filter((question) => question.stages.query.cacheHit === true).length, + }, + read: { + completed: count("read", "completed"), + failed: count("read", "failed"), + cacheHits: questions.filter((question) => question.stages.read.cacheHit === true).length, + }, + evaluate: { + completed: count("evaluate", "completed"), + failed: count("evaluate", "failed"), + blocked: count("evaluate", "blocked"), + }, + } +} + +function questionListItem(question: BuildAwareQuestionCheckpoint) { + return { + questionId: question.questionId, + questionType: question.questionType, + question: question.question, + buildId: question.buildId, + stages: { + query: { status: question.stages.query.status }, + read: { status: question.stages.read.status }, + evaluate: { status: question.stages.evaluate.status }, + }, + ...(question.evaluationArtifact + ? { + evaluationArtifact: { + score: question.evaluationArtifact.score, + label: question.evaluationArtifact.label, + }, + } + : {}), + } +} + +async function resolveRecordedRoot( + recordedRoot: string | undefined, + allowedRoot: string +): Promise<{ + status: "available" | "missing" | "rejected" + absolutePath?: string +}> { + const allowedAbsolute = resolve(allowedRoot) + const recordedAbsolute = resolve(recordedRoot ?? allowedAbsolute) + if (!isInside(allowedAbsolute, recordedAbsolute)) return { status: "rejected" } + try { + const [allowedReal, recordedReal] = await Promise.all([ + realpath(allowedAbsolute), + realpath(recordedAbsolute), + ]) + if (!isInside(allowedReal, recordedReal)) return { status: "rejected" } + const metadata = await stat(recordedReal) + return metadata.isDirectory() + ? { status: "available", absolutePath: recordedReal } + : { status: "rejected" } + } catch { + return { status: "missing" } + } +} + +async function loadBuildPlan(runRoot: string, buildId: string): Promise { + if (!validateEntityId(buildId)) return null + const path = await resolveExistingFileWithin(runRoot, `builds/${buildId}.plan.json`) + if (!path) return null + const metadata = await stat(path) + if (metadata.size > MAX_JSON_ARTIFACT_BYTES) return null + const plan = JSON.parse(await readFile(path, "utf8")) as Record + if (plan.buildId !== buildId || typeof plan.buildFingerprint !== "string") { + return null + } + return { + buildId, + buildFingerprint: plan.buildFingerprint, + containerTag: typeof plan.containerTag === "string" ? plan.containerTag : undefined, + provider: typeof plan.provider === "string" ? plan.provider : undefined, + domain: typeof plan.domain === "string" ? plan.domain : undefined, + trajectoryCount: Array.isArray(plan.orderedSourceIds) + ? plan.orderedSourceIds.length + : undefined, + documentCount: Array.isArray(plan.documents) ? plan.documents.length : undefined, + } +} + +async function resolveBuildCheckpoint( + buildRoot: { + status: "available" | "missing" | "rejected" + absolutePath?: string + }, + provider: string, + buildFingerprint: string | undefined +): Promise { + if (buildRoot.status !== "available" || !buildRoot.absolutePath) { + return { + status: buildRoot.status, + reason: + buildRoot.status === "rejected" + ? "Recorded build root is outside the allowed build root" + : "Build root is not available", + } + } + if (!validateEntityId(provider) || !buildFingerprint || !validateEntityId(buildFingerprint)) { + return { + status: "missing", + reason: "Build fingerprint is not available", + } + } + const relativePath = `${provider}/${buildFingerprint}/checkpoint.sqlite` + const absolutePath = await resolveExistingFileWithin(buildRoot.absolutePath, relativePath) + return absolutePath + ? { + status: "available", + scope: "builds", + relativePath, + absolutePath, + } + : { + status: "missing", + scope: "builds", + relativePath, + reason: "Durable build checkpoint is not available", + } +} + +function readBuildSqliteSummary(path: string, buildId: string): Record { + const db = new Database(path, { readonly: true, strict: true }) + try { + const build = db + .query( + `SELECT build_id, build_fingerprint, container_tag, provider, status, error + FROM builds WHERE build_id = ?` + ) + .get(buildId) as { + build_id: string + build_fingerprint: string + container_tag: string + provider: string + status: string + error: string | null + } | null + if (!build) return { available: true, buildFound: false } + + const trajectoryRows = db + .query( + "SELECT status, COUNT(*) AS count FROM trajectories WHERE build_id = ? GROUP BY status" + ) + .all(buildId) as Array<{ status: string; count: number }> + const documentRows = db + .query("SELECT status, COUNT(*) AS count FROM documents WHERE build_id = ? GROUP BY status") + .all(buildId) as Array<{ status: string; count: number }> + + return { + available: true, + buildFound: true, + buildFingerprint: build.build_fingerprint, + containerTag: build.container_tag, + provider: build.provider, + status: build.status, + error: build.error ?? undefined, + trajectories: Object.fromEntries(trajectoryRows.map((row) => [row.status, row.count])), + documents: Object.fromEntries(documentRows.map((row) => [row.status, row.count])), + } + } finally { + db.close() + } +} + +async function buildSummaries( + checkpoint: BuildAwareRunCheckpoint, + runRoot: string, + buildRoot: { + status: "available" | "missing" | "rejected" + absolutePath?: string + }, + report: BuildAwareReport | null +): Promise>> { + const buildIds = Array.from( + new Set([ + ...checkpoint.buildIds, + ...Object.values(checkpoint.questions).map((question) => question.buildId), + ]) + ) + + return Promise.all( + buildIds.map(async (buildId) => { + const questions = Object.values(checkpoint.questions).filter( + (question) => question.buildId === buildId + ) + const plan = await loadBuildPlan(runRoot, buildId) + const reportBuild = report?.builds?.find((build) => build.buildId === buildId) + const checkpointFingerprint = questions.find( + (question) => question.queryArtifact?.buildFingerprint + )?.queryArtifact?.buildFingerprint + const buildFingerprint = + checkpointFingerprint ?? plan?.buildFingerprint ?? reportBuild?.buildFingerprint + const provider = plan?.provider ?? checkpoint.config.provider + const link = await resolveBuildCheckpoint(buildRoot, provider, buildFingerprint) + let stateStore: Record = { + available: false, + reason: link.status === "rejected" ? link.reason : "No readable build state link", + } + if (link.status === "available" && link.absolutePath) { + try { + stateStore = readBuildSqliteSummary(link.absolutePath, buildId) + } catch (error) { + stateStore = { + available: false, + reason: + error instanceof Error + ? "Build state is not a supported readable SQLite checkpoint" + : "Could not inspect build state", + } + } + } + const storeFingerprint = + typeof stateStore.buildFingerprint === "string" ? stateStore.buildFingerprint : undefined + return { + buildId, + buildFingerprint: buildFingerprint ?? storeFingerprint, + containerTag: plan?.containerTag ?? reportBuild?.containerTag, + domain: plan?.domain ?? reportBuild?.domain, + trajectoryCount: plan?.trajectoryCount ?? reportBuild?.trajectoryCount, + documentCount: plan?.documentCount ?? reportBuild?.documentCount, + questionCount: questions.length, + questionIds: questions.map((question) => question.questionId), + questionLinkMismatches: questions + .filter((question) => checkpoint.buildLinks[question.questionId] !== buildId) + .map((question) => question.questionId), + reused: questions.length > 1, + reuseCount: questions.length, + priorBuildReuse: reportBuild?.reused, + checkpointLink: { + status: link.status, + scope: link.scope, + relativePath: link.relativePath, + reason: link.reason, + }, + stateStore, + } + }) + ) +} + +function artifactDescriptorFor( + question: BuildAwareQuestionCheckpoint, + kind: string +): { descriptor?: ArtifactDescriptor; embedded?: unknown } { + if (kind === "query-raw") { + return { descriptor: question.queryArtifact?.rawArtifact } + } + if (kind === "query-normalized") { + return { descriptor: question.queryArtifact?.normalizedArtifact } + } + if (kind === "reader") { + return { + descriptor: question.stages.read.artifactPath + ? { relativePath: question.stages.read.artifactPath } + : undefined, + embedded: question.readerArtifact, + } + } + return { + descriptor: question.stages.evaluate.artifactPath + ? { relativePath: question.stages.evaluate.artifactPath } + : undefined, + embedded: question.evaluationArtifact, + } +} + +function artifactLinks( + runId: string, + question: BuildAwareQuestionCheckpoint, + artifactRootAvailable: boolean +): Record { + return Object.fromEntries( + [...ARTIFACT_KINDS].map((kind) => { + const source = artifactDescriptorFor(question, kind) + return [ + kind, + { + available: Boolean(source.embedded || (source.descriptor && artifactRootAvailable)), + href: `/api/runs/${encodeURIComponent(runId)}/questions/${encodeURIComponent( + question.questionId + )}/artifacts/${kind}`, + provenance: source.descriptor + ? { + relativePath: source.descriptor.relativePath, + sha256: source.descriptor.sha256, + byteLength: source.descriptor.byteLength, + } + : { source: "checkpoint" }, + }, + ] + }) + ) +} + +async function readArtifact( + artifactRoot: string, + descriptor: ArtifactDescriptor +): Promise<{ data: unknown; provenance: Record }> { + if ( + !descriptor.relativePath || + isAbsolute(descriptor.relativePath) || + descriptor.relativePath === ".." || + descriptor.relativePath.startsWith(`..${sep}`) + ) { + throw new Error("Artifact path is outside the artifact root") + } + const artifactPath = await resolveExistingFileWithin(artifactRoot, descriptor.relativePath) + if (!artifactPath) throw new Error("Artifact is missing or outside the artifact root") + const metadata = await stat(artifactPath) + if (metadata.size > MAX_JSON_ARTIFACT_BYTES) { + throw new Error(`Artifact exceeds the ${MAX_JSON_ARTIFACT_BYTES}-byte inspection limit`) + } + if (descriptor.byteLength !== undefined && metadata.size !== descriptor.byteLength) { + throw new Error("Artifact byte length does not match its checkpoint provenance") + } + const bytes = await readFile(artifactPath) + const actualHash = createHash("sha256").update(bytes).digest("hex") + if (descriptor.sha256 && actualHash !== descriptor.sha256) { + throw new Error("Artifact hash does not match its checkpoint provenance") + } + return { + data: sanitizeForResponse(JSON.parse(bytes.toString("utf8"))), + provenance: { + relativePath: descriptor.relativePath, + sha256: actualHash, + byteLength: metadata.size, + integrity: descriptor.sha256 ? "verified" : "computed", + }, + } +} + +function pageParameters(url: URL): { page: number; limit: number } { + const rawPage = Number.parseInt(url.searchParams.get("page") ?? "1", 10) + const rawLimit = Number.parseInt(url.searchParams.get("limit") ?? "50", 10) + return { + page: Number.isFinite(rawPage) ? Math.max(1, rawPage) : 1, + limit: Number.isFinite(rawLimit) ? Math.min(200, Math.max(1, rawLimit)) : 50, + } +} + +export function createBuildAwareInspectionHandler(options: BuildAwareInspectionRouteOptions = {}) { + const runsRoot = resolve(options.runsRoot ?? "data/runs-v2") + const buildsRoot = resolve(options.buildsRoot ?? "data/memory-builds-v2") + const artifactsRoot = resolve(options.artifactsRoot ?? "data/artifacts-v2") + + return async function handleBuildAwareInspectionRoutes( + req: Request, + url: URL + ): Promise { + if (req.method !== "GET") return null + const pathname = url.pathname + + const routeMatch = pathname.match(/^\/api\/runs\/([^/]+)(?:\/.*)?$/) + if (!routeMatch) return null + + let runId: string + try { + runId = decodeURIComponent(routeMatch[1]) + } catch { + return null + } + if (!validateRunId(runId)) { + return null + } + + const loaded = await loadCheckpoint(runsRoot, runId) + if (!loaded) return null + const { checkpoint, runRoot } = loaded + const [artifactStorage, buildStorage] = await Promise.all([ + resolveRecordedRoot(checkpoint.artifactRoot, artifactsRoot), + resolveRecordedRoot(checkpoint.buildRoot, buildsRoot), + ]) + const storageRoots = { + artifacts: artifactStorage.status, + builds: buildStorage.status, + } + + if (pathname === `/api/runs/${encodeURIComponent(runId)}`) { + const report = await loadReport(runRoot, runId) + const builds = await buildSummaries(checkpoint, runRoot, buildStorage, report) + const control = await loadControlHistory(runRoot, runId) + const uiManaged = control.events.some( + (event) => event.action === "start" || event.action === "resume" + ) + const publicRun = publicCheckpoint(checkpoint, storageRoots, uiManaged) + const compact = url.searchParams.get("compact") === "true" + const runPayload = compact + ? (({ questions: _questions, ...rest }) => rest)(publicRun) + : publicRun + return json( + sanitizeForResponse({ + ...runPayload, + summary: questionStageSummary(checkpoint), + inspection: { + builds, + control, + reportAvailable: Boolean(report), + metricNamespaces: { + official: report?.official ?? null, + diagnostics: report?.diagnostics ?? null, + }, + }, + }) + ) + } + + if (pathname === `/api/runs/${encodeURIComponent(runId)}/inspection`) { + const report = await loadReport(runRoot, runId) + const builds = await buildSummaries(checkpoint, runRoot, buildStorage, report) + const control = await loadControlHistory(runRoot, runId) + const uiManaged = control.events.some( + (event) => event.action === "start" || event.action === "resume" + ) + return json( + sanitizeForResponse({ + checkpoint: publicCheckpoint(checkpoint, storageRoots, uiManaged), + summary: questionStageSummary(checkpoint), + builds, + control, + report, + metricNamespaces: { + official: report?.official ?? null, + diagnostics: report?.diagnostics ?? null, + }, + }) + ) + } + + if (pathname === `/api/runs/${encodeURIComponent(runId)}/report`) { + const report = await loadReport(runRoot, runId) + return report ? json(sanitizeForResponse(report)) : json({ error: "Report not found" }, 404) + } + + if (pathname === `/api/runs/${encodeURIComponent(runId)}/builds`) { + const report = await loadReport(runRoot, runId) + return json( + sanitizeForResponse({ + builds: await buildSummaries(checkpoint, runRoot, buildStorage, report), + }) + ) + } + + const buildMatch = pathname.match(/^\/api\/runs\/[^/]+\/builds\/([^/]+)$/) + if (buildMatch) { + let buildId: string + try { + buildId = decodeURIComponent(buildMatch[1]) + } catch { + return json({ error: "Invalid build ID encoding" }, 400) + } + if (!validateEntityId(buildId)) return json({ error: "Invalid build ID" }, 400) + const report = await loadReport(runRoot, runId) + const builds = await buildSummaries(checkpoint, runRoot, buildStorage, report) + const build = builds.find((item) => item.buildId === buildId) + return build ? json(sanitizeForResponse(build)) : json({ error: "Build not found" }, 404) + } + + if (pathname === `/api/runs/${encodeURIComponent(runId)}/questions`) { + const { page, limit } = pageParameters(url) + const status = url.searchParams.get("status") + const type = url.searchParams.get("type") + let questions = Object.values(checkpoint.questions) + if (status) { + questions = questions.filter((question) => { + const evaluationStatus = question.stages.evaluate.status + if (status === "completed") return evaluationStatus === "completed" + if (status === "failed") return evaluationStatus === "failed" + if (status === "pending") { + return evaluationStatus !== "completed" && evaluationStatus !== "failed" + } + return true + }) + } + if (type) questions = questions.filter((question) => question.questionType === type) + const total = questions.length + const start = (page - 1) * limit + return json( + sanitizeForResponse({ + questions: questions.slice(start, start + limit).map(questionListItem), + questionTypeRegistry: {}, + pagination: { + page, + limit, + total, + totalPages: Math.ceil(total / limit), + }, + }) + ) + } + + const artifactMatch = pathname.match( + /^\/api\/runs\/[^/]+\/questions\/([^/]+)\/artifacts\/([^/]+)$/ + ) + if (artifactMatch) { + let questionId: string + try { + questionId = decodeURIComponent(artifactMatch[1]) + } catch { + return json({ error: "Invalid question ID encoding" }, 400) + } + const kind = artifactMatch[2] + if (!validateEntityId(questionId)) return json({ error: "Invalid question ID" }, 400) + if (!ARTIFACT_KINDS.has(kind)) return json({ error: "Unknown artifact kind" }, 404) + const question = checkpoint.questions[questionId] + if (!question) return json({ error: "Question not found" }, 404) + const source = artifactDescriptorFor(question, kind) + if (source.embedded) { + return json({ + kind, + data: sanitizeForResponse(source.embedded), + provenance: { source: "checkpoint", integrity: "embedded" }, + }) + } + if (!source.descriptor) return json({ error: "Artifact not available" }, 404) + if (artifactStorage.status !== "available" || !artifactStorage.absolutePath) { + return json( + { + error: + artifactStorage.status === "rejected" + ? "Recorded artifact root is outside the allowed artifact root" + : "Artifact root is not available", + }, + 422 + ) + } + try { + const artifact = await readArtifact(artifactStorage.absolutePath, source.descriptor) + return json({ kind, ...artifact }) + } catch (error) { + return json( + { error: error instanceof Error ? error.message : "Artifact could not be read" }, + 422 + ) + } + } + + const assetMatch = pathname.match(/^\/api\/runs\/[^/]+\/questions\/([^/]+)\/assets\/([^/]+)$/) + if (assetMatch) { + let questionId: string + let assetId: string + try { + questionId = decodeURIComponent(assetMatch[1]) + assetId = decodeURIComponent(assetMatch[2]) + } catch { + return json({ error: "Invalid asset URL encoding" }, 400) + } + if (!validateEntityId(questionId) || !validateEntityId(assetId)) { + return json({ error: "Invalid question or asset ID" }, 400) + } + const question = checkpoint.questions[questionId] + if (!question) return json({ error: "Question not found" }, 404) + const candidates = referencedAssets(question).filter(({ asset }) => asset.assetId === assetId) + if (candidates.length === 0) return json({ error: "Image asset not found" }, 404) + + const errors: string[] = [] + for (const candidate of candidates.sort( + (left, right) => Number(right.scope === "artifacts") - Number(left.scope === "artifacts") + )) { + let root: string | undefined + if (candidate.scope === "artifacts") { + root = artifactStorage.status === "available" ? artifactStorage.absolutePath : undefined + } else { + try { + const datasetRoot = resolve(checkpoint.config.datasetPath) + const datasetReal = await realpath(datasetRoot) + if (datasetReal !== parse(datasetReal).root) root = datasetReal + } catch { + root = undefined + } + } + if (!root) { + errors.push(`Recorded ${candidate.scope} root is not available`) + continue + } + try { + const bytes = await verifiedImageAsset(root, candidate.asset) + return new Response(Uint8Array.from(bytes).buffer, { + status: 200, + headers: { + "Content-Type": candidate.asset.mimeType, + "Content-Length": String(bytes.byteLength), + "Cache-Control": "private, max-age=31536000, immutable", + ETag: `"${candidate.asset.sha256}"`, + "X-Content-Type-Options": "nosniff", + }, + }) + } catch (error) { + errors.push(error instanceof Error ? error.message : "Image verification failed") + } + } + return json({ error: errors[0] ?? "Image asset could not be verified" }, 422) + } + + const questionMatch = pathname.match(/^\/api\/runs\/[^/]+\/questions\/([^/]+)$/) + if (questionMatch) { + let questionId: string + try { + questionId = decodeURIComponent(questionMatch[1]) + } catch { + return json({ error: "Invalid question ID encoding" }, 400) + } + if (!validateEntityId(questionId)) return json({ error: "Invalid question ID" }, 400) + const question = checkpoint.questions[questionId] + if (!question) return json({ error: "Question not found" }, 404) + const reuseCount = Object.values(checkpoint.questions).filter( + (item) => item.buildId === question.buildId + ).length + const buildPlan = await loadBuildPlan(runRoot, question.buildId) + return json( + sanitizeForResponse({ + ...question, + buildReuseCount: reuseCount, + buildLinkMatchesCheckpoint: checkpoint.buildLinks[questionId] === question.buildId, + buildFingerprint: question.queryArtifact?.buildFingerprint ?? buildPlan?.buildFingerprint, + artifactLinks: artifactLinks(runId, question, artifactStorage.status === "available"), + metricNamespace: { + evaluation: "longmemeval-v2-official", + retrievalAndLatency: "memorybench-diagnostics", + }, + }) + ) + } + + return null + } +} + +export const handleBuildAwareInspectionRoutes = createBuildAwareInspectionHandler() diff --git a/src/server/routes/longmemeval-v2-control.test.ts b/src/server/routes/longmemeval-v2-control.test.ts new file mode 100644 index 0000000..5900974 --- /dev/null +++ b/src/server/routes/longmemeval-v2-control.test.ts @@ -0,0 +1,794 @@ +import { afterEach, describe, expect, test } from "bun:test" +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises" +import { tmpdir } from "node:os" +import { dirname, join, resolve } from "node:path" +import type { LongMemEvalV2RunnerOptions } from "../../orchestrator/longmemeval-v2" +import { BuildAwareRunStore } from "../../orchestrator/build-aware-run-store" +import { + supermemoryPreflightGatePath, + type SupermemoryPreflightReport, +} from "../../providers/supermemory/advanced" +import type { BuildAwareRunCheckpoint, BuildAwareRunConfig } from "../../types/build-aware" +import { createLongMemEvalV2ControlHandler } from "./longmemeval-v2-control" + +const temporaryDirectories: string[] = [] + +afterEach(async () => { + await Promise.all( + temporaryDirectories.splice(0).map((path) => rm(path, { recursive: true, force: true })) + ) +}) + +async function temporaryRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), "memorybench-v2-control-")) + temporaryDirectories.push(root) + return root +} + +function checkpoint( + config: BuildAwareRunConfig, + status: BuildAwareRunCheckpoint["status"] = "completed" +) { + return { + schemaVersion: 1, + executionModel: "shared-memory-build-v1", + runId: "ui-v2-run", + configFingerprint: "fingerprint", + status, + currentStage: "plan", + config, + targetQuestionIds: [], + buildIds: [], + buildLinks: {}, + questions: {}, + createdAt: "2026-07-28T00:00:00.000Z", + updatedAt: "2026-07-28T00:00:01.000Z", + } satisfies BuildAwareRunCheckpoint +} + +function config(datasetPath: string): BuildAwareRunConfig { + return { + provider: "supermemory", + benchmark: "longmemeval-v2", + mode: "benchmark", + datasetPath, + datasetRevision: "f152293e235517d504809563c833d7190b8c713b", + tier: "small", + domain: "all", + seed: "memorybench-longmemeval-v2", + retrieval: { + topK: 20, + threshold: 0, + searchMode: "hybrid", + rerank: true, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: true, + metadataFilter: {}, + }, + reader: { + model: "gpt-5", + reasoningEffort: "high", + maxCompletionTokens: 20_000, + maxContextTokens: 200_000, + evidenceTopK: 20, + maxImages: 100, + maxImageBytes: 20 * 1024 * 1024, + malformedResponseAttempts: 3, + }, + evaluator: { + model: "gpt-5", + reasoningEffort: "high", + maxCompletionTokens: 4096, + }, + build: { + serviceBaseUrl: "https://api.supermemory.ai", + dreaming: "instant", + rootFilterMode: "self", + maxDocumentChars: 200_000, + trajectoryConcurrency: 4, + maxInFlightRequests: 20, + maxTrajectoryAttempts: 4, + indexingTimeoutMs: 30 * 60_000, + pollIntervalMs: 2_000, + preflightMaxAgeMs: 24 * 60 * 60_000, + continueOnIndexingTimeout: true, + }, + execution: { buildConcurrency: 2, questionConcurrency: 5 }, + } +} + +function startBody(overrides: Record = {}) { + return { + runId: "ui-v2-run", + datasetPath: "../LongMemEval-V2/data/longmemeval-v2", + runThrough: "plan", + ...overrides, + } +} + +async function call( + handler: ReturnType, + method: string, + path: string, + body?: unknown, + origin?: string +) { + const headers = new Headers() + if (body !== undefined) headers.set("content-type", "application/json") + if (origin) headers.set("origin", origin) + const request = new Request(`http://localhost${path}`, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }) + return handler(request, new URL(request.url)) +} + +async function waitFor(predicate: () => boolean | Promise): Promise { + for (let attempt = 0; attempt < 100; attempt += 1) { + if (await predicate()) return + await Bun.sleep(5) + } + throw new Error("Timed out waiting for test condition") +} + +describe("LongMemEval-V2 control routes", () => { + test("starts an offline plan with CLI-equivalent defaults and server-side credentials", async () => { + const root = await temporaryRoot() + const captured: LongMemEvalV2RunnerOptions[] = [] + const executeOptions: unknown[] = [] + const events: Array> = [] + const handler = createLongMemEvalV2ControlHandler({ + runsRoot: join(root, "runs"), + buildsRoot: join(root, "builds"), + artifactsRoot: join(root, "artifacts"), + preflightRoot: join(root, "preflights"), + runnerFactory: (options) => { + captured.push(options) + return { + async execute(input) { + executeOptions.push(input) + return checkpoint(options.config) + }, + } + }, + broadcast: (event) => events.push(event), + }) + + const response = await call(handler, "POST", "/api/runs-v2/start", startBody()) + expect(response?.status).toBe(202) + expect(await response!.json()).toMatchObject({ + message: "Run started", + runId: "ui-v2-run", + statusUrl: "/api/runs-v2/ui-v2-run/status", + runUrl: "/api/runs/ui-v2-run", + }) + const immediateCheckpoint = await new BuildAwareRunStore("ui-v2-run", join(root, "runs")).load() + expect(immediateCheckpoint).toMatchObject({ + runId: "ui-v2-run", + status: "running", + currentStage: "plan", + config: { benchmark: "longmemeval-v2" }, + }) + await waitFor(() => events.some((event) => event.type === "run_complete")) + expect(captured).toHaveLength(1) + expect(captured[0].supermemoryApiKey).toBeUndefined() + expect(captured[0].openAIApiKey).toBeUndefined() + expect(captured[0].signal).toBeInstanceOf(AbortSignal) + expect(captured[0].config).toMatchObject({ + provider: "supermemory", + benchmark: "longmemeval-v2", + tier: "small", + domain: "all", + datasetRevision: "f152293e235517d504809563c833d7190b8c713b", + retrieval: { topK: 20 }, + reader: { evidenceTopK: 20, model: "gpt-5", reasoningEffort: "high" }, + evaluator: { model: "gpt-5", reasoningEffort: "high" }, + build: { + maxTrajectoryAttempts: 4, + indexingTimeoutMs: 1_800_000, + continueOnIndexingTimeout: true, + }, + }) + expect(executeOptions).toEqual([{ through: "plan", forceBuild: false, freshQuery: false }]) + + const status = await call(handler, "GET", "/api/runs-v2/ui-v2-run/status") + expect(status?.status).toBe(200) + const statusBody = await status!.json() + expect(statusBody.active).toBe(false) + expect(statusBody.stopping).toBe(false) + expect(statusBody.control.events.map((event: { action: string }) => event.action)).toEqual([ + "start", + "completed", + ]) + }) + + test("persists the selected provider, provider retrieval profile, concurrency, and launch flags", async () => { + const root = await temporaryRoot() + const captured: LongMemEvalV2RunnerOptions[] = [] + const handler = createLongMemEvalV2ControlHandler({ + runsRoot: join(root, "runs"), + runnerFactory: (options) => { + captured.push(options) + return { execute: async () => checkpoint(options.config) } + }, + }) + const response = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ + provider: "filesystem", + trajectoryConcurrency: 3, + maxInFlightRequests: 7, + forceBuild: true, + freshQuery: true, + }) + ) + expect(response?.status).toBe(202) + await waitFor(() => captured.length === 1) + expect(captured[0].config).toMatchObject({ + provider: "filesystem", + retrieval: { searchMode: "memories", rerank: false }, + build: { trajectoryConcurrency: 3, maxInFlightRequests: 7 }, + }) + await waitFor(async () => { + const current = await call(handler, "GET", "/api/runs-v2/ui-v2-run/status") + return current?.status === 200 && (await current.json()).active === false + }) + const status = await call(handler, "GET", "/api/runs-v2/ui-v2-run/status") + const body = await status!.json() + expect(body.control.events[0]).toMatchObject({ + action: "start", + provider: "filesystem", + forceBuild: true, + freshQuery: true, + }) + }) + + test("keeps providers without a safe build adapter plan-only", async () => { + const root = await temporaryRoot() + const handler = createLongMemEvalV2ControlHandler({ runsRoot: join(root, "runs") }) + const response = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ provider: "mem0", runThrough: "build", questionIds: ["q-1"] }) + ) + expect(response?.status).toBe(400) + expect(await response!.json()).toEqual({ + error: "mem0 does not yet have a safe LongMemEval-V2 adapter; use Plan only", + }) + }) + + test("accepts a bounded haystack selection and configurable OpenAI models", async () => { + const root = await temporaryRoot() + const captured: LongMemEvalV2RunnerOptions[] = [] + const handler = createLongMemEvalV2ControlHandler({ + runsRoot: join(root, "runs"), + runnerFactory: (options) => { + captured.push(options) + return { execute: async () => checkpoint(options.config) } + }, + }) + + const response = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ + haystackLimit: 1, + runThrough: "query", + readerModel: "gpt-4o", + evaluatorModel: "gpt-5.2", + reasoningEffort: "none", + evaluatorReasoningEffort: "none", + }) + ) + expect(response?.status).toBe(202) + await waitFor(() => captured.length === 1) + expect(captured[0].config).toMatchObject({ + haystackLimit: 1, + reader: { + model: "gpt-4o", + reasoningEffort: "none", + maxCompletionTokens: 8_000, + maxContextTokens: 120_000, + }, + evaluator: { model: "gpt-5.2", reasoningEffort: "none" }, + }) + await waitFor(async () => { + const status = await call(handler, "GET", "/api/runs-v2/ui-v2-run/status") + return status?.status === 200 && (await status.json()).active === false + }) + }) + + test("rejects cross-site, unknown, key-bearing, and unsafe configurations", async () => { + const root = await temporaryRoot() + const handler = createLongMemEvalV2ControlHandler({ + runsRoot: join(root, "runs"), + runnerFactory: () => ({ execute: async () => checkpoint(config(root)) }), + }) + + const crossSite = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody(), + "https://evil.example" + ) + expect(crossSite?.status).toBe(403) + + const keyBearing = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ supermemoryApiKey: "sm_secret-value-must-not-echo" }), + "http://localhost:3000" + ) + expect(keyBearing?.status).toBe(400) + expect(JSON.stringify(await keyBearing!.json())).not.toContain("secret-value") + + const invalidTopK = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ topK: 5, evidenceTopK: 6 }) + ) + expect(invalidTopK?.status).toBe(400) + expect((await invalidTopK!.json()).error).toContain("cannot exceed topK") + + const medium = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ tier: "medium", runThrough: "build", questionIds: ["q-1"] }) + ) + expect(medium?.status).toBe(400) + expect((await medium!.json()).error).toContain("allowMedium") + + const scoringCanary = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ + mode: "one-trajectory-canary", + questionIds: ["question-1"], + runThrough: "evaluate", + }) + ) + expect(scoringCanary?.status).toBe(400) + expect((await scoringCanary!.json()).error).toContain("may only plan, build, or query") + + const accidentalFullRun = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ runThrough: "query" }) + ) + expect(accidentalFullRun?.status).toBe(400) + expect((await accidentalFullRun!.json()).error).toContain("allowFullRun") + + const mixedSelectors = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ questionIds: ["question-1"], haystackLimit: 1 }) + ) + expect(mixedSelectors?.status).toBe(400) + expect((await mixedSelectors!.json()).error).toContain("mutually exclusive") + }) + + test("prevents duplicate active IDs and aborts the exact run on stop", async () => { + const root = await temporaryRoot() + let signal: AbortSignal | undefined + const events: Array> = [] + const handler = createLongMemEvalV2ControlHandler({ + runsRoot: join(root, "runs"), + runnerFactory: (options) => { + signal = options.signal + return { + execute: async () => + new Promise((_resolve, reject) => { + if (options.signal?.aborted) return reject(options.signal.reason) + options.signal?.addEventListener("abort", () => reject(options.signal?.reason), { + once: true, + }) + }), + } + }, + broadcast: (event) => events.push(event), + }) + + const started = await call(handler, "POST", "/api/runs-v2/start", startBody()) + expect(started?.status).toBe(202) + const duplicate = await call(handler, "POST", "/api/runs-v2/start", startBody()) + expect(duplicate?.status).toBe(409) + + const keyBearingStop = await call(handler, "POST", "/api/runs-v2/ui-v2-run/stop", { + apiKey: "sm_secret-value-must-not-echo", + }) + expect(keyBearingStop?.status).toBe(400) + expect(signal?.aborted).toBe(false) + + const stopped = await call(handler, "POST", "/api/runs-v2/ui-v2-run/stop") + expect(stopped?.status).toBe(202) + expect(signal?.aborted).toBe(true) + await waitFor(() => events.some((event) => event.type === "run_stopped")) + const status = await call(handler, "GET", "/api/runs-v2/ui-v2-run/status") + const body = await status!.json() + expect(body.active).toBe(false) + expect(body.control.events.map((event: { action: string }) => event.action)).toEqual([ + "start", + "stop-request", + "stopped", + ]) + }) + + test("resumes from the exact stored config and enforces monotonic stage targets", async () => { + const root = await temporaryRoot() + const runsRoot = join(root, "runs") + const storedConfig = { + ...config(resolve(root, "dataset")), + questionIds: ["question-1"], + } + const store = new BuildAwareRunStore("resume-run", runsRoot) + const stored = await store.createOrLoad(storedConfig) + stored.status = "failed" + stored.currentStage = "query" + stored.error = "network interruption" + await store.save(stored) + + const captured: LongMemEvalV2RunnerOptions[] = [] + const executions: unknown[] = [] + const handler = createLongMemEvalV2ControlHandler({ + runsRoot, + runnerFactory: (options) => { + captured.push(options) + return { + async execute(input) { + executions.push(input) + return { ...stored, status: "completed", currentStage: "report" } + }, + } + }, + }) + + const earlier = await call(handler, "POST", "/api/runs-v2/resume-run/resume", { + runThrough: "plan", + }) + expect(earlier?.status).toBe(400) + expect((await earlier!.json()).error).toContain("cannot be earlier") + + const resumed = await call(handler, "POST", "/api/runs-v2/resume-run/resume", {}) + expect(resumed?.status).toBe(202) + await waitFor(() => executions.length === 1) + expect(captured[0].config).toEqual(storedConfig) + expect(executions).toEqual([{ through: "query", forceBuild: false, freshQuery: false }]) + await waitFor(async () => { + const status = await call(handler, "GET", "/api/runs-v2/resume-run/status") + return status?.status === 200 && (await status.json()).active === false + }) + + const explicitContinuation = await call(handler, "POST", "/api/runs-v2/resume-run/resume", { + runThrough: "report", + }) + expect(explicitContinuation?.status).toBe(202) + await waitFor(() => executions.length === 2) + expect(executions[1]).toEqual({ through: "report", forceBuild: false, freshQuery: false }) + await waitFor(async () => { + const status = await call(handler, "GET", "/api/runs-v2/resume-run/status") + return status?.status === 200 && (await status.json()).active === false + }) + }) + + test("resumes a failed canary through its checkpoint stage and exposes stale runs as resumable", async () => { + const root = await temporaryRoot() + const runsRoot = join(root, "runs") + const canaryConfig = { + ...config(resolve(root, "dataset")), + mode: "one-trajectory-canary" as const, + questionIds: ["question-1"], + } + const store = new BuildAwareRunStore("canary-resume", runsRoot) + const stored = await store.createOrLoad(canaryConfig) + stored.status = "running" + stored.currentStage = "query" + await store.save(stored) + await writeFile( + join(store.runRoot, "control.json"), + JSON.stringify({ + schemaVersion: 1, + runId: "canary-resume", + events: [ + { + action: "start", + at: "2026-07-28T00:00:00.000Z", + through: "query", + }, + ], + }) + ) + + const executions: unknown[] = [] + const handler = createLongMemEvalV2ControlHandler({ + runsRoot, + runnerFactory: () => ({ + async execute(input) { + executions.push(input) + return { ...stored, status: "completed" } + }, + }), + }) + + const staleStatus = await call(handler, "GET", "/api/runs-v2/canary-resume/status") + expect(await staleStatus!.json()).toMatchObject({ + active: false, + checkpoint: { + status: "failed", + currentStage: "query", + error: "Run process is no longer active; resume from the durable checkpoint", + }, + }) + + const resumed = await call(handler, "POST", "/api/runs-v2/canary-resume/resume", {}) + expect(resumed?.status).toBe(202) + await waitFor(() => executions.length === 1) + expect(executions).toEqual([{ through: "query", forceBuild: false, freshQuery: false }]) + await waitFor(async () => { + const status = await call(handler, "GET", "/api/runs-v2/canary-resume/status") + return status?.status === 200 && (await status.json()).active === false + }) + }) + + test("releases a resume reservation when lifecycle history cannot be persisted", async () => { + const root = await temporaryRoot() + const runsRoot = join(root, "runs") + const store = new BuildAwareRunStore("history-failure", runsRoot) + const stored = await store.createOrLoad({ + ...config(resolve(root, "dataset")), + questionIds: ["question-1"], + }) + stored.status = "failed" + stored.currentStage = "query" + await store.save(stored) + await mkdir(join(store.runRoot, "control.json")) + + const handler = createLongMemEvalV2ControlHandler({ + runsRoot, + runnerFactory: () => ({ execute: async () => stored }), + }) + const first = await call(handler, "POST", "/api/runs-v2/history-failure/resume", {}) + expect(first?.status).toBe(500) + const second = await call(handler, "POST", "/api/runs-v2/history-failure/resume", {}) + expect(second?.status).toBe(500) + expect((await second!.json()).error).not.toContain("already active") + }) + + test("rejects start for an existing checkpoint and resume for a completed report", async () => { + const root = await temporaryRoot() + const runsRoot = join(root, "runs") + const store = new BuildAwareRunStore("existing-run", runsRoot) + const existing = await store.createOrLoad(config(root)) + existing.status = "completed" + existing.currentStage = "report" + await store.save(existing) + const handler = createLongMemEvalV2ControlHandler({ + runsRoot, + runnerFactory: () => ({ execute: async () => existing }), + }) + + const start = await call( + handler, + "POST", + "/api/runs-v2/start", + startBody({ runId: "existing-run" }) + ) + expect(start?.status).toBe(409) + const resume = await call(handler, "POST", "/api/runs-v2/existing-run/resume", {}) + expect(resume?.status).toBe(409) + expect((await resume!.json()).error).toContain("cannot be resumed") + }) + + test("requires fresh explicit confirmation before a full-scope plan can continue live", async () => { + const root = await temporaryRoot() + const runsRoot = join(root, "runs") + const store = new BuildAwareRunStore("full-plan", runsRoot) + const planned = await store.createOrLoad(config(resolve(root, "dataset"))) + planned.status = "completed" + planned.currentStage = "plan" + await store.save(planned) + const executions: unknown[] = [] + const handler = createLongMemEvalV2ControlHandler({ + runsRoot, + runnerFactory: () => ({ + async execute(input) { + executions.push(input) + return { ...planned, status: "completed", currentStage: "build" } + }, + }), + }) + + const rejected = await call(handler, "POST", "/api/runs-v2/full-plan/resume", { + runThrough: "build", + }) + expect(rejected?.status).toBe(400) + expect((await rejected!.json()).error).toContain("allowFullRun") + expect(executions).toHaveLength(0) + + const confirmed = await call(handler, "POST", "/api/runs-v2/full-plan/resume", { + runThrough: "build", + allowFullRun: true, + }) + expect(confirmed?.status).toBe(202) + await waitFor(() => executions.length === 1) + expect(executions[0]).toEqual({ through: "build", forceBuild: false, freshQuery: false }) + await waitFor(async () => { + const status = await call(handler, "GET", "/api/runs-v2/full-plan/status") + return status?.status === 200 && (await status.json()).active === false + }) + }) + + test("runs one bounded server-key preflight at a time and publishes the passing gate", async () => { + const root = await temporaryRoot() + let finishPreflight!: (report: SupermemoryPreflightReport) => void + const pendingPreflight = new Promise((resolve) => { + finishPreflight = resolve + }) + const handler = createLongMemEvalV2ControlHandler({ + preflightRoot: join(root, "preflights"), + serviceBaseUrl: "https://api.supermemory.ai", + preflightRunner: () => pendingPreflight, + now: () => Date.parse("2026-07-28T00:00:00.000Z"), + }) + + const started = await call(handler, "POST", "/api/runs-v2/preflight", { topK: 25 }) + expect(started?.status).toBe(202) + const duplicate = await call(handler, "POST", "/api/runs-v2/preflight", { topK: 25 }) + expect(duplicate?.status).toBe(409) + + finishPreflight({ + schemaVersion: 1, + generatedAt: "2026-07-28T00:00:00.000Z", + baseUrl: "https://api.supermemory.ai", + identity: { + buildId: "preflight-ui", + containerTag: "preflight-ui", + runFingerprint: "preflight-ui", + }, + searchContract: { + searchMode: "hybrid", + standaloneChunksExpected: true, + deprecatedIncludeChunks: false, + requestedTopK: 25, + }, + checks: [], + allPassed: true, + blockers: [], + requestBudget: { + configuredCap: 20, + effectiveCap: 20, + inFlight: 0, + peakInFlight: 1, + throttleEvents: 0, + successStreak: 1, + notBeforeMs: 0, + }, + }) + + await waitFor(async () => { + const response = await call(handler, "GET", "/api/runs-v2/options") + const body = await response!.json() + return body.preflightActivity.status === "passed" && body.preflight.status === "passing" + }) + const options = await call(handler, "GET", "/api/runs-v2/options") + expect(await options!.json()).toMatchObject({ + preflightActivity: { status: "passed", topK: 25 }, + preflight: { status: "passing", testedTopK: 25 }, + }) + }) + + test("reports bounded prepared-dataset and non-secret prerequisite options", async () => { + const root = await temporaryRoot() + const dataset = join(root, "longmemeval-v2") + await mkdir(join(dataset, "haystacks"), { recursive: true }) + await mkdir(join(dataset, "screenshots"), { recursive: true }) + await writeFile(join(dataset, "questions.jsonl"), "{}\n") + await writeFile(join(dataset, "trajectories.jsonl"), "{}\n") + await writeFile(join(dataset, "haystacks/lme_v2_small.json"), "{}\n") + const preflightRoot = join(root, "preflights") + const gatePath = supermemoryPreflightGatePath(preflightRoot, "https://api.supermemory.ai") + await mkdir(dirname(gatePath), { recursive: true }) + await writeFile( + gatePath, + JSON.stringify({ + schemaVersion: 1, + generatedAt: "2026-07-28T00:00:00.000Z", + baseUrl: "https://api.supermemory.ai", + identity: { + buildId: "preflight-test", + containerTag: "preflight-test", + runFingerprint: "preflight-test", + }, + searchContract: { + searchMode: "hybrid", + standaloneChunksExpected: true, + deprecatedIncludeChunks: false, + requestedTopK: 50, + }, + checks: [], + allPassed: true, + blockers: [], + requestBudget: {}, + }) + ) + const handler = createLongMemEvalV2ControlHandler({ + runsRoot: join(root, "runs"), + preflightRoot, + datasetCandidates: [{ path: dataset, source: "env" }], + now: () => Date.parse("2026-07-28T00:00:01.000Z"), + }) + + const response = await call(handler, "GET", "/api/runs-v2/options") + expect(response?.status).toBe(200) + const body = await response!.json() + expect(body.defaults.datasetPath).toBe(resolve(dataset)) + expect(body.defaults.runThrough).toBe("plan") + expect(body.haystacks).toEqual({ + small: { all: 2, web: 1, enterprise: 1, trajectoriesPerBuild: 100 }, + medium: { all: 447, web: 236, enterprise: 211 }, + }) + expect(body.datasets).toEqual([ + { + path: resolve(dataset), + source: "env", + exists: true, + coreFiles: true, + pinnedMarker: false, + screenshots: true, + prepared: true, + }, + ]) + expect(body.preflight).toEqual({ + status: "passing", + baseUrl: "https://api.supermemory.ai", + generatedAt: "2026-07-28T00:00:00.000Z", + expiresAt: "2026-07-29T00:00:00.000Z", + testedTopK: 50, + blockers: [], + }) + expect(typeof body.credentials.supermemoryConfigured).toBe("boolean") + expect(typeof body.credentials.openAIConfigured).toBe("boolean") + expect(body.providers.map((provider: { name: string }) => provider.name)).toEqual([ + "supermemory", + "filesystem", + "rag", + "mem0", + "zep", + ]) + expect( + body.providers.find((provider: { name: string }) => provider.name === "filesystem") + ).toMatchObject({ + adapterAvailable: true, + searchMode: "memories", + requiresPreflight: false, + }) + expect( + body.providers.find((provider: { name: string }) => provider.name === "rag") + ).toMatchObject({ + adapterAvailable: true, + searchMode: "hybrid", + requiresPreflight: false, + }) + expect( + body.providers.find((provider: { name: string }) => provider.name === "mem0") + ).toMatchObject({ + adapterAvailable: false, + capabilities: { plan: true, build: false }, + }) + expect(JSON.stringify(body)).not.toMatch(/(?:sk|sm)_[A-Za-z0-9_-]{12,}/) + }) +}) diff --git a/src/server/routes/longmemeval-v2-control.ts b/src/server/routes/longmemeval-v2-control.ts new file mode 100644 index 0000000..a225c64 --- /dev/null +++ b/src/server/routes/longmemeval-v2-control.ts @@ -0,0 +1,1143 @@ +import { lstat, mkdir, readFile } from "node:fs/promises" +import { resolve } from "node:path" +import { z } from "zod" +import { LONGMEMEVAL_V2_COMPLETION_MARKER } from "../../benchmarks/longmemeval-v2/download" +import { LONGMEMEVAL_V2_PINNED_REVISION } from "../../benchmarks/longmemeval-v2/source" +import { atomicWriteJson } from "../../core/canonical" +import { + LongMemEvalV2Runner, + type LongMemEvalV2ExecuteOptions, + type LongMemEvalV2RunnerOptions, + type LongMemEvalV2RunThrough, +} from "../../orchestrator/longmemeval-v2" +import { BuildAwareRunStore } from "../../orchestrator/build-aware-run-store" +import { + AdvancedSupermemoryProvider, + supermemoryPreflightGatePath, + validateSupermemoryPreflightReport, + type SupermemoryPreflightReport, +} from "../../providers/supermemory/advanced" +import { + createLongMemEvalV2BuildProvider, + isLongMemEvalV2BuildProviderName, +} from "../../providers/build-aware" +import type { BuildAwareRunCheckpoint, BuildAwareRunConfig } from "../../types/build-aware" +import type { ProviderName } from "../../types/provider" +import { config as serverConfig } from "../../utils/config" +import { + endRun as unregisterSharedRun, + isRunActive as isLegacyRunActive, + requestStop as requestSharedStop, + startRun as registerSharedRun, +} from "../runState" + +const SAFE_RUN_ID = /^[A-Za-z0-9_-]{1,100}$/ +const DEFAULT_PREFLIGHT_MAX_AGE_MS = 24 * 60 * 60_000 +const RUN_THROUGH_VALUES = ["plan", "build", "query", "read", "evaluate", "report", "run"] as const + +const positiveInteger = z.number().int().positive() +const boundedString = z + .string() + .trim() + .min(1) + .max(4096) + .refine((value) => !value.includes("\0"), { + message: "must not contain a null byte", + }) +const reasoningEffort = z.enum(["none", "minimal", "low", "medium", "high", "xhigh"]) +const runThrough = z.enum(RUN_THROUGH_VALUES) +const providerName = z.enum(["supermemory", "filesystem", "rag", "mem0", "zep"]) + +const startSchema = z + .object({ + runId: z.string().regex(SAFE_RUN_ID, "must match [A-Za-z0-9_-]+ and be <= 100 characters"), + provider: providerName.default("supermemory"), + datasetPath: boundedString, + tier: z.enum(["small", "medium"]).default("small"), + allowMedium: z.boolean().default(false), + domain: z.enum(["web", "enterprise", "all"]).default("all"), + questionIds: z + .array( + z + .string() + .trim() + .min(1) + .max(200) + .regex(/^[A-Za-z0-9._:-]+$/) + ) + .min(1) + .max(451) + .optional(), + limit: positiveInteger.max(451).optional(), + perCategory: positiveInteger.max(451).optional(), + haystackLimit: positiveInteger.max(447).optional(), + seed: z.string().min(1).max(200).default("memorybench-longmemeval-v2"), + mode: z.enum(["benchmark", "one-trajectory-canary"]).default("benchmark"), + topK: positiveInteger.max(100).default(20), + evidenceTopK: positiveInteger.max(100).default(20), + threshold: z.number().finite().default(0), + readerModel: z.string().trim().min(1).max(100).default("gpt-5"), + evaluatorModel: z.string().trim().min(1).max(100).default("gpt-5"), + reasoningEffort: reasoningEffort.default("high"), + evaluatorReasoningEffort: reasoningEffort.default("high"), + buildConcurrency: positiveInteger.max(20).default(2), + questionConcurrency: positiveInteger.max(100).default(5), + trajectoryConcurrency: positiveInteger.max(100).default(4), + maxInFlightRequests: positiveInteger.max(100).default(20), + maxTrajectoryAttempts: positiveInteger.max(20).default(4), + indexingTimeoutMs: positiveInteger.max(24 * 60 * 60_000).default(30 * 60_000), + strictIngestion: z.boolean().default(false), + runThrough: runThrough.default("plan"), + allowFullRun: z.boolean().default(false), + forceBuild: z.boolean().default(false), + freshQuery: z.boolean().default(false), + }) + .strict() + .superRefine((value, context) => { + if (value.tier === "medium" && value.runThrough !== "plan" && !value.allowMedium) { + context.addIssue({ + code: z.ZodIssueCode.custom, + path: ["allowMedium"], + message: "Medium is an explicit high-cost tier; allowMedium must be true", + }) + } + if (value.evidenceTopK > value.topK) { + context.addIssue({ + code: z.ZodIssueCode.custom, + path: ["evidenceTopK"], + message: "cannot exceed topK", + }) + } + const selectors = [ + value.questionIds, + value.limit, + value.perCategory, + value.haystackLimit, + ].filter((item) => item !== undefined) + if (selectors.length > 1) { + context.addIssue({ + code: z.ZodIssueCode.custom, + path: ["questionIds"], + message: "questionIds, limit, perCategory, and haystackLimit are mutually exclusive", + }) + } + if (selectors.length === 0 && value.runThrough !== "plan" && !value.allowFullRun) { + context.addIssue({ + code: z.ZodIssueCode.custom, + path: ["allowFullRun"], + message: "must be true for a non-plan run with the complete tier selection", + }) + } + if (value.questionIds && new Set(value.questionIds).size !== value.questionIds.length) { + context.addIssue({ + code: z.ZodIssueCode.custom, + path: ["questionIds"], + message: "must not contain duplicates", + }) + } + if (value.mode === "one-trajectory-canary") { + if (value.questionIds?.length !== 1) { + context.addIssue({ + code: z.ZodIssueCode.custom, + path: ["questionIds"], + message: "Canary requires exactly one question ID", + }) + } + if (!["plan", "build", "query"].includes(value.runThrough)) { + context.addIssue({ + code: z.ZodIssueCode.custom, + path: ["runThrough"], + message: "Canary may only plan, build, or query", + }) + } + } + }) + +const resumeSchema = z + .object({ + runThrough: runThrough.optional(), + allowFullRun: z.boolean().default(false), + forceBuild: z.boolean().default(false), + freshQuery: z.boolean().default(false), + }) + .strict() +const emptyBodySchema = z.object({}).strict() +const preflightSchema = z + .object({ + topK: positiveInteger.max(100).default(20), + }) + .strict() + +type StartInput = z.infer +type ResumeInput = z.infer + +interface RunnerLike { + execute(options?: LongMemEvalV2ExecuteOptions): Promise +} + +interface ActiveRun { + controller: AbortController + status: "running" | "stopping" + startedAt: string +} + +interface DatasetCandidate { + path: string + source: "env" | "repo" | "sibling" +} + +export interface LongMemEvalV2ControlEvent { + action: "start" | "resume" | "stop-request" | "completed" | "failed" | "stopped" + at: string + through?: LongMemEvalV2RunThrough + message?: string + provider?: ProviderName + forceBuild?: boolean + freshQuery?: boolean +} + +export interface LongMemEvalV2ControlHistory { + schemaVersion: 1 + runId: string + events: LongMemEvalV2ControlEvent[] +} + +export interface LongMemEvalV2ControlRouteOptions { + runsRoot?: string + buildsRoot?: string + artifactsRoot?: string + preflightRoot?: string + serviceBaseUrl?: string + datasetCandidates?: DatasetCandidate[] + runnerFactory?: (options: LongMemEvalV2RunnerOptions) => RunnerLike | Promise + preflightRunner?: (input: { topK: number }) => Promise + broadcast?: (message: Record) => void + isLegacyRunActive?: (runId: string) => boolean + now?: () => number +} + +function json(data: unknown, status = 200): Response { + return new Response(JSON.stringify(data), { + status, + headers: { "Content-Type": "application/json" }, + }) +} + +function redact(message: string): string { + return message.replace(/\b(?:sk|sm)_[A-Za-z0-9_-]{12,}\b/g, "[REDACTED]") +} + +function errorMessage(error: unknown): string { + return redact(error instanceof Error ? error.message : String(error)) +} + +function safeDecodeRunId(raw: string): string | null { + try { + const decoded = decodeURIComponent(raw) + return SAFE_RUN_ID.test(decoded) ? decoded : null + } catch { + return null + } +} + +function validLocalOrigin(request: Request): boolean { + const raw = request.headers.get("origin") + if (!raw) return true + try { + const origin = new URL(raw) + return ( + origin.protocol === "http:" && ["localhost", "127.0.0.1", "[::1]"].includes(origin.hostname) + ) + } catch { + return false + } +} + +async function parseBody( + request: Request, + schema: T +): Promise> { + let value: unknown + try { + value = await request.json() + } catch { + throw new Error("Request body must be valid JSON") + } + const result = schema.safeParse(value) + if (!result.success) { + const issue = result.error.issues[0] + const field = issue.path.length > 0 ? `${issue.path.join(".")}: ` : "" + throw new Error(`Invalid request body: ${field}${issue.message}`) + } + return result.data +} + +async function parseOptionalEmptyBody(request: Request): Promise { + const text = await request.text() + if (!text.trim()) return + let value: unknown + try { + value = JSON.parse(text) + } catch { + throw new Error("Request body must be valid JSON") + } + const result = emptyBodySchema.safeParse(value) + if (!result.success) throw new Error("Stop request body must be empty") +} + +function nativeThrough(value: (typeof RUN_THROUGH_VALUES)[number]): LongMemEvalV2RunThrough { + return value === "run" ? "report" : value +} + +function readerBudgets(model: string): { + maxCompletionTokens: number + maxContextTokens: number +} { + if (/^gpt-4o(?:-|$)/i.test(model)) { + return { maxCompletionTokens: 8_000, maxContextTokens: 120_000 } + } + if (/^gpt-4(?:\.|-|$)/i.test(model)) { + return { maxCompletionTokens: 16_000, maxContextTokens: 200_000 } + } + return { maxCompletionTokens: 20_000, maxContextTokens: 200_000 } +} + +function retrievalProfile(provider: ProviderName): { + searchMode: "hybrid" | "memories" + rerank: boolean +} { + if (provider === "filesystem" || provider === "mem0" || provider === "zep") { + return { searchMode: "memories", rerank: provider === "zep" } + } + return { searchMode: "hybrid", rerank: provider === "supermemory" } +} + +function configFrom(input: StartInput, serviceBaseUrl: string): BuildAwareRunConfig { + const budgets = readerBudgets(input.readerModel) + const retrieval = retrievalProfile(input.provider) + return { + provider: input.provider, + benchmark: "longmemeval-v2", + mode: input.mode, + datasetPath: resolve(input.datasetPath), + datasetRevision: LONGMEMEVAL_V2_PINNED_REVISION, + tier: input.tier, + domain: input.domain, + questionIds: input.questionIds, + limit: input.limit, + perCategory: input.perCategory, + haystackLimit: input.haystackLimit, + seed: input.seed, + retrieval: { + topK: input.topK, + threshold: input.threshold, + searchMode: retrieval.searchMode, + rerank: retrieval.rerank, + rewriteQuery: false, + includeSummaries: true, + includeChunks: true, + includeDocuments: true, + includeRelatedMemories: true, + metadataFilter: {}, + }, + reader: { + model: input.readerModel, + reasoningEffort: input.reasoningEffort, + maxCompletionTokens: budgets.maxCompletionTokens, + maxContextTokens: budgets.maxContextTokens, + evidenceTopK: input.evidenceTopK, + maxImages: 100, + maxImageBytes: 20 * 1024 * 1024, + malformedResponseAttempts: 3, + }, + evaluator: { + model: input.evaluatorModel, + reasoningEffort: input.evaluatorReasoningEffort, + maxCompletionTokens: 4096, + }, + build: { + serviceBaseUrl, + dreaming: "instant", + rootFilterMode: "self", + maxDocumentChars: 200_000, + trajectoryConcurrency: input.trajectoryConcurrency, + maxInFlightRequests: input.maxInFlightRequests, + maxTrajectoryAttempts: input.maxTrajectoryAttempts, + indexingTimeoutMs: input.indexingTimeoutMs, + pollIntervalMs: 2_000, + preflightMaxAgeMs: DEFAULT_PREFLIGHT_MAX_AGE_MS, + continueOnIndexingTimeout: !input.strictIngestion, + }, + execution: { + buildConcurrency: input.buildConcurrency, + questionConcurrency: input.questionConcurrency, + }, + } +} + +async function checkpointStatus(store: BuildAwareRunStore, active: boolean, uiManaged: boolean) { + try { + const checkpoint = await store.load() + const interrupted = checkpoint.status === "running" && !active && uiManaged + return { + status: interrupted ? "failed" : checkpoint.status, + currentStage: checkpoint.currentStage, + updatedAt: checkpoint.updatedAt, + ...(interrupted + ? { error: "Run process is no longer active; resume from the durable checkpoint" } + : checkpoint.error + ? { error: redact(checkpoint.error) } + : {}), + } + } catch { + return null + } +} + +async function pathKind(path: string): Promise<"file" | "directory" | "missing"> { + try { + const metadata = await lstat(path) + if (metadata.isFile()) return "file" + if (metadata.isDirectory()) return "directory" + return "missing" + } catch { + return "missing" + } +} + +async function inspectDataset(candidate: DatasetCandidate) { + const path = resolve(candidate.path) + const markerPath = resolve(path, LONGMEMEVAL_V2_COMPLETION_MARKER) + let pinnedMarker = false + try { + const marker = JSON.parse(await readFile(markerPath, "utf8")) as Record + pinnedMarker = marker.schemaVersion === 1 && marker.revision === LONGMEMEVAL_V2_PINNED_REVISION + } catch { + // A marker is an optimization signal; the loader still performs authoritative validation. + } + const [rootKind, questions, trajectories, haystack, screenshots] = await Promise.all([ + pathKind(path), + pathKind(resolve(path, "questions.jsonl")), + pathKind(resolve(path, "trajectories.jsonl")), + pathKind(resolve(path, "haystacks/lme_v2_small.json")), + pathKind(resolve(path, "screenshots")), + ]) + const coreFiles = questions === "file" && trajectories === "file" && haystack === "file" + const hasScreenshots = screenshots === "directory" + return { + path, + source: candidate.source, + exists: rootKind === "directory", + coreFiles, + pinnedMarker, + screenshots: hasScreenshots, + prepared: coreFiles && hasScreenshots, + } +} + +async function inspectPreflight(input: { root: string; baseUrl: string; now: number }): Promise<{ + status: "passing" | "missing" | "invalid" | "expired" + baseUrl: string + generatedAt?: string + expiresAt?: string + testedTopK?: number + blockers?: string[] +}> { + const path = supermemoryPreflightGatePath(input.root, input.baseUrl) + let report: SupermemoryPreflightReport + try { + report = JSON.parse(await readFile(path, "utf8")) as SupermemoryPreflightReport + } catch { + return { status: "missing", baseUrl: input.baseUrl } + } + const generated = Date.parse(report.generatedAt) + const expiresAt = Number.isFinite(generated) + ? new Date(generated + DEFAULT_PREFLIGHT_MAX_AGE_MS).toISOString() + : undefined + const common = { + baseUrl: input.baseUrl, + ...(typeof report.generatedAt === "string" ? { generatedAt: report.generatedAt } : {}), + ...(expiresAt ? { expiresAt } : {}), + ...(typeof report.searchContract?.requestedTopK === "number" + ? { testedTopK: report.searchContract.requestedTopK } + : {}), + ...(Array.isArray(report.blockers) ? { blockers: report.blockers.map(String) } : {}), + } + if ( + Number.isFinite(generated) && + (input.now < generated || input.now - generated > DEFAULT_PREFLIGHT_MAX_AGE_MS) + ) { + return { status: "expired", ...common } + } + try { + validateSupermemoryPreflightReport(report, { + baseUrl: input.baseUrl, + requiredTopK: 20, + maxAgeMs: DEFAULT_PREFLIGHT_MAX_AGE_MS, + now: input.now, + }) + return { status: "passing", ...common } + } catch { + return { status: "invalid", ...common } + } +} + +function configuredDatasetCandidates(): DatasetCandidate[] { + const candidates: DatasetCandidate[] = [] + const environmentPath = process.env.LONGMEMEVAL_V2_DATASET_PATH?.trim() + if (environmentPath && !environmentPath.includes("\0")) { + candidates.push({ path: environmentPath, source: "env" }) + } + candidates.push( + { path: "data/benchmarks/longmemeval-v2", source: "repo" }, + { path: "../LongMemEval-V2/data/longmemeval-v2", source: "sibling" } + ) + const unique = new Set() + return candidates.filter((candidate) => { + const path = resolve(candidate.path) + if (unique.has(path)) return false + unique.add(path) + return true + }) +} + +export function createLongMemEvalV2ControlHandler(options: LongMemEvalV2ControlRouteOptions = {}) { + const runsRoot = resolve(options.runsRoot ?? "data/runs-v2") + const buildsRoot = resolve(options.buildsRoot ?? "data/memory-builds-v2") + const artifactsRoot = resolve(options.artifactsRoot ?? "data/artifacts-v2") + const preflightRoot = resolve(options.preflightRoot ?? "data/preflights-v2") + const serviceBaseUrl = options.serviceBaseUrl ?? serverConfig.supermemoryBaseUrl + const runnerFactory = options.runnerFactory + const broadcast = options.broadcast ?? (() => {}) + const legacyActive = options.isLegacyRunActive ?? isLegacyRunActive + const now = options.now ?? Date.now + const activeRuns = new Map() + const controlQueues = new Map>() + + const createRunner = async ( + runnerOptions: LongMemEvalV2RunnerOptions, + through: LongMemEvalV2RunThrough + ): Promise => { + if (through === "plan") { + return runnerFactory ? runnerFactory(runnerOptions) : new LongMemEvalV2Runner(runnerOptions) + } + if (!isLongMemEvalV2BuildProviderName(runnerOptions.config.provider)) { + throw new Error( + `${runnerOptions.config.provider} does not yet have a safe LongMemEval-V2 adapter; use Plan only` + ) + } + if (runnerFactory) return runnerFactory(runnerOptions) + const provider = await createLongMemEvalV2BuildProvider({ + provider: runnerOptions.config.provider, + serviceBaseUrl: runnerOptions.config.build.serviceBaseUrl, + maxInFlightRequests: runnerOptions.config.build.maxInFlightRequests, + operationTimeoutMs: runnerOptions.config.build.indexingTimeoutMs, + signal: runnerOptions.signal, + }) + return new LongMemEvalV2Runner({ + ...runnerOptions, + provider, + requirePreflight: runnerOptions.config.provider === "supermemory", + }) + } + let preflightActivity: + | { status: "idle" } + | { status: "running"; startedAt: string; topK: number } + | { status: "passed"; startedAt: string; completedAt: string; topK: number } + | { status: "failed"; startedAt: string; completedAt: string; topK: number; error: string } = { + status: "idle", + } + const runPreflight = + options.preflightRunner ?? + (async ({ topK }: { topK: number }) => { + const apiKey = process.env.SUPERMEMORY_API_KEY?.trim() + if (!apiKey) throw new Error("SUPERMEMORY_API_KEY is required for preflight") + const provider = new AdvancedSupermemoryProvider({ + apiKey, + baseUrl: serviceBaseUrl, + maxInFlightRequests: 20, + }) + return provider.preflight({ + searchTopK: topK, + readinessTimeoutMs: 5 * 60_000, + searchVisibilityTimeoutMs: 2 * 60_000, + searchPollMs: 5_000, + keepDocuments: false, + }) + }) + + const readControlHistory = async (runId: string): Promise => { + const store = new BuildAwareRunStore(runId, runsRoot) + try { + const history = JSON.parse( + await readFile(resolve(store.runRoot, "control.json"), "utf8") + ) as LongMemEvalV2ControlHistory + if (history.schemaVersion === 1 && history.runId === runId && Array.isArray(history.events)) { + return history + } + } catch { + // A control history is supplementary to the durable benchmark checkpoint. + } + return { schemaVersion: 1, runId, events: [] } + } + + const appendControlEvent = async ( + runId: string, + event: LongMemEvalV2ControlEvent + ): Promise => { + const previous = controlQueues.get(runId) ?? Promise.resolve() + const next = previous.then(async () => { + const store = new BuildAwareRunStore(runId, runsRoot) + await mkdir(store.runRoot, { recursive: true }) + const history = await readControlHistory(runId) + history.events = [...history.events, event].slice(-100) + await atomicWriteJson(resolve(store.runRoot, "control.json"), history) + }) + controlQueues.set(runId, next) + try { + await next + } finally { + if (controlQueues.get(runId) === next) controlQueues.delete(runId) + } + } + + const release = (runId: string, controller: AbortController): void => { + if (activeRuns.get(runId)?.controller === controller) { + activeRuns.delete(runId) + unregisterSharedRun(runId) + } + } + + const reserve = (runId: string, controller: AbortController): void => { + activeRuns.set(runId, { + controller, + status: "running", + startedAt: new Date(now()).toISOString(), + }) + registerSharedRun(runId, "longmemeval-v2", controller) + } + + const execute = ( + runId: string, + controller: AbortController, + runner: RunnerLike, + executeOptions: LongMemEvalV2ExecuteOptions, + through: LongMemEvalV2RunThrough + ): void => { + void Promise.resolve() + .then(() => runner.execute(executeOptions)) + .then(async () => { + await appendControlEvent(runId, { + action: "completed", + at: new Date(now()).toISOString(), + through, + }) + broadcast({ type: "run_complete", runId }) + }) + .catch(async (error) => { + const stopped = controller.signal.aborted + await appendControlEvent(runId, { + action: stopped ? "stopped" : "failed", + at: new Date(now()).toISOString(), + through, + message: stopped ? "Run stopped by user" : errorMessage(error), + }) + broadcast({ + type: stopped ? "run_stopped" : "error", + runId, + message: stopped ? "Run stopped by user" : errorMessage(error), + }) + }) + .finally(() => release(runId, controller)) + } + + return async function handleLongMemEvalV2ControlRoutes( + request: Request, + url: URL + ): Promise { + const method = request.method + const pathname = url.pathname + + if (method === "GET" && pathname === "/api/runs-v2/options") { + const candidates = options.datasetCandidates ?? configuredDatasetCandidates() + const datasets = await Promise.all(candidates.map(inspectDataset)) + const selected = datasets.find((candidate) => candidate.prepared) ?? datasets[0] + const preflight = await inspectPreflight({ + root: preflightRoot, + baseUrl: serviceBaseUrl, + now: now(), + }) + const supermemoryConfigured = Boolean(process.env.SUPERMEMORY_API_KEY?.trim()) + const openAIConfigured = Boolean(process.env.OPENAI_API_KEY?.trim()) + const mem0Configured = Boolean(process.env.MEM0_API_KEY?.trim()) + const zepConfigured = Boolean(process.env.ZEP_API_KEY?.trim()) + const providerReady = supermemoryConfigured && preflight.status === "passing" + const providerDescriptors = [ + { + name: "supermemory", + displayName: "Supermemory", + adapterAvailable: true, + configured: supermemoryConfigured, + requiresPreflight: true, + searchMode: "hybrid", + rerank: true, + note: "Reference V2 adapter with remote reconciliation and verified screenshot provenance.", + capabilities: { + plan: true, + build: providerReady, + query: providerReady, + read: providerReady && openAIConfigured, + evaluate: providerReady && openAIConfigured, + report: providerReady && openAIConfigured, + }, + }, + ...[ + ["filesystem", "Filesystem", "memories"], + ["rag", "Local RAG", "hybrid"], + ].map(([name, displayName, searchMode]) => ({ + name, + displayName, + adapterAvailable: true, + configured: openAIConfigured, + requiresPreflight: false, + searchMode, + rerank: false, + note: + name === "filesystem" + ? "Durable MEMORY.md-style extraction with exact sidecar reconciliation." + : "Durable hybrid retrieval backed by a per-build SQLite/WAL index.", + capabilities: { + plan: true, + build: openAIConfigured, + query: openAIConfigured, + read: openAIConfigured, + evaluate: openAIConfigured, + report: openAIConfigured, + }, + })), + { + name: "mem0", + displayName: "Mem0", + adapterAvailable: false, + configured: mem0Configured, + requiresPreflight: false, + searchMode: "memories", + rerank: false, + note: "Plan only: async event identity and exact interrupted-ingestion cleanup still need a live contract adapter.", + capabilities: { + plan: true, + build: false, + query: false, + read: false, + evaluate: false, + report: false, + }, + }, + { + name: "zep", + displayName: "Zep", + adapterAvailable: false, + configured: zepConfigured, + requiresPreflight: false, + searchMode: "memories", + rerank: true, + note: "Plan only: exact episode reconciliation, provenance, and individual cleanup are not yet proven.", + capabilities: { + plan: true, + build: false, + query: false, + read: false, + evaluate: false, + report: false, + }, + }, + ] + return json({ + defaults: { + provider: "supermemory", + datasetPath: selected?.path ?? null, + tier: "small", + domain: "all", + mode: "benchmark", + topK: 20, + evidenceTopK: 20, + reasoningEffort: "high", + readerModel: "gpt-5", + evaluatorModel: "gpt-5", + buildConcurrency: 2, + questionConcurrency: 5, + trajectoryConcurrency: 4, + maxInFlightRequests: 20, + maxTrajectoryAttempts: 4, + indexingTimeoutMs: 30 * 60_000, + strictIngestion: false, + runThrough: "plan", + }, + haystacks: { + small: { all: 2, web: 1, enterprise: 1, trajectoriesPerBuild: 100 }, + medium: { all: 447, web: 236, enterprise: 211 }, + }, + datasets, + providers: providerDescriptors, + credentials: { + supermemoryConfigured, + openAIConfigured, + mem0Configured, + zepConfigured, + }, + preflight, + preflightActivity, + capabilities: { + plan: true, + build: providerReady, + query: providerReady, + read: providerReady && openAIConfigured, + evaluate: providerReady && openAIConfigured, + report: providerReady && openAIConfigured, + }, + }) + } + + if (method === "POST" && pathname === "/api/runs-v2/preflight") { + if (!validLocalOrigin(request)) return json({ error: "Origin is not allowed" }, 403) + let input: z.infer + try { + input = await parseBody(request, preflightSchema) + } catch (error) { + return json({ error: errorMessage(error) }, 400) + } + if (preflightActivity.status === "running") { + return json({ error: "A Supermemory preflight is already running" }, 409) + } + if (!options.preflightRunner && !process.env.SUPERMEMORY_API_KEY?.trim()) { + return json({ error: "SUPERMEMORY_API_KEY is not configured on the server" }, 400) + } + const startedAt = new Date(now()).toISOString() + preflightActivity = { status: "running", startedAt, topK: input.topK } + void runPreflight({ topK: input.topK }) + .then(async (report) => { + if (!report.allPassed) { + throw new Error( + `Supermemory preflight failed${report.blockers.length ? `: ${report.blockers.join(", ")}` : ""}` + ) + } + await atomicWriteJson(supermemoryPreflightGatePath(preflightRoot, serviceBaseUrl), report) + preflightActivity = { + status: "passed", + startedAt, + completedAt: new Date(now()).toISOString(), + topK: input.topK, + } + broadcast({ type: "longmemeval_v2_preflight_complete", topK: input.topK }) + }) + .catch((error) => { + preflightActivity = { + status: "failed", + startedAt, + completedAt: new Date(now()).toISOString(), + topK: input.topK, + error: errorMessage(error), + } + broadcast({ + type: "error", + scope: "longmemeval-v2-preflight", + message: errorMessage(error), + }) + }) + return json( + { + message: "Supermemory preflight started", + statusUrl: "/api/runs-v2/options", + }, + 202 + ) + } + + if (method === "POST" && pathname === "/api/runs-v2/start") { + if (!validLocalOrigin(request)) return json({ error: "Origin is not allowed" }, 403) + let input: StartInput + try { + input = await parseBody(request, startSchema) + } catch (error) { + return json({ error: errorMessage(error) }, 400) + } + if (activeRuns.has(input.runId) || legacyActive(input.runId)) { + return json({ error: "Run is already active" }, 409) + } + const controller = new AbortController() + reserve(input.runId, controller) + const store = new BuildAwareRunStore(input.runId, runsRoot) + if (await store.exists()) { + release(input.runId, controller) + return json({ error: `Run ${input.runId} already exists; use resume` }, 409) + } + const config = configFrom(input, serviceBaseUrl) + const requestedThrough = nativeThrough(input.runThrough) + let runner: RunnerLike + try { + runner = await createRunner( + { + runId: input.runId, + config, + runRoot: runsRoot, + buildRoot: buildsRoot, + cacheRoot: artifactsRoot, + preflightRoot, + signal: controller.signal, + }, + requestedThrough + ) + } catch (error) { + release(input.runId, controller) + return json({ error: errorMessage(error) }, 400) + } + try { + // The UI redirects as soon as this endpoint returns. Persist the initial + // checkpoint first so the run-detail route is immediately inspectable. + await store.createOrLoad(config) + } catch (error) { + release(input.runId, controller) + return json({ error: errorMessage(error) }, 500) + } + try { + await appendControlEvent(input.runId, { + action: "start", + at: new Date(now()).toISOString(), + through: requestedThrough, + provider: config.provider, + forceBuild: input.forceBuild, + freshQuery: input.freshQuery || input.forceBuild, + }) + } catch (error) { + try { + const checkpoint = await store.load() + await store.fail(checkpoint, error) + } catch { + // Preserve the original control-history error in the response. + } + release(input.runId, controller) + return json({ error: errorMessage(error) }, 500) + } + broadcast({ + type: "run_started", + runId: input.runId, + provider: config.provider, + benchmark: "longmemeval-v2", + }) + execute( + input.runId, + controller, + runner, + { + through: requestedThrough, + forceBuild: input.forceBuild, + freshQuery: input.freshQuery || input.forceBuild, + }, + requestedThrough + ) + return json( + { + message: "Run started", + runId: input.runId, + statusUrl: `/api/runs-v2/${encodeURIComponent(input.runId)}/status`, + runUrl: `/api/runs/${encodeURIComponent(input.runId)}`, + }, + 202 + ) + } + + const resumeMatch = pathname.match(/^\/api\/runs-v2\/([^/]+)\/resume$/) + if (method === "POST" && resumeMatch) { + if (!validLocalOrigin(request)) return json({ error: "Origin is not allowed" }, 403) + const runId = safeDecodeRunId(resumeMatch[1]) + if (!runId) return json({ error: "Invalid run ID" }, 400) + let input: ResumeInput + try { + input = await parseBody(request, resumeSchema) + } catch (error) { + return json({ error: errorMessage(error) }, 400) + } + if (activeRuns.has(runId) || legacyActive(runId)) { + return json({ error: "Run is already active" }, 409) + } + const controller = new AbortController() + reserve(runId, controller) + const store = new BuildAwareRunStore(runId, runsRoot) + let checkpoint: BuildAwareRunCheckpoint + try { + checkpoint = await store.load() + } catch { + release(runId, controller) + return json({ error: "Run not found" }, 404) + } + const control = await readControlHistory(runId) + const priorTarget = [...control.events] + .reverse() + .find( + (event) => + (event.action === "start" || event.action === "resume") && event.through !== undefined + )?.through + const requestedThrough = nativeThrough( + input.runThrough ?? priorTarget ?? checkpoint.currentStage + ) + if ( + checkpoint.config.mode === "one-trajectory-canary" && + !["plan", "build", "query"].includes(requestedThrough) + ) { + release(runId, controller) + return json({ error: "Canary may only plan, build, or query" }, 400) + } + const stageOrder: LongMemEvalV2RunThrough[] = [ + "plan", + "build", + "query", + "read", + "evaluate", + "report", + ] + if (checkpoint.status === "completed" && checkpoint.currentStage === "report") { + release(runId, controller) + return json({ error: "Completed report runs cannot be resumed" }, 409) + } + if (stageOrder.indexOf(requestedThrough) < stageOrder.indexOf(checkpoint.currentStage)) { + release(runId, controller) + return json( + { + error: `Resume target ${requestedThrough} cannot be earlier than checkpoint stage ${checkpoint.currentStage}`, + }, + 400 + ) + } + if ( + checkpoint.status === "completed" && + stageOrder.indexOf(requestedThrough) === stageOrder.indexOf(checkpoint.currentStage) + ) { + release(runId, controller) + return json({ error: `Stage ${checkpoint.currentStage} is already completed` }, 409) + } + const fullScope = + checkpoint.config.questionIds === undefined && + checkpoint.config.limit === undefined && + checkpoint.config.perCategory === undefined && + checkpoint.config.haystackLimit === undefined + if (fullScope && requestedThrough !== "plan" && !input.allowFullRun) { + release(runId, controller) + return json( + { + error: + "allowFullRun must be true to resume or continue a complete-tier selection beyond Plan", + }, + 400 + ) + } + let runner: RunnerLike + try { + runner = await createRunner( + { + runId, + config: checkpoint.config, + runRoot: runsRoot, + buildRoot: buildsRoot, + cacheRoot: artifactsRoot, + preflightRoot, + signal: controller.signal, + }, + requestedThrough + ) + } catch (error) { + release(runId, controller) + return json({ error: errorMessage(error) }, 400) + } + try { + await appendControlEvent(runId, { + action: "resume", + at: new Date(now()).toISOString(), + through: requestedThrough, + provider: checkpoint.config.provider, + forceBuild: input.forceBuild, + freshQuery: input.freshQuery || input.forceBuild, + }) + } catch (error) { + release(runId, controller) + return json({ error: errorMessage(error) }, 500) + } + broadcast({ + type: "run_started", + runId, + provider: checkpoint.config.provider, + benchmark: "longmemeval-v2", + resumed: true, + }) + execute( + runId, + controller, + runner, + { + through: requestedThrough, + forceBuild: input.forceBuild, + freshQuery: input.freshQuery || input.forceBuild, + }, + requestedThrough + ) + return json( + { + message: "Run resumed", + runId, + statusUrl: `/api/runs-v2/${encodeURIComponent(runId)}/status`, + runUrl: `/api/runs/${encodeURIComponent(runId)}`, + }, + 202 + ) + } + + const stopMatch = pathname.match(/^\/api\/runs-v2\/([^/]+)\/stop$/) + if (method === "POST" && stopMatch) { + if (!validLocalOrigin(request)) return json({ error: "Origin is not allowed" }, 403) + try { + await parseOptionalEmptyBody(request) + } catch (error) { + return json({ error: errorMessage(error) }, 400) + } + const runId = safeDecodeRunId(stopMatch[1]) + if (!runId) return json({ error: "Invalid run ID" }, 400) + const active = activeRuns.get(runId) + if (!active) return json({ error: "Run is not active" }, 404) + active.status = "stopping" + requestSharedStop(runId) + await appendControlEvent(runId, { + action: "stop-request", + at: new Date(now()).toISOString(), + }) + broadcast({ type: "run_stopping", runId }) + return json({ message: "Stop requested", runId }, 202) + } + + const statusMatch = pathname.match(/^\/api\/runs-v2\/([^/]+)\/status$/) + if (method === "GET" && statusMatch) { + const runId = safeDecodeRunId(statusMatch[1]) + if (!runId) return json({ error: "Invalid run ID" }, 400) + const active = activeRuns.get(runId) + const control = await readControlHistory(runId) + const uiManaged = control.events.some( + (event) => event.action === "start" || event.action === "resume" + ) + const checkpoint = await checkpointStatus( + new BuildAwareRunStore(runId, runsRoot), + Boolean(active), + uiManaged + ) + if (!active && !checkpoint && control.events.length === 0) { + return json({ error: "Run not found" }, 404) + } + return json({ + runId, + active: Boolean(active), + stopping: active?.status === "stopping" || active?.controller.signal.aborted === true, + checkpoint, + control, + }) + } + + return null + } +} diff --git a/src/server/routes/runs.ts b/src/server/routes/runs.ts index 1aaab7b..5e26f32 100644 --- a/src/server/routes/runs.ts +++ b/src/server/routes/runs.ts @@ -10,6 +10,10 @@ import type { BenchmarkName } from "../../types/benchmark" import type { PhaseId, SamplingConfig } from "../../types/checkpoint" import type { ConcurrencyConfig } from "../../types/concurrency" import { getPhasesFromPhase, PHASE_ORDER } from "../../types/checkpoint" +import { + handleBuildAwareInspectionRoutes, + listBuildAwareRunSummaries, +} from "./build-aware-inspection" const checkpointManager = new CheckpointManager() @@ -34,10 +38,15 @@ export async function handleRunsRoutes(req: Request, url: URL): Promise { const checkpoint = checkpointManager.load(runId) if (!checkpoint) return null @@ -68,7 +77,9 @@ export async function handleRunsRoutes(req: Request, url: URL): Promise (b.createdAt || "").localeCompare(a.createdAt || "")) + const runDetails = [...legacyRunDetails, ...(await listBuildAwareRunSummaries())].sort( + (a: any, b: any) => (b.createdAt || "").localeCompare(a.createdAt || "") + ) return json(runDetails) } diff --git a/src/server/runState.ts b/src/server/runState.ts index 663ac3b..499bfe1 100644 --- a/src/server/runState.ts +++ b/src/server/runState.ts @@ -5,6 +5,7 @@ export type RunState = { status: "running" | "stopping" startedAt: string benchmark?: string + abortController?: AbortController } // In-memory map of active runs @@ -21,15 +22,23 @@ export function requestStop(runId: string): boolean { const state = activeRuns.get(runId) if (!state) return false state.status = "stopping" + if (!state.abortController?.signal.aborted) { + state.abortController?.abort(new Error("Run stopped by user")) + } return true } // Start tracking a run -export function startRun(runId: string, benchmark?: string): void { +export function startRun( + runId: string, + benchmark?: string, + abortController?: AbortController +): void { activeRuns.set(runId, { status: "running", startedAt: new Date().toISOString(), benchmark, + abortController, }) } diff --git a/src/types/build-aware.ts b/src/types/build-aware.ts new file mode 100644 index 0000000..5151f6e --- /dev/null +++ b/src/types/build-aware.ts @@ -0,0 +1,155 @@ +import type { + EvaluationArtifact, + QueryArtifact, + ReaderArtifact, + RetrievalConfig, +} from "./migration" +import type { LongMemEvalV2OfficialAggregate } from "../benchmarks/longmemeval-v2/evaluation" +import type { ProviderName } from "./provider" + +export type BuildAwareStage = "plan" | "build" | "query" | "read" | "evaluate" | "report" +export type BuildAwareStageStatus = "pending" | "running" | "completed" | "failed" | "blocked" + +export interface StageState { + status: BuildAwareStageStatus + fingerprint?: string + artifactPath?: string + error?: string + startedAt?: string + completedAt?: string + durationMs?: number + cacheHit?: boolean +} + +export interface BuildAwareQuestionCheckpoint { + questionId: string + questionType: string + question: string + groundTruth: string + evalFunction: string + buildId: string + questionImageHash?: string + stages: { + query: StageState + read: StageState + evaluate: StageState + } + queryArtifact?: QueryArtifact + readerArtifact?: ReaderArtifact + evaluationArtifact?: EvaluationArtifact +} + +export interface BuildAwareRunConfig { + provider: ProviderName + benchmark: "longmemeval-v2" + mode: "benchmark" | "one-trajectory-canary" + datasetPath: string + datasetRevision: string + tier: "small" | "medium" + domain: "web" | "enterprise" | "all" + questionIds?: string[] + limit?: number + perCategory?: number + /** Keep the first N exact builds in pinned question order. */ + haystackLimit?: number + seed: string + retrieval: RetrievalConfig + reader: { + model: string + reasoningEffort: "none" | "minimal" | "low" | "medium" | "high" | "xhigh" + maxCompletionTokens: number + maxContextTokens: number + evidenceTopK: number + maxImages: number + maxImageBytes: number + malformedResponseAttempts: number + } + evaluator: { + model: string + reasoningEffort: string + maxCompletionTokens: number + } + build: { + serviceBaseUrl: string + dreaming: "instant" + rootFilterMode: "self" + maxDocumentChars: number + trajectoryConcurrency: number + maxInFlightRequests: number + maxTrajectoryAttempts: number + indexingTimeoutMs: number + pollIntervalMs: number + preflightMaxAgeMs: number + continueOnIndexingTimeout?: boolean + } + execution: { + buildConcurrency: number + questionConcurrency: number + } +} + +export interface BuildAwareRunCheckpoint { + schemaVersion: 1 + executionModel: "shared-memory-build-v1" + runId: string + configFingerprint: string + status: BuildAwareStageStatus + currentStage: BuildAwareStage + config: BuildAwareRunConfig + datasetManifestPath?: string + datasetFingerprint?: string + artifactRoot?: string + buildRoot?: string + preflightGate?: { + schemaVersion: 1 + reportFingerprint: string + generatedAt: string + baseUrl: string + testedTopK: number + } + targetQuestionIds: string[] + buildIds: string[] + buildLinks: Record + questions: Record + createdAt: string + updatedAt: string + error?: string +} + +export interface BuildAwareReport { + schemaVersion: 1 + protocol: "longmemeval-v2-official" + runId: string + benchmark: "longmemeval-v2" + provider: ProviderName + converter: "Structured Accessibility Converter" + targetQuestionCount: number + completedQuestionCount: number + failedQuestionCount: number + officiallyComparable: boolean + ineligibilityReasons: string[] + buildIds: string[] + builds: Array<{ + buildId: string + buildFingerprint: string + containerTag: string + domain: string + trajectoryCount: number + documentCount: number + linkedQuestionIds: string[] + reused: boolean + status: "ready" | "degraded" + skippedTrajectoryCount: number + skippedDocumentCount: number + }> + official: LongMemEvalV2OfficialAggregate + diagnostics: { + queryCacheHits: number + readerCacheHits: number + remoteSearchLatencyMs: number[] + queryWallLatencyMs: number[] + contextImagesSent: number + failedQuestions: Array<{ questionId: string; stage: string; error: string }> + } + createdAt: string +} diff --git a/src/types/index.ts b/src/types/index.ts index 27aafe8..4c4a644 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -3,3 +3,5 @@ export * from "./provider" export * from "./benchmark" export * from "./judge" export * from "./checkpoint" +export * from "./migration" +export * from "./build-aware" diff --git a/src/types/migration.ts b/src/types/migration.ts new file mode 100644 index 0000000..9ce8698 --- /dev/null +++ b/src/types/migration.ts @@ -0,0 +1,233 @@ +export type BenchmarkDomain = "web" | "enterprise" | string +export type DatasetTier = "small" | "medium" | string + +export interface AssetRef { + assetId: string + kind: "question-image" | "trajectory-screenshot" + /** Runtime-only resolved path. It is excluded from persisted identities. */ + absolutePath?: string + relativePath: string + mimeType: string + sha256: string + byteLength: number +} + +export interface DatasetFileManifest { + relativePath: string + sha256: string + byteLength: number +} + +export interface DatasetManifest { + schemaVersion: number + benchmark: string + source: string + revision: string + dataRoot: string + tier: DatasetTier + domain?: BenchmarkDomain + files: DatasetFileManifest[] + /** Assets used by this manifest/run. Every entry is content addressed. */ + assets: AssetRef[] + assetScope?: "selected-run" | "full-dataset" + assetsFingerprint?: string + questionOrder: string[] + trajectoryOrder: string[] + expectedCounts: { + questions: number + trajectories: number + states: number + assets: number + uniqueBuilds: number + } + fingerprint: string +} + +export type MetadataValue = string | number | boolean | string[] + +export interface DocumentSpec { + logicalDocumentId: string + content: string + metadata: Record + sourceStateIndices: number[] + localAttachmentPaths: string[] + dependsOn: string[] + allowParallelUpload: boolean + documentType: string + stateIndex?: number + step?: number + screenshotRef?: AssetRef + allowDuplicateContent: boolean +} + +export interface DocumentPlan { + trajectoryId: string + documents: DocumentSpec[] + batchUpload: boolean + declaredInvariants: string[] + notes?: string +} + +export interface ValidatedDocument { + spec: DocumentSpec + documentOrdinal: number + contentHash: string + dependsOnOrdinals: number[] +} + +export interface ValidatedDocumentPlan { + trajectoryId: string + planHash: string + documents: ValidatedDocument[] + batchUpload: boolean + declaredInvariants: string[] +} + +export interface PhysicalDocument { + trajectoryId: string + logicalDocumentId: string + documentOrdinal: number + partIndex: number + partCount: number + content: string + contentHash: string + customId: string + documentType: string + stateIndex?: number + step?: number + screenshotRef?: AssetRef + metadata: Record +} + +export interface MemoryBuildPlan { + schemaVersion: number + buildId: string + benchmark: string + provider: string + datasetFingerprint: string + tier: DatasetTier + domain: BenchmarkDomain + orderedSourceIds: string[] + sourceContentHashes: string[] + converter: { + name: string + version: number + sourceHash: string + } + providerBuildConfig: Record + buildFingerprint: string + containerTag: string + documentPlans: ValidatedDocumentPlan[] + documents: PhysicalDocument[] +} + +export interface RetrievalConfig { + topK: number + threshold: number + searchMode: "hybrid" | "memories" + rerank: boolean + rewriteQuery: boolean + includeSummaries: boolean + includeChunks: boolean + includeDocuments: boolean + includeRelatedMemories: boolean + metadataFilter: Record +} + +export interface NormalizedRetrievalResult { + rank: number + score?: number + kind: string + text: string + summary?: string + chunks: string[] + providerResultId?: string + documentIds: string[] + trajectoryId?: string + stateIndex?: number + screenshotRefs: AssetRef[] + provenanceValid: boolean +} + +export interface QueryArtifact { + schemaVersion: number + questionId: string + buildId: string + buildFingerprint: string + queryFingerprint: string + query: string + questionImage?: AssetRef + config: RetrievalConfig + request: Record + rawArtifact: { relativePath: string; sha256: string; byteLength: number } + normalizedArtifact: { relativePath: string; sha256: string; byteLength: number } + normalizedResults: NormalizedRetrievalResult[] + remoteDurationMs: number + wallDurationMs: number + cacheHit: boolean + createdAt: string +} + +export type ReaderMessagePart = + | { type: "text"; text: string; provenance?: Record } + | { + type: "image" + asset: AssetRef + caption?: string + provenance?: Record + } + +export interface ReaderArtifact { + schemaVersion: number + questionId: string + readerFingerprint: string + model: string + reasoningEffort?: string + systemPrompt: string + parts: ReaderMessagePart[] + sentAssetIds: string[] + omittedItems: number + responseText: string + parsedAnswer: string + rawAttempts?: unknown[] + usage?: Record + durationMs: number + cacheHit: boolean + createdAt: string +} + +export interface EvaluationArtifact { + schemaVersion: number + questionId: string + evaluatorFingerprint: string + evalFunction: string + answer: string + groundTruth: string + score: 0 | 1 + label: "correct" | "incorrect" + evaluatorModel?: string + promptVersion: string + implementationVersion: string + request?: Record + rawResponse?: unknown + rationale?: string + error?: string + durationMs: number + createdAt: string +} + +export interface ProviderCapabilities { + deterministicExternalIds: boolean + batchUpload: boolean + documentDependencies: boolean + ingestionMetadataFilters: boolean + searchMetadataFilters: boolean + searchModes: ReadonlyArray<"hybrid" | "memories"> + reranking: boolean + queryRewriting: boolean + remoteClear: boolean + readinessStates: boolean + mediaIngestion: boolean + durableLocalPersistence: boolean + splitPhaseSafe: boolean +} diff --git a/src/types/provider.ts b/src/types/provider.ts index cdc0228..cec16a4 100644 --- a/src/types/provider.ts +++ b/src/types/provider.ts @@ -1,6 +1,13 @@ import type { UnifiedSession } from "./unified" import type { ProviderPrompts } from "./prompts" import type { ConcurrencyConfig } from "./concurrency" +import type { ProviderCapabilities } from "./migration" +import type { + MemoryBuildPlan, + NormalizedRetrievalResult, + PhysicalDocument, + RetrievalConfig, +} from "./migration" export interface ProviderConfig { apiKey: string @@ -11,12 +18,14 @@ export interface ProviderConfig { export interface IngestOptions { containerTag: string metadata?: Record + signal?: AbortSignal } export interface SearchOptions { containerTag: string limit?: number threshold?: number + signal?: AbortSignal } export interface IngestResult { @@ -34,6 +43,7 @@ export type IndexingProgressCallback = (progress: IndexingProgress) => void export interface Provider { name: string + capabilities: ProviderCapabilities prompts?: ProviderPrompts concurrency?: ConcurrencyConfig initialize(config: ProviderConfig): Promise @@ -47,4 +57,65 @@ export interface Provider { clear(containerTag: string): Promise } +/** + * Optional exact-session bridge used by build-aware adapters around legacy + * MemoryBench providers. Providers implementing this contract can prove that a + * deterministic session exists after a process restart and can remove only the + * requested sessions without disturbing the rest of a shared build. + */ +export interface BuildAwareSessionBridge { + inspectSessions( + containerTag: string, + sessionIds: string[] + ): Promise< + Array<{ + sessionId: string + status: "ready" | "absent" + metadata?: Record + }> + > + deleteSessions(containerTag: string, sessionIds: string[]): Promise +} + export type ProviderName = "supermemory" | "mem0" | "zep" | "filesystem" | "rag" + +export type RemoteDocumentStatus = "absent" | "pending" | "ready" | "failed" | "unknown" + +export interface RemoteDocumentState { + customId: string + remoteId?: string + status: RemoteDocumentStatus + raw?: unknown + error?: string +} + +export interface BuildBatchRequest { + build: MemoryBuildPlan + trajectoryId: string + documents: PhysicalDocument[] +} + +export interface BuildSearchRequest { + build: MemoryBuildPlan + questionId: string + query: string + config: RetrievalConfig +} + +export interface BuildSearchResponse { + request: Record + rawResponse: unknown + normalizedResults: NormalizedRetrievalResult[] + remoteDurationMs: number +} + +export interface BuildProvider { + name: string + capabilities: ProviderCapabilities + submitDocumentBatch(request: BuildBatchRequest): Promise + reconcileDocuments(build: MemoryBuildPlan, customIds: string[]): Promise + searchBuild(request: BuildSearchRequest): Promise + verifyBuildHealth(build: MemoryBuildPlan): Promise + deleteDocuments(build: MemoryBuildPlan, customIds: string[]): Promise + clearBuild(build: MemoryBuildPlan): Promise +} diff --git a/src/utils/models.ts b/src/utils/models.ts index b29ac80..1589536 100644 --- a/src/utils/models.ts +++ b/src/utils/models.ts @@ -66,6 +66,15 @@ export const MODEL_CONFIGS: Record = { maxTokensParam: "max_completion_tokens", defaultMaxTokens: 1000, }, + "gpt-5.2": { + id: "gpt-5.2", + provider: "openai", + displayName: "GPT-5.2", + supportsTemperature: false, + defaultTemperature: 1, + maxTokensParam: "max_completion_tokens", + defaultMaxTokens: 1000, + }, "gpt-5-mini": { id: "gpt-5-mini", provider: "openai", diff --git a/ui/app/runs/[runId]/page.tsx b/ui/app/runs/[runId]/page.tsx index 79e49a7..c707318 100644 --- a/ui/app/runs/[runId]/page.tsx +++ b/ui/app/runs/[runId]/page.tsx @@ -3,10 +3,20 @@ import { useState, useEffect, useRef, useCallback } from "react" import Link from "next/link" import { useParams, useSearchParams, useRouter } from "next/navigation" -import { getRun, getRunReport, stopRun, startRun, type RunDetail } from "@/lib/api" +import { + getRun, + getRunReport, + isBuildAwareRunDetail, + stopRun, + startRun, + type BuildAwareReport, + type BuildAwareRunDetail, + type RunDetail, +} from "@/lib/api" import { formatDate, getStatusColor, cn } from "@/lib/utils" import { PhaseProgress } from "@/components/phase-progress" import { QuestionList } from "@/components/question-list" +import { BuildAwareRunInspection } from "@/components/build-aware-run-inspection" import { StatsGrid, AccuracyByType, @@ -30,7 +40,7 @@ export default function RunDetailPage() { const initialTab: Tab = tabFromUrl && ["overview", "results"].includes(tabFromUrl) ? tabFromUrl : "overview" - const [run, setRun] = useState(null) + const [run, setRun] = useState(null) const [report, setReport] = useState(null) const [loading, setLoading] = useState(true) const [error, setError] = useState(null) @@ -52,7 +62,7 @@ export default function RunDetailPage() { const [continuing, setContinuing] = useState(false) async function handleContinue() { - if (continuing || !run) return + if (continuing || !run || isBuildAwareRunDetail(run)) return setContinuing(true) try { await startRun({ @@ -167,6 +177,16 @@ export default function RunDetailPage() { ) } + if (isBuildAwareRunDetail(run)) { + return ( + + ) + } + // Show initializing state while benchmark is loading/downloading if (isInitializing) { return ( diff --git a/ui/app/runs/[runId]/questions/[questionId]/page.tsx b/ui/app/runs/[runId]/questions/[questionId]/page.tsx index 125a6ff..093a564 100644 --- a/ui/app/runs/[runId]/questions/[questionId]/page.tsx +++ b/ui/app/runs/[runId]/questions/[questionId]/page.tsx @@ -4,8 +4,9 @@ import { useState, useEffect } from "react" import Link from "next/link" import { useParams } from "next/navigation" import { Highlight, themes } from "prism-react-renderer" -import { getQuestion } from "@/lib/api" +import { getQuestion, isBuildAwareQuestionDetail } from "@/lib/api" import { cn } from "@/lib/utils" +import { BuildAwareQuestionInspection } from "@/components/build-aware-question-inspection" export default function QuestionDetailPage() { const params = useParams() @@ -51,6 +52,10 @@ export default function QuestionDetailPage() { ) } + if (isBuildAwareQuestionDetail(question)) { + return + } + const isCorrect = question.phases?.evaluate?.label === "correct" const searchResults = question.searchResultsFile?.results || question.phases?.search?.results || [] diff --git a/ui/app/runs/new/page.tsx b/ui/app/runs/new/page.tsx index ae3b222..39fd031 100644 --- a/ui/app/runs/new/page.tsx +++ b/ui/app/runs/new/page.tsx @@ -18,6 +18,7 @@ import { type Provider, } from "@/lib/api" import { SingleSelect } from "@/components/single-select" +import { LongMemEvalV2Launcher } from "@/components/longmemeval-v2-launcher" type Tab = "new" | "advanced" @@ -323,8 +324,22 @@ export default function NewRunPage() { const allModels = [...Object.values(models).flat()] as { alias: string; displayName: string }[] const providerOptions = providers.map((p) => ({ value: p.name, label: p.displayName })) - const benchmarkOptions = benchmarks.map((b) => ({ value: b.name, label: b.displayName })) + const benchmarkOptions = [ + ...benchmarks.map((b) => ({ value: b.name, label: b.displayName })), + ...(!benchmarks.some((benchmark) => benchmark.name === "longmemeval-v2") + ? [{ value: "longmemeval-v2", label: "LongMemEval-V2" }] + : []), + ] const modelOptions = allModels.map((m) => ({ value: m.alias, label: m.displayName || m.alias })) + const openAIModelOptions = ( + (models.openai ?? []) as Array<{ + alias: string + displayName?: string + }> + ).map((model) => ({ + value: model.alias, + label: model.displayName || model.alias, + })) const runOptions = completedRuns.map((r) => ({ value: r.runId, @@ -341,7 +356,9 @@ export default function NewRunPage() { } return ( -
+
Runs @@ -395,7 +412,28 @@ export default function NewRunPage() {
-
+ {activeTab === "new" && form.benchmark === "longmemeval-v2" && ( +
+
+ + setForm({ ...form, benchmark: value })} + placeholder="Select benchmark" + /> +
+ router.push(`/runs/${encodeURIComponent(runId)}`)} + /> +
+ )} + {activeTab === "advanced" && ( <>

@@ -833,7 +871,13 @@ export default function NewRunPage() { label="Select benchmark" options={benchmarkOptions} selected={form.benchmark} - onChange={(value) => setForm({ ...form, benchmark: value })} + onChange={(value) => + setForm({ + ...form, + benchmark: value, + provider: value === "longmemeval-v2" ? "supermemory" : form.provider, + }) + } placeholder="Select benchmark" />

diff --git a/ui/app/runs/page.tsx b/ui/app/runs/page.tsx index a92a7b4..7fd5a05 100644 --- a/ui/app/runs/page.tsx +++ b/ui/app/runs/page.tsx @@ -3,7 +3,15 @@ import { useState, useEffect, useMemo, useRef, useCallback } from "react" import Link from "next/link" import { useRouter } from "next/navigation" -import { getRuns, deleteRun, stopRun, startRun, addToLeaderboard, type RunSummary } from "@/lib/api" +import { + getRuns, + deleteRun, + stopRun, + startRun, + addToLeaderboard, + stopLongMemEvalV2Run, + type RunSummary, +} from "@/lib/api" import { formatDate, getStatusColor, cn } from "@/lib/utils" import { FilterBar } from "@/components/filter-bar" import { DataTable, type Column } from "@/components/data-table" @@ -29,7 +37,11 @@ export default function RunsPage() { // Check if any run is in progress const hasRunningRuns = useMemo(() => { return runs.some( - (r) => r.status === "running" || r.status === "pending" || r.status === "initializing" + (r) => + r.status === "running" || + r.status === "pending" || + r.status === "initializing" || + r.status === "stopping" ) }, [runs]) @@ -91,9 +103,10 @@ export default function RunsPage() { } } - async function handleTerminate(runId: string) { + async function handleTerminate(run: RunSummary) { try { - await stopRun(runId) + if (run.readOnlyInspection) await stopLongMemEvalV2Run(run.runId) + else await stopRun(run.runId) await refreshRuns() } catch (e) { alert(e instanceof Error ? e.message : "Failed to terminate run") @@ -274,18 +287,40 @@ export default function RunsPage() { header: "", width: "40px", align: "right", - render: (run) => ( - handleAddToLeaderboard(run.runId, data)} - onDelete={() => handleDelete(run.runId)} - onTerminate={() => handleTerminate(run.runId)} - onContinue={() => handleContinue(run)} - /> - ), + render: (run) => + run.readOnlyInspection ? ( +
+ + {["failed", "partial", "blocked"].includes(run.status) + ? "inspect / resume" + : "inspect"} + + {["running", "pending", "initializing", "stopping"].includes(run.status) && ( + + )} +
+ ) : ( + handleAddToLeaderboard(run.runId, data)} + onDelete={() => handleDelete(run.runId)} + onTerminate={() => handleTerminate(run)} + onContinue={() => handleContinue(run)} + /> + ), }, ], [] diff --git a/ui/components/benchmark-results.tsx b/ui/components/benchmark-results.tsx index 0fb6bb6..4294e2f 100644 --- a/ui/components/benchmark-results.tsx +++ b/ui/components/benchmark-results.tsx @@ -201,7 +201,15 @@ export function LatencyTable({ latency }: LatencyTableProps) { export interface RetrievalMetricsProps { retrieval?: RetrievalStats | null - byQuestionType?: Record | null + byQuestionType?: Record< + string, + { + total?: number + correct?: number + accuracy?: number + retrieval?: RetrievalStats + } + > | null } export function RetrievalMetrics({ retrieval, byQuestionType }: RetrievalMetricsProps) { diff --git a/ui/components/build-aware-question-inspection.tsx b/ui/components/build-aware-question-inspection.tsx new file mode 100644 index 0000000..fad1308 --- /dev/null +++ b/ui/components/build-aware-question-inspection.tsx @@ -0,0 +1,695 @@ +"use client" + +import { useState } from "react" +import Link from "next/link" +import { + getBuildAwareAssetUrl, + getBuildAwareArtifact, + type BuildAwareAssetRef, + type BuildAwareArtifactResponse, + type BuildAwareQuestionDetail, +} from "@/lib/api" +import { cn, formatDuration, getStatusColor } from "@/lib/utils" + +type ArtifactKind = BuildAwareArtifactResponse["kind"] + +interface BuildAwareQuestionInspectionProps { + runId: string + question: BuildAwareQuestionDetail +} + +function shortHash(value: string | undefined): string { + if (!value) return "not recorded" + return value.length > 20 ? `${value.slice(0, 12)}…${value.slice(-6)}` : value +} + +export function BuildAwareQuestionInspection({ + runId, + question, +}: BuildAwareQuestionInspectionProps) { + const [artifacts, setArtifacts] = useState< + Partial> + >({}) + const [artifactErrors, setArtifactErrors] = useState>>({}) + const [loadingArtifact, setLoadingArtifact] = useState(null) + + const query = question.queryArtifact + const reader = question.readerArtifact + const evaluation = question.evaluationArtifact + const sentImages = + reader?.parts + .filter( + (part): part is Extract<(typeof reader.parts)[number], { type: "image" }> => + part.type === "image" && reader.sentAssetIds.includes(part.asset.assetId) + ) + .map((part) => part.asset) ?? [] + + async function toggleArtifact(kind: ArtifactKind) { + if (artifacts[kind]) { + setArtifacts((current) => { + const next = { ...current } + delete next[kind] + return next + }) + return + } + try { + setLoadingArtifact(kind) + const artifact = await getBuildAwareArtifact(runId, question.questionId, kind) + setArtifacts((current) => ({ ...current, [kind]: artifact })) + setArtifactErrors((current) => ({ ...current, [kind]: undefined })) + } catch (error) { + setArtifactErrors((current) => ({ + ...current, + [kind]: error instanceof Error ? error.message : "Artifact could not be loaded", + })) + } finally { + setLoadingArtifact(null) + } + } + + return ( +
+
+ + Runs + + / + + {runId} + + / + {question.questionId} +
+ +
+
+

+ {question.questionId} +

+ {question.questionType} + + {evaluation?.label ?? question.stages.evaluate.status} + +
+

+ Build-aware LongMemEval-V2 inspection with checkpointed retrieval, reader, and evaluator + provenance. +

+
+ +
+
+
+
Memory Build
+
{question.buildId}
+
+ {question.buildFingerprint ?? + query?.buildFingerprint ?? + "fingerprint not recorded yet"} +
+
+ 1 ? "badge-success" : "badge-neutral" + )} + > + {question.buildReuseCount > 1 + ? `reused by ${question.buildReuseCount} questions` + : "single-question build"} + +
+ {!question.buildLinkMatchesCheckpoint && ( +

+ Checkpoint provenance mismatch: the question-to-build link does not match this + question's recorded build ID. +

+ )} +
+ +
+
+
Question
+

{question.question}

+
+ Evaluation function: {question.evalFunction} +
+
+
+
Ground truth
+

{question.groundTruth}

+
+
+ +
+
+
+

+ Official LongMemEval-V2 evaluation +

+

+ Only the evaluator result in this panel contributes to the official benchmark + aggregate. +

+
+ longmemeval-v2-official +
+ {evaluation ? ( +
+
+
+ {evaluation.score} +
+
{evaluation.label}
+
+
+
+
Model answer
+

{evaluation.answer}

+
+ {evaluation.rationale && ( +
+
+ Evaluator rationale +
+

{evaluation.rationale}

+
+ )} +
+ + + + + +
+ {(evaluation.request || evaluation.rawResponse !== undefined) && ( +
+ + Evaluator request and raw response + + 
+                    {JSON.stringify(
+                      {
+                        request: evaluation.request,
+                        rawResponse: evaluation.rawResponse,
+                      },
+                      null,
+                      2
+                    )}
+
+
+ )} + + +
+
+ ) : ( +
+

+ Evaluation is {question.stages.evaluate.status}. No score is shown. +

+ {question.stages.evaluate.error && ( +

{question.stages.evaluate.error}

+ )} + {question.artifactLinks.evaluation.available && ( +
+ + +
+ )} +
+ )} +
+ +
+
+
+

+ Retrieval diagnostics +

+

+ Retrieval content, latency, cache state, and provenance are MemoryBench + diagnostics—not official benchmark metrics. Referenced screenshots are served only + after the server verifies their allowlist entry, hash, size, and MIME type. +

+
+ not an official score +
+ +
+ + + + + +
+ + {query && ( +
+ + + + +
+ )} + +
+ {(["query-raw", "query-normalized"] as ArtifactKind[]).map((kind) => { + const link = question.artifactLinks[kind] + return ( + + ) + })} +
+ + {(["query-raw", "query-normalized"] as ArtifactKind[]).map((kind) => ( + + ))} + +
+ {query?.normalizedResults.map((result) => ( +
+
+ #{result.rank} + {result.kind} + {result.score !== undefined && ( + + score {result.score.toFixed(4)} + + )} + + provenance {result.provenanceValid ? "valid" : "invalid"} + +
+

+ {result.text} +

+
+ documents: {result.documentIds.join(", ") || "none"} + {result.trajectoryId ? ` · trajectory: ${result.trajectoryId}` : ""} + {result.stateIndex !== undefined ? ` · state: ${result.stateIndex}` : ""} +
+ {result.screenshotRefs.length > 0 && ( +
+ {result.screenshotRefs.map((asset, assetIndex) => ( + + ))} +
+ )} +
+ ))} + {!query?.normalizedResults.length && ( +

No normalized retrieval results recorded.

+ )} +
+
+ +
+
+
+

+ Reader provenance +

+

+ Exact reader model, prompt fingerprint, answer, omissions, and image inputs recorded + for this question. +

+
+ + reader cache{" "} + {reader + ? reader.cacheHit || question.stages.read.cacheHit + ? "hit" + : "miss" + : "not run"} + +
+ {reader ? ( +
+
+ + + + +
+
+
+
Parsed answer
+

{reader.parsedAnswer}

+
+
+
+ Reader response +
+

{reader.responseText}

+
+
+
+ {reader.parts.length} message parts · {reader.omittedItems} omitted items + {reader.usage ? ` · usage ${JSON.stringify(reader.usage)}` : ""} +
+
+ + Reader system prompt + + 
+                {reader.systemPrompt}
+
+
+ + +
+ ) : ( +
+

+ Reader stage is {question.stages.read.status}. No reader artifact exists. +

+ {question.stages.read.error && ( +

{question.stages.read.error}

+ )} +
+ )} +
+ +
+
+
+

+ Screenshots sent to the reader +

+

+ Only assets in the reader's sent-asset list appear here. Each image is loaded + from the hash-verified asset endpoint. +

+
+ {sentImages.length} sent +
+ {sentImages.length ? ( +
+
+ {sentImages.map((asset, index) => ( + + ))} +
+
+ + + + + + + + + + + + + {sentImages.map((asset, index) => ( + + + + + + + + + ))} + +
OrderAsset IDSHA-256MIMEBytesArtifact path
{index + 1}{asset.assetId} + {shortHash(asset.sha256)} + {asset.mimeType} + {asset.byteLength.toLocaleString()} + + {asset.relativePath} +
+
+
+ ) : ( +

+ No screenshot assets were sent for this reader request. +

+ )} +
+
+ ) +} + +function VerifiedScreenshot({ + runId, + questionId, + asset, + orderLabel, + alt, +}: { + runId: string + questionId: string + asset: BuildAwareAssetRef + orderLabel: string + alt: string +}) { + const [failed, setFailed] = useState(false) + const assetUrl = getBuildAwareAssetUrl(runId, questionId, asset.assetId) + + return ( +
+ {failed ? ( +
+ Screenshot could not be loaded or failed server-side integrity verification. +
+ ) : ( + + {alt} setFailed(true)} + className="max-h-96 w-full object-contain" + /> + + )} +
+
+ {orderLabel} + {asset.kind} +
+
{asset.assetId}
+
+ sha256 {shortHash(asset.sha256)} · {asset.byteLength.toLocaleString()} bytes +
+
+
+ ) +} + +function Metric({ label, value }: { label: string; value: string }) { + return ( +
+
{label}
+
{value}
+
+ ) +} + +function Provenance({ label, value }: { label: string; value: string }) { + return ( +
+
{label}
+
{value}
+
+ ) +} + +function ArtifactPanel({ + kind, + artifact, + error, + checkpointProvenance, +}: { + kind: ArtifactKind + artifact?: BuildAwareArtifactResponse + error?: string + checkpointProvenance: BuildAwareQuestionDetail["artifactLinks"][ArtifactKind]["provenance"] +}) { + if (!artifact && !error) return null + return ( +
+
+ {kind} + + {"relativePath" in checkpointProvenance + ? `${checkpointProvenance.relativePath} · ${shortHash(checkpointProvenance.sha256)} · ${checkpointProvenance.byteLength?.toLocaleString() ?? "?"} bytes` + : "embedded in checkpoint"} + +
+ {error ? ( +

{error}

+ ) : ( + 
+          {JSON.stringify(artifact?.data, null, 2)}
+
+ )} +
+ ) +} diff --git a/ui/components/build-aware-run-inspection.tsx b/ui/components/build-aware-run-inspection.tsx new file mode 100644 index 0000000..b2c4419 --- /dev/null +++ b/ui/components/build-aware-run-inspection.tsx @@ -0,0 +1,782 @@ +"use client" + +import { useCallback, useEffect, useRef, useState } from "react" +import Link from "next/link" +import { + getBuildAwareRunQuestions, + getLongMemEvalV2RunStatus, + resumeLongMemEvalV2Run, + stopLongMemEvalV2Run, + type BuildAwareReport, + type BuildAwareQuestionSummary, + type BuildAwareRunDetail, + type LongMemEvalV2RunStatusResponse, + type PaginatedResponse, +} from "@/lib/api" +import { + requiresFullScopeResumeConfirmation, + type LongMemEvalV2ResumeTarget, +} from "@/lib/longmemeval-v2-form" +import { cn, formatDate, formatDuration, getStatusColor } from "@/lib/utils" + +interface BuildAwareRunInspectionProps { + run: BuildAwareRunDetail + report: BuildAwareReport | null + onRefresh?: () => Promise +} + +function percent(value: number | null | undefined): string { + return value === null || value === undefined ? "—" : `${(value * 100).toFixed(1)}%` +} + +function average(values: number[] | undefined): number | null { + if (!values?.length) return null + return values.reduce((total, value) => total + value, 0) / values.length +} + +function stageLabel(status: string): string { + return status.replace(/_/g, " ") +} + +function nextStage( + stage: BuildAwareRunDetail["currentStage"], + canary: boolean +): BuildAwareRunDetail["currentStage"] | null { + const next: Partial< + Record + > = { + plan: "build", + build: "query", + query: "read", + read: "evaluate", + evaluate: "report", + } + const target = next[stage] ?? null + return canary && target && !["build", "query"].includes(target) ? null : target +} + +export function BuildAwareRunInspection({ run, report, onRefresh }: BuildAwareRunInspectionProps) { + const [runtime, setRuntime] = useState(null) + const [controlAction, setControlAction] = useState<"stop" | "resume" | null>(null) + const [controlError, setControlError] = useState(null) + const [allowFullContinuation, setAllowFullContinuation] = useState(false) + const [resumeForceBuild, setResumeForceBuild] = useState(false) + const [resumeFreshQuery, setResumeFreshQuery] = useState(false) + const [questionPage, setQuestionPage] = useState(1) + const [questionPageData, setQuestionPageData] = + useState | null>(null) + const [questionsLoading, setQuestionsLoading] = useState(true) + const [questionsError, setQuestionsError] = useState(null) + const lastStatusSignature = useRef("") + const official = report?.official ?? run.inspection.metricNamespaces.official + const diagnostics = report?.diagnostics ?? run.inspection.metricNamespaces.diagnostics + const questions = + questionPageData?.questions ?? + (Object.values(run.questions ?? {}) as BuildAwareQuestionSummary[]) + const checkpointStatus = runtime?.checkpoint?.status ?? run.status + const checkpointStage = runtime?.checkpoint?.currentStage ?? run.currentStage + const lifecycleEvents = runtime?.control?.events ?? [] + const isActive = runtime?.active ?? ["pending", "running"].includes(run.status) + const isStopping = runtime?.stopping ?? false + const canResume = + !isActive && !isStopping && ["failed", "partial", "blocked"].includes(checkpointStatus) + const continuationStage = + checkpointStatus === "completed" + ? nextStage(checkpointStage, run.config.mode === "one-trajectory-canary") + : null + const canContinue = !isActive && !isStopping && continuationStage !== null + const priorTarget = [...lifecycleEvents] + .reverse() + .find( + (event) => + (event.action === "start" || event.action === "resume") && event.through !== undefined + )?.through + const resumeTarget: LongMemEvalV2ResumeTarget | null = canContinue + ? continuationStage + : canResume + ? (priorTarget ?? checkpointStage) + : null + const needsFullContinuationConfirmation = requiresFullScopeResumeConfirmation( + run.config, + resumeTarget + ) + + const refreshRuntime = useCallback(async () => { + try { + const next = await getLongMemEvalV2RunStatus(run.runId) + const signature = `${next.active}:${next.stopping}:${next.checkpoint?.status ?? "none"}:${next.checkpoint?.currentStage ?? "none"}:${next.checkpoint?.updatedAt ?? "none"}` + const changed = + lastStatusSignature.current !== "" && lastStatusSignature.current !== signature + lastStatusSignature.current = signature + setRuntime(next) + if (changed) await onRefresh?.() + } catch (error) { + setControlError(error instanceof Error ? error.message : "Could not read run lifecycle") + } + }, [onRefresh, run.runId]) + + const refreshQuestions = useCallback(async () => { + try { + setQuestionsLoading(true) + const next = await getBuildAwareRunQuestions(run.runId, { page: questionPage, limit: 25 }) + setQuestionPageData(next) + setQuestionsError(null) + } catch (error) { + setQuestionsError(error instanceof Error ? error.message : "Could not load questions") + } finally { + setQuestionsLoading(false) + } + }, [questionPage, run.runId, run.updatedAt]) + + useEffect(() => { + void refreshRuntime() + const interval = window.setInterval(() => void refreshRuntime(), 2_000) + return () => window.clearInterval(interval) + }, [refreshRuntime]) + + useEffect(() => { + void refreshQuestions() + }, [refreshQuestions]) + + useEffect(() => { + setAllowFullContinuation(false) + }, [resumeTarget, run.runId]) + + async function handleStop() { + if (controlAction || !isActive || isStopping) return + try { + setControlAction("stop") + setControlError(null) + await stopLongMemEvalV2Run(run.runId) + await refreshRuntime() + await onRefresh?.() + } catch (error) { + setControlError(error instanceof Error ? error.message : "Failed to stop run") + } finally { + setControlAction(null) + } + } + + async function handleResume() { + if ( + controlAction || + (!canResume && !canContinue) || + (needsFullContinuationConfirmation && !allowFullContinuation) + ) + return + try { + setControlAction("resume") + setControlError(null) + await resumeLongMemEvalV2Run(run.runId, { + ...(continuationStage ? { runThrough: continuationStage } : {}), + ...(needsFullContinuationConfirmation ? { allowFullRun: true } : {}), + forceBuild: resumeForceBuild, + freshQuery: resumeFreshQuery, + }) + setAllowFullContinuation(false) + setResumeForceBuild(false) + setResumeFreshQuery(false) + await refreshRuntime() + await onRefresh?.() + } catch (error) { + setControlError(error instanceof Error ? error.message : "Failed to resume run") + } finally { + setControlAction(null) + } + } + + return ( +
+
+ + Runs + + / + {run.runId} +
+ +
+
+

{run.runId}

+ + {stageLabel(checkpointStatus)} + + shared memory build + {run.config.mode} + {isActive && ( + + )} + {(canResume || canContinue) && ( + + )} +
+
+ + Provider: {run.config.provider} + + + Benchmark: LongMemEval-V2 + + + Stage: {checkpointStage} + + + Runtime:{" "} + {isStopping ? "stopping" : isActive ? "active" : "inactive"} + + + Created: {formatDate(run.createdAt)} + +
+
+ + {(canResume || canContinue) && needsFullContinuationConfirmation && ( +
+
+ Full-tier continuation requires confirmation +
+

+ This run has no question, haystack, limit, or per-category selector. Continuing through{" "} + {resumeTarget} can process the + complete {run.config.tier}/{run.config.domain} selection ({run.targetQuestionIds.length}{" "} + planned questions) and may ingest its full memory build. +

+ +
+ )} + + {(canResume || canContinue) && ( +
+
Resume options
+
+ + +
+
+ )} + + {controlError && ( +
+ {controlError} +
+ )} + + {lifecycleEvents.length > 0 && ( +
+ + Lifecycle history ({lifecycleEvents.length}) + +
+ + + + + + + + + + + {lifecycleEvents + .slice(-10) + .reverse() + .map((event, index) => ( + + + + + + + ))} + +
ActionThroughTimeMessage
{event.action} + {event.through ?? "—"} + + {formatDate(event.at)} + + {[ + event.message, + event.provider ? `provider ${event.provider}` : undefined, + event.forceBuild ? "force rebuild" : undefined, + event.freshQuery ? "fresh retrieval" : undefined, + ] + .filter(Boolean) + .join(" · ") || "—"} +
+
+
+ )} + + {(runtime?.checkpoint?.error || run.error) && ( +
+
Run error
+

+ {runtime?.checkpoint?.error ?? run.error} +

+
+ )} + + {report?.officiallyComparable === false && ( +
+
+ Degraded build — not an official comparison +
+

+ Ingestion continued after bounded failures or indexing timeouts. Any score remains + diagnostic because the exact haystack was incomplete. +

+ {(report.ineligibilityReasons?.length ?? 0) > 0 && ( +
    + {report.ineligibilityReasons!.map((reason) => ( +
  • {reason}
    • + ))} +
+ )} +
+ )} + + {(run.storageRoots.artifacts !== "available" || run.storageRoots.builds !== "available") && ( +
+
+ Partial local provenance +
+

+ Artifact root: {run.storageRoots.artifacts}; build root: {run.storageRoots.builds}. + Checkpoint-embedded data remains visible, but unavailable roots are never followed + outside the server's allowlisted directories. +

+
+ )} + +
+
+

+ Memory Build reuse +

+

+ Questions sharing a build query one ingested haystack. Reuse counts below come from + checkpoint question-to-build links. +

+
+
+ + + + + + + + + + + + + {run.inspection.builds.map((build) => ( + + + + + + + + + ))} + +
Build IDFingerprintPlanQuestion sharingIngestionBuild state
{build.buildId} + {build.buildFingerprint ?? "not recorded yet"} + +
{build.domain ?? "unknown domain"}
+
+ {build.trajectoryCount ?? "?"} trajectories · {build.documentCount ?? "?"}{" "} + documents +
+ 		+ {build.reused ? ( + + shared by {build.reuseCount} questions + + ) : ( + 1 question + )} + {build.questionLinkMismatches.length > 0 && ( +
+ {build.questionLinkMismatches.length} checkpoint link mismatch +
+ )} + 		+ {build.priorBuildReuse === undefined ? ( + not reported yet + ) : build.priorBuildReuse ? ( + checkpoint reused + ) : ( + built for this run + )} + + {build.stateStore.available ? ( +
+ + {build.stateStore.status ?? "readable"} + + {build.stateStore.documents && ( +
+ documents ·{" "} + {Object.entries(build.stateStore.documents) + .map(([status, count]) => `${status}: ${count}`) + .join(" · ")} +
+ )} + {build.stateStore.trajectories && ( +
+ trajectories ·{" "} + {Object.entries(build.stateStore.trajectories) + .map(([status, count]) => `${status}: ${count}`) + .join(" · ")} +
+ )} +
+ ) : ( + + checkpoint summary only + + )} +
+
+
+ +
+
+
+

+ Official LongMemEval-V2 metrics +

+

+ This is the official benchmark protocol namespace. Failed, pending, and blocked + questions remain in the full-set denominator. +

+
+ longmemeval-v2-official +
+ {official ? ( + <> +
+ + + + +
+
+ {Object.entries(official.non_abstention_by_category).map(([category, breakdown]) => ( + + ))} +
+ + ) : ( +

+ No official report exists yet. Checkpoint progress is not presented as a benchmark + score. +

+ )} +
+ +
+
+
+

+ MemoryBench diagnostics +

+

+ Operational evidence for cache, retrieval latency, and screenshots. These values are + not official LongMemEval-V2 scores. +

+
+ not an official score +
+
+ + + + + +
+
+ +
+

+ Run provenance +

+
+ + + + + + + + + + + + + +
+
+ +
+
+
+

Questions

+

+ {run.summary.evaluate.completed}/{run.summary.total} officially evaluated +

+
+
+ {questionsError && ( +
+ {questionsError} +
+ )} +
+ {questionsLoading && questions.length === 0 && ( +
+ Loading questions… +
+ )} + {questions.map((question, index) => { + const evaluation = question.evaluationArtifact + return ( + + + + {question.questionId} + + {question.questionType} + + {question.question} + + {question.buildId} + + {evaluation?.label ?? question.stages.evaluate.status} + + + ) + })} +
+ {questionPageData && questionPageData.pagination.totalPages > 1 && ( +
+ + Page {questionPageData.pagination.page} of {questionPageData.pagination.totalPages} ·{" "} + {questionPageData.pagination.total} questions + +
+ + +
+
+ )} +
+
+ ) +} + +function Metric({ label, value, detail }: { label: string; value: string; detail?: string }) { + return ( +
+
{label}
+
{value}
+ {detail &&
{detail}
} +
+ ) +} + +function Provenance({ label, value }: { label: string; value: string }) { + return ( +
+
{label}
+
{value}
+
+ ) +} diff --git a/ui/components/longmemeval-v2-launcher.tsx b/ui/components/longmemeval-v2-launcher.tsx new file mode 100644 index 0000000..492c725 --- /dev/null +++ b/ui/components/longmemeval-v2-launcher.tsx @@ -0,0 +1,1004 @@ +"use client" + +import { useEffect, useMemo, useState } from "react" +import { SingleSelect } from "@/components/single-select" +import { + getLongMemEvalV2Options, + startLongMemEvalV2Preflight, + startLongMemEvalV2Run, + type LongMemEvalV2OptionsResponse, + type LongMemEvalV2RunThrough, +} from "@/lib/api" +import { + parseLongMemEvalV2QuestionIds, + toStartLongMemEvalV2RunParams, + validateLongMemEvalV2Launch, + type LongMemEvalV2LaunchValues, +} from "@/lib/longmemeval-v2-form" + +interface SelectOption { + value: string + label: string + sublabel?: string +} + +interface LongMemEvalV2LauncherProps { + onStarted: (runId: string) => void + modelOptions?: SelectOption[] +} + +const inputClass = + "w-full rounded border border-[#333333] bg-[#222222] px-3 py-2.5 text-sm text-text-primary placeholder-text-muted focus:border-accent focus:outline-none" + +const fallbackModels: SelectOption[] = [ + { value: "gpt-5", label: "GPT-5", sublabel: "Reasoning model · benchmark default" }, + { value: "gpt-5.2", label: "GPT-5.2", sublabel: "Reasoning model" }, + { value: "gpt-5-mini", label: "GPT-5 Mini", sublabel: "Reasoning model" }, + { value: "gpt-4.1", label: "GPT-4.1", sublabel: "No reasoning control" }, + { value: "gpt-4.1-mini", label: "GPT-4.1 Mini", sublabel: "No reasoning control" }, + { value: "gpt-4o", label: "GPT-4o (Legacy)", sublabel: "No reasoning control" }, + { value: "gpt-4o-mini", label: "GPT-4o Mini (Legacy)", sublabel: "No reasoning control" }, +] + +const runThroughOptions: SelectOption[] = [ + { value: "plan", label: "Plan only", sublabel: "Offline validation; no API calls" }, + { value: "build", label: "Ingest memories", sublabel: "Build and index selected haystacks" }, + { value: "query", label: "Retrieve", sublabel: "Build, then save retrieval results" }, + { + value: "evaluate", + label: "Answer and evaluate", + sublabel: "Save per-question scores; no aggregate report", + }, + { value: "run", label: "Full report", sublabel: "Complete pipeline and report" }, +] + +const reasoningOptions: SelectOption[] = ["none", "minimal", "low", "medium", "high", "xhigh"].map( + (value) => ({ value, label: value.charAt(0).toUpperCase() + value.slice(1) }) +) + +function generateRunId(): string { + const date = new Date().toISOString().slice(0, 10).replace(/-/g, "") + const random = Math.random().toString(36).slice(2, 6) + return `lme-v2-${date}-${random}` +} + +function supportsReasoning(model: string): boolean { + return /^(?:gpt-5|o1|o3|o4)/i.test(model) +} + +function availableHaystacks( + options: LongMemEvalV2OptionsResponse | null, + tier: LongMemEvalV2LaunchValues["tier"], + domain: LongMemEvalV2LaunchValues["domain"] +): number | null { + return options?.haystacks[tier][domain] ?? null +} + +export function LongMemEvalV2Launcher({ + onStarted, + modelOptions = fallbackModels, +}: LongMemEvalV2LauncherProps) { + const supportedModels = useMemo(() => { + const allowed = new Set(fallbackModels.map((model) => model.value)) + const discovered = modelOptions.filter((model) => allowed.has(model.value)) + return discovered.length > 1 ? discovered : fallbackModels + }, [modelOptions]) + const [values, setValues] = useState(() => ({ + runId: generateRunId(), + provider: "supermemory", + datasetPath: "", + tier: "small", + allowMedium: false, + domain: "all", + selectionMode: "all-haystacks", + haystackLimit: 1, + questionIds: "", + canary: false, + topK: 20, + evidenceTopK: 20, + readerModel: "gpt-5", + evaluatorModel: "gpt-5", + reasoningEffort: "high", + evaluatorReasoningEffort: "high", + buildConcurrency: 2, + questionConcurrency: 5, + trajectoryConcurrency: 4, + maxInFlightRequests: 20, + indexingTimeoutMinutes: 30, + maxTrajectoryAttempts: 4, + strictIngestion: false, + runThrough: "plan", + allowFullRun: false, + forceBuild: false, + freshQuery: false, + })) + const [options, setOptions] = useState(null) + const [optionsLoading, setOptionsLoading] = useState(true) + const [optionsError, setOptionsError] = useState(null) + const [showSetup, setShowSetup] = useState(false) + const [showAdvanced, setShowAdvanced] = useState(false) + const [submitting, setSubmitting] = useState(false) + const [preflightSubmitting, setPreflightSubmitting] = useState(false) + const [error, setError] = useState(null) + + useEffect(() => { + let cancelled = false + getLongMemEvalV2Options() + .then((response) => { + if (cancelled) return + const preparedDataset = response.datasets.find((dataset) => dataset.prepared) + setOptions(response) + setValues((current) => ({ + ...current, + datasetPath: + response.defaults.datasetPath ?? preparedDataset?.path ?? current.datasetPath, + provider: response.defaults.provider, + tier: response.defaults.tier, + domain: response.defaults.domain, + topK: response.defaults.topK, + evidenceTopK: response.defaults.evidenceTopK, + readerModel: response.defaults.readerModel, + evaluatorModel: response.defaults.evaluatorModel, + reasoningEffort: response.defaults.reasoningEffort, + evaluatorReasoningEffort: response.defaults.reasoningEffort, + buildConcurrency: response.defaults.buildConcurrency, + questionConcurrency: response.defaults.questionConcurrency, + trajectoryConcurrency: response.defaults.trajectoryConcurrency, + maxInFlightRequests: response.defaults.maxInFlightRequests, + indexingTimeoutMinutes: response.defaults.indexingTimeoutMs / 60_000, + maxTrajectoryAttempts: response.defaults.maxTrajectoryAttempts, + strictIngestion: response.defaults.strictIngestion, + })) + }) + .catch((cause) => { + if (!cancelled) { + setOptionsError( + cause instanceof Error ? cause.message : "Could not inspect local prerequisites" + ) + } + }) + .finally(() => { + if (!cancelled) setOptionsLoading(false) + }) + return () => { + cancelled = true + } + }, []) + + const questionCount = + values.selectionMode === "questions" + ? parseLongMemEvalV2QuestionIds(values.questionIds).length + : 0 + const haystackCount = availableHaystacks(options, values.tier, values.domain) + const requestedHaystacks = + values.selectionMode === "haystack-limit" ? values.haystackLimit : haystackCount + const isFullScopeLiveRun = + values.selectionMode === "all-haystacks" && values.runThrough !== "plan" + const selectedDataset = options?.datasets.find( + (dataset) => dataset.path === values.datasetPath.trim() + ) + const selectedProvider = options?.providers.find((provider) => provider.name === values.provider) + const preflightCoversTopK = + options?.preflight.status === "passing" && (options.preflight.testedTopK ?? 0) >= values.topK + const capability = values.canary + ? selectedProvider?.capabilities.query + : values.runThrough === "plan" + ? selectedProvider?.capabilities.plan + : values.runThrough === "build" + ? selectedProvider?.capabilities.build + : values.runThrough === "query" + ? selectedProvider?.capabilities.query + : values.runThrough === "evaluate" + ? selectedProvider?.capabilities.evaluate + : selectedProvider?.capabilities.report + const liveStageBlocked = + values.runThrough !== "plan" && + (capability !== true || + selectedDataset?.prepared === false || + (selectedProvider?.requiresPreflight && !preflightCoversTopK)) + const setupReady = + Boolean(selectedDataset?.prepared) && + Boolean(selectedProvider?.configured) && + Boolean(options?.credentials.openAIConfigured) && + (!selectedProvider?.requiresPreflight || preflightCoversTopK) + + function updateDatasetSlice(next: Partial>) { + setValues((current) => { + const tier = next.tier ?? current.tier + const domain = next.domain ?? current.domain + const count = availableHaystacks(options, tier, domain) + return { + ...current, + ...next, + allowMedium: tier === "medium" ? current.allowMedium : false, + haystackLimit: count ? Math.min(current.haystackLimit, count) : current.haystackLimit, + allowFullRun: false, + } + }) + } + + async function handlePreflight() { + if (preflightSubmitting || options?.preflightActivity.status === "running") return + try { + setPreflightSubmitting(true) + setOptionsError(null) + await startLongMemEvalV2Preflight(values.topK) + for (let attempt = 0; attempt < 240; attempt += 1) { + await new Promise((resolve) => window.setTimeout(resolve, attempt === 0 ? 500 : 2_000)) + const next = await getLongMemEvalV2Options() + setOptions(next) + if (next.preflightActivity.status === "passed") return + if (next.preflightActivity.status === "failed") { + throw new Error(next.preflightActivity.error) + } + } + throw new Error("Preflight did not finish within the bounded eight-minute UI wait") + } catch (cause) { + setOptionsError(cause instanceof Error ? cause.message : "Supermemory preflight failed") + } finally { + setPreflightSubmitting(false) + } + } + + async function handleSubmit(event: React.FormEvent) { + event.preventDefault() + const validationError = validateLongMemEvalV2Launch(values) + if (validationError) { + setError(validationError) + return + } + if ( + values.selectionMode === "haystack-limit" && + haystackCount !== null && + values.haystackLimit > haystackCount + ) { + setError( + `Only ${haystackCount} exact haystack${haystackCount === 1 ? " is" : "s are"} available` + ) + return + } + if (liveStageBlocked) { + setError("Live prerequisites are not ready. Open Setup details or use Plan only.") + return + } + try { + setSubmitting(true) + setError(null) + const response = await startLongMemEvalV2Run(toStartLongMemEvalV2RunParams(values)) + onStarted(response.runId || values.runId.trim()) + } catch (cause) { + setError(cause instanceof Error ? cause.message : "Failed to start LongMemEval-V2 run") + setSubmitting(false) + } + } + + return ( + +
+
+
+
+

+ LongMemEval-V2 +

+ build-aware +
+

+ Choose the haystacks, models, and stopping point. MemoryBench reuses each exact + ingested haystack across all of its questions. +

+
+ + {selectedProvider?.displayName ?? values.provider} memory layer + +
+
+ +
+ + ({ + value: provider.name, + label: provider.displayName, + sublabel: provider.note, + }))} + selected={values.provider} + onChange={(provider) => + setValues((current) => ({ + ...current, + provider: provider as LongMemEvalV2LaunchValues["provider"], + runThrough: + options?.providers.find((candidate) => candidate.name === provider) + ?.adapterAvailable === false + ? "plan" + : current.runThrough, + allowFullRun: false, + forceBuild: false, + freshQuery: false, + })) + } + wide + /> + {selectedProvider && ( +
+ {selectedProvider.note} Search: {selectedProvider.searchMode}; reranking{" "} + {selectedProvider.rerank ? "on" : "off"}. +
+ )} +
+ +
+ + {showSetup && ( +
+ {options && ( +
+ + + + +
+ )} +
+ + {options?.datasets.map((dataset) => ( + + ))} + + setValues((current) => ({ ...current, datasetPath: event.target.value })) + } + aria-label="Prepared dataset path" + spellCheck={false} + /> +
+ {selectedProvider?.requiresPreflight && ( +
+ + + Synthetic probes only; exact probe document IDs are deleted. + +
+ )} + {(optionsError || options?.preflightActivity.status === "failed") && ( +

+ {optionsError ?? + (options?.preflightActivity.status === "failed" + ? options.preflightActivity.error + : "")} +

+ )} +
+ )} +
+ +
+ + setValues((current) => ({ ...current, runId: event.target.value }))} + autoComplete="off" + /> +
+ +
+ +
+
+ + + updateDatasetSlice({ tier: tier as LongMemEvalV2LaunchValues["tier"] }) + } + /> +
+
+ + + updateDatasetSlice({ domain: domain as LongMemEvalV2LaunchValues["domain"] }) + } + /> +
+
+ +
+ +
+ {( + [ + ["all-haystacks", "All haystacks"], + ["haystack-limit", "Limit haystacks"], + ["questions", "Specific questions"], + ] as const + ).map(([mode, label]) => ( + + ))} +
+
+ + {values.selectionMode === "haystack-limit" && ( +
+ +
+ + setValues((current) => ({ + ...current, + haystackLimit: Number(event.target.value), + })) + } + /> + + of {haystackCount ?? "?"} available + +
+

+ Uses the first N exact haystacks in pinned dataset order. Every chosen haystack keeps + all of its trajectories and all linked questions. +

+
+ )} + + {values.selectionMode === "questions" && ( +
+ +