refactor(config): make [embedding] and [rerank] soft dependencies

[Kendrick-Song]

Summary

Make [embedding] and [rerank] soft runtime dependencies so a freshly-onboarded user can run EverOS end-to-end with only [llm] configured. Previously the server refused to start without embedding, locking out anyone who just wanted keyword-only search.

Note on history: the core work was squashed into 07a7cd5; round-4 review fixes land as follow-up commits (561b5fe, 67758ec). Four rounds of Fable 5 code review (10 + 13 + 15 + 26 = 64 findings surveyed, 59 addressed as fixes, 4 accepted-as-designed, 1 refuted). The complete audit trail lives in the commit history and the "Files changed" view.

Capability tiers

Tier	Configured	Enabled features
1	[llm] only	KEYWORD search, add/flush, md writes, cascade sync
2	[llm] + [embedding]	+ VECTOR / HYBRID search, reflection, skill extraction, backfill
3	[llm] + [embedding] + [rerank]	+ AGENTIC search, knowledge (write + search)

• Tier upgrades require a server restart — capability accessors cache for the process lifetime.
• Tier downgrades are read-safe: a Tier-3 user who drops [rerank] can still read / rename / delete existing knowledge documents; only write / search endpoints return 422.

What changed

• Component accessors — component/{embedding,rerank,llm,multimodal}/accessor.py are the single process-wide provider singletons. service/* never maintains parallel singletons; it consumes get_*_capability() / get_llm_client() directly. Build-time ValueError from the factory is logged as *_capability_build_failed on all four accessors (was silently swallowed on 3 of 4 prior to round-4).
• Error mapping — ProviderNotConfiguredError → HTTP 422 CAPABILITY_UNAVAILABLE with everos.toml section hints (never EVEROS_* env-var strings). LanceDBMigrationError fails loud with escalating recovery guidance (restart → wipe index, in that order). LLMNotConfiguredError in search maps to None for KEYWORD degradation.
• Nullable-vector LanceDB migration — schema v2 makes the vector column nullable so Tier-1 rows can land without embeddings. Migration is guarded by a cross-process memory_root_lock (fcntl.flock + anyio.to_thread) and runs optimize(cleanup_older_than=timedelta(0)) per table after Phase-1 backfill so the manifest chain is physically pruned (not just compacted). _TABLE_SPECS has an import-time drift assertion against BUSINESS_SCHEMAS_WITH_VECTOR.
• Cascade — all handlers register unconditionally (Tier-3 → Tier-2/1 downgrade no longer strands DELETE); embed-requiring branches inside each handler body-guard on capability.available at execution time. Handler skip logs demoted INFO → DEBUG so Tier-1 imports don't flood the sink.
• everos cascade backfill CLI — Phase 1 (embed missing vectors) / Phase 2 (emit synthetic events for cascaded processing) / Phase 3 (sync new skill files). Exit codes: 0/1/2/3 (server-running preflight, now checked in Phase 1 too) / 4 (COMPLETED_WITH_FAILURES — includes rows where any needed side embed is still NULL post-write) / 130 (SIGINT). OMEConfig.crash_recovery_enabled = False in backfill engines prevents stale-RUNNING rows re-enqueuing into a smaller strategy registry. Phase 2 cluster_repo.find_cluster_id_for_member is scoped by (app_id, project_id, owner_id) — same-day entry_id collisions across owners are no longer possible (entry_id is per-owner unique by design).
• /health — reports capabilities + disabled_features per tier so ops can distinguish "boots but degraded" from "boots and full". Return type is HealthResponse (Pydantic model), so OpenAPI clients get real field shapes instead of additionalProperties: true.
• Presentation split — memory / service / infra don't import typer / click (verified by git grep, see below). TyperPresenter + run_backfill() live in entrypoints/cli/commands/_backfill_cmd.py; NullBackfillPresenter in memory/cascade/_backfill.py for tests. nothing_to_backfill(..., *, scan_failed: bool) — colour is picked from the flag, not a substring match on the message.
• Startup hint — unconditional count_rows(filter="vector IS NULL") sweep emits unbackfilled_memory_rows (event name + hint text pinned) when Tier-1 rows exist. ParserLifespanProvider warms the everalgo.parser import at boot so /health doesn't block on first call.
• Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text() routes explicit plaintext mimes (text/plain, text/markdown, text/x-rst, text/x-markdown) and known extensions (md, txt, rst, markdown, text) straight to UTF-8 decode. text/html is deliberately excluded so HTML uploads keep going through everalgo's clean_html_for_llm (strips <script>/<style>/<nav> + 1 MiB cap). Prevents 503 «Multimodal not configured» when a Tier-3 user without [multimodal] uploads a markdown doc.
• Upload filename hardening — _safe_original_filename rejects NUL bytes and > 255-byte UTF-8 filenames up front (400) instead of letting them surface as 500 at write_bytes after the md was already written.
• Backfill Ctrl-C at prompt — the interrupt-catch tuple now covers typer.Abort (which is typer._click.exceptions.Abort under typer 0.15+ — a distinct class from the standalone click.exceptions.Abort the vendored copy diverged from). click>=8.1 is now an explicit dependency (was only pulled in transitively via uvicorn).

Sync with main

Two origin/main merges collapsed into 07a7cd5:

1. 6dcd3eb (v1.1.4 → v1.2.0) — adds OTel tracing, /api/v2 alias, TracingLifespanProvider, per-cascade-embedding span fix, memory-op instrumentation.
2. 42629df (PR fix: backfill missing 1.1.4 fixes into main #366) — v1.1.4 CWE-22 knowledge path traversal fix + cascade retry-budget rework + cascade/errors.py → core.errors.ExternalServiceError.

Merge decisions worth flagging for review:

• service/search.py adopts single wrap site — component.llm accessor already applies UsageRecordingClient when observability is on; service layer never keeps a parallel LLM singleton (Round-1 CR rule: "service layer never maintains parallel singletons").
• Knowledge router prefix — main established the /knowledge prefix + create_app() mounting under /api/v1 and /api/v2; this PR keeps main's design (docstring on _require_knowledge_capabilities rewritten to describe the current gate location — the earlier build_handlers gate was removed there per Round-2 finding B3).
• Cascade retry classification uses ExternalServiceError from core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12 cross-cycle budget preserved.
• MemoryRoot.default() → resolve() — this branch renamed the MemoryRoot classmethod for clearer semantics (the old name was ambiguous with "default location"). MemoryRoot is a publicly-exported class, so a default() backward-compatibility alias is kept in this PR, forwarding to resolve() with a DeprecationWarning. Removal is left to a future major release. CHANGELOG.md under Unreleased carries the migration note. The rename is orthogonal to soft-dependency work but touches 26 files; reviewers wanting to isolate its impact can start from git log --oneline --diff-filter=M -- src/everos/core/persistence/memory_root.py in the squashed commit.

Layering verification

$ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra}
# empty
$ git grep -l "^import click\|^from click" src/everos/{memory,service,infra}
# empty

memory / service / infra layers are clean of CLI presentation libraries. This is a grep-verified invariant, not an import-linter contract — memory-layer forbidden-import contracts still only cover infra.persistence.** internals; extending the contract to typer / click is a followup.

Test plan

• make lint — ruff + import-linter (3 contracts) + assets + deprecated-names + github-docs + datetime + OpenAPI drift
• Hermetic env EVEROS_ROOT=/tmp/… EVEROS_EMBEDDING__API_KEY="" EVEROS_RERANK__API_KEY="" EVEROS_LLM__API_KEY="" full pytest = 2037 passed / 7 deselected (7 = slow + live_llm markers)
• Manual e2e — 21/21 assertions across Tier 1/2/3 + downgrade + v1/v2 double-mount
• /health reports correct capabilities + disabled_features per tier
• Four rounds of Fable 5 (opus) code review — 59 findings closed (10 + 13 + 15 + 21). See Round-4 review outcomes below.

Round-4 review outcomes

Round 4 surfaced 26 additional findings after the squash landed. Verdict / disposition:

Fixed in 561b5fe + 67758ec (21):

Severity	Finding	Fix
Blocker	N1	cluster_repo.find_cluster_id_for_member gained (app_id, project_id, owner_id) scoping via JOIN — entry_id is per-owner-unique, so unscoped reverse lookups silently mis-attributed same-day episodes across owners
Blocker	N2	Ctrl-C at the y/N prompt now dispatches to the exit-130 path (was falling through to exit-2 with rich traceback) — typer.Abort and click.exceptions.Abort are distinct classes under typer 0.15+
Blocker	N3	_looks_like_utf8_text mime whitelist tightened from startswith("text/") to an explicit set — text/html no longer bypasses everalgo's HTML sanitiser
Major	J3	MemoryRoot.default() restored as deprecation alias; PR body errata corrected
Major	J4	list_by_owner_after_ts(limit=…) docstring now spells out the "keeps oldest N" trap
Major	J5.2 / J5.5	Presenter nothing_to_backfill gains explicit scan_failed kwarg; dead phase_header Protocol method + implementations removed
Major	J6.3	3 inline ProviderNotConfiguredError imports promoted to top-level
Major	J7	Rows with any needed side still NULL after write count as rows_failed — exit code 4 now honest instead of silent 0
Major	J9	test_migration_cross_process.py → test_migration_lock_wiring.py; docstring scopes to "wiring only" and points at test_locking.py for real flock
Major	J10	Phase 1 gained the _probe_ome_lock_available preflight Phase 2/3 already had — --phase all against a live server halts before Phase 1 burns any embed budget
Minor	M1	NUL / > 255-byte filenames rejected up front (400) instead of surfacing as 500 with a half-written md
Minor	M2	Backfill optimize() passes cleanup_older_than=timedelta(0) to prune older manifest versions
Minor	M3	verify_business_schemas remediation walks restart → wipe (was jumping straight to rm -rf)
Minor	M5	Multimodal accessor logs multimodal_llm_capability_build_failed (was silent)
Minor	M7	parser_available.cache_clear() in the 2 tests that swap sys.modules['everalgo.parser']
Minor	M8	6 handler cascade_handler_embed_skipped logs demoted INFO → DEBUG
Minor	M10.1	Drift-assertion test now reloads _backfill with a synthetic-schema override and asserts on the actual RuntimeError
Minor	M11	test_cascade_verbose_position.py subprocess calls scrub EVEROS_* from env=
Minor	M14	/health return type is HealthResponse Pydantic model; OpenAPI schema regenerated
Minor	M16	routes/knowledge.py:_require_knowledge_capabilities docstring rewritten to describe the current design (was still citing the removed build_handlers gate)

Refuted (1):

• M12 — review claimed cascade_backfill_row_embed_failed doesn't exist; it does, at _backfill.py:501.

Deferred to followup (13):

• J1 — root-cause fix for parser/_core.py:74 needs everalgo signature change to support lazy LLM factory; UTF-8 short-circuit covers the user-facing path meanwhile
• J2 — Tier definition confirms "knowledge = Tier 3 whole" (Y2 was a design proposal, rejected as-designed)
• J5.1 / J5.3 / J5.4 — broader presentation-split refactor (domain-layer user text, auto_yes percolation, import-linter private-name contract)
• J6.1 / J6.2 — run_backfill phase-dispatch and _backfill_table refactor (both ~100+ lines of restructuring)
• J8 — Phase 1 keyset pagination for genuine bulk migration (500k+ rows)
• M4 — fcntl.flock timeout / waiting log / re-entry guards (core/persistence refactor)
• M6 — UTF-8 codec strategy (BOM / UTF-16 / GBK fallback)
• M9 — count_by_owner non-monotonicity (INTERVAL=1 short-circuits it today; latent)
• M13 — --phase all multi-phase combined-outcome coverage (priority is sequential-loop-defined; test additions only)
• M15 — PR-marker rationale comments (16 occurrences, all inert)

Known follow-ups (beyond round-4)

• _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version dispatch table
• extract_user_profile.py throttle-counter block — replace LanceDB count_by_owner with a SQLite memcell count
• PDF/DOCX uploads still require [multimodal] by design; the 415/503 error text could be more targeted

🤖 Generated with Claude Code