docs(contexts): establish canonical design contexts and the Paper E2E contract

[keli-wen]

Summary

Make contexts/design/ the canonical home for QuantMind engineering design, and define the Paper end-to-end contract there before implementing PageIndex or making further structural changes to paper_flow.

This issue intentionally combines the context organization with its first concrete consumer. The directory shape should be established by real Paper design work, not by empty scaffolding.

Motivation

Design context is currently split: contexts/ acts only as a routing layer while the canonical component documents live under docs/design/. At the same time, Paper work is being discussed through individual implementation bugs and proposals without one contract that says what the full pipeline must do, where deterministic code ends and model extraction begins, and which guarantees future PageIndex integration may rely on.

We should establish that contract before choosing more schema fixes. PageIndex is not a prerequisite for Paper Flow; the Paper design should preserve the source and define a clean tree-building seam so PageIndex can be added later as one compatible implementation.

Canonical Context Layout

Organize design directly by domain under contexts/design/:

contexts/
└── design/
    ├── README.md
    ├── flow/
    │   ├── paper.md
    │   └── news.md
    ├── preprocess/
    │   └── <topic>.md
    ├── knowledge/
    │   └── tree.md
    ├── library/
    │   └── <topic>.md
    └── operations/
        └── naming.md

The exact files beyond flow/paper.md should follow the design content that actually exists. This layout has the following rules:

• Use direct domain namespaces such as preprocess/, knowledge/, library/, and flow/; do not add an intermediate components/ directory.
• Keep one global index at contexts/design/README.md; do not require a README.md in every domain directory.
• Do not create empty directories, placeholder documents, or speculative component pages.
• Move and refine canonical design content from docs/design/ rather than copying it. Remove the old canonical path after links are updated.
• Keep docs/ for user-facing documentation, examples, and public catalogs; it may link to design context but must not maintain a second copy.
• Treat flow/ as a documentation grouping for cross-domain E2E behavior. It does not override the public operation naming contract or require new *_flow APIs.
• Review migrated documents against the current code and accepted architecture. In particular, do not blindly preserve stale storage-layer descriptions if quantmind/library/ now owns that responsibility.

Paper E2E Design Contract

Add contexts/design/flow/paper.md as the canonical description of the target Paper pipeline:

Paper input
  -> resolve and fetch
  -> page-preserving source document plus authoritative metadata
  -> structural extraction draft
  -> deterministic canonical assembly
  -> tree and provenance validation
  -> Paper

The document must define:

• supported inputs, explicitly unsupported inputs, and source-resolution behavior;
• the page-preserving intermediate representation between PDF parsing and semantic extraction;
• authoritative metadata ownership, including source URL, publication time, availability/fetch time, authors, and extraction provenance;
• the boundary between model-produced semantic structure and code-owned IDs, parent/child edges, page spans, citations, and source content;
• deterministic and nondeterministic stages, retry/failure behavior, and the meaning of a successful result;
• canonical Paper and Tree invariants, including root validity, bidirectional parent/child consistency, reachability, acyclicity, span bounds, and safe traversal;
• whether branch nodes may retain source content and how content is sliced from source pages;
• the output boundary: Paper extraction returns canonical knowledge; persistence, semantic indexing, retrieval, and answer synthesis remain separate consumers;
• current behavior and known gaps, so the target contract is not confused with what the repository already guarantees.

Future PageIndex Compatibility

The Paper design must reserve a narrow compatibility seam without implementing PageIndex:

• Preserve ordered source pages before any tree builder runs.
• Represent structural locations with an explicit span unit and documented indexing convention; use 1-based inclusive page spans for the PDF golden contract.
• Keep external PageIndex node IDs adapter-local; canonical IDs and graph links remain code-owned.
• Allow a future tree builder or adapter to provide titles, summaries, and page spans without becoming the canonical Paper schema.
• Separate outline/navigation from fetching the tight source page range for a selected node.
• Do not assume sibling spans are non-overlapping or that every child range must be strictly contained by its parent unless the canonical contract independently requires that rule.

This issue should make it explicit that Paper Flow can be implemented first and PageIndex support can be added later against this seam.

Golden Fixture

Add one small, repository-owned synthetic fixture:

tests/fixtures/paper/golden/
├── paper.pdf
└── expected.json

The PDF should be small enough for fast offline tests while containing multiple pages, at least one nested subsection, and at least one section spanning multiple pages.

expected.json should define only stable, reviewable facts:

• PDF page count;
• key section titles and paths;
• 1-based inclusive page spans;
• distinctive text anchors needed to verify those spans;
• expected parent/child topology;
• Tree invariants to validate.

Do not assert exact model summaries or other wording that can vary between model runs. Add a deterministic unittest.TestCase-based contract test that reads the fixed PDF and manifest, checks the page count and anchors, and validates the expected topology and invariants without network access or an LLM. Later Paper and PageIndex implementation issues should reuse this fixture rather than create competing fixtures.

Deliverables

• Update contexts/README.md so contexts/design/ is a canonical design source, not only a routing layer.
• Add contexts/design/README.md as the single design index.
• Move and refine applicable content from docs/design/ into direct domain directories under contexts/design/.
• Remove the superseded docs/design/ canonical copies and update all repository links.
• Add the complete Paper E2E contract at contexts/design/flow/paper.md.
• Add the synthetic Paper golden PDF and expected.json.
• Add an offline deterministic contract test for the fixture using unittest.TestCase.
• Ensure the design explicitly defines the future PageIndex seam without making PageIndex a prerequisite.
• Ensure scripts/verify.sh passes.

Non-goals

• Fixing the current structured-output failure tracked by paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91.
• Implementing PageIndex or tree-navigation behavior tracked by feat: page-preserving TreeKnowledge build + agentic navigation #95.
• Implementing Paper-to-library semantic search tracked by feat(example): paper extraction to semantic knowledge search #112.
• Renaming public operations or moving runtime packages.
• Finalizing every possible component design document.
• Creating placeholder directories or per-domain README files.
• Treating live LLM output as a deterministic golden result.

Related Issues

• paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91: current paper_flow structured-output failure.
• feat: page-preserving TreeKnowledge build + agentic navigation #95: future PageIndex-compatible tree navigation.
• feat(library): local semantic knowledge library MVP #111: canonical library and semantic-index foundation.
• feat(example): paper extraction to semantic knowledge search #112: Paper extraction to semantic knowledge search E2E.