Skip to content

feat(example): paper extraction to semantic knowledge search #112

Description

@keli-wen

Feature Summary

Add one end-to-end example and integration test that proves the complete financial knowledge path:

paper input
  -> paper_flow
  -> canonical Paper / TreeKnowledge
  -> LocalKnowledgeLibrary.put
  -> semantic search
  -> typed SemanticHit evidence

This issue stays separate from #111 so the semantic library can be developed with deterministic fixtures while the existing paper-flow bug in #91 is resolved.

Motivation

A local semantic index is not sufficient unless real extraction output can enter the library and be found again without losing:

  • information-cutoff and source-availability time;
  • source and extraction provenance;
  • tree/node identity;
  • citations;
  • typed financial query filters.

Proposed Implementation

Add one focused example:

examples/library/search_paper.py
paper = await paper_flow(input, cfg=paper_cfg)

library = await LocalKnowledgeLibrary.open(
    ".quantmind/library.db",
    embedding_model="text-embedding-3-small",
)
await library.put(paper)

hits = await library.search(
    SemanticQuery(
        text="What are the paper's main methodological limitations?",
        item_types=["paper"],
        available_at_before=research_cutoff,
        top_k=5,
    )
)
stored_paper = await library.get(hits[0].item_id)

Keep the example to this single use case. Do not add conversational RAG, PageIndex, reranking, or multiple providers.

Acceptance Criteria

  • Depends on the completed semantic-library contract from feat(library): local semantic knowledge library MVP #111.
  • Depends on a working default paper_flow from paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91.
  • Runs from a supported paper input to typed semantic hits.
  • Prints hit title/path, score, source, and citations without generating a final answer.
  • Uses a controlled model and embedding seam for an offline deterministic integration test.
  • Verifies both document/root and tree-node retrieval.
  • Verifies as_of_before as information-cutoff filtering.
  • Verifies available_at_before excludes unavailable and unknown-time knowledge at the research cutoff.
  • paper_flow propagates Paper.available_at and SourceRef.fetched_at from the actual source version. A later arXiv revision must not inherit the original publication time as its availability timestamp.
  • Resolves retrieved node IDs through library.get() to canonical nodes in the stored Paper.
  • Preserves source and citation metadata through extraction, storage, indexing, and retrieval.
  • Re-putting the same extracted Paper is idempotent and does not duplicate knowledge. Cross-run extraction deduplication is not claimed.
  • scripts/verify.sh passes.

Non-goals

Related Issues

Implementation Considerations

Breaking Changes

  • This feature introduces breaking changes.
  • This feature is backward compatible.

Dependencies

Metadata

Metadata

Assignees

Labels

area: examplesExamples are the primary deliverablearea: flowsPublic operation implementations under quantmind/flows/type: featureAdds a new capability or observable behavior

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions