Skip to content

fix(flows): paper_flow draft-schema + code-assembled UUIDs#88

Open
bluezdot wants to merge 8 commits into
LLMQuant:masterfrom
bluezdot:fix/paper-flow-draft-assembly
Open

fix(flows): paper_flow draft-schema + code-assembled UUIDs#88
bluezdot wants to merge 8 commits into
LLMQuant:masterfrom
bluezdot:fix/paper-flow-draft-assembly

Conversation

@bluezdot

@bluezdot bluezdot commented Jul 7, 2026

Copy link
Copy Markdown

Problem

paper_flow could not produce a Paper end-to-end against a live model — a defect the test suite never caught because it mocks the SDK (scripts/verify.sh stayed green). Running it for real surfaced three failure modes:

  1. Strict JSON schema rejected Paper. Agent(output_type=Paper) fails at Runner.run with UserError: Strict JSON schema is enabled… — key-independent. PaperTreeKnowledge.nodes: dict[UUID, TreeNode] maps to an open-ended additionalProperties object, which strict mode forbids.
  2. The model cannot reliably emit UUIDs. With a non-strict wrap, it returned semantic ids ("paper_001", "introduction") for the many UUID-typed fields (id, root_node_id, node_id, the nodes keys, Citation.tree_id/node_id).
  3. Even when instructed to emit UUIDs, it corrupts them. Prompt-patching only reduced the errors: the model hallucinated an invalid hex char (…-bdle-… for …-bd1e-…) and reused one UUID for two nodes (a duplicate JSON key that silently drops a node).

Asking an LLM to hand-generate dozens of unique, valid, cross-referenced UUIDs is structurally unreliable.

Fix

Stop the LLM from ever touching an identifier:

  • New leaf module quantmind/flows/_paper_draft.py defines an id-less DraftPaper (it is the root node: carries title/summary/content/citations/children directly) with nested DraftNode/DraftCitation.
  • assemble_paper(...) is a pure function that walks the draft, mints every uuid4(), wires parent_id/children_ids/root_node_id and each Citation.node_id/tree_id, and builds the real Paper.
  • paper_flow sets the Agent's output_type to AgentOutputSchema(DraftPaper, strict_json_schema=False) and derives provenance (SourceRef/as_of/ExtractionRef(flow="paper_flow", model=cfg.model)) in code rather than trusting the LLM.

Design rationale: docs/design/en/2026-07-07-paper-flow-draft-assembly.md.

Verification

  • scripts/verify.sh green: ruff, basedpyright, import-linter (apex contract intact), pytest — 247 tests, 89.79% coverage (floor 75%).
  • Live end-to-end on a real PDF now returns a valid multi-node Paper with no extra_instructions, code-owned provenance, and no schema/UUID errors.
  • A test pins the Agent's output_type to the non-strict DraftPaper schema so this regression cannot silently reappear behind mocks.

Notes

  • DoiIdentifier still raises NotImplementedError (OA-PDF resolver is a separate follow-up).
  • SourceRef.fetched_at/content_hash intentionally left None in this MVP; documented as follow-ups.

🤖 Generated with Claude Code

bluezdot and others added 8 commits July 7, 2026 14:52
Root-cause and fix design for paper_flow being unable to produce a Paper
end-to-end (strict-schema rejection + LLM UUID hallucination/reuse).
Chosen approach: nested id-less draft schema; code assembles all UUIDs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Transitional: unblocks Runner.run past strict-JSON-schema rejection of
Paper's dict[UUID, TreeNode]. Superseded by the draft-schema design.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes end-to-end extraction: the Agent no longer emits UUIDs (eliminating
strict-schema rejection and UUID hallucination/reuse); provenance is now
code-owned.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… validates

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ale doc

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@keli-wen keli-wen added type: bug Existing behavior violates its contract or documented expectation area: flows Public operation implementations under quantmind/flows/ labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: flows Public operation implementations under quantmind/flows/ type: bug Existing behavior violates its contract or documented expectation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants