Import PT9 interlinearizations end-to-end (parser → InterlinearProject)

[imnasnainaec]

Umbrella for the goal raised in #94: fully support importing PT9 interlinearizations into an InterlinearProject — the end-to-end pipeline from PT9 data to a working, editable project.

Design principles this follows

• Simplify project creation — import produces an ordinary project, not a locked "PT9 mode." The model/output relationships it discovers are attached to the result (per Attach the discovered model and output projects to an imported PT9 project #215), never baked into a rigid creation mode.
• Generalize the interlinearizer — an imported PT9 project behaves like any other: its source, model, and output can be changed or dropped afterward.
• First-contact discoverable — import is offered where a user would look for it (e.g. from the project picker / "open" flow), not as a required up-front decision.

Current state

The XML parsers this issue started from are gone. #272 replaced them with a converter over the platform-parsed Pt9InterlinearProjectData payload, fetched through the read-only Pt9Interlinear projectInterface — which settles where PT9 data comes from: a PAPI-exposed project addressed by its Platform.Bible sourceProjectId, so there is no ScrTextName mapping and no file picker. src/parsers/pt9/ now holds only lexemeKey.ts and the schema notes in pt9-xml.md. The pipeline lives in src/converters/pt9/ and pt9ImportService.ts. Earlier parser groundwork, now superseded: #237, #238.

Checklist against the scope items below — link the PR that completes an item as you check it.

• 1. Import command + UI — Add the Paratext 9 import service, storage, provenance, and commands #273 (service, storage, provenance, commands), Add the Paratext 9 import WebView experience with a first-open offer #274 (import WebView with a first-open offer), Seal the read-only import and make its sync land #292 (seal the read-only import, make sync land). Shipped as interlinearizer.importPt9Project taking a sourceProjectId, not the interlinearizer.importPt9 + file picker the scope item guessed.
• 2. Cluster → Token mapping — clusterAnchoring.ts (Convert the platform-parsed PT9 interlinear payload instead of raw XML #272). Clusters that cannot be anchored drop and are counted (clusterDrops, ambiguousAnchors).
• 3. Gloss-text resolution — gloss text is inlined from the payload's lexicon (pt9GlossSource.ts, Convert the platform-parsed PT9 interlinear payload instead of raw XML #272). The lexicon-ref half is deliberately unimplemented: the Pt9LexiconResolver seam is wired at every ref site but no implementation is supplied, so imports carry zero refs — Add lexicon ref resolution when copying a Paratext 9 import for editing #280.
• 4. Punctuation / Excluded / Hash — languageAnalysisBuilder.ts + analysisMerger.ts (Convert the platform-parsed PT9 interlinear payload instead of raw XML #272). Punctuation entries are ignored and counted (punctuationEntriesIgnored); Excluded → 'rejected'; Hash → 'approved'/'suggested'.
• 5. Keys & offsets — clusterAnchoring.ts (Convert the platform-parsed PT9 interlinear payload instead of raw XML #272), but not as this issue specified: a cluster whose range does not match the tokenized baseline is dropped and counted (clusterDrops.formMismatch, versesNotFound), never marked stale. staleKept in pt9ImportService is a different notion — the source's interlinear files vanished, so an earlier import is kept rather than replaced.
• 6. Multi-file merge — languageAnalysisBuilder.ts, glossLanguageTags.ts, convertPt9Project.ts (Convert the platform-parsed PT9 interlinear payload instead of raw XML #272). Per-language reports plus merge.sameTagCollisions cover files that disagree on a tag.
• 7. Link model & output — Attach the discovered model and output projects to an imported PT9 project #215.

Open follow-ups on what shipped: #280 (lexicon refs), #283 (test projects, PR open), #285 (morphemes with blank forms), #296 (report prints the same language tag twice).

Scope — the pipeline

1. Import command + UI. An interlinearizer.importPt9 command and an entry point (file/folder picker for Interlinear_<lang>_<book>.xml, or selection of a PT9 project). Creates an InterlinearProject via projectStorage / interlinearizer.createProject.

2. Cluster → Token mapping (the crux). PT9 Cluster.Range is an Index/Length span in the verse text; our tokens carry charStart/charEnd within Segment.baselineText from bookTokenizer. Reconcile the two so each cluster resolves to the token(s) it covers. A single-lexeme cluster over one token → a TokenAnalysis; a cluster spanning multiple tokens → a PhraseAnalysis; a multi-lexeme cluster → a morpheme breakdown (TokenAnalysis.morphemes).

3. Gloss-text resolution. The XML stores only LexemeId + SenseId — not gloss text. PT9's
   own lexicon files hold the text, and inlined gloss text is what every imported record renders
   from.

   A lexicon reference is stored when a resolver maps a PT9 lexical identity onto a lexicon that
   can actually resolve it (Pt9LexiconResolver, Convert the platform-parsed PT9 interlinear payload instead of raw XML #272). Under Namespace lexicon refs by provider before any ref is persisted #225 every ref carries a required
   authority naming the id space that minted its ids, so:

   • The ref a resolver returns carries the authority of the lexicon it resolved against, not
     PT9's.
   • projectId is likewise that lexicon's project — not the PT9 project, and not the
     Platform.Bible sourceProjectId.
   • Refs are stamped at construction. Namespace lexicon refs by provider before any ref is persisted #225's persistence guards reject a ref that names no
     authority, and one unstamped ref fails validation for the whole draft rather than just that
     record.

   The resolver shipped today (unresolvedPt9LexiconResolver) resolves nothing, so import persists
   no refs at all yet. Whether an identity that resolves against no lexicon is additionally stored
   as a foreign PT9-authority ref is open, and owned by Add lexicon ref resolution when copying a Paratext 9 import for editing #280.

4. Punctuation / Excluded / Hash. Punctuation tokens come from the tokenizer already; PT9
   Punctuation change records are likely informational. Excluded clusters import as
   status: 'rejected'. Verse Hash present ⇒ 'approved', absent ⇒ 'suggested' — PT9 writes
   the hash only for an approved verse, so absence is the not-approved state (pt9-xml.md:49).

   Two further rules the shipped merger applies, neither derivable from a single cluster: gloss
   languages that disagree on a record reduce it to 'suggested' (mergeStatus), and a
   would-be-approved record whose tokens are already claimed by an approved one is demoted to
   'candidate' (counted as approvedDemotedToCandidate).

   confidence is not derived from approval. The interlinear XML carries no confidence source
   — no IsGuess, no Score, only Hash — and the importer spends the field on anchoring
   instead: confidence: 'low' marks a record whose token placement was a guess among equal
   folded forms (analysisMerger.ts:339, :406).

5. Keys & offsets. Verse key ("MAT 1:1") → Segment.id; cluster ranges → token offsets (step 2). Non-matching ranges (baseline drift) → stale handling.

6. Multi-file merge. One file per book per gloss language (Interlinear_<lang>_<book>.xml). Merge across books and languages into one project whose analysisLanguages union the GlossLanguages.

7. Link model & output. Split out into Attach the discovered model and output projects to an imported PT9 project #215.

Dependencies

• A lexicon — for refs, not gloss text (item 3 inlines the text). PT9-minted ids are their own
  id space, so resolution means mapping a PT9 lexical identity onto a lexicon holding its own ids
  for the same thing, and the first real Pt9LexiconResolver needs that lexicon's authority
  constant. Define the lexicon port and capability descriptor #226 defined the port and the authority/provider split (completed 2026-08-31); this
  model defines no authority values (interlinearizer.d.ts:403), so the constant comes from
  whichever lexicon is chosen — Add lexicon ref resolution when copying a Paratext 9 import for editing #280.
• bookTokenizer — the token grid clusters must map onto.

Out of scope

• Linking the model and output projects (Attach the discovered model and output projects to an imported PT9 project #215), and the engines behind them — the model-text suggestion engine (Wire a model project into the suggestion engine (surface-form tier) #148) and the output/export pipeline (Wire the output project (back-translation / adaptation) into export, not creation #149).
• Exporting back to PT9 XML (separate, if ever needed).

Open questions

• Gloss resolution: inlined gloss text always; a stored ref where a resolver resolves. Open,
  both in Add lexicon ref resolution when copying a Paratext 9 import for editing #280: who implements the first real Pt9LexiconResolver and against which lexicon, and
  whether an unresolved PT9 identity is stored as a foreign ref so it survives into an editable
  copy.
• Should this go to user-questions.md for review outside the dev team? (per AGENTS.md UX-decisions guidance)

Size: L (multi-stage pipeline; the cluster↔token mapping and gloss resolution are each substantial).
Priority: P1 — this is the concrete goal ("fully support importing PT9 interlinearizations").

[imnasnainaec]

Related: #144

[alex-rawlings-yyc]

My thought on the open questions:

• Both; import ref to lexicon extension and literal gloss text to our model
• Seems outside of our jurisdiction on first read
• Not quite sure about this one
• Probably not a user-facing issue

[alex-rawlings-yyc]

Absorbing #144 ("Import PT9 project"), closed as a duplicate of this issue. Its only content beyond the title was a scheduling note:

Further support of RTL in core may be necessary to begin on this

Recorded here so it is not lost. Note that the note refers to RTL support in paranext-core, not to our own #97 — a blocked by #97 edge has been added as the closest trackable stand-in, but if the real constraint is upstream, that edge should be replaced with a link to the relevant paranext-core issue.

Assignee carried over from #144.