Split divergent token histories by default

[bradhilton]

Summary

Make protocol-native trajectory histories preserve divergent served token identities by default instead of automatically reconciling text-equivalent prefixes.

• split Chat Completions, Responses, Anthropic Messages, and string Completions histories when a later prompt does not extend the exact sampled-token lineage
• add reconcile_text_equivalent_tokenizations=True as an explicit opt-in on history projection, Trajectory.tokenize(), and TrajectoryGroup.tokenize()
• preserve exact sampled token IDs, SAMPLED provenance, and rollout logprobs when an unchanged generated suffix appears in a later split history
• remove the reconciliation option from token-based Completions histories, whose token identity is already explicit
• require exchange-based preprocessing/PipelineTrainer inputs to contain only inference-provided TokenFlag.EXACT tokens
• cache parsed generation evidence so divergent long-history projection scales with captured input size instead of repeatedly reparsing prior generations

Why

Rendered text can remain identical while its token IDs differ from the sequence originally sampled by the policy. Treating those turns as one history silently changes the RL sequence. The safe default is now to expose separate histories; callers may explicitly opt into best-effort text-equivalent reconciliation.

Splitting must not discard valid rollout evidence. When a later exact prompt proves that an earlier sampled output is retained, that output keeps its original token IDs, logprobs, EXACT, and SAMPLED flags in the descendant history.

The preprocessing path fails closed when exchange tokenization required local rendering, matching its strict tokens-in/tokens-out requirement.

Validation

• focused trajectory suites: 201 passed
• CPU-compatible broader unit suite: 591 passed, 6 skipped
• two unrelated OpenAI stream-close tests fail only in full-suite order on this workstation and pass 2/2 in isolation; CI is authoritative for their configured environment
• Ruff lint/format, ty, prek, and lock consistency: passed
• source distribution and wheel build: passed
• divergent 100-turn Chat projection improved from the reviewer-observed ~24 seconds to ~0.20 seconds; regression coverage asserts each exact generation is parsed once

The full GitHub quality, package-install, and TrainerRank classification checks will rerun against the updated head.