Add protocol-native trajectory histories and tokenization

[bradhilton]

Summary

• add mutable protocol-native history views for Chat Completions, Anthropic Messages, Responses, and token/string Completions
• move tokenization onto histories, trajectories, and trajectory groups with explicit single- versus multi-history overloads
• capture Chat Completions, Completions, Responses, and Messages transparently through HTTPX, aiohttp, and requests, including streaming reconstruction
• preserve provider-sampled token IDs/logprobs while rendering only missing prompt, scaffold, or completion regions
• consume ordered Responses token_generations, retaining server-side tool turns and splitting non-append-only reasoning continuations into separate histories
• make Completions exhaustive across batched prompts, multiple choices, token/string prompts, mixed evidence, textual logprobs, and echo=True
• keep direct, validated source provenance for mutable views and fail loudly on sourced-item contradictions
• require unambiguous concrete model selection for training and apply loss to each sampled generation once

Behavior

Exchange trajectories identify their model from captured protocol data. Legacy trajectories require explicit model= when tokenized. base_model=, tokenizer, chat template, and template kwargs remain optional and are inferred where possible.

Exact provider evidence wins. ART preserves sampled IDs/logprobs across later prompt retokenization, reasoning-stripped branches, hidden EOS/control tokens, tool turns, trimmed whitespace, and protocol conversion. Fallback textual evidence uses provider byte reconstruction, validates contextual tokenizer boundaries when available, and otherwise falls back to independently proven one-token pieces.

Converted Chat sources identify every contributing Responses output index. Singular tokenization raises when a trajectory resolves to multiple histories; multi_history=True returns all selected histories. Mixed protocols and unresolved opaque cross-protocol conversions fail closed.

Validation

• final CPU-installable unit suite: 700 passed, 15 skipped
• trajectory-focused suite: 264+ passed across capture, histories, tokenization, serialization, and training traces
• regressions cover multi-choice source selection, repeated-text attribution, request capture transparency, generic-group NaN serialization, Responses generation provenance, refusal/content evidence, batched/echo Completions, model selection, split UTF-8 token bytes, noncanonical tokenizer boundaries, whitespace-preserving/trimming templates, immediate EOS, tool-probe collisions, and long-history scaling
• real cached Qwen 3/3.5 tokenizer checks preserve exact/textual evidence across whitespace, refusal, reasoning, tools, continuation subwords, and noncanonical token boundaries
• Ruff, formatting, ty, lock consistency, sdist/wheel build, and package import pass

Previously completed live validation includes Qwen base/LoRA across all four protocols and both engines, OpenRouter across all four protocols and streaming modes, and native OpenAI/Anthropic SDK parsing. Companion OpenPipe/caladan#87 provides the authoritative generation and prompt evidence consumed here.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
openpipe-art	🟢 Ready	View Preview	Jul 23, 2026, 11:02 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.