Add exact token content attribution

[bradhilton]

Summary

• add TokenFlag.CONTENT plus sequence-level content_attribution_complete
• mark caller/model body tokens only after the renderer reproduces every engine-consumed prompt ID exactly
• preserve fail-closed behavior for missing, conflicting, malformed, ambiguous, or non-matching prompt metadata
• document safe consumer guards: require complete attribution and a nonempty mask before taking a mean

Proven support matrix

Surface	Qualified models	Evidence requirement
Chat Completions, Anthropic Messages, OpenAI Responses (streaming and non-streaming)	ART-registry Qwen3 dense/MoE chat models, Qwen3.5/Qwen3.6 dense/MoE models, OpenPipe/Qwen3-14B-Instruct with its documented tool-history limitation, and openai/gpt-oss-20b / openai/gpt-oss-120b	Exact full equality with engine-consumed prompt_token_ids for every exchange and exact sampled output token IDs

Gemma 4, DeepSeek V4, explicit -Base variants, raw Completions, arbitrary custom templates, unknown models, multi-generation Responses requests, and any incomplete or divergent token metadata remain unavailable and leave CONTENT unset.

Validation

• ART focused trajectories + pinned renderer parity: 100 passed
• ART backend unit partition: 386 passed, 3 skipped
• all ART repository hooks: ruff, format, ty, lock check
• disposable H200 live test with vLLM 0.24.0 / Qwen3-0.6B: exact prompt-ID parity for Chat Completions, Messages, and Responses, both streaming and non-streaming
• persisted Claude Fable 5 adversarial review: no remaining high- or medium-severity findings

Known gaps

• Megatron/Transformer Engine-specific unit partition was not runnable locally; the PR's CI quality job subsequently covered it and passed
• live SGLang engine parity was not run; pinned SGLang 0.5.15 protocol shapes are covered by fail-loud bootstrap guards and focused protocol tests
• no dedicated cached CI job currently installs the optional renderer parity dependency; the pinned parity suite was run locally

Companion Caladan PR OpenPipe/caladan#82 depends on this contract and supplies the exact engine prompt-token carriers.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
openpipe-art	🟢 Ready	View Preview	Jul 22, 2026, 1:22 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
openpipe-art	🟡 Building	–	Jul 22, 2026, 1:22 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.