[codex] Support raw image refs for multimodal rendering

[eligotts]

Design update — inline/offload image storage

This PR now supports both raw image transport modes used by prime-rl:

• offload: existing behavior, raw image bytes are written to run-scoped image assets and refs carry a file-backed image id.
• inline: data-image URIs remain inline and raw refs carry the inline source instead of requiring raw_image_id.

This repo adds inline-capable mmraw:v3 refs while preserving mmraw:v2 parsing, keeps Qwen image hashes aligned to the raw decoded bytes, and emits raw descriptor items with either raw_image_id or raw_uri.

Validation after latest push: uv run pytest tests/test_client.py -q passed (14 passed).

Design update — dropped the None/cache-only image path

This PR and its companions (prime-rl #2836 / verifiers #1746 / renderers #89) no longer use the "send None for already-cached images" mechanism. Every image carries its raw descriptor ref at every slot (current and prior turns); /inference/v1/generate rematerializes each ref from disk every request.

Why: the None path coupled correctness to deployment (LRU cache present, single replica / DP-affinity, no eviction) and surfaced a miss as a hard vLLM EngineDeadError (qwen3-vl mrope dereferences a None image_grid_thw) that the retry net couldn't catch across the engine→API IPC. Dropping it is deployment-agnostic (a miss is impossible) and non-hacky. vLLM's mm_hash encoder cache still skips the expensive GPU re-encode for free — we only forgo the cheap IPC/CPU-reprocess dedup.

Validated: color-codeword (Qwen3-VL-4B) under DP=2, no affinity / no cache reliance: 0 crashes, 0 data=None, multi-turn accumulation correct, reward ~0.84. Also confirmed under TP.

This repo: every image emits a raw descriptor ref at every slot. _descriptor_only_mm_data no longer strips the pointer (pixel_values were never present in v1, so the strip was both stale and the root cause of the descriptor-only/rebuild churn). Removed the materialize_all_image_refs flag and the now-orphaned materialize_image_refs / materialize_kimi_image_refs.

---

Original description

Summary

• adds generic mmraw: raw multimodal refs in renderers.mm_store, parsed as RawMMRef objects with family, modality, hash, image locator, and adapter-owned payload
• emits strict prime_raw_mm_item envelopes instead of processed image payloads for Qwen-VL and Kimi K2.5 image rendering
• keeps adapter-specific layout details in renderer-owned payloads (image_grid_thw for Qwen, grid_thws/media token metadata for Kimi)
• supports materializing all raw image refs for retry paths after vLLM multimodal cache misses
• keeps run-scoped image asset refs file-backed so downstream Prime-RL trainer materializes images with its own processor

Companion PRs

• Prime-RL: Support v1 raw multimodal image offload prime-rl#2836
• Verifiers: [codex] Support raw image offload in v1 train client verifiers#1746

Notes

• Draft/WIP: stacked with the Verifiers and Prime-RL raw image offload PRs.
• Verifiers is expected to offload image content to file://.../assets/images/... refs before rendering.
• This intentionally treats raw image refs as the supported path, not processed multimodal feature sidecars.

Validation

• uvx ruff@0.15.18 check . passed.
• uvx ruff@0.15.18 format --check . passed.
• uvx 'ty<0.0.22' check . exited 0; remaining diagnostics are warning-level advisories under the repo config.
• PYTHONPATH=/home/ubuntu/renderers uv run --no-project --active pytest -q tests/test_client.py passed: 14 passed.
• End-to-end hosted-style smoke through Prime-RL with /home/ubuntu/renderers, /home/ubuntu/verifiers, and /home/ubuntu/prime-rl-v1-raw-mm-offload completed inference, env rollouts, train batch creation, trainer step 0, and decoded strict trainer-bound raw image refs.

[!NOTE]

Add raw image ref support for multimodal rendering in Qwen and Kimi renderers

• Adds a new multimodal_output field (default "raw") to BaseRendererConfig that controls whether renderers emit processed tensors or raw multimodal reference envelopes (JSON-encoded mmraw: strings).
• Introduces renderers/mm_store.py with helpers for encoding/decoding raw multimodal refs, offloading data:image URLs to content-addressed files on disk, and fetching image processor configs from Hugging Face Hub.
• Introduces renderers/mm_image.py with shared utilities for loading PIL images (from file URIs, data URIs, or PIL objects), hashing, and dimension extraction — remote HTTP URLs are rejected.
• Refactors Qwen3-VL, Qwen3.5, and Kimi K2.5 renderers to support both modes: raw mode derives token counts from layout config without running a processor; processed mode lazily loads a processor only when needed. Removes image_cache_max from all renderer configs.
• Updates renderers/client.py to serialize raw multimodal items into features (mm_hashes, mm_placeholders, kwargs_data) and optionally offload image assets before the request; drops model-specific tensor preprocessing from the client.
• Risk: raw mode requires images to be pre-offloaded as file:// URIs — passing remote URLs or unencoded bytes will raise at render time.

^{Macroscope summarized 2fbf8bd.}

Update: review hardening (e3c12e9)

• Fixed render_completion_update mutating the caller's previous_multi_modal_data in place (shallow dict copy + setdefault().extend()); the merge now lives in a shared merge_multi_modal_data helper in base.py that copies inner lists, and the bridge test asserts the previous sidecar is unmutated.
• Qwen resize math is imported from transformers (torch-free PIL-backend module, with a fallback for older layouts) instead of maintaining a port.
• Raw layout describes resolve and read each image asset once; mm_store uses full sha256 content-addressed filenames and raises on undecodable base64.
• Added test_raw_layout_math_matches_image_processor: parity against the real Qwen3-VL-4B and Kimi-K2.5 (pinned revision) processors at rounding-boundary dimensions. Full suite: 2182 passed.

Update: merged main (e64cc58)

Reconciled with renderers main (12 commits: Hy3/PrimeQwen3/LagunaXS21 configs, kept-tokens parsing, SFT stop-token opt-in, a qwen thinking-wrapper fix, and bdb96b0 — main's own independent fix for the same bridge-mutation bug this PR's merge_multi_modal_data() already covers).

• kimi_k25.py/qwen35.py/qwen3_vl.py: kept this branch's shared merge_multi_modal_data() helper over main's three duplicated inline fixes for the identical bug — same behavior, one implementation instead of three.
• test_multimodal.py: re-injected main's _skip_for_disabled_thinking_deviation skip guard into this branch's parallel renderer_cases/processor_cases structure at all 4 call sites; took main's tuple-based prior_mm/prior_counts assertion over this branch's redundant duplicate check.
• test_client.py, __init__.py, configs.py: non-overlapping additions on both sides, no semantic conflict.
• client.py: no textual conflict (main's fix landed near, not on, this branch's rewritten _build_vllm_mm_features) — manually verified the merged generate() is consistent (prompt_attr/mm_data naming intact).

Full suite: 2578 passed, 153 skipped, 1 xfailed.

Update: unwrapped the ref payload (7c58fd6)

raw_mm_ref serializes its payload as compact JSON instead of base64-wrapping it. The ref travels as a string inside a JSON request body, so the wrapper's 33% inflation bought nothing. The saving is small in offload mode (the payload is a short file:// URI), but it keeps the ref format identical to the inline bundle, where the payload carries the image source and the wrapper cost ~32 KiB per image slot. Refs parse with a single partition(":") now that the payload contains colons.

Update: synced with main (1af6589)

Merged renderers main (5 commits: vLLM token-serving import fix #115, GLM tool-name validation #114, setuptools bump, ruff-action pin).

One conflict, in client.py: main's #115 updated the vLLM mm_serde import path inside the processed-tensor encoder that this PR deletes by design. Kept this branch's raw-ref _build_vllm_mm_features; main's fix applies to code that no longer exists here.

Full suite: 2622 passed, 125 skipped, 1 xfailed.

Update: single raw-mode image route (66505d2)

Raw mode now accepts exactly one image source: an offloaded file:// URL. The bare-filesystem-path fallback was a second route the rest of the stack couldn't honor anyway — prime-rl's trainer-side file_uri_to_path already rejects non-file:// sources and verifiers ingress only ever emits as_uri(). Anything else raises the existing "requires offloaded file:// image assets" error.

Update: layout knobs sourced from the checkpoint config (f0d1dd9)

Deleted the baked QWEN_VL_IMAGE_LAYOUT / KIMI_K25_IMAGE_LAYOUT constants. The layout spec dataclass is now the canonical knob list per family:

• Renderers fill it from the checkpoint's preprocessor_config.json — a cached raw JSON read (hub_image_processor_config), never a processor instantiation, so render hosts still avoid torch and (for Kimi) trust-remote-code execution. No baked fallback: a missing knob fails loudly.
• One dual-source extractor per family (qwen_layout_from / kimi_layout_from) fills the spec from either the config dict or a live image processor object (prime-rl's adapters use the latter, deleting their hand-duplicated knob lists). Qwen honors both config shapes (top-level min_pixels/max_pixels, or nested size.shortest_edge/longest_edge) and live-processor SizeDict attributes.
• fingerprint() on the spec hashes exactly its own fields, so the knob list and the hash can't drift. Fingerprint values are byte-identical to before.
• The parity test now also asserts our grid math against HF's metadata-only oracle get_number_of_image_patches (Qwen), and asserts config-sourced spec == live-processor spec for both families (this caught a SizeDict extraction bug in review).

New runtime requirement: the tokenizer must carry name_or_path and the checkpoint's preprocessor_config.json must be resolvable (local path, HF cache, or hub); failures surface at first image render.

Full suite: 2622 passed, 125 skipped, 1 xfailed.

Update: fingerprints deleted; layout resolved once per renderer (29d443c)

Companion to prime-rl's "trust the train-time checkpoint contract" change: materialize no longer re-checks layout knobs (it keeps output-level grid asserts, which need no knob extraction), so the fingerprint machinery is deleted end to end — no image_layout_fingerprint, no fingerprint() on the layout specs, no fingerprint/layout_fingerprint fields in mmraw: refs or descriptor envelopes (a ref format change; nothing is merged, both bundles move together).

Render-side resolution is also simplified to the cleanest shape:

• hub_image_processor_config is a local-path check plus one cache-aware hf_hub_download (the previous local_files_only two-step re-implemented hub internals).
• Extractors (qwen_layout_from / kimi_layout_from) are Mapping-only — config JSON is the single knob source; the live-processor/SizeDict duck-typing is gone with its only caller.
• Each VL renderer resolves its layout spec lazily once on first image render and stashes it (_raw_image_layout), mirroring the existing lazy _get_processor idiom; describe_*_layout / *_image_item_for_render take the spec directly instead of a model name.

Full suite: 2622 passed, 125 skipped, 1 xfailed.

Update: pruned unread layout knobs and envelope fields (aa4b0c6)

Review follow-up to the fingerprint deletion — removed everything only fingerprints consumed:

• Kimi spec drops image_mean/image_std (+ the float_triple validator); Qwen spec drops temporal_patch_size (grid_t is 1 for images). Specs now carry exactly what the geometry math reads.
• Kimi's ref payload drops num_media_tokens (no consumer; adapters read grid_thws only). The descriptor field stays — the parity test asserts it against the processor's media_tokens_calculator.
• The descriptor envelope drops its modality key: an item's modality is its key in MultiModalData.mm_items; only the standalone mmraw: ref carries its own (consumed by the vLLM-front routing check).

Full suite: 2622 passed, 125 skipped, 1 xfailed.

---

Note

High Risk
Changes core multimodal wire format, vLLM client encoding, and image input requirements (file offload only in raw mode), with companion PRs required for end-to-end rollout.

Overview
Multimodal inference path is reworked so Qwen-VL, Qwen3.5, and Kimi K2.5 no longer run HF image processors on the default path. They predict placeholder counts from checkpoint preprocessor_config.json and emit prime_raw_mm_item envelopes (family payload + raw_image_uri). multimodal_output on base renderer config selects raw (default, inference) vs processed (SFT-style pixel_values). Raw mode requires offloaded file:// assets; shared mm_store / mm_image handle offload, refs, and local image I/O.

renderers.client.generate drops vLLM/torch-specific Qwen tensor encoding and builds a generic features payload of hashes, placeholders, and mmraw: refs per slot (including prior turns—no cache-only None slots). image_cache_max and constructor processor args are removed; processors load lazily only for processed.

Bridge turns use merge_multi_modal_data so prior sidecars are merged without mutating caller state. AutoRendererConfig propagates multimodal_output. Optional renderers[vision] extra adds Pillow/torch for processed paths. Tests and docs shift parity fixtures to file:// images and assert raw-ref serialization and layout math vs real processors.

^{Reviewed by Cursor Bugbot for commit 2fbf8bd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new multimodal_output config option that changes how multimodal images are serialized by default (from processed tensors to raw image refs). The changes span multiple new modules, refactor three renderers, modify the inference client wire format, and remove existing caching logic — substantial new capability that warrants human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e64cc58. Configure here.}