MLX backend: large, deterministic logit divergence between Apple Silicon Mac and iPhone on identical inputs (~0.5x logit compression, ~18% answer flips)

[vinceecws]

Description

TL;DR. We run yes/no visual questions against LFM2.5-VL-1.6B (MLX backend, int4 .pte) and read p("yes") from first-answer-token logprobs. The same model file, same image bytes, same prompt, greedy decoding produce systematically different logits on an iPhone 17 Pro than on an M3 Pro Mac — both platforms perfectly deterministic individually, but the iPhone's logits come out compressed to roughly half of the Mac's, which flips ~18% of near-threshold answers. We've ruled out our image pipeline (details below) and believe this happens inside the MLX-delegated forward pass. We'd like to know if this magnitude is known/expected and whether there's a precision knob we're missing.

A single concrete example (one photo, prompt Is this a photo of a retail purchase receipt? Answer yes or no., greedy, identical everything):

	first-token logprobs	No↔Yes margin	p_yes
Mac (M3 Pro)	No: −0.003, Yes: −5.941	5.9 nats	0.003
iPhone 17 Pro	No: −0.915, Yes: −1.415	0.5 nats	0.361

Same answer here, but the margin collapsed ~12×. On rows where the Mac is less confident, this collapse crosses 0.5 and the answer flips (e.g. another row: Mac p_yes 0.234 → iPhone 0.642). Across our 268-row benchmark the effect is a clean linear map, not noise:

logit_device ≈ 0.52 + 0.50 · logit_mac      (R² = 0.85, n = 129 single-call rows;
the full 268-row fit including multi-call recipes: a = 0.22, b = 0.54, R² = 0.73)

Expected: cross-device numeric differences on the order of fp16 rounding, no systematic answer flips.
Actual: a directional ~0.5× logit compression, always toward uncertainty on the phone.

Setup (exact pins)

• iPhone side: react-native-executorch 0.9.3 in-app. The MLX delegate executing the model is the unmodified prebuilt ExecutorchLib binary that 0.9.3 ships — we do not touch it.
• Mac side: a small CLI that compiles RNE 0.9.3's unmodified common/ runner sources (same vision_encoder.cpp, same OpenCV preprocessing) against software-mansion-labs/executorch@24f0f69 — the fork commit RNE 0.9.3's prebuilt binary was built from (the newest fork commit preceding RNE bca2a54, which last updated the binary; the later "compiled-execution" MLX update 10f535e4/28d8af6 is not in 0.9.3 and not in our build).
• Model: lfm_2_5_vl_1_6b_mlx_int4.pte from software-mansion/react-native-executorch-lfm-2.5 @ v0.9.0, same file both sides; same tokenizer files.
• Decoding: greedy on both sides (temperature 0, topP 1, minP 0, repetitionPenalty 1).
• Disclosure: both sides carry a small patch to the runner sources (not the delegate): (a) a backport of RNE main's fp16-input dtype fix in vision_encoder — stable 0.9.3 fails prefill with InvalidArgument on this MLX VL export without it; (b) a read-only tap that reports top-k logprobs of the first answer token. Determinism results below show the tap doesn't perturb decisions.

We're aware the Mac CLI is not a supported configuration — but the claim here is not "the Mac is right." It's that the same delegated graph, built from the same commit, executed on two Apple GPUs, disagrees with itself far beyond fp16-rounding levels, deterministically. The Mac just makes the second data point easy to instrument.

What we observed

• Answer agreement Mac vs iPhone: 81.7% (219/268), mean |Δp_yes| = 0.135; the same 49 rows flip on every run.
• Both platforms individually deterministic: two device runs 10 days apart — across an iOS update (26.5.2 → 26.6) and an app rebuild — gave 268/268 identical answers, max |Δp_yes| = 0.0023. The Mac CLI is exactly reproducible run-to-run.

What we ruled out before blaming the backend

1. Pixels (the decisive experiment). We captured the exact JPEGs the device feeds its encoder and fed those same bytes to the Mac CLI. The cross-feed reproduced the Mac's probabilities (mean |Δp| = 0.037, divergence-vs-Mac slope ≈ 0 flat), not the device's (mean |Δp| = 0.124, slope −0.5). Of 20 disagreeing rows in that subset, the cross-feed sided with the Mac 17 times. The gap does not travel with the image bytes.
2. Image decode. Identical JPEGs decode near-identically on both sides (max per-pixel diff 3/255). Much larger upstream pixel perturbations showed zero correlation with answer flips.
3. Build drift. Both binaries rebuilt from one clean commit of our repo; the gap reproduced to the row.
4. Sampling. Greedy both sides; determinism above.

Questions

1. Is a systematic ~0.5× logit compression between M-series and A-series GPUs a known behavior of the MLX backend (fp16 accumulation, different kernel paths, tile sizes…)? It is much larger than fp16 rounding folklore suggests, and it is directional.
2. Is there a knob we're missing — higher-precision accumulation, fp32 fallback for sensitive ops, or a vision-encoder precision option?
3. If this belongs with the ExecuTorch MLX delegate or MLX kernels rather than RNE, we're happy to file upstream — a pointer to the right repo/owners would help.

We can share the full minimal repro (the JPEG above + prompt + complete top-64 logprob dumps from both platforms, and the 268-row paired dataset) — happy to attach or run further diagnostics on our hardware.

Steps to reproduce

1. Export LFM2.5-VL-1.6B to .pte with the MLX backend (int4), or use the published software-mansion/react-native-executorch-lfm-2.5 @ v0.9.0 artifact.
2. Run a yes/no visual prompt with greedy decoding on an iPhone 17 Pro via react-native-executorch 0.9.3 and capture first-answer-token logprobs.
3. Run the identical image bytes + prompt through the same runner sources built against software-mansion-labs/executorch@24f0f69 on an Apple Silicon Mac (or any second Apple GPU).
4. Compare first-token logprobs: device logits come out compressed ~0.5× relative to the Mac; near-threshold answers flip. Repeating either side reproduces its own outputs exactly.

Snack or a link to a repository

No response (private repo; minimal repro artifacts — JPEG, prompt, both full logprob dumps — available on request)

React Native Executorch version

0.9.3

React Native version

0.81.5

Platforms

iOS

JavaScript runtime

Hermes

Workflow

Expo Dev Client

Architecture

Fabric (New Architecture)

Build type

Release mode

Device

Real device

Device model

iPhone 17 Pro (iOS 26.6); compared against MacBook Pro M3 Pro (macOS)

AI model

LFM2.5-VL-1.6B (lfm_2_5_vl_1_6b_mlx_int4.pte)

Performance logs

No response

Acknowledgements

Yes

[vinceecws]

Follow-up — cross-backend control run. We repeated the same 268-row benchmark on the same two machines with the XNNPACK export (lfm_2_5_vl_1_6b_xnnpack_8da4w.pte from the published registry; runner sources and prompts identical, greedy).

Result: the directional compression does not appear on CPU. Regressing (logit_iphone − logit_mac) on logit_mac:

backend	slope	R²
MLX (int4)	−0.50	0.85
XNNPACK (8da4w)	+0.014	0.001

So the ~0.5× logit shrink between M3 Pro and A19 Pro is specific to the MLX delegate's execution, not a general property of running this model on two different Apple chips.

For completeness/honesty: XNNPACK is not bit-identical across the two machines either — 0/267 rows have exactly equal p_yes, with mean |Δlogit| ≈ 0.73 of non-directional per-row noise (which still flips knife-edge answers). One caveat on that noise figure: our macOS XNNPACK build enables KleidiAI and different threading than the iOS prebuilt, so kernel selection may account for part of it. The MLX slope, by contrast, is directional, large, and survives every control we've run.

Both XNNPACK reports (full per-row logprobs) are available alongside the MLX ones if useful.

[vinceecws]

Update — Mac reference rebuilt at the prebuilt's exact source state; results bit-identical.

To close a validity hole in our own comparison, we fingerprinted the ExecutorchLib binary shipped in 0.9.3 (symbol inventory + metallib kernel set + build-path strings) to determine which software-mansion-labs/executorch commit it was actually built from. Findings, in case they're useful to you:

• The 0.9.3 (and 0.9.1) prebuilt's MLX C++ matches fork commit 5133162 ("Free MLX temp tensors at last use") — it contains compute_temp_lifetimes but none of the compiled-execution strings from 10f535e. The xcframework blob is unchanged from RNE 140bf84 (feat: ExecuTorch 1.3 with MLX (iOS) and Vulkan (Android) backends + Gemma 4 E2B #1223) through the v0.9.3 tag.
• The Metal kernel inventory (364 kernels) and MLX pin (0.31.1, ce45c52) are identical between that commit and its neighbors.

Our Mac CLI had been built from 24f0f69, one C++ commit earlier. We rebuilt it at 5133162 — the phone's exact MLX C++ state — and re-ran the full 268-row benchmark: bit-identical output to the previous Mac build (0/268 rows changed by any amount). So the temp-lifetime/floor_divide delta is numerically neutral on macOS, and the ~0.5× logit compression reported above is now measured between two builds of the same MLX delegate source, differing only in target (macOS/M3 Pro vs iOS/A19 Pro).

Combined with the XNNPACK control (flat slope, previous comment), we're fairly confident the effect lives in MLX GPU execution differences between the two chips (kernel selection/precision), not in source-version skew, our pipeline, or CPU-side code.

[msluszniak]

Hi @vinceecws,

Thank you for letting us know. I will investigate this one.

[vinceecws]

Additional data point: the logit compression slope is prompt-dependent.

Fitting logit_device = a + b·logit_mac per prompt reveals b is not a constant of the hardware pair:

• Controlled comparison (same 45 images, one device session, three different yes/no prompts): b = 0.32, 0.44, 0.35 for the three prompts respectively. Same pixels, same run, only the prompt text differs.
• Broader corroboration (a fresh 191-row paired run over never-before-used photos, grouped by prompt template; note images differ per group, so this is partially confounded with image population):

prompt template (group)	n	b	R²
cooler stem	42	0.45	0.85
shelf stem	40	0.46	0.82
endcap/side-cap stem	50	0.42	0.69
free-standing-display stem	20	0.39	0.82
micro-fixture stems	9	0.33	0.68
brand-presence stem	30	0.50	0.78

• Across runs/datasets we've now measured b from 0.32 to 0.57 depending on prompt, while any single (prompt, dataset) pair reproduces its own slope deterministically.

Implications for triage:

1. This rules out a fixed post-hoc rescaling of logits (a single global scale/temperature difference).
2. Since the lm_head is prompt-invariant, a precision difference confined to the final-norm/lm_head chain can't fully explain it either. The damping must arise at least partly earlier in the network (attention/KV path), where the prompt tokens actually differ.

The remaining hypotheses we can't distinguish from our side are (a) reduced-precision attention softmax on the A-series kernel path vs (b) int4 dequant scale math at reduced precision. A non-int4 MLX export run on both machines would separate them (slope flattens → dequant path; persists → general kernel precision), we're happy to run that on our hardware if an 8-bit MLX artifact of LFM2.5-VL exists or can be exported. Per-prompt paired logprob sets available on request.

[msluszniak]

@vinceecws I can export INT8 variant for you to check if it works. I'll ping you once it will be published on HF. Otherwise, I will research this case thoroughly.