Skip to content

fix(llm): normalize multimodal image paths to file:// URIs#1090

Merged
msluszniak merged 1 commit intomainfrom
fix/multimodal-image-path-normalize
Apr 22, 2026
Merged

fix(llm): normalize multimodal image paths to file:// URIs#1090
msluszniak merged 1 commit intomainfrom
fix/multimodal-image-path-normalize

Conversation

@msluszniak
Copy link
Copy Markdown
Member

@msluszniak msluszniak commented Apr 22, 2026

Description

LLMController.forward passed imagePaths straight through to nativeModule.generateMultimodal with no normalization. The native side requires the file:// prefix; without it, native throws "Read image error: invalid argument" with no further context. Callers can plausibly arrive with either form:

  • ResourceFetcher.fetch returns raw paths without file:// (per its own docstring on the fetch method).
  • Platform image-picker APIs (e.g. expo-image-picker) typically return file:///... URIs.
  • The same path string passed to a vision module's forward(...) works either way; the asymmetry between vision modules and multimodal LLM is undocumented.

This PR normalizes each image path inside LLMController.forward so both forms work, and updates the JSDoc on Message.mediaPath and LLMModule.forward.imagePaths to document the new contract.

Introduces a breaking change?

  • Yes
  • No

Strictly additive: previously-working calls (paths with file://) keep working unchanged. Previously-failing calls (paths without file://) now succeed.

Type of change

  • Bug fix (change which fixes an issue)
  • New feature
  • Documentation update
  • Other

Tested on

  • iOS
  • Android

The bare-path failure was reproduced on Android (Samsung Galaxy S24 Ultra) with LFM2-VL-1.6B while building a downstream consumer; both forms tested manually post-fix on the same device. Re-verification of both forms on iOS is recommended.

Testing instructions

import { LLMModule, LFM2_VL_1_6B_QUANTIZED } from 'react-native-executorch';
const llm = await LLMModule.fromModelName(LFM2_VL_1_6B_QUANTIZED);

// Both should now work; previously only the first did.
await llm.generate([
  { role: 'user', content: 'Describe.', mediaPath: 'file:///absolute/path/to/img.jpg' },
]);
await llm.generate([
  { role: 'user', content: 'Describe.', mediaPath: '/absolute/path/to/img.jpg' },
]);

Related issues

Addresses item 3 of #1086.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

The normalizer is module-scope (matching messagesForChatTemplate from #1089) rather than a class method because it doesn't depend on controller state.

The native multimodal pipeline expects image paths to be `file://`
URIs. `ResourceFetcher.fetch` returns raw filesystem paths without
that prefix (per its own docstring), and platform image-picker APIs
typically return `file:///...` URIs, so callers can plausibly arrive
at `LLMModule.generate`/`sendMessage` with either form. Today only the
prefixed form works — the bare path gets `"Read image error: invalid
argument"` from native, with no explanation in the error.

Normalize each image path inside `LLMController.forward` so both
forms are accepted, and document the new contract on the relevant
JSDoc (`Message.mediaPath` and `LLMModule.forward`'s `imagePaths`
parameter).

Refs #1086 (item 3).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@msluszniak msluszniak self-assigned this Apr 22, 2026
@msluszniak msluszniak added the bug fix PRs that are fixing bugs label Apr 22, 2026
@msluszniak msluszniak requested review from NorbertKlockiewicz and barhanc and removed request for NorbertKlockiewicz April 22, 2026 12:06
@msluszniak msluszniak merged commit e8d4305 into main Apr 22, 2026
4 checks passed
@msluszniak msluszniak deleted the fix/multimodal-image-path-normalize branch April 22, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug fix PRs that are fixing bugs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants