[RNE Rewrite] feat: Add OCR implementation - #1322
Merged
Merged
Conversation
…TC, rectifyQuad, warpByGrid, rotate)
…essive OCR screen
…OcrOptions.tables) Tables were the odd one out — orientation/dewarp had per-run toggles, tables fired unconditionally when a table region was found. Add RunOcrOptions.tables (default true when the model supports them; a per-run false skips the recognizer), gate the table pass on it, and surface a Table-structure sub-toggle in the demo alongside orientation/dewarp.
…ailing | _ glyphs)
Flat-gray strip padding left a content→pad seam EasyOCR's CRNN read as a
trailing glyph. Add OcrModelOptions.recognizerPadMode ('constant' default,
'cornerMean' = background-matched), thread it through the recognizer, default
EasyOCR to cornerMean. Not a decode bug — the argmax was predicting the seam.
…f composed strip Composing a vertical stack into one horizontal strip fed the recognizer a non-word column with its horizontal-word context, making it hallucinate words (and the vertical-digit phantoms). Recognize each glyph on its own and join top-to-bottom — drops the word bias, gives per-glyph confidence. Cost: one recognizer pass per glyph (fine for the opt-in vertical path).
Vertical reads skip the horizontal drop-score gate because they run lower-confidence. Expose RunOcrOptions.verticalMinConfidence so callers can raise it to filter the low-confidence hallucinations the vertical path emits.
Hardens the pipeline — encodeConstraint variant handling, snapUpDim lattice clamp, span-aware table fill, tagged shape-constraint union, dead compose-mode removal, JSI length guards and demo polish. Also pins the document pipeline to the now-published 0.9.0 model tag.
…chema contract Replaces the removed get_dynamic_dims_/get_enum_shapes_ companion API with validateSpec over model.schema: - detect/recognize input sizes are now per-dimension ConcreteDim domains, so one snap routine covers constant/range/enum and the CoreML enumerated grid needs no cross-dimension shape set. - the recognizer's width-to-CTC-timestep relation is read from the linear runtime constraint the model declares instead of inferred as a width/timestep ratio, which was wrong for EasyOCR's CRNN (512 = 4*127 + 4, not a multiple). - TextBoxExtractor declares the detect output layout it decodes, so the allowed spec covers the outputs instead of wildcarding them. - documentModels validates its four static methods in one spec; repeated symbols now enforce the vocab/hidden/feature agreements the manual checks did. - useOcr resolves its nested models through the single-pass resource download.
…HF repos The four OCR repos now follow the MODEL_SPEC layout the rest of the ecosystem uses — per-backend directories, a root config.json, and <model>[_<size>]_<backend>_<precision>.pte file names — published under v0.10.0, so they share NEXT_VERSION_TAG with the other rewrite models instead of the unprefixed 0.9.0 tag they had. Precision is now part of the path, so each preset spells out what it actually ships rather than hiding it behind a bare backend name.
PP-DocLayoutV3's RT-DETR head is set prediction, so the preset runs it with NMS off — but it still emits nested duplicates, e.g. a region for one line inside the region for the paragraph that contains it. Every region is OCR'd on its own crop, so those duplicates transcribed the same text twice: a page with "Lekcja / Indywidualna" came back as three blocks, one per nesting. IoU suppression cannot fix this (a small box inside a large one has low IoU), so the regions are now merged on containment, as PaddleX's `layout_merge_bboxes_mode: 'large'` does: largest first, drop any region a kept one already covers by >=80% of its own area. Visual regions never suppress, so a caption inside a figure is still read. It also stops recognizing those crops twice.
…very OCR backend The OCR and Model Inspector screens never read the safe-area insets the other eight screens already pad by, so "Run OCR" sat under the navigation bar, and the main menu was a fixed View whose tenth button was clipped by it. The menu scrolls now, still centered while it fits. The OCR screen also filtered its model list by platform instead of disabling what the platform cannot run, so CoreML was invisible on Android. All six variants are listed now, with CoreML disabled off iOS and Vulkan disabled on it, matching every other screen.
msluszniak
reviewed
Aug 13, 2026
msluszniak
left a comment
msluszniak
left a comment
Member
There was a problem hiding this comment.
All 15 points from the previous round are resolved. I also checked the new fetch path end to end: all eight EasyOCR charset.txt files, the PP-OCRv6 one and the retagged pp_ocrv6_xnnpack_int8.pte return 200 on the v0.10.0 tag, and the payloads really are JSON arrays of strings. collectRemoteSources picks up charsetPath automatically and download weights progress by byte size, so the sidecar does not skew the progress bar. Typecheck, ESLint and clang-format are clean.
Four small things left, none blocking.
…able one Addresses review on #1322: - The published charset is a JSON array of strings, so name it charset.json. Renamed on both HF repos and re-pointed v0.10.0; better now than after the release, when the URLs are load-bearing. - Failing to READ the charset is the re-fetch-the-asset case that model.cpp already reports as LOAD_FAILED; only malformed content is the caller's argument to fix. Split the try so each maps to its own code. - modelOpts.charset now documents the same indexing convention as charsetPath. - The filename precision tag names the DETECTOR only, so say that rather than leaving pp_ocrv6_xnnpack_int8.pte reading as a contradiction.
Member
|
Two comments for you @barhanc:
|
Member
Yes, let's drop the EasyOCR and make the pipeline PaddleOCR specific, instead of trying to cram two different pipelines into one. |
Member
|
@barhanc advantage of easyOCR comes from better vertical ocr feature. So I will keep export script and the model itself. I just wondering how we would add support for these features without breaking changes in the core package. |
Member
|
We can implement the code for OCR in separate package directly on top of rn-executorch. The TS code will just use our TS API and we can add ocr-specific native code in that lib that can directly include the rn-executorch headers (same way as we include JSI) for all the TensorHostObjects, etc. definitions, so I don't think there is any problem in this regard. |
Member
|
Ok, if we will use our core native code then I think it's fine. |
PP-OCRv6 is faster and reads better on ordinary pages, and without vertical OCR there is nothing EasyOCR does better, so core carries one detector family instead of two. EasyOCR comes back in a dedicated OCR package later. Removes the 8 language presets and, with them, everything only they used: the CRAFT extractor and its native decode, the component-box grouping and de-skew geometry, and the cornerMean strip padding. The published artifacts are untouched, so reintroducing EasyOCR is a matter of restoring this code. Also fixes the postinstall lib map, which still described OCR as CRAFT+CRNN on xnnpack only. PP-OCRv6 ships xnnpack, coreml and vulkan, so an app opting into the ocr feature was not getting the backends its models need.
barhanc
reviewed
Aug 20, 2026
… registry C++ (ocr_ops, image_ops): - Declare the real export shapes to fromJs ([1,1,H,W] detect, [1,T,V] probs) instead of accepting [..,H,W] and re-deriving the rank by hand. That also retires three checks that could not fire: numel % vocab is always 0 when vocab is the last dim, and the rank/numel guards are now the contract. - getRequiredProperty<float> directly rather than casting from double. - ctcGreedyDecode walks a std::span instead of raw pointer arithmetic. - Order extractDbnetTextBoxes as options, tensor, lock, data; move ctx down to its use; single-line the host-function registrations, matching the rest of image_ops; break up the two dense function bodies. Registry: - Spell the three PP-OCRv6 entries out as consts, like every other model here, instead of generating them through a factory. - Drop the normalization rationale that only made sense while the pipeline served two model families. Exports: group the OCR task with the other CV tasks, and stop re-exporting NormalizeOptions from the package root.
… pipeline
The engine/wrapper split and the generic detector abstraction were both there
to let one implementation serve EasyOCR and PaddleOCR. Only PP-OCRv6 is left,
so tasks/ocr/{engine,detectors,geometry,ocr}.ts collapse into a single
tasks/paddleOcr.ts, matching how every other CV task is laid out.
What that removes:
- TextBoxExtractor and the makeDbnet* factory. DBNet is the detector, its
output shape and decode thresholds are constants.
- Three of the four spec variants. PP-OCRv6 binds dynamic detect + dynamic
recognize on every backend, so the fixed-size combinations were dead.
- Everything configurable that the export already decides: detector and
recognizer norms, pad value, the custom decode hook. Only the confidence
threshold survives, and inference now takes a per-call override for it.
Kept, contrary to the review: the enum snapping and the CTC lattice check.
CoreML has no RangeDim, so both methods ship enumerated sizes there, and
recognize declares its width-to-timesteps constraint on all three backends.
Dropping either would break PPOCRV6_SMALL.COREML.
Renames for the public API: createOcr -> createPaddleOcr, useOcr ->
useOpticalCharacterRecognizer, runOcr -> recognizeCharacters, ocrUtils ->
paddleOcrUtils, and extractDbnetTextBoxes -> extractDbnetTextQuads, which now
returns Quad[] rather than a flat array for the caller to parse.
ops/quad.ts: Quad is a four-element tuple instead of a Point[], so quadSize
and splitWideQuad stop casting and orderQuad is Quad -> Quad. mapQuadToImage
takes from/to sizes like the other image helpers. orderByReadingOrder moves
here from the deleted geometry.ts, which is where it belonged.
Also drops fetcher.readTextFile again and reads the charset through
RNBlobUtil directly, as Kokoro and Supertonic do.
barhanc
reviewed
Aug 23, 2026
ops/quad.ts is now just the quad type and the operations on it: - Quad no longer claims its corners are ordered. The return type was lying wherever a detector handed one back, so callers had to know out of band which quads had been through orderQuad. Helpers that need the order say so. - boundsOfPoints -> boundingBoxOfPoints, mapQuadToImage -> scaleQuad with a from/to/resizeMode options object mirroring scaleBox. - rectifyQuad moves in from ops/image. restrictToBox is the precedent: a native op that takes a geometry type lives with that type. - quadsFromFlat, splitWideQuad and orderByReadingOrder move out of the public API. The first is now private to paddleOcrUtils, the other two to paddleOcr. paddleOcr.ts: inline property docs on OcrDetection, PaddleOcrModelOptions and PaddleOcrModel, and defaultConfidenceThreshold is required now that it is the documented default rather than something the task substitutes. models.ts: PPOCRV6_OPTS untyped and inline, registry comment dropped, and only PaddleOcrModel imported. index.ts stops re-exporting Quad, which is still reachable as cv.Quad. C++: rectifyQuad's locks sit next to fromJs like every other op in the file, and two comments lose their trailing half.
…iant Follows the registry convention the other twelve families already use: spread the default variant into the family object so it doubles as a model config.
The recognizer's greedy CTC decode needs, per timestep, both the argmax index and the probability at it. That was a dedicated ctcGreedyDecode C++ op tied to the OCR pipeline; gather is the general form of its second half, and its shape contract is exactly argmax's output, so argmax then gather composes into the same result with a reusable op.
Adopts barhanc's implementation from the PR review. The contract resolution shrinks to one validateSpec call plus a snap helper: the CTC stride is hardcoded and asserted through constr.linear rather than discovered from the schema, which drops the lattice machinery that existed to pre-size the probs output for a stride nobody knew. Inference moves inline and the detector goes through createImagePreprocessor, so the pipeline reads like the other tasks. 874 lines to 535. Two things carry over from the old code: the aspect-preserving scale before the detector snap, so a wide page does not pay for padding on its short side, and the charset-vs-vocab length check, so a mismatched charset fails at load instead of silently producing wrong text. orderByReadingOrder stays as a task-private helper. createImagePreprocessor and its dispose gain a worklet directive: the detector size varies per image, so the preprocessor is now built inside the worklet.
Brings in the LLM chat session (#1305). Three conflicts, all two blocks landing in the same list: the cspell wordlist, the barrel export in index.ts, and models.ts, where the OCR and LLM model blocks and their type imports sit side by side.
barhanc
approved these changes
Aug 24, 2026
Drops the worklet directives added to createImagePreprocessor: the detector input size varies per page, so the preprocessor had to be built inside the worklet, which is not how the other tasks use it. The detector now runs the same resize/convert/normalize chain inline, the way the recognizer branch already does, and preprocessing.ts is untouched by this branch again. Also hoists the CTC decode's typed-array allocations out of the try block and unwraps a line in the DBNet unclip guard.
msluszniak
added a commit
that referenced
this pull request
Aug 24, 2026
Reverts the changes 4ed38b7 made, which landed with #1322. That commit was meant to apply three small review notes but also rewrote the PaddleOCR detector path to stop using createImagePreprocessor, which was not part of the review. The three review notes are un-applied again by this revert and need redoing at the intended scope.
12 tasks
msluszniak
added a commit
that referenced
this pull request
Aug 24, 2026
Part of 4ed38b7, which landed with #1322, rewrote the PaddleOCR detector path to stop using createImagePreprocessor and inline the resize/convert/normalize chain instead. That was not part of the review it claimed to address. Reverts that path only. The two review notes from the same commit stay applied: the unclip guard in ocr_ops.cpp is untouched, and greedyCtcDecode keeps its hoisted typed-array allocations. The worklet directives on createImagePreprocessor come back with the detector, since it builds the preprocessor per page inside the worklet.
msluszniak
added a commit
that referenced
this pull request
Aug 24, 2026
Part of 4ed38b7, which landed with #1322, rewrote the PaddleOCR detector path to stop using createImagePreprocessor and inline the resize/convert/normalize chain instead. That was not part of the review it claimed to address. Reverts that path only. Everything else from that commit stays: the unclip guard in ocr_ops.cpp, the hoisted typed-array allocations in greedyCtcDecode, and the removal of the worklet directives from createImagePreprocessor.
msluszniak
added a commit
that referenced
this pull request
Aug 24, 2026
## Description Reverts the part of 4ed38b7, which landed with #1322, that acted on the review note about the worklet directives on `createImagePreprocessor`. The note does not hold: the OCR detector builds a preprocessor per page inside the worklet, so the constructor and its `dispose` have to be callable there. Restores the directives and the detector's use of `createImagePreprocessor`. The other two notes from the same commit stay applied: - `ocr_ops.cpp` unclip guard keeps its single-line form, this PR does not touch the file - `greedyCtcDecode` keeps its hoisted `const indices` / `const maxima` ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [x] Bug fix (change which fixes an issue) - [ ] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [ ] iOS - [x] Android ### Testing instructions Open the OCR screen in `apps/computer-vision`, pick an image with text and run PaddleOCR. Without this revert the run fails on the worklet runtime with `[Worklets] Tried to synchronously call a Remote Function. Called "createImagePreprocessor" on the ExecuTorchDefaultRuntime Runtime.` ### Screenshots ### Related issues #1322 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a unified, OCR and document understanding pipeline to react-native-executorch. Two OCRs - EasyOCR and PP-OCRv6, plus a higher-level document pipeline that orchestrates orientation correction, UVDoc dewarp, PP-DocLayoutV3 region layout, SLANet table-structure recognition, and reading-order assembly into HTML.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Try OCRing on different images, use different OCR options. See if anything is ill behaved.
Screenshots
Related issues
Checklist
Additional notes