[RNE Rewrite] feat: add Whisper STT pipeline#1302
Draft
barhanc wants to merge 17 commits into
Draft
Conversation
Add int64/Long tensor dtype support and text/image embeddings tasks, hooks, and model registry entries, plus an interactive text-embeddings demo screen in apps/nlp. Closes #1247
model.execute now validates dynamically-shaped forward inputs against the model-declared [min, max, step] bounds exposed by an optional get_dynamic_dims method, instead of requiring an exact shape match; models without it keep exact per-dimension validation. Text embeddings feed the exact token length with no padding, which fixes scale-sensitive pooling heads (e.g. DistilUSE's tanh projection). Point DistilUSE at v0.10.0 (re-exported with get_dynamic_dims).
…mbeddings demo - Simplify text-embeddings cosine to a dot product (all models L2-normalize) and drop redundant inline comments. - Move the get_dynamic_dims / input-validation contract into the ModelHostObject class docs; trim the inline narration in model.cpp. - Add an Image Embeddings example to the computer-vision app: pick two images and compare their CLIP embeddings by cosine similarity.
Rework the computer-vision Image Embeddings screen (based on main's CLIP demo): pick an image and rank editable text labels by CLIP image/text embedding similarity, instead of the uninformative two-image score. Pads the scroll content past the Android nav bar. Point CLIP text + image at v0.10.0 (text re-exported with get_dynamic_dims; image unchanged) and declare the textEmbeddings feature in the app.
- model.{h,cpp}: read get_dynamic_dims once per model and cache it instead
of re-executing the method on every forward() call; reject a present-but-
malformed declaration (wrong dtype/rank/shape, bad min/max/step, or row
count not matching forward's tensor input dims) with an explicit error
instead of silently falling back to exact validation.
- textEmbeddings: throw a clear error when input tokenizes to zero tokens
(was BigInt(undefined)); fix docstring to match no-padding behavior.
- useTextEmbeddings: expose localPath/tokenizerPath like sibling hooks.
- computer-vision: extract shared skImageToBuffer helper, dedup from
classification and imageEmbeddings screens.
Rebase onto rne-rewrite adopted #1296's rewritten model.cpp, which delegates tensor dtype/shape checks to tensor::fromJs and already supports RangeDim [min, max, step] bounds. Re-implement variable-length forward inputs on top of it: parse get_dynamic_dims once per method into cached bounds, build a SymbolicShape of RangeDims, and pass it to fromJs. Statically shaped methods keep exact validation.
Align the text/image embeddings tasks with the add-task-pipeline skill (and every other task): allocate the static output tensor in a `[...] as const` array, destructure it, and dispose via `tensors.forEach`.
…r meta validation
…ynamic vocab validation
… isolate task exports
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Screenshots
Related issues
Checklist
Additional notes