feat(recorder): on-device recorder host + whisper transcription + chat/RAG#433
Open
dishit-wednesday wants to merge 116 commits into
Open
feat(recorder): on-device recorder host + whisper transcription + chat/RAG#433dishit-wednesday wants to merge 116 commits into
dishit-wednesday wants to merge 116 commits into
Conversation
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…, stop, native log) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…option (core) whisperService.transcribeFile gains offset/duration so a recording can be transcribed in chunks, and streams segments live via onNewSegments (cumulative result + per-segment t0/t1). loadModel takes useGpu/useFlashAttn/useCoreML and passes them to initWhisper (useCoreMLIos on iOS). Bumps the pro submodule to the chunked/resumable transcription experience. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…header Replaces the Settings bottom tab with a Memory tab. MemoryTabScreen renders the Pro recorder when it is registered (pro.activate) and a paywall otherwise, read reactively so unlocking Pro swaps it in without a restart - the tab shell stays in core, the recorder injects from the submodule. Settings becomes a pushed RootStack screen reached from a gear in the Home header. Updates nav types and the onboarding step that pointed at the old SettingsTab, and the SettingsScreen nav-prop type (no longer a tab). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Adds a map-reduce summarizer (transcriptSummarizer) for text too large for the context window: split into context-sized chunks, summarize each, then combine (recursing if needed) - so the whole transcript is read instead of truncated to the tail like contextCompaction does. A Summarize action on a document/transcript attachment chip streams progress (part i/N, combining) into an assistant message, then the final summary. MediaAttachment gains recordingId + transcript range metadata, and a one-shot chatAttachmentInbox lets the Pro recorder hand a transcript to the composer across the navigation boundary. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Summarizing an oversized transcript used to show a static "part i of N" counter for minutes and only stream on the final combine pass, and the Summarize button was clipped invisible inside the 60x60 attachment chip. - generateWithMaxTokens gains an optional onToken callback. - transcriptSummarizer streams each map part as it is written (new 'combining' phase marks the final pass), so text flows from part 1, not part 3. - useSummarizeAttachment renders the live work (parts stack while streaming, then the combined answer streams over the top) on a 50ms flush so the JS thread is not saturated by per-token deep store updates. - Document/transcript chips get a larger 168x76 layout with a filled, visible Summarize button and a spinner+label busy state. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…er.rn 0.5.5) Hoist the segment-callback user_data to the dispatch-block scope so live segment streaming on the iOS file-transcribe path does not deref a dangling pointer; add an NSLog marker to verify the patch is compiled into a build. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…d project Thread per-chunk metadata (recordingId/startMs/eventTitle) through chunking, the SQLite store, and retrieval; add indexText() to index a raw transcript; seed a 'Recordings' project via ensureProject so recordings can be asked about. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Move the exact timestamp out of the system prompt (which busted llama.rn's prefix cache every turn) and onto the latest user message; keep only the stable date in the system prompt so the ~800-token system+tools prefix is reused. Add tool-loop diagnostic logging. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… model Guard CoreML so it falls back to CPU when the per-model encoder asset is absent (was crashing on A12); force 'en' for English-only ggml-*.en models; add the small.en-tdrz model entry; re-enable iOS live segment streaming. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Used by transcribeChunked to log per-chunk memory footprint. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…tream fast On CPU-only low-RAM devices, reasoning/preamble burned the whole token budget before any summary streamed. Force the thinking channel off for internal generations (summarize, tool-selection), instruct the prompts to emit no preamble, strip any leaked control tokens, and cap each MAP chunk to ~1500 input tokens so each part prefills in under a minute and starts streaming sooner. Reduce/combine passes still use the full context budget. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Snapshot RAM before and after loadModel. On 4 GB iOS devices a large whisper model can push the app past the jetsam limit and the OS kills it mid-load, which presents as a crash; the before/after pair localizes a kill to load vs transcription. Fire-and-forget so it adds no await points to the load critical path. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The tab and paywall are about on-device recording/transcription, so call it Recorder. Tab label, testID (recorder-tab), and paywall title updated. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
loadModel gained an options param (CoreML/lang config); update the store tests to match the new call signature. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Turn the bare gating card into a feature list - always-on recording (Android), on-device transcription, summaries, calendar context - with a privacy line and the unlock CTA. Copy follows the brand voice (privacy-as-mechanism, no jargon). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Points the pro submodule at the locket recorder feature set: continuous recorder, on-device transcription, calendar matching, summaries, knowledge-base sync, and the recording-detail/list UI polish. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Update tests left stale by intended code changes, and add coverage for new
behavior. No production code changed; each assertion was verified against the
current implementation (the source of truth), not weakened to pass.
- nav/onboarding: Memory tab renamed to Recorder (recorder-tab), and Settings
moved off the tab bar to a Home-header route, so STEP_TAB_MAP.exploredSettings
is 'Settings' and the 5th tab label/testID is Recorder.
- whisper: loadModel now forwards {useGpu,useFlashAttn,useCoreMLIos}; .en models
force language 'en' (plain .bin auto-detects).
- rag: schema gained a metadata column (4th CREATE = ALTER migration; 4th insert
bind = metadata, null when absent).
- generationToolLoop: the precise time-of-day note now rides on the latest user
message (keeps the system+tools prefix cacheable), not the system prompt.
- whisperService.branches: reset RNFS exists/stat mocks per test - clearAllMocks
does not drain the mockResolvedValueOnce queue, so onces leaked between tests.
Added: CoreML-encoder-missing -> CPU fallback guard, and RAG metadata JSON
serialization on the write path.
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
RNWhisper.mm's vadDetectSpeech/vadDetectSpeechFile decode the WAV to a malloc'd float PCM buffer and pass it to detectSpeech synchronously, but never free it. Over a long recording that leaks ~one slice of PCM per call and OOMs. Add free(data) after the synchronous detectSpeech consumes it, in both methods. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Add ungated controls to clear the cached/stored Pro license so the free -> activate flow can be re-tested on a release build (where the __DEV__ toggle is inert). The Keygen machine slot stays claimed (fingerprint persists), so re-pasting the key re-activates instantly. Marked TEMPORARY - re-gate behind __DEV__ or remove before shipping. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Point at the pro commits adding the recorder transcription/archive UI polish and the speech/noise/silence cleanup classifier. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The ChatInput render fn was 358 lines (limit 350) - the #431 lint blocker. Lift the send/stop/voice action button into a module-scope ActionButton component (derives colors/styles from the same hooks; behaviour identical to the previous inline ternary). Render fn now ~343 lines. 91/91 ChatInput tests pass, tsc clean. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The submodule URL still referenced the old wednesday-solutions/offgrid-pro repo. Point it at the canonical off-grid-ai/mobile-pro so CI and fresh clones fetch the pro recorder code from the right place. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
feat(core): summarizer streaming/dynamic-context + whisper model fallback + pro bump [WIP/divergent]
dishit-wednesday
force-pushed
the
feat/locket-pro
branch
from
July 15, 2026 14:08
217a2a0 to
79f0ee8
Compare
Catch feat/locket-pro up to dev (595 commits, ~490 files). Conflict
resolutions (dev = canonical current; locket features preserved additively):
- whisperService.ts: HEAD base (locket transcribeFile chunking/extractWavSlice,
CoreML, diarize, vocabulary, WhisperBusyError, native logging) + dev's disjoint
deltas grafted in: activeDownloadModelId delete-scoping, B26 realtime→file
fallback, and file-ops delegated to whisperModelFiles (dev's extraction).
- whisperModels.ts: HEAD's rich catalogue (WhisperModel interface, CoreML urls,
tdrz) + single-sourced cleanTranscription with the non-ASCII \p{L}\p{N} fix.
- llm.ts: dev's version + HEAD's generateWithMaxTokens superset (streaming/GBNF/
repeatPenalty) the insights summarizer depends on.
- activeModelService: combined textOnly mmproj-skip (locket) + GPU-aware overhead
(dev) in the RAM estimate.
- ChatInput Attachments: combined document Summarize (locket) + image tap-to-view (dev).
- remoteServer: kept the 'whisper' provider arm (locket cloud transcribe).
- constants/logger/ChatScreenComponents/llmToolGeneration: took dev's canonical
versions (DevGrammar harness dropped from the chat path).
- whisperStore: kept locket additions; reconciled refreshPresentModels to not
auto-adopt a model when none is active (restores dev's explicit-select semantics).
- Deleted the two core whisper tests dev removed (pro-importing core tests).
tsc + lint green. Remaining test failures are all pro-submodule-coupled (TTS/audio
tests expecting pro-main's newer Kokoro engine) — resolved by the coupled pro
catch-up + gitlink bump, tracked as the follow-up step.
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…deps in tests Point the pro submodule at the catch-up merge (recorder branch feat/continuous-smart-recorder merged with core-dev's pinned pro 3599b5b) so the pin carries BOTH the locket recorder work AND dev's newer TTS/Kokoro audio engine. Without this the core pin lagged pro-main and every voice-mode/TTS test failed. The locket recorder introduced native deps the core test env didn't mock, so requiring @offgrid/pro threw during pro activation (no slots registered → EngineBridge never mounts → TTS isReady never fires → 13 voice/TTS tests failed). Add the mocks: - @notifee/react-native (meeting reminders): full method surface + real enum values. - react-native-device-info: usePowerState hook + getPowerState (recorder power gating). Restore the onAttachPress prop in the AudioModeLayout test now that the caught-up pro has it. Full core suite green (7831 passed); the intermittent single-suite failures are pre-existing parallelism flakiness (0 individual test failures, shifts between runs). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The previous bump targeted the pin core happened to hold (10f49a18), which was 13 commits behind the live recorder branch (feat/continuous-smart-recorder = 5b5921cc) — all recent locket polish (LoudnessEnhancer, header search, honest insights load, single-source RAM budget, hide-no-speech, unified feed, etc.). Repoint the submodule to the catch-up built on 5b5921cc + core-dev's pro (3599b5b), so the merge includes every recent locket commit, not a stale snapshot. tsc clean; voice/TTS/whisper suites green against the new pin. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… in catch-up) The catch-up merge based whisperService on the locket branch, which predated dev's forceReset fix, so dev's cancel of the parallel B26/B28 fallback recorder was silently dropped. On an aborted/cancelled chat voice-input session, forceReset stopped the realtime stream but left the backup file recorder capturing the mic (B11-class leak). Re-add dev's guard: cancel the fallback recorder on forceReset when it's mid-flight. Chat voice mode only; the Locket recorder uses a different path. Caught by the catch-up audit (Finding 2). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ion's own recorder The prior restore used a blunt audioRecorderService.isCurrentlyRecording() check, but that recorder is a shared singleton — Voice.ts's direct-audio and file-path modes start it too. forceReset must not cancel a recording it doesn't own. Track ownership with a fallbackRecorderActive flag (set only when startRealtimeTranscription starts the fallback recorder, cleared on stop/cancel). forceReset now cancels the recorder ONLY when the realtime session owns it, and leaves any other recording untouched. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
chore(locket): catch feat/locket-pro up to dev (merge-conflict resolution)
…dev-2 # Conflicts: # pro
Dev's knip de-export sweep (bd34f2e) dropped its export as 'internal-only', but the pro AudioMessageBubble imports it — knip can't see the submodule. Restore the export so the combined core+pro tree typechecks. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ingle-source + TTS Load-Anyway Dev's 80-commit advance also moved dev's pro pin 3599b5b→ff0d8742 (2 audio fixes: prepareMessageForSpeech single-source + TTS memory-refusal Load-Anyway). Merged those into the recorder branch and repointed the pin so the new dev audio tests pass. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… finding) The new pro pin (55006e4) refactored the audio bubble to route through prepareMessageForSpeech, so pro no longer imports stripMarkdownForSpeech. The re-export added earlier is now unused — revert it to match dev's knip intent (avoids an unused-export flag). tsc stays clean; prepareMessageForSpeech still calls it internally. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
chore(locket): top-up feat/locket-pro to latest dev (80 commits) — merge-conflict resolution
…ase-sync feat/locket now contains the full recorder + audio work (base-sync merge, 0 conflicts), so the submodule pin tracks the branch that actually reaches pro main via PR #11, not a throwaway catch-up branch. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…t/locket tip Pro's locket code imports AlertButton (from CustomAlert) and Chunk (from rag/chunking) via the @offgrid/core barrels, which didn't re-export them (dev's knip sweep). Add the re-exports so pro typechecks against feat/locket-pro. Bump the pro submodule pin to the current feat/locket tip (pro type/lint fixes). Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
fix(core): re-export AlertButton + Chunk for pro + pin bump
…led in pro) Removes android/app/src/main/assets/whisper-vad/ggml-silero-v5.1.2.bin from the public core repo — it's relocated into the pro submodule's android assets (paired pro commit). Keeps the pro recorder's VAD implementation out of open-core; vadDetect.ts still resolves it (pro asset merge, or HuggingFace download fallback). Bumps the pro pin. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
chore(recorder): move Silero VAD model out of public core into pro
forceReset() reset only the realtime stopFn; if it ran while a whole-file
transcription was in flight, fileTranscribeStop stayed non-null and every
later transcribeFile threw WhisperBusyError ("a transcription is already in
progress") until app restart. Clear it with the same atomic grab-and-clear
plus a best-effort native stop of the orphaned job.
Add an integration test that drives the real whisperService over a faked
native context and proves a second transcription runs after a mid-flight
forceReset. Also drop a stale screen-name reference in a Memory tab comment.
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
fix(whisper): clear file-transcription stop handle in forceReset
…fc8745) CI (uat-dev.yml) builds the committed pro pointer; advance feat/locket-pro to the merged RPS/recorder stack so the internal build ships the recorder work. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
CI's pod install regenerates Podfile.lock against the merged pro (feat/locket), adding the OffgridPro pod; commit it so scripts/uat.sh sees a clean tree. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Prior lock was generated by system CocoaPods 1.16.2; CI uses the Gemfile-pinned 1.15.2, so its pod install rewrote the version line and tripped uat.sh's clean-tree check. Regenerated with 1.15.2 so CI's pod install is a no-op. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ty file) Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ostic CI's ruby/setup-ruby rewrites .bundle/config during setup, dirtying an otherwise clean checkout and tripping uat.sh's clean-tree guard. Whitelist it like fastlane/README.md. Also revert the temporary git-status diagnostic. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The pod install step restored the committed Podfile.lock after 'pod install' to keep the tree clean, but that made Podfile.lock diverge from the freshly generated Pods/Manifest.lock, failing the archive's Check Pods Manifest.lock phase (sandbox not in sync). Stop restoring it; whitelist ios/Podfile.lock in uat.sh's clean-tree check instead, like .bundle/config. Regenerated lock == Manifest.lock -> archive passes. Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



TL;DR - Open-core host that wires in the private
pro/recorder submodule and shipsthe shared on-device transcription + retrieval engine it relies on. Recorder feature code
stays closed-source in
pro/; this PR is the public wiring + shared services. Pairs withoff-grid-ai/mobile-pro#11 (the submodule pointer references a commit on that pro branch, so
they land together).
flowchart TB subgraph pro["pro/ submodule (private, autolinked)"] rec["Recorder + AudioNormalizer"] end subgraph core["core (public host)"] slot["MemoryTabScreen<br/>paywall + registry slot"] ws["whisperService<br/>windowed transcribe + CoreML guard"] rag["RAG 'Recordings' KB<br/>metadata column + migration"] chat["chat: summarize attachment"] end slot -. "registered-screen slot" .-> rec rec -->|"16k mono WAV"| ws ws -->|"transcript + t0/t1 segments"| rag ws -->|"transcript"| chatWhat's in this PR
native autolink (
react-native.config.js); iOS background-audio mode..enlanguage forcing; iOS streaming-lifetime patch (whisper.rn+0.5.5).metadatacolumn + migration);summarize oversized transcript/document attachments.
.gitmodulesre-pointed to canonicaloff-grid-ai/mobile-pro;submodule pointer bumped to the merged pro commit.
How it works (expand for detail)
Transcription pipeline (whisper.cpp via whisper.rn)
transcribeFiledrives whisper.rn with an{offset, duration}window so long recordings transcribe in chunks and resume from acheckpoint (last finished chunk) instead of reloading the whole file. Live cumulative
segments surface through whisper.rn's
onNewSegments(nativenew_segment_callback),wired only when
nProcessors == 1(parallel decode cannot stream in order).(
transcribeFile-> ObjCtranscribeData) crashed becausenew_segment_callback'suser_datawas a stack struct freed before the callback fired. The patch hoists it so itoutlives the call; compiled into the iOS binary, so live segments work on both platforms.
ggml-<model>-encoder.mlmodelcbundle makes whisper.rn fail CoreML init, fall back toCPU, then crash at transcribe 0% on some devices (A12 / iPhone XS).
loadModelchecks theencoder asset exists and downgrades
useCoreMLIosto false when missing, so the toggle isa no-op instead of a crash.
.enmodels carry only English tokens; auto-detect(returns codes like "tg") or an explicit non-English pick makes whisper unstable on iOS
(crash at 0%, 762s for 13%, garbled).
transcribeFileforceslanguage: 'en'for anymodel whose catalogue
lang === 'en', falling back to the.en.binfilename convention.Retrieval (recordings knowledge base)
Transcripts index into a local RAG "Recordings" KB.
rag_chunksgains ametadata TEXTcolumn (recordingId, startMs, eventTitle, ...) via a guarded
ALTER TABLEmigration forolder installs; chunk metadata is JSON-serialized on write.
Integration / build
react-native.config.js(onlywhen the pro android/iOS sources are present), so closed-source code never leaks into the
public repo but builds when the submodule is checked out.
.gitmodulesre-pointed from the oldwednesday-solutions/offgrid-proto the canonicaloff-grid-ai/mobile-pro; submodule pointer bumped to the merged pro commit.Testing
npm run lint(0 errors),tsc --noEmit(clean), and the full jest suite pass; thepre-push hook ran them, no
--no-verifyused. Added CoreML-fallback + RAG-metadata coverage.Follow-ups (not blocking)
MemoryTabScreenlooks up the recorder by the concrete proscreen name; move it behind an abstract slot key so public never names a pro internal.
max-lines: deferred with a TODO; clearing it means moving modeldownload/management into its own module (~11 call sites across core + pro).
Reviewer's guide (commits by theme)
The branch is chronological; this groups the commits by area.
recorder/paywall) + move Settings to Home header; autolink pro native + pod install; iOS
background-audio mode.
segments + CoreML; CoreML asset guard +
.enforcing + tdrz; iOS streaming lifetimepatch; memory-snapshot logging;
whisperremote-server provider type.oversized attachments; stream summaries live.
off-grid-ai/mobile-pro.