feat(tts): add a Core ML Kokoro synthesizer variant - #1383
Closed
msluszniak wants to merge 1 commit into
Closed
Conversation
Adds KOKORO_STANDARD_COREML, which keeps the duration predictor on
XNNPACK and runs the synthesizer on Core ML. On an iPhone 16 the
synthesizer produces 7.4 s of audio in 627 ms against 4741 ms for the
XNNPACK build, after a one-time compile on the first call that is cached
across launches.
The registry voice factories now take an optional { backend } argument,
so `models.text_to_speech.kokoro.en_us.heart({ backend: 'coreml' })`
returns the same voice pointed at the Core ML synthesizer. Core ML only
covers the standard model, so the polish and german variants throw a
clear error rather than silently falling back.
The speech demo gets a synthesizer backend picker, shown for Kokoro on
iOS only.
msluszniak
marked this pull request as draft
August 25, 2026 13:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
KOKORO_STANDARD_COREML: the Kokoro synthesizer on Core ML, with the duration predictor left on XNNPACK. The synthesizer is the expensive half of Kokoro, and on an iPhone 16 it produces 7.4 s of audio in 627 ms against 4741 ms for the XNNPACK build (warm median of 6 runs, real af_heart voice and the real duration predictor output). The first call pays a 12 s Core ML compile, which is cached across launches (0.46 s on a relaunch).Quality against eager on the same inputs: spectral correlation 0.9993, RMS ratio 0.9991.
The registry voice factories now take an optional
{ backend }argument, somodels.text_to_speech.kokoro.en_us.heart({ backend: 'coreml' })returns the same voice pointed at the Core ML synthesizer. Core ML only covers the standard model, so the polish and german variants throw a clear error instead of silently falling back.The duration predictor stays on XNNPACK because its contract is a single dynamic
forwardand coremltools cannot express that today (dynamicF.padon a rank > 1 tensor is unsupported, and thecatworkaround makestorch.exportderive a non-equality constraint on the token dim). It is also the cheap half, at 233 ms.The artifact is not published yet.
synthesizer_coreml_fp32.pte(296 MB) has to land at<hf>/react-native-executorch-kokoro/resolve/v0.10.0/coreml/standard/before this config resolves. The export lives in export-scripts MR !17.Introduces a breaking change?
Type of change
Tested on
Testing instructions
The
.ptewas benchmarked and verified on an iPhone 16 through a standalone harness (ExecutorchModuleloading the artifact from the app container), not yet through this config, since the artifact is not on HF. Once it is published:apps/speechon an iOS device.Screenshots
Related issues
Core ML export and the two model fixes it needed: export-scripts MR !17. One of them is an upstream coremltools bug, apple/coremltools#2836 (
x % 1converts to zeros), which silently broke the excitation until it was worked around.Checklist
Additional notes
Core ML is pinned to
CPU_ONLYin the export.ComputeUnit.ALLon this model gave a 25 minute compile for one duration predictor method and an execute failure for another, and fp16 drops spectral correlation to 0.973 against 0.9993 for fp32, so the shipped artifact is fp32 CPU_ONLY.