Skip to content

feat(tts): add a Core ML Kokoro synthesizer variant - #1383

Closed
msluszniak wants to merge 1 commit into
mainfrom
@ms/kokoro-coreml-synthesizer
Closed

feat(tts): add a Core ML Kokoro synthesizer variant#1383
msluszniak wants to merge 1 commit into
mainfrom
@ms/kokoro-coreml-synthesizer

Conversation

@msluszniak

Copy link
Copy Markdown
Member

Description

Adds KOKORO_STANDARD_COREML: the Kokoro synthesizer on Core ML, with the duration predictor left on XNNPACK. The synthesizer is the expensive half of Kokoro, and on an iPhone 16 it produces 7.4 s of audio in 627 ms against 4741 ms for the XNNPACK build (warm median of 6 runs, real af_heart voice and the real duration predictor output). The first call pays a 12 s Core ML compile, which is cached across launches (0.46 s on a relaunch).

Quality against eager on the same inputs: spectral correlation 0.9993, RMS ratio 0.9991.

The registry voice factories now take an optional { backend } argument, so models.text_to_speech.kokoro.en_us.heart({ backend: 'coreml' }) returns the same voice pointed at the Core ML synthesizer. Core ML only covers the standard model, so the polish and german variants throw a clear error instead of silently falling back.

The duration predictor stays on XNNPACK because its contract is a single dynamic forward and coremltools cannot express that today (dynamic F.pad on a rank > 1 tensor is unsupported, and the cat workaround makes torch.export derive a non-equality constraint on the token dim). It is also the cheap half, at 233 ms.

The artifact is not published yet. synthesizer_coreml_fp32.pte (296 MB) has to land at <hf>/react-native-executorch-kokoro/resolve/v0.10.0/coreml/standard/ before this config resolves. The export lives in export-scripts MR !17.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

The .pte was benchmarked and verified on an iPhone 16 through a standalone harness (ExecutorchModule loading the artifact from the app container), not yet through this config, since the artifact is not on HF. Once it is published:

  1. Run apps/speech on an iOS device.
  2. Text to speech screen, pick Kokoro, then switch "Synthesizer backend" to Core ML.
  3. The first generation is slow (one-time compile), later ones are noticeably faster than XNNPACK.

Screenshots

Related issues

Core ML export and the two model fixes it needed: export-scripts MR !17. One of them is an upstream coremltools bug, apple/coremltools#2836 (x % 1 converts to zeros), which silently broke the excitation until it was worked around.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

Core ML is pinned to CPU_ONLY in the export. ComputeUnit.ALL on this model gave a 25 minute compile for one duration predictor method and an execute failure for another, and fp16 drops spectral correlation to 0.973 against 0.9993 for fp32, so the shipped artifact is fp32 CPU_ONLY.

Adds KOKORO_STANDARD_COREML, which keeps the duration predictor on
XNNPACK and runs the synthesizer on Core ML. On an iPhone 16 the
synthesizer produces 7.4 s of audio in 627 ms against 4741 ms for the
XNNPACK build, after a one-time compile on the first call that is cached
across launches.

The registry voice factories now take an optional { backend } argument,
so `models.text_to_speech.kokoro.en_us.heart({ backend: 'coreml' })`
returns the same voice pointed at the Core ML synthesizer. Core ML only
covers the standard model, so the polish and german variants throw a
clear error rather than silently falling back.

The speech demo gets a synthesizer backend picker, shown for Kokoro on
iOS only.
@msluszniak
msluszniak marked this pull request as draft August 25, 2026 13:01
@msluszniak msluszniak closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant