feat: add Supertonic 3 text-to-speech model#1300
Conversation
IgorSwat
requested review from
barhanc and
Copilot
and removed request for
Copilot
barhanc
left a comment
barhanc
left a comment
There was a problem hiding this comment.
Tested example app and it works fine.
| auto partition = partitioner_.partition(phonemes, context_.inputTokensLimit, | ||
| Partitioner::Mode::MIN_BREAKS); |
There was a problem hiding this comment.
The new TextPartitioner only supports a single partitioning logic (equivalent to Kokoro's old MIN_LATENCY mode), the offline/batch generate call originally used Partitioner::Mode::MIN_BREAKS to split the text. Is this deliberate or accidental regression?
| void Supertonic::unload() noexcept { | ||
| durationPredictor_.unload(); | ||
| textEncoder_.unload(); | ||
| vectorEstimator_.unload(); | ||
| vocoder_.unload(); | ||
| } |
There was a problem hiding this comment.
unload() does not wait for or join the detached thread, and instead immediately frees the models, the background thread may still be running synthesize(...) or processing the loop. When the background thread invokes the next ExecuTorch model run, it will access freed/unloaded memory and crash the application instantly.
| - `model` of type [`TextToSpeechModelSources`](../../06-api-reference/type-aliases/TextToSpeechModelSources.md) — model configuration. | ||
| - [`voiceSource`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#voicesource) of type [`ResourceSource`](../../06-api-reference/type-aliases/ResourceSource.md) — the voice tensor used for synthesis. | ||
| - [`phonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#phonemizerconfig) of type [`TextToSpeechPhonemizerConfig`](../../06-api-reference/interfaces/TextToSpeechPhonemizerConfig.md) — Kokoro only: phonemizer configuration. Unused by Supertonic. | ||
| - [`lang`](../../06-api-reference/interfaces/TextToSpeechModelConfig.md#lang) of type [`TextToSpeechSupertonicLanguage`](../../06-api-reference/type-aliases/TextToSpeechSupertonicLanguage.md) — Supertonic only: default language token (e.g. `'en'`, `'na'`). Unused by Kokoro. |
There was a problem hiding this comment.
Language which is typed as sth strictly related to supertonic, generally incorrect abstraction. If adding this in a correct way causes huge rewrite please ignore it since we will introduce the model in rewritten flow in v0.10.0 either way.
| The module provides two ways to generate speech. The available parameters differ by model family: | ||
|
|
||
| | Parameter | Kokoro | Supertonic | | ||
| | ------------ | ------ | ---------- | | ||
| | `speed` | ✓ | | | ||
| | `phonemize` | ✓ | | |
| // Kokoro — language-specific voice bundle: | ||
| // const tts = await TextToSpeechModule.fromModelName( | ||
| // models.text_to_speech.kokoro.en_us.heart(), | ||
| // (progress) => console.log(progress) | ||
| // ); |
There was a problem hiding this comment.
Do we really want to keep this one for Kokoro as comment? Maybe drop it. Same for all such cases in this file.
| 'Hello from ExecuTorch!', | ||
| 1.0, | ||
| true, | ||
| 8, | ||
| 'en' |
There was a problem hiding this comment.
maybe adding args names as inline commets next to the passed values might be more readable. Same for all such cases in this file.
| if constexpr (meta::SameAs< | ||
| Model, models::text_to_speech::supertonic::Supertonic>) { | ||
| addFunctions( | ||
| JSI_EXPORT_FUNCTION(ModelHostObject<Model>, unload, "unload")); | ||
| addFunctions(JSI_EXPORT_FUNCTION(ModelHostObject<Model>, | ||
| promiseHostFunction<&Model::stream>, | ||
| "stream")); | ||
| addFunctions(JSI_EXPORT_FUNCTION( | ||
| ModelHostObject<Model>, synchronousHostFunction<&Model::streamStop>, | ||
| "streamStop")); | ||
| addFunctions(JSI_EXPORT_FUNCTION( | ||
| ModelHostObject<Model>, synchronousHostFunction<&Model::streamInsert>, | ||
| "streamInsert")); | ||
| addFunctions(JSI_EXPORT_FUNCTION( | ||
| ModelHostObject<Model>, synchronousHostFunction<&Model::streamFlush>, | ||
| "streamFlush")); | ||
| } | ||
|
|
There was a problem hiding this comment.
These are exactly the same as in kokoro, you can just modify the if condition.
| return {audio.begin(), audio.end()}; | ||
| } | ||
|
|
||
| std::vector<float> Supertonic::generate(std::u32string input, float speed, |
There was a problem hiding this comment.
Isn't this one const method?
| */ | ||
| class TextEncoder : public BaseModel { | ||
| public: | ||
| TextEncoder(const std::string &modelSource, |
| } | ||
| } | ||
|
|
||
| bool isAsciiSpace(char32_t c) { |
| for (char32_t c : s) { | ||
| if (isAsciiSpace(c)) { | ||
| if (!prevSpace) { | ||
| cleaned.push_back(U' '); | ||
| } | ||
| prevSpace = true; | ||
| } else { | ||
| cleaned.push_back(c); | ||
| prevSpace = false; |
There was a problem hiding this comment.
| for (char32_t c : s) { | |
| if (isAsciiSpace(c)) { | |
| if (!prevSpace) { | |
| cleaned.push_back(U' '); | |
| } | |
| prevSpace = true; | |
| } else { | |
| cleaned.push_back(c); | |
| prevSpace = false; | |
| for (char32_t c : s) { | |
| prevSpace = isAsciiSpace(c); | |
| if (prevSpace) { | |
| if (!prevSpace) { | |
| cleaned.push_back(U' '); | |
| } | |
| } else { | |
| cleaned.push_back(c); |
|
|
||
| // Mask to silence invalid latent frames during the flow loop. | ||
| std::vector<float> latentMask(static_cast<size_t>(L), 0.0F); | ||
| std::fill(latentMask.begin(), latentMask.begin() + latentValid, 1.0F); |
3 participants
Description
This PR adds a full pipeline and API for Supertonic 3 model. It also refactors the native TTS code to make it better organized across 2 different text-to-speech pipelines.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Run demo app in apps/speech.
Screenshots
Related issues
Checklist
Additional notes
Needs to be tested on iOS. Also, the typescript API is in a somewhat PoC state and will most likely need some changes.