Add OpenAI-compatible TTS web speech provider (#2691) - #2843
Conversation
|
A fun aside with this, since OpenAI servers are fairly simple to implement, it opens up a big range of fun stuff, so you can also write a text translation layer that intercepts the text TTS and sends the text off to another AI (eg, Claude, ChatGPT, Gemini) before it then sends to TTS. so you can do fun stuff like this:
And the best part is this will be different every time. |
The text translation layer sounds interesting but as far as I can see it looks like it would something set up outside of EDDI or that would need to be included in a separate PR. Might be worth writing up a separate how-to for this if you're already doing this outside of EDDI. :-) |
Tkael
left a comment
Tkael
left a comment
There was a problem hiding this comment.
Thank you for this PR. Please review the comments and let me know what you think?
| ProviderTypeId, | ||
| ProviderDisplayName, | ||
| [ | ||
| new WebSpeechProviderProfileField( ApiKeySetting, "API key" ), |
There was a problem hiding this comment.
Change to new WebSpeechProviderProfileField( ApiKeySetting, "API key", true ), to use a PasswordBox instead of a TextBox for this setting.
| private const string ModelSetting = "model"; | ||
| private const string SpeedSetting = "speed"; | ||
| private const string CustomVoicesSetting = "customVoices"; | ||
| private const string SetupUrl = "https://github.com/EDCD/EDDI/wiki/OpenAI"; |
There was a problem hiding this comment.
This wiki documentation appears to be missing from the PR?
| private const string SpeedSetting = "speed"; | ||
| private const string CustomVoicesSetting = "customVoices"; | ||
| private const string SetupUrl = "https://github.com/EDCD/EDDI/wiki/OpenAI"; | ||
| private const string AccountUrl = "https://platform.openai.com/"; |
There was a problem hiding this comment.
This may not be accurate for OpenAI compatible servers (rather than an OpenAI hosted server).
| { CustomVoicesSetting, "alloy" } | ||
| } | ||
| }; | ||
| } |
There was a problem hiding this comment.
CreateProfile() should probably avoid fake configured values.
- Do not default apiKey to ENTER_API_KEY.
- Probably do not default customVoices to
alloy(unless the intended provider is specifically “OpenAI hosted” rather than “OpenAI-compatible.”). It isn't a given that an OpenAI compatible endpoint will include this custom voice. - Defaults like baseUrl and model are reasonable only if the provider is explicitly OpenAI. For OpenAI-compatible servers, these should probably be flexible blank values
It might make sense to create separate providers for "OpenAI Hosted" vs "OpenAI Compatible" instead of trying to make this provider straddle both?
| public async Task ValidateAsync ( WebSpeechProvider profile, CancellationToken ct ) | ||
| { | ||
| using var timeoutSource = CancellationTokenSource.CreateLinkedTokenSource( ct ); | ||
| timeoutSource.CancelAfter( TimeSpan.FromSeconds( 15 ) ); | ||
| var voices = await GetVoicesAsync( profile, timeoutSource.Token ).ConfigureAwait( false ); | ||
| if ( voices.Count == 0 ) | ||
| { | ||
| Logging.Warn( $"OpenAI profile '{profile.DisplayName}' did not return any voices. Check the API key and locale filters." ); | ||
| throw new InvalidOperationException(); | ||
| } | ||
| } |
There was a problem hiding this comment.
ValidateAsync() can succeed without validating credentials or the endpoint. With the default profile, GetVoicesAsync() returns the configured alloy voice locally, so “Verify” can report success even when the API key is still ENTER_API_KEY or the configured server is unreachable.
Since OpenAI-compatible TTS does not appear to expose a standard voices endpoint, validation should synthesize a tiny test phrase with the selected/default voice and fail on auth, connection, model, voice, or invalid audio errors. Local voice-list generation alone is not sufficient validation.
| // OpenAI voices are multilingual; default to English | ||
| var culture = CultureInfo.GetCultureInfo( "en-US" ); | ||
| var displayName = voiceId; | ||
| var friendlyName = $"{voiceId} [{profile.DisplayName}]"; | ||
| return new VoiceDetails( | ||
| displayName, | ||
| gender, | ||
| culture, | ||
| ProviderTypeId, | ||
| profile.Id, | ||
| profile.DisplayName, | ||
| isMultilingual: true, | ||
| supportedLocales: [ "en" ], | ||
| providerVoiceId: voiceId, | ||
| friendlyName: friendlyName ); |
There was a problem hiding this comment.
Please avoid claiming "en-US" / "en" unless the provider has actually told us that voice is US English.
EDDI has a substantial number of non-English speaking users.
I've revised the filtering logic so that you can use a null culture and empty list of supportedLocales if those values are not available.
| internal static double GetSpeed ( WebSpeechProvider profile ) | ||
| { | ||
| if ( double.TryParse( profile?.GetSetting( SpeedSetting ), NumberStyles.Float, CultureInfo.InvariantCulture, out var speed ) ) | ||
| { | ||
| return Math.Clamp( speed, 0.25, 4.0 ); | ||
| } | ||
| return DefaultSpeed; | ||
| } | ||
|
|
||
| internal static void SetSpeed ( WebSpeechProvider profile, double speed ) => | ||
| profile?.SetSetting( SpeedSetting, Math.Clamp( speed, 0.25, 4.0 ).ToString( "G", CultureInfo.InvariantCulture ) ); |
There was a problem hiding this comment.
EDDI already has TTS rate controls.
Adding a provider-specific speech rate setting creates two independent speed knobs (the native EDDI TTS speech rate and this provider specific speech rate), with this provider speech rate bypassing pre-existing speech rate settings.
I would have made this a direct conversion from SpeechServiceConfiguration.Rate but if you do think that a separate speed setting still makes sense, it needs to be documented as a provider-native multiplier and composed deliberately as a modifier on EDDI’s SpeechServiceConfiguration.Rate value (so that it still ties back to EDDI's pre-existing speech rate controls).
2 participants
Adds an OpenAI-compatible TTS web speech provider that works with any server implementing the
/v1/audio/speechendpoint, including:Details
HttpClientdirectly — no additional NuGet packagesalloy)Tested against
Closes #2691