feat(ultravox): support server-side external TTS voices via externalVoice - #6850
feat(ultravox): support server-side external TTS voices via externalVoice#6850ssgutierrez42 wants to merge 6 commits into
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
longcw
left a comment
longcw
left a comment
There was a problem hiding this comment.
looks good to me. something nit:
| voice : str | UltravoxVoice | ||
| The voice to use for TTS. | ||
| external_voice : dict[str, Any], optional | ||
| The Ultravox configuration for an external TTS provider. Mutually exclusive with ``voice``. |
There was a problem hiding this comment.
should we log a warning when both voice and external_voice are given?
There was a problem hiding this comment.
external_voice cannot work with output_medium="text" too?
There was a problem hiding this comment.
@longcw
good catch !
voice always has a default value ("Mark"), so we may need a bigger change to distinguish “explicitly set” from “default.” Do you think we should log a warning when voice != DEFAULT_VOICE and is_given(external_voice) ?
There was a problem hiding this comment.
external_voicecannot work withoutput_medium="text"too?
@longcw not as documented currently - because externalVoice tells UV to route its audio output through an external TTS provider, but in output_medium="text" the model emits text tokens only (no audio for external_voice to synthesize)
| "selectedTools": parse_tools(list(self._tools.function_tools.values())), | ||
| } | ||
|
|
||
| if is_given(self._realtime_model._opts.external_voice): |
There was a problem hiding this comment.
should we skip a given but empty external_voice or raise an error, or fallback to voice?
There was a problem hiding this comment.
will update to raise an error 📝
I realize that silently falling back to voice may mask the bug (caller thinks they’re using an external voice but hears the default “Mark” instead), and sending it through would produce a confusing 400 from Ultravox.
Reject an empty external_voice dict in __init__ with a ValueError rather than silently falling back to the built-in voice, which would mask a caller mistake as a wrong-voice call. Warn when both external_voice and a non-default voice are given, since external_voice takes precedence and the voice is ignored.
|
@longcw ready for re-review |
3 participants
Summary
external_voiceparameter toRealtimeModelto pass an external TTS provider configuration (e.g. ElevenLabs, Cartesia, MiniMax) through Ultravox's/callsAPI via theexternalVoicefieldexternal_voiceis provided, theexternalVoicekey is sent in the/callspayload instead ofvoice, keeping the model in voice mode so native barge-in is preserveddict[str, Any]to accommodate all provider variants without coupling to Ultravox's evolving schemaWhy
Ultravox's
output_medium="text"path (text-out half-cascade) cannot support barge-in — the plugin's interruption signal (playback_clear_buffer) only fires in voice mode, and Ultravox exposes no input-side speech-onset event. Server-sideexternalVoicesolves this by having Ultravox call the TTS provider directly, keeping the session in voice mode with working interruption.Changes
_UltravoxOptions: addedexternal_voice: NotGivenOr[dict[str, Any]]RealtimeModel.__init__: addedexternal_voicekeyword argument (defaults toNOT_GIVEN), documented as mutually exclusive withvoice/callspayload construction:voiceis now conditional — whenexternal_voiceis given,externalVoiceis sent insteadReferences
/callsAPI: https://docs.ultravox.ai/api-reference/calls/calls-post