Add FireRedTTS3-Base as an audio.cpp TTS engine - #14599
Merged
Merged
Conversation
FireRedTTS3-Base (FireRed Team, Apache-2.0, arXiv 2608.17492) is a zero-shot voice-cloning model covering 24 languages with the best published speaker similarity of the open cloning models (Seed-TTS-eval avg 78.8, MiniMax MLS-Test avg 84.8). audio.cpp registers it as the `fireredtts3` family; this wires it up as a fourth engine on the shared audio.cpp runtime next to IndexTTS 2.5, Higgs Audio v3 and Fish Audio S2 Pro. Same shape as the Higgs engine (per-request voice_ref, so per-line "Clone from video" works and a voice change never restarts the server; optional .txt transcript sidecar passed as reference_text; shared settings dialog through the adapter record), with three differences: - No licence gate: both the runtime and the weights are Apache-2.0, so the adapter and installer helper now accept a null ModelLicenseDefinition. - A mandatory language combo (FireRedTts3Languages): the model has no language detection and audio.cpp's default for an unset tag is Chinese, so every request carries an explicit tag and English is the pre-selected pick. The 21 Chinese dialect tags are left out on purpose. - The Base package declares the clone task only, so the server config uses task "clon" like IndexTTS 2.5. Weights come from audio-cpp/audio.cpp-gguf/FireRedTTS3-Base-GGUF: q8_0 (4,180,334,848 B, default) and orig (12,301,253,120 B); LFS oids pinned in DownloadHashManager. Smoke-tested on M4 Metal against a support-files build of upstream main b0757573 with the family compiled in: English clone round-trips word-perfect through whisper.cpp tiny.en; cold 25 s incl. model load, warm 8.6 s for 4.2 s of audio. The runtime archives still need the family compiled in; that release pin is bumped separately. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rebuilt in support-files from upstream main b0757573 with index_tts2,higgs_audio_tts,fish_audio,fireredtts3. The FireRedTTS3 engine only works against this build; older archives reject it with "unsupported model family hint: fireredtts3". New archive hashes go in at index 0, so existing installs show the amber 'update available' status and the Update button in the audio.cpp settings, and the running server is stopped before the new archive is extracted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds FireRedTTS3-Base (FireRed Team, Apache-2.0, arXiv 2608.17492) as a fourth engine on the shared audio.cpp runtime, next to IndexTTS 2.5, Higgs Audio v3 and Fish Audio S2 Pro. Zero-shot voice cloning across 24 languages; the model reports the best speaker similarity of the open cloning models on Seed-TTS-eval (avg 78.8) and MiniMax MLS-Test (avg 84.8).
What is different from the Higgs engine it is modelled on
AudioCppTtsSettingsAdapterandEnsureAudioCppEngineWithLicensenow accept a nullModelLicenseDefinition.FireRedTts3Languages). The model has no language detection and audio.cpp's default for an unset tag is Chinese, so every request carries an explicit tag and English is pre-selected. The 21 Chinese dialect tags are left out."task": "clon"like IndexTTS 2.5.Everything else follows the Higgs shape: per-request
voice_ref(so "Clone from video" per-line works and a voice change never restarts the server), optional.txttranscript passed asreference_text, shared settings dialog, seeded reference voices at 24 kHz.Weights and runtime
audio-cpp/audio.cpp-gguf/FireRedTTS3-Base-GGUF: q8_0 (~3.9 GB, default) and orig (~11.5 GB). SHA-256 pinned from the HF LFS oids; the q8_0 download was verified locally.b0757573, familiesindex_tts2,higgs_audio_tts,fish_audio,fireredtts3). Hashes of all seven archives are pinned at index 0, so existing installs get the amber update prompt; FireRedTTS3 needs this build.Testing
dotnet build src/ui/UI.csprojgreen; TTS UI tests green (414 incl. the new engine inAudioCppPerLineCloneTestsandVoiceCloningConsentTests).Caveats worth knowing
experimental; streaming/batching is an open upstream issue.🤖 Generated with Claude Code