Add Google Cloud Speech-to-Text plugin (word level timings) - #289
Conversation
|
Thanks for the detailed write-up and testing! Since SE already ships the Google gRPC/auth stack (for Google TTS), I've added Speech-to-Text v2 as a built-in engine over plain REST in SubtitleEdit/subtitleedit#14561 - no extra packages, word level timings, same bucket/BatchRecognize flow, and your word-offset guard ported. I don't have a Google Cloud project to test against, so it would be great if you could try it with your setup and report back. Also, for future plugins I'd prefer the source to live in this repository under |
|
Thanks for looking at this so quickly, and for building it properly into the core instead. Closing this in favour of SubtitleEdit/subtitleedit#14561. I will keep the repository Noted on |
This adds one index entry for a Subtitle Edit 5 plugin that transcribes the open video
with Google Cloud Speech-to-Text v2. The plugin, its source and its releases live in my
own repository, so nothing here needs building or hosting on your side.
https://github.com/muaz978/subtitleedit-gcloud-stt-plugin
Why this is a plugin and not a pull request against Subtitle Edit
My company moved to Google Cloud for transcription because it is the most accurate engine
we have measured for this work. The obvious next step would have been to add it to
Subtitle Edit as another speech to text engine, and I looked at that first. I do not think
it belongs there, for three reasons that are about your users rather than about the code:
gRPC and protobuf, and they carry that weight for every user, including everyone who
will never touch this engine.
keys outright (
401, API keys are not supported by this API. Expected OAuth2 access token), so it needs a Google Cloud project, a service account JSON key, two IAM roles,and a Cloud Storage bucket. Every existing online engine in Subtitle Edit is a single
API key textbox. That is a materially different onboarding, and putting it in the core
would put that complexity in front of people who did not ask for it.
through BatchRecognize, which is an asynchronous job model reading from Cloud Storage.
It does not map onto
ISttTranscriber.TranscribeAsyncthe way the current engines do.So a plugin is the better home. People who want this accuracy can opt in, and everyone
else pays nothing for it.
What it does differently
Online engines that return text only force cue times to be inferred from character counts,
which cannot represent silence. Measured on the same 145 minute episode:
97.4% speech density is not possible for TV drama. 54.3% is what the audio actually
contains.
Testing
Three full episodes through the live API, plus roughly 40 hours of comparison runs.
Costs differ because the middle two runs used dynamic batching, at $0.003 per minute
instead of $0.016.
For comparison, the same work through OpenRouter, counted from Subtitle Edit's own logs:
The selected model does not support response_formatProvider returned 400That path is worth fixing on its own and I have opened
SubtitleEdit/subtitleedit#14542 for it, but even when it works it returns text with no
timings, which is the limitation this plugin exists to get past.
Defects found in Google's API, and what the plugin does about them
Testing at full episode length surfaced three problems that a smaller trial would not have.
Each one has a guard with a unit test built from the real observed values:
chirp_3page lists word level timestamps under features the model does not support, yet returns them on every run.Full detail, including the exact configuration and the ffmpeg pitfalls, is in
docs/testing-evidence.md.
Packaging
Six platforms, self contained so no .NET runtime is needed, roughly 40 MB each. macOS
binaries are ad-hoc signed, without which the kernel kills them on Apple Silicon and the
user only sees "exited with code 137". The build runs the published binary's self test
before packaging it, because both the JSON layer and the credential loader fail at runtime
rather than at build time. All six download URLs in this entry return 200.
Happy to move the source into this repository under
se5/instead if you would ratherhost it yourself. I went this way to avoid handing you the build and release burden for a
plugin with Google Cloud and Avalonia dependencies, but it is your call.