Skip to content

Add Google Cloud Speech-to-Text plugin (word level timings) - #289

Closed
muaz978 wants to merge 1 commit into
SubtitleEdit:mainfrom
muaz978:add-google-cloud-stt
Closed

Add Google Cloud Speech-to-Text plugin (word level timings)#289
muaz978 wants to merge 1 commit into
SubtitleEdit:mainfrom
muaz978:add-google-cloud-stt

Conversation

@muaz978

@muaz978 muaz978 commented Sep 5, 2026

Copy link
Copy Markdown

This adds one index entry for a Subtitle Edit 5 plugin that transcribes the open video
with Google Cloud Speech-to-Text v2. The plugin, its source and its releases live in my
own repository, so nothing here needs building or hosting on your side.

https://github.com/muaz978/subtitleedit-gcloud-stt-plugin

Why this is a plugin and not a pull request against Subtitle Edit

My company moved to Google Cloud for transcription because it is the most accurate engine
we have measured for this work. The obvious next step would have been to add it to
Subtitle Edit as another speech to text engine, and I looked at that first. I do not think
it belongs there, for three reasons that are about your users rather than about the code:

  1. It would make Subtitle Edit noticeably heavier. The Google client libraries pull in
    gRPC and protobuf, and they carry that weight for every user, including everyone who
    will never touch this engine.
  2. It would ask a lot of the user before anything works. Speech-to-Text v2 refuses API
    keys outright (401, API keys are not supported by this API. Expected OAuth2 access token), so it needs a Google Cloud project, a service account JSON key, two IAM roles,
    and a Cloud Storage bucket. Every existing online engine in Subtitle Edit is a single
    API key textbox. That is a materially different onboarding, and putting it in the core
    would put that complexity in front of people who did not ask for it.
  3. Its transport does not fit the existing engine contract. Long audio has to go
    through BatchRecognize, which is an asynchronous job model reading from Cloud Storage.
    It does not map onto ISttTranscriber.TranscribeAsync the way the current engines do.

So a plugin is the better home. People who want this accuracy can opt in, and everyone
else pays nothing for it.

What it does differently

Online engines that return text only force cue times to be inferred from character counts,
which cannot represent silence. Measured on the same 145 minute episode:

Metric Text only engine Speech-to-Text v2
Cue timing source character count measured word offsets
Pauses longer than 2 s 0 334
Largest gap 0.85 s 138.4 s
Consecutive cues touching at 1 ms 35 of 50 none
Speech density 97.4% 54.3%

97.4% speech density is not possible for TV drama. 54.3% is what the audio actually
contains.

Testing

Three full episodes through the live API, plus roughly 40 hours of comparison runs.

Gönül Dağı 220 Teşkilat 182 Mehmed (trailer)
Audio length 145.28 min 139.74 min 0.89 min
Cost about $2.79 $0.42 $0.003
Words with timings 13,175 9,432 78
Cues 2,978 2,370 16

Costs differ because the middle two runs used dynamic batching, at $0.003 per minute
instead of $0.016.

For comparison, the same work through OpenRouter, counted from Subtitle Edit's own logs:

Observed Count
The selected model does not support response_format 211
Automatic plain json retries fired 203
Hard failures 17
Opaque Provider returned 400 12

That path is worth fixing on its own and I have opened
SubtitleEdit/subtitleedit#14542 for it, but even when it works it returns text with no
timings, which is the limitation this plugin exists to get past.

Defects found in Google's API, and what the plugin does about them

Testing at full episode length surfaced three problems that a smaller trial would not have.
Each one has a guard with a unit test built from the real observed values:

Defect What happened Handling
Silent truncation An 18 minute chunk stopped transcribing 6.6 minutes in and discarded the remaining 11.4 minutes while reporting success. Coverage is checked per chunk; a short chunk has its tail re-cut and resubmitted.
Corrupt word offsets 79 of 9,432 words (0.8%) carried impossible timings, one claiming 6,324 s inside a 1,080 s chunk. Unfiltered, that alone produced a negative total speech time. Every word is range checked against its own chunk.
Undocumented timing support Google's chirp_3 page lists word level timestamps under features the model does not support, yet returns them on every run. If timings ever stop arriving the plugin fails loudly rather than silently falling back to character proportional cues.

Full detail, including the exact configuration and the ffmpeg pitfalls, is in
docs/testing-evidence.md.

Packaging

Six platforms, self contained so no .NET runtime is needed, roughly 40 MB each. macOS
binaries are ad-hoc signed, without which the kernel kills them on Apple Silicon and the
user only sees "exited with code 137". The build runs the published binary's self test
before packaging it, because both the JSON layer and the credential loader fail at runtime
rather than at build time. All six download URLs in this entry return 200.

Happy to move the source into this repository under se5/ instead if you would rather
host it yourself. I went this way to avoid handing you the build and release burden for a
plugin with Google Cloud and Avalonia dependencies, but it is your call.

@niksedk

niksedk commented Sep 5, 2026

Copy link
Copy Markdown
Member

Thanks for the detailed write-up and testing!

Since SE already ships the Google gRPC/auth stack (for Google TTS), I've added Speech-to-Text v2 as a built-in engine over plain REST in SubtitleEdit/subtitleedit#14561 - no extra packages, word level timings, same bucket/BatchRecognize flow, and your word-offset guard ported. I don't have a Google Cloud project to test against, so it would be great if you could try it with your setup and report back.

Also, for future plugins I'd prefer the source to live in this repository under se5/, so builds and releases stay in one place.

@muaz978

muaz978 commented Sep 5, 2026

Copy link
Copy Markdown
Author

Thanks for looking at this so quickly, and for building it properly into the core instead.
You are right that it is the better home if the auth stack is already there: I had assumed
adding Google's client libraries meant new packages for everyone, and folding it in over
REST for a few tens of KB is clearly the better trade.

Closing this in favour of SubtitleEdit/subtitleedit#14561. I will keep the repository
published for now, since it carries a few behaviours the built in engine does not have
yet, and I would rather contribute those to the core than maintain a parallel plugin.
I am reviewing #14561 against my runs and will follow up there, including the live testing
you asked for once I have a project key in front of me.

Noted on se5/ for future plugins, I will lay the next one out that way.

@muaz978 muaz978 closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants