fix(realtime): support externally transcribed user turns

[TonyG-FWE]

Fixes #5408

Problem

Realtime agents could treat raw provider audio and externally finalized STT text as two independent inputs for the same user turn. That split authority caused duplicate input, lost turns, incorrect interruption, and cleanup/reconnect races.

Mode	Turn input owner	Provider input
"audio" (default)	Realtime provider	Raw audio; existing behavior remains the default
"text" (opt-in)	External STT / LiveKit turn pipeline	One finalized ChatMessage, including on_user_turn_completed edits

Text mode never also commits the same raw audio as model input.

Ownership architecture

• One immutable resolved policy decides detector validity, input/interruption ownership, and timeout behavior before live state changes.
• One FIFO framework transaction owns client-detected audio from capture through hook processing, provider submission, generation, cancellation, and cleanup. Provider-owned turns stay outside that seal/defer/advance machinery.
• Finalized-message synchronization is explicit: ACCEPTED, REJECTED, or UNKNOWN. Provider chat-mirror equality is not a correctness gate.
• Google retains input sequence, provider epoch, deferred media, and tool-result ownership across safe resumable reconnects; fresh sessions reject stale provider-owned results.
• Fallback replacement uses short-held phase/epoch state and never holds a state lock across interruption, child synchronization/closure, or provider setup.

The default remains backward-compatible. AgentActivity contains no provider-name checks or provider-specific types.

Final confirmed fixes

The final review hypotheses were reproduced on the prior head before production changes:

• 7ec384a51 adds five fail-before ownership-boundary regressions.
• 1245db908 restores explicit provider-owned clear_user_turn() and activity notifications without transferring ownership into framework transactions. It also preserves a per-reply allow_interruptions=True override when the session default is disabled.
• 946be984e observes eventual provider-close failures after caller cancellation and converts unexpected automatic fallback-swap failures into one terminal RealtimeModelError; explicitly awaited restart failures still propagate normally.

Guarded STT segment recreation, the internal empty-transcript timeout, bounded bookkeeping/replay sets, intentional Gemini fresh-restart behavior, and the terminal all-models-failed window remain unchanged because focused lifecycle evidence did not establish production defects.

Validation

Current head: 946be984eeb1a340c600a434a6d0bfcd5331c277

• Final fail-before regressions: 5/5 passed after the fixes
• Complete directly affected modules: 145 passed
• Adjacent external-text, AudioRecognition, false-interruption, and AgentSession/realtime suites: 224 passed; the two remaining AudioRecognition failures reproduce on exact upstream 49bfd8b31
• Google realtime: 120 passed
• OpenAI/xAI realtime: 62 passed
• Ruff formatting/lint: 943 files clean
• Python 3.13/Linux-platform mypy for affected Agents/Google/OpenAI/xAI packages: no issues in 248 source files
• git diff --check: passed
• GitHub Linux unit workflow: 2,228 passed, 5 skipped
• GitHub Ruff, Python 3.10 and 3.13 typing, BlockGuard on Ubuntu/macOS/Windows, aggregate dumps, release gate, and CLA: passed

Environmental limitation

The local Windows full-unit run reached 1,815 passed and 7 skipped before the known event-loop/native-resource cascade; GitHub Linux completed the authoritative suite. No live-provider credentials were available, requested, or used. Provider lifecycle coverage is credential-free and hermetic.

This PR remains a draft with maintainer edits enabled. A final Devin rerun is pending on this exact green head; no additional Codex review was requested.

[CLAassistant]

All committers have signed the CLA.

[TonyG-FWE]

@codex review

[TonyG-FWE]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 90f92f31db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[TonyG-FWE]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 9dcf6ebf94

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[TonyG-FWE]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 8c249b8ce6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: f96f43792b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[TonyG-FWE]

Dashboard-only Devin findings on f96f43792 were rechecked against the exact head before production changes:

• Confirmed: Python 3.10 shutdown compatibility and unbounded callback settlement. Fixed in 99f48a4c; focused Python 3.10 and bounded-cleanup regressions pass.
• Confirmed: prospective text-mode validation reused stale detector explicitness. Fixed transactionally in 3af88ef60; rejected updates preserve policy and listeners.
• False positive: both VAD- and STT-based late-final/empty-turn races already preserve the newer transcript. Event-driven regressions pass without a production edit.
• Intended / bounded debt: realtime preemptive-generation gating, replay-exclusion retention, and duplicate VAD/STT onset ordering remain unchanged because their lifecycle reproductions preserve the documented behavior.

The follow-up set also preserves the established commit-helper contract in 204c6dfe5 and replaces scheduling sleeps with deterministic events in 744b9370c.

[TonyG-FWE]

Fresh Devin shutdown finding on 744b9370c was split by lifecycle evidence:

• Confirmed: an empty realtime-audio candidate could finish its hook after close blocked new work and then be appended as ChatMessage(content=[""]). test_close_does_not_commit_empty_bounded_realtime_audio_turn in 4dfcadf09 fails on the untouched reviewed head and also verifies no generation starts and owned audio is cleared.
• Not reproducible under valid ownership: the duplicate-provider-transcript half. Server-detected and manually submitted audio turns exit before this callback path. In the close race, provider audio is still unsubmitted; the existing regression verifies commit_audio_calls == 0, so retaining non-empty externally bounded text prevents data loss rather than duplicating a provider item.
• Fix: a53327677 routes every shutdown-bounded local commit through one idempotent helper that rejects empty/whitespace candidates while retaining a non-empty unsubmitted turn exactly once.

Post-fix evidence: the close module passes 9 tests; close + external-input + reply-context + AgentSession pass 189 tests; full Ruff and strict package typing pass.

[TonyG-FWE]

Fresh Devin follow-up (e890e9dc2)

I rechecked all 2 bug cards and 12 investigation flags against the exact prior head before changing production code.

Confirmed and fixed:

• Provider-owned shutdown duplication: a server-detected audio turn could be followed by a second, externally recognized item because close-time local retention ran before the provider-ownership check. The fail-before regression in 6ade6f5d8 produced two user items; e890e9dc2 now keeps the provider-owned item only. The complete close/fallback/late-final set is 59/59.
• Zero-cooldown fallback cycling: a replacement failure could immediately re-enable and retry the same model, bypassing a later healthy backup. The fail-before regression in 6ade6f5d8 landed back on model 0; 479d228c0 attempts each model once per swap and lands on model 2.

Not production defects:

• The late-final/empty-bounce card does not occur through legal recognition events. A VAD late final replaces the pending bounce; an STT final synchronously schedules its replacement before the old bounce can clean up. Both lifecycle tests pass.
• Mixed Gemini user/tool updates exclude function-call items before computing the trailing provider turn; fresh-session tool results are intentionally rejected, while resumable restarts retain FIFO ownership.
• Complete deferred audio is intentionally retained; capping it would truncate a genuine next turn. Swap-time audio is intentionally dropped rather than replayed with permanent latency.
• The STT reconnect remains the guarded segment-boundary compatibility workaround. The id(event) and replay-exclusion observations remain bounded implementation debt, not reproduced lifecycle failures.
• Preemptive realtime generation remains excluded by the existing llm.LLM guard. Exhausted fallback replacement emits a terminal error and immediately starts AgentSession close; no continuing child is selected.

Validation on this head: 260 affected ownership tests passed (the two remaining AudioRecognition failures reproduce on exact upstream 49bfd8b31), 187 hermetic provider tests passed, Ruff is clean across 943 files, and Python 3.13 mypy is clean across 206 files. GitHub Linux CI is now authoritative for the full unit and Python 3.10 lanes.

[TonyG-FWE]

Final Devin ownership-boundary follow-up (946be984e)

The remaining findings were confirmed with deterministic fail-before coverage in 7ec384a51 and fixed without changing the established turn-policy, FIFO transaction, reconnect, or replay architecture:

• 1245db908 delegates explicit clear/activity controls to provider-owned sessions without creating framework ownership, and restores per-reply interruption overrides when the session default is disabled.
• 946be984e observes provider-close failures that outlive caller-cancelled teardown and emits one terminal error for an unexpected automatic fallback-swap failure while preserving normal cancellation and explicitly awaited restart() propagation.

Post-fix evidence: all five exact regressions pass; the four directly affected modules pass 145/145; Google realtime passes 120/120; OpenAI/xAI realtime passes 62/62. GitHub is green with 2,228 unit tests passed and 5 skipped, native Python 3.10/3.13 typing, Ruff, BlockGuard on all three operating systems, aggregate dumps, release gate, and CLA.

The six documented compatibility/debt flags listed in the PR description remain unchanged because no focused lifecycle reproduction established a defect.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 16 additional findings in Devin Review.

[devin-ai-integration]

🟡 A declined short user turn can still wipe the speech recognizer and leave the model holding stale audio

A user turn that the session deliberately declined is torn down anyway (if committed or allow_empty_transcript: at livekit-agents/livekit/agents/voice/audio_recognition.py:1876) instead of being kept open, so the recognizer is restarted while the model still holds the old audio.

Impact: With a minimum-word interruption threshold configured, brief noise while the agent is talking can drop in-flight speech recognition and let the leftover audio bleed into the user's next sentence.

Mechanism: `allow_empty_transcript` overrides the `committed=False` verdict from `on_end_of_turn`

_run_eou_detection is invoked with allow_empty_transcript=True from two places when a realtime model uses client-side turn detection and finalize_empty_transcript_on_timeout is set: audio_recognition.py:1490-1491 (STT end-of-speech with no text) and audio_recognition.py:2112-2117 (transcription timeout).

Inside _bounce_eou_task, the cleanup block is now gated on committed or allow_empty_transcript (audio_recognition.py:1876). It ends the user-turn span, resets _user_turn_start, clears _audio_transcript/_audio_interim_transcript/_audio_preflight_transcript, and calls _restart_stt_input() (audio_recognition.py:1898-1901), which tears down and recreates the STT stream (audio_recognition.py:1141-1147).

However AgentActivity.on_end_of_turn can return False for that same turn via the minimum-word interruption gate at agent_activity.py:3184-3198. That branch explicitly does not discard the buffered provider audio, with the comment "AudioRecognition retains this transcript for the next endpointing verdict. Keep the matching provider audio under the same logical input owner as well."

The two layers therefore disagree: the activity keeps the realtime turn transaction open and the provider audio buffered, while AudioRecognition closes the turn and recreates the STT stream. Two consequences follow:

1. The in-flight STT stream is destroyed, so any audio still buffered inside it (which could have produced the transcript the activity said it would re-evaluate) is lost.
2. The provider-side empty turn is never settled — the exact failure the finalize_empty_transcript_on_timeout path was added to fix — so that audio prefixes the next committed turn.

Reachability: _current_speech active and non-interruptible-yet, interruption["min_words"] > 0 (opt-in; default is 0 per voice/turn.py:197), self.stt is not None, _turn_detection != "manual", and an empty transcript at timeout. The sibling backchannel branch at agent_activity.py:3202-3216 cannot collide because it requires self.stt is None.

Prompt for agents

In `AudioRecognition._run_eou_detection`'s inner `_bounce_eou_task`, the turn-teardown block is gated on `committed or allow_empty_transcript`. `allow_empty_transcript` is decided when `_run_eou_detection` is called, before the end-of-turn hook runs, so it can force teardown even when `self._hooks.on_end_of_turn(...)` returned `False`.

`AgentActivity.on_end_of_turn` returns `False` from the minimum-word interruption gate (see the branch guarded by `self._session.options.interruption["min_words"] > 0` in `livekit-agents/livekit/agents/voice/agent_activity.py`). That branch intentionally leaves the realtime turn transaction open and the buffered provider audio owned by the current turn, expecting AudioRecognition to keep accumulating for the next endpointing verdict. But the teardown block ends the user-turn span, resets `_user_turn_start`, clears the transcript buffers, and calls `_restart_stt_input()`, destroying the live STT stream.

Decide which layer owns the decision and make them agree. Options to consider: (a) only run the teardown block when `committed` is true, and handle the empty-turn settlement separately (e.g. only reset span/transcript state, without `_restart_stt_input()`, when the hook declined); or (b) have the activity treat an `allow_empty_transcript` turn as authoritative so the minimum-word gate does not decline it (for example by discarding the provider audio and returning `True` in that case). Ensure the existing tests `test_vad_empty_bounce_is_replaced_by_late_final` and `test_stt_empty_bounce_consumes_late_final_before_cleanup` in `tests/test_audio_recognition_turn_detection.py` still pass, and add coverage for the declined + `allow_empty_transcript` combination.

---

Was this helpful? React with 👍 or 👎 to provide feedback.