fix(interruption): tolerate websocket teardown race on session.close write - #6893
Open
BhaskarKapri07 wants to merge 1 commit into
Open
fix(interruption): tolerate websocket teardown race on session.close write#6893BhaskarKapri07 wants to merge 1 commit into
BhaskarKapri07 wants to merge 1 commit into
Conversation
β¦write
send_task writes a final session.close frame once the audio input channel
closes. If the transport is already closing at that moment β the inference
server won the close race, or the local shutdown path called ws.close()
first β aiohttp's writer guard raises ClientConnectionResetError
("Cannot write to closing transport"). That exception is not an APIError,
so neither _main_task's retry/emit path nor AudioRecognition's
except APIError swallows it, and it escapes as an unhandled asyncio task
exception during agent-activity teardown (e.g. embedded agent handoffs,
call end), surfacing as spurious error reports. The barge-in capability
for that activity silently stops.
- send_task: skip the session.close write when ws.closed and swallow the
connection reset when the peer wins the close race (best-effort).
- _run finally: cancel the tasks before ws.close(), so the local close
cannot arm the writer guard while send_task is mid-write.
- Mid-stream audio send failures intentionally still surface as an
unrecoverable detector error (documented by test).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes an unhandled
ClientConnectionResetError: Cannot write to closing transportthat escapes the adaptive interruption (barge-in) WebSocket during teardown, surfacing as spurious error reports and silently killing barge-in detection for the affected activity.Problem
InterruptionWebSocketStream.send_taskwrites a finalsession.closeframe once the audio input channel closes. If the transport is already closing at that moment β the inference server won the close race, or the local shutdown path calledws.close()first β aiohttp's writer guard raises:Because
ClientConnectionResetErroris not anAPIError, neither_main_task's retry/emit-error path norAudioRecognition._interruption_task'sexcept APIError: returnswallows it. It escapes as an unhandled asyncio task exception whenever an agent activity is torn down around the interruption stream (embedded agent handoffs, call end) β and the adaptive barge-in capability for that activity silently stops.Observed in production on LiveKit Cloud: 3 occurrences in ~3 weeks, all at the
session.closewrite during teardown, all with zero impact on the call itself β pure crash-report noise plus lost false-interruption filtering.Fix
send_task: skip thesession.closewrite whenws.closed, and swallow the connection reset when the peer wins the close race. The close message is best-effort β if the transport is already gone, there is nothing worth sending._runfinally: cancel the send/recv/forward tasks beforews.close(), so the local close can no longer arm the writer guard whilesend_taskis mid-write. This removes the local half of the race; thetry/exceptcovers the remaining peer-initiated half.Tests
tests/test_interruption/test_interruption_failover.py::TestWsConnectionReset::test_session_close_reset_is_ignored_during_teardownβ the teardown close race produces no error event and no task exception.tests/test_interruption/test_interruption_failover.py::TestWsConnectionReset::test_audio_send_reset_fails_instead_of_hangingβ a mid-stream send reset surfaces as exactly one unrecoverable error.Validated locally: full
tests/test_interruption/suite (17 tests) green,ruff checkandruff format --checkclean,scripts/check_types.py(strict mypy) reports no issues.