feat(voice): add user_away_signal for transcript-based away tracking - #6880
feat(voice): add user_away_signal for transcript-based away tracking#6880longcw wants to merge 1 commit into
Conversation
The "away" user state follows detected speech. On a noisy line VAD and STT report speech that never becomes text, and each report re-arms the full user_away_timeout. Noise also ends "away" as soon as it starts. user_away_signal="transcript" trusts only transcribed speech: activity that produces no text does not hold off "away", and only a transcript ends it. Interim transcripts count, so a long answer does not trip "away" before its final arrives. The default "audio" is unchanged.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if ( | ||
| self._opts.user_away_signal == "transcript" | ||
| and self._user_state == "away" | ||
| and not by_transcript | ||
| ): | ||
| # only a transcript ends "away"; noise would otherwise clear it at once | ||
| return |
There was a problem hiding this comment.
🟡 Typing a message while marked away leaves the user stuck as away for the rest of the call
Speech-independent user turns such as a typed chat message are refused (_update_user_state guard at livekit-agents/livekit/agents/voice/agent_session.py:1992-1998) whenever the user is already marked away under the transcript-based setting, so the user stays marked away even while actively conversing.
Impact: With the transcript away signal enabled, a user who returns via text (the default text-input path) is still reported as away for the remainder of the session, so "are you still there?" style logic and any app code keyed on the user state behave as if nobody is present.
Why the away state can never be cleared by a text turn
_default_text_input_cb (livekit-agents/livekit/agents/voice/room_io/types.py:46-49) wraps the turn in AgentSession._claim_user_turn (livekit-agents/livekit/agents/voice/agent_session.py:1537-1560), whose contract is to pin user_state to "speaking" for the duration and re-derive it on release.
Both calls it makes — _update_user_state("speaking", last_speaking_time=...) at line 1551 and _update_user_state("speaking" if speaking else "listening") at line 1560 — pass by_transcript=False. With user_away_signal == "transcript" and _user_state == "away", the new guard returns early for both, so the state remains "away".
Nothing later restores it: under the transcript signal _update_user_state no longer touches the away timer, _update_agent_state only arms it when the user is listening/speaking (line 1961-1968), and only _user_input_transcribed passes by_transcript=True. A text-only user therefore stays "away" indefinitely.
| if ( | |
| self._opts.user_away_signal == "transcript" | |
| and self._user_state == "away" | |
| and not by_transcript | |
| ): | |
| # only a transcript ends "away"; noise would otherwise clear it at once | |
| return | |
| if ( | |
| self._opts.user_away_signal == "transcript" | |
| and self._user_state == "away" | |
| and not by_transcript | |
| and self._user_turn_claims == 0 | |
| ): | |
| # only a transcript (or a programmatic user turn) ends "away"; | |
| # noise would otherwise clear it at once | |
| return |
Was this helpful? React with 👍 or 👎 to provide feedback.
davidzhao
left a comment
davidzhao
left a comment
There was a problem hiding this comment.
this lg, but I think we should not use an additional toggle, but automatically count any presence of user transcript as activity. if we can make dealing with VAD signals more resilient, then we could simplify the amount of toggles a user has to figure out and tune.
this has been added, the problem is that the noise causes the VAD to false alarm and therefore never recognizes if the user is |
| user_away_signal (Literal["audio", "transcript"], optional): Which user | ||
| activity holds off the "away" state. ``"audio"`` (default) trusts any | ||
| detected speech. ``"transcript"`` trusts only transcribed speech, so | ||
| noise that never becomes text is ignored; it needs a streaming STT. |
There was a problem hiding this comment.
We should mention the audio signal can come from either vad or STT.
| user_away_signal (Literal["audio", "transcript"], optional): Which user | ||
| activity holds off the "away" state. ``"audio"`` (default) trusts any | ||
| detected speech. ``"transcript"`` trusts only transcribed speech, so | ||
| noise that never becomes text is ignored; it needs a streaming STT. |
There was a problem hiding this comment.
This would also suppress the timer for some STTs that send ghost transcripts. Observed with 11labs Scribe v2 from #4043
There was a problem hiding this comment.
Same failure on Deepgram SIP, not only Scribe v2. Noise often yields non-empty interims (uh, the, punctuation). If those re-arm the full away window, a drip every few seconds never reaches away.
Separate hole on the same path: if away swallows VAD start and end, an interim promotes to speaking and a later final does not demote. The user stays speaking until the next timeout.
4 participants
Problem
The
awayuser state follows detected speech. On a noisy phone line, VAD and DeepgramSpeechStartedreport speech that never becomes text, and each report re-arms the fulluser_away_timeout. Noise also endsawayas soon as it starts, which stops the "are you still there?" flows that the state exists for.Fix
This change adds
user_away_signal. The default value"audio"keeps the current behavior, so a session that does not set the option is unchanged. The value"transcript"trusts only transcribed speech: activity that produces no text does not hold offaway, and only a transcript endsaway.The
"transcript"value reads interim transcripts, so a long answer does not tripawaybefore its final arrives. This needs a streaming STT, becauseStreamAdapteremits no interim transcripts.Fixes #6030. Supersedes #6499, and thanks to @dorukdumlu, whose analysis in that issue found the root cause.