fix(voice): preserve turns across activity handoffs - #6844
Conversation
|
ian seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| await self._schedule_reply_after_user_turn( | ||
| info=user_turn.info, | ||
| user_message=user_turn.user_message, | ||
| chat_ctx=user_turn.chat_ctx, | ||
| on_user_turn_completed_delay=user_turn.on_user_turn_completed_delay, | ||
| handoff_user_turn=user_turn, | ||
| ) |
There was a problem hiding this comment.
🔴 After an agent handoff, the new agent answers the carried-over question using the previous agent's persona and history
The carried-over question is answered using the previous agent's conversation snapshot (chat_ctx=user_turn.chat_ctx at livekit-agents/livekit/agents/voice/agent_activity.py:2627) instead of the new agent's own, so the new agent replies with the old agent's personality and past conversation.
Impact: The first reply after a handoff sounds like the old agent (wrong system prompt, wrong history), which can produce off-topic or incorrect answers and defeats the purpose of switching agents.
Mechanism: the source activity's `temp_mutable_chat_ctx` is replayed on the successor activity
In _user_turn_completed_task the source activity builds temp_mutable_chat_ctx = self._agent.chat_ctx.copy() (livekit-agents/livekit/agents/voice/agent_activity.py:2556). That copy contains the source agent's history plus the system message injected by update_instructions(self._agent._chat_ctx, instructions=self._agent.instructions, add_if_missing=True) in _start_session (livekit-agents/livekit/agents/voice/agent_activity.py:1065-1069).
The copy is stored on the _PreparedUserTurn (agent_activity.py:2573-2578), forwarded to the successor activity, and finally passed to _schedule_reply_after_user_turn(chat_ctx=user_turn.chat_ctx, ...) from _prepared_user_turn_completed_task. _schedule_reply_after_user_turn hands it straight to _generate_reply(chat_ctx=chat_ctx, ...), which uses it verbatim as the LLM input (chat_ctx or self._agent._chat_ctx at agent_activity.py:1625).
_pipeline_reply_task_impl only rewrites instructions when self._agent.instructions is an Instructions object (agent_activity.py:3223-3229); for the common str case nothing removes the source agent's system message, so the successor's LLM call runs with the old instructions while receiving the new agent's tools (all_tools = self.tools.copy() at agent_activity.py:1576). The AgentHandoff marker inserted by _update_activity (livekit-agents/livekit/agents/voice/agent_session.py:1767) is also absent from that snapshot.
Note the non-prepared (_EndOfTurnInfo) forwarding path is unaffected: the successor re-runs _user_turn_completed_task, which rebuilds temp_mutable_chat_ctx from its own agent.
tests/test_agent_task_handoff_turn.py:547-553 actually asserts the source snapshot reaches the successor's llm_node, so this leak is currently locked in by the tests.
Prompt for agents
A pipeline user turn whose `on_user_turn_completed` callback already ran on the source activity is forwarded to the successor activity as a `_PreparedUserTurn`. `_prepared_user_turn_completed_task` then calls `_schedule_reply_after_user_turn(chat_ctx=user_turn.chat_ctx, ...)`, and that chat context is the *source* agent's `temp_mutable_chat_ctx` snapshot.
Because `_generate_reply` passes that chat context straight through to `_pipeline_reply_task_impl`, and `_pipeline_reply_task_impl` only re-renders instructions when `Agent.instructions` is an `Instructions` object (the plain-`str` case is left untouched), the successor agent's first LLM call runs with the previous agent's system instructions and the previous agent's message history, while being given the successor's tool set. The `AgentHandoff` item inserted by `AgentSession._update_activity` is also missing from that snapshot.
The goal of preserving the callback's edits (rewritten user message, extra context messages added by the source's `on_user_turn_completed`) is reasonable, but it should be rebased onto the successor agent's own chat context rather than replaying the source's snapshot wholesale. Consider capturing only the delta the callback introduced (or just the finalized `user_message`) and applying it to `self._agent.chat_ctx.copy()` on the successor when the prepared turn resumes. At minimum, the source's instructions system message must be stripped/replaced with the successor's instructions.
The existing assertion in tests/test_agent_task_handoff_turn.py (test_callback_handoff_continues_without_invoking_the_callback_twice) that checks the successor's `llm_context` contains the source's system message will need to be revisited alongside the fix.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
What changed
Why
During an
AgentTaskorupdate_agent()handoff, the source activity blocks scheduling before the successor is fully active. An EOU landing in that window currently follows thescheduling_pausedbranch, returnsTrue, and clears the recognition transcript without adding it to either activity's history. The user turn disappears.The queue is owned by the activity and moves across the concrete activity boundary. Successful successors consume it through
_user_turn_completed_task; terminal paths use the same history/metrics semantics as session close. Realtime models are intentionally excluded because their input commit lifecycle is remote and is covered separately by #6662.Impact
Cascade/
AgentTaskand serializedupdate_agent()handoffs retain a crossing user turn exactly once instead of silently dropping it.Design feedback requested
This PR implements the existing TODO in
on_end_of_turnby transferring ownership to the successor activity. Feedback is especially welcome on whether activity-owned buffering is the preferred framework boundary for chained handoffs.Validation
skipping user input, speech scheduling is pausedAgentTask, consecutiveupdate_agent(), successor close rejection, successor startup failure, and source closepytest tests/test_agent_task_handoff_turn.py tests/test_nested_agent_task.py tests/test_update_agent_long_on_enter.py tests/test_agent_task_close_race.py tests/test_foreground_run_tracking.py tests/test_agent_session.py --unit -q— 98 passedlivekit.agents) — 206 files passedRelated but distinct: #6662 fixes Realtime paused-speech/overlap behavior; this PR handles local pipeline turn ownership across activity replacement.