feat(voice): let agents reply with a voice recording - #173
Conversation
Agents could only produce text; the fork's message-listening TTS was a manual per-message action. This adds a voice_reply MCP tool on the injected t3-code server: the agent writes a script for the ear, the server synthesizes it through ElevenLabs and stages the MP3, and provider-runtime ingestion attaches it to the turn's final assistant message when the turn completes (a voice-only turn publishes the transcript as the message). The projector owns the speech row via the message-sent event, so live clients get the update and replay rebuilds it. Web and desktop render the player as the main message content with the written reply behind a toggle; mobile does the same. MCP credentials now carry per-capability scopes, gated by ELEVENLABS_API_KEY and a new Settings -> Extras switch. Implemented by Claude Fable 5 via Claude Code.
Bind each staged recording to the turn that was active when voice_reply ran, so a stale completion for another turn can no longer consume or delete it and an interrupted turn cannot leak its recording into the next one; turn.aborted now discards a matching entry. The staged entry is cleared only after the attach commands land instead of being taken up front. The projector no longer deletes replaced audio files (unsafe under transaction rollback and event replay), and the web primary player drops the Regenerate action that would have replaced an agent recording with re-synthesized text. Review findings by gpt-5.6-sol; fixes by Claude Fable 5 via Claude Code.
|
An independent gpt-5.6-sol review of the diff flagged turn-ownership and file-lifecycle problems; aec9804 addresses them: staged recordings are now bound to the turn that was active when voice_reply ran (stale or foreign turn completions leave them alone, turn.aborted discards them), the entry is cleared only after the attach commands land, the projector no longer deletes replaced audio files, and the web primary player no longer offers a Regenerate that would replace an agent recording with re-synthesized text. Accepted as known limitations: global MCP tool listing per server, no crash-recovery for staged MP3 orphans, and last-synthesis-wins on concurrent voice_reply calls. |
Written replies now surface when the recording's audio file is missing, voice-only turns no longer offer a written reply that duplicates the transcript, mobile no longer downloads every visible recording on mount (the player fetches on first play), empty assistant messages that carry only a recording render on mobile instead of disappearing, VoiceOver labels say voice reply, and the Settings toggle is searchable.
…ping Recordings are claimed atomically (a re-stage can no longer cross-wire or delete an in-flight attach), staging requires the active turn and revalidates it after synthesis, stale completions for superseded turns are gated out, the attach is one command whose fallback transcript the decider publishes only when the message has no text, user-requested synthesis can no longer overwrite an agent recording, and each MCP capability set gets its own endpoint so tools/list only advertises what the credential can call. voice_reply now states when to use it, and an empty script gets its own error reason.
… fallback The user-synthesis upsert now refuses to replace a row an agent recording claimed mid-synthesis and serves that recording instead, and a mobile voice-only message whose MP3 is gone shows its transcript as the message text instead of nothing.
|
Second review round, run against dd5a3c8: an opus pass over the client surfaces, contracts, and projection read path, and an independent gpt-5.6-sol pass attacking server lifecycle and concurrency. Fixed across e0e168a, b33bf98, and 512a848:
A third gpt-5.6-sol pass verified the fix commits and found two residuals (the mid-synthesis overwrite race and the mobile voice-only dead-audio case); 512a848 closes both. Accepted limitations stand: staging is memory-only across restarts, and concurrent |
The thread reducer dropped the speech payload carried on the completion event, so a recording only appeared after a page reload. Found in the integrated web pass; the player now shows up the moment the turn ends.
|
Integrated verification, both surfaces, against a disposable environment with a real Claude session: Web (paired browser, three voice turns): the agent received Mobile (iPhone 17 Pro simulator, dev client + Metro): all recordings render as primary players showing
|
Agents could only answer in text; the fork's TTS was a manual per-message "listening version". This gives agents a
voice_replyMCP tool so a reply can arrive as a recording you listen to, which is the point of discussing conceptual work on the go.How it works:
t3-codeMCP server gains avoicetoolkit.voice_reply(script)synthesizes the script through ElevenLabs (reusing the message-listening model/voice settings), writes the MP3 to the attachments dir, and stages it bound to the thread's active turn. Staging refuses when no active turn can be identified, and re-checks the turn after the synthesis call so a steered thread cannot end up with a recording bound to the wrong turn. The tool description says when to use it (only when the user asked to hear the answer), teaches writing for the ear, and keeps the normal text reply.thread.message.assistant.completecommand carrying the new optionalspeechpayload plus the transcript asfallbackText. The decider publishes the fallback text only when the target message is missing or has no text of its own, so a voice-only turn lands as a normal message with the transcript as its text, in the same atomic command. The projector materializes theprojection_message_speechrow (neworigincolumn, migration 050), so live clients get the pushed update and event replay rebuilds the row. Failed/interrupted turns, aborts, and session exits discard the staged audio.previewfrom agent browser access as before,voicewhenELEVENLABS_API_KEYis set and the new Settings → Extras switch (default on) allows it. Each capability combination is served by its own MCP server at its own endpoint (/mcp,/mcp/preview,/mcp/voice), sotools/listonly advertises what the credential can call; capability checks at call time remain as a second gate.Known limitations: staging is in-memory, so a recording staged when the server crashes mid-turn leaves an orphaned MP3 and is not recovered on restart; concurrent
voice_replycalls in one turn resolve last-synthesis-wins.Focused tests: ingestion attach/voice-only/discard/turn-mismatch paths (mutation-verified), snapshot origin gate, registry capability endpoints, settings patch round-trip.
vp checkand repovp run typecheckpass.Implemented by Claude Fable 5 via Claude Code.