Skip to content

feat(voice): let agents reply with a voice recording - #173

Merged
pandec merged 7 commits into
devfrom
fork/agent-voice-reply
Aug 24, 2026
Merged

feat(voice): let agents reply with a voice recording#173
pandec merged 7 commits into
devfrom
fork/agent-voice-reply

Conversation

@pandec

@pandec pandec commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Agents could only answer in text; the fork's TTS was a manual per-message "listening version". This gives agents a voice_reply MCP tool so a reply can arrive as a recording you listen to, which is the point of discussing conceptual work on the go.

How it works:

  • The injected t3-code MCP server gains a voice toolkit. voice_reply(script) synthesizes the script through ElevenLabs (reusing the message-listening model/voice settings), writes the MP3 to the attachments dir, and stages it bound to the thread's active turn. Staging refuses when no active turn can be identified, and re-checks the turn after the synthesis call so a steered thread cannot end up with a recording bound to the wrong turn. The tool description says when to use it (only when the user asked to hear the answer), teaches writing for the ear, and keeps the normal text reply.
  • Provider-runtime ingestion claims the staged recording atomically when its turn completes normally (stale completions for superseded turns are gated out) and attaches it with one thread.message.assistant.complete command carrying the new optional speech payload plus the transcript as fallbackText. The decider publishes the fallback text only when the target message is missing or has no text of its own, so a voice-only turn lands as a normal message with the transcript as its text, in the same atomic command. The projector materializes the projection_message_speech row (new origin column, migration 050), so live clients get the pushed update and event replay rebuilds the row. Failed/interrupted turns, aborts, and session exits discard the staged audio.
  • MCP credentials carry per-capability scopes: preview from agent browser access as before, voice when ELEVENLABS_API_KEY is set and the new Settings → Extras switch (default on) allows it. Each capability combination is served by its own MCP server at its own endpoint (/mcp, /mcp/preview, /mcp/voice), so tools/list only advertises what the credential can call; capability checks at call time remain as a second gate.
  • User-requested synthesis treats an agent recording as immutable: asking for a listening version of a voice-reply message returns the recording instead of overwriting it, which an older client could otherwise trigger.
  • Web/desktop and mobile render agent recordings as the message's main content, player first with transcript, written reply behind a "Show written reply" toggle. The toggle is skipped when it would only duplicate the transcript, the written reply is forced visible when the audio file is gone, and mobile defers downloading the MP3 until the first play so a feed full of recordings does not fetch audio on scroll. User-requested listening versions are unchanged, and the snapshot staleness gate is bypassed for agent recordings since their transcript is authored independently of the message text.

Known limitations: staging is in-memory, so a recording staged when the server crashes mid-turn leaves an orphaned MP3 and is not recovered on restart; concurrent voice_reply calls in one turn resolve last-synthesis-wins.

Focused tests: ingestion attach/voice-only/discard/turn-mismatch paths (mutation-verified), snapshot origin gate, registry capability endpoints, settings patch round-trip. vp check and repo vp run typecheck pass.

Implemented by Claude Fable 5 via Claude Code.

pandec added 2 commits August 24, 2026 15:57
Agents could only produce text; the fork's message-listening TTS was a
manual per-message action. This adds a voice_reply MCP tool on the
injected t3-code server: the agent writes a script for the ear, the
server synthesizes it through ElevenLabs and stages the MP3, and
provider-runtime ingestion attaches it to the turn's final assistant
message when the turn completes (a voice-only turn publishes the
transcript as the message). The projector owns the speech row via the
message-sent event, so live clients get the update and replay rebuilds
it. Web and desktop render the player as the main message content with
the written reply behind a toggle; mobile does the same. MCP credentials
now carry per-capability scopes, gated by ELEVENLABS_API_KEY and a new
Settings -> Extras switch.

Implemented by Claude Fable 5 via Claude Code.
Bind each staged recording to the turn that was active when voice_reply
ran, so a stale completion for another turn can no longer consume or
delete it and an interrupted turn cannot leak its recording into the
next one; turn.aborted now discards a matching entry. The staged entry
is cleared only after the attach commands land instead of being taken
up front. The projector no longer deletes replaced audio files (unsafe
under transaction rollback and event replay), and the web primary
player drops the Regenerate action that would have replaced an agent
recording with re-synthesized text.

Review findings by gpt-5.6-sol; fixes by Claude Fable 5 via Claude Code.
@pandec

pandec commented Aug 24, 2026

Copy link
Copy Markdown
Owner Author

An independent gpt-5.6-sol review of the diff flagged turn-ownership and file-lifecycle problems; aec9804 addresses them: staged recordings are now bound to the turn that was active when voice_reply ran (stale or foreign turn completions leave them alone, turn.aborted discards them), the entry is cleared only after the attach commands land, the projector no longer deletes replaced audio files, and the web primary player no longer offers a Regenerate that would replace an agent recording with re-synthesized text. Accepted as known limitations: global MCP tool listing per server, no crash-recovery for staged MP3 orphans, and last-synthesis-wins on concurrent voice_reply calls.

pandec added 4 commits August 24, 2026 16:39
Written replies now surface when the recording's audio file is missing,
voice-only turns no longer offer a written reply that duplicates the
transcript, mobile no longer downloads every visible recording on mount
(the player fetches on first play), empty assistant messages that carry
only a recording render on mobile instead of disappearing, VoiceOver
labels say voice reply, and the Settings toggle is searchable.
…ping

Recordings are claimed atomically (a re-stage can no longer cross-wire or
delete an in-flight attach), staging requires the active turn and
revalidates it after synthesis, stale completions for superseded turns
are gated out, the attach is one command whose fallback transcript the
decider publishes only when the message has no text, user-requested
synthesis can no longer overwrite an agent recording, and each MCP
capability set gets its own endpoint so tools/list only advertises what
the credential can call. voice_reply now states when to use it, and an
empty script gets its own error reason.
… fallback

The user-synthesis upsert now refuses to replace a row an agent recording
claimed mid-synthesis and serves that recording instead, and a mobile
voice-only message whose MP3 is gone shows its transcript as the message
text instead of nothing.
@pandec

pandec commented Aug 24, 2026

Copy link
Copy Markdown
Owner Author

Second review round, run against dd5a3c8: an opus pass over the client surfaces, contracts, and projection read path, and an independent gpt-5.6-sol pass attacking server lifecycle and concurrency. Fixed across e0e168a, b33bf98, and 512a848:

  • Staged recordings are claimed atomically instead of peek-then-clear, so a concurrent voice_reply can no longer be cross-wired with an in-flight attach or have its file deleted under it.
  • Staging requires the thread's active turn (no more wildcard "attach to whatever completes next") and re-checks it after the synthesis call; a steered thread gets a turn_unavailable error instead of a recording bound to the wrong turn.
  • The voice block is gated on the same lifecycle guard the rest of turn handling uses, so a stale completion for a superseded turn leaves the staged entry alone.
  • The attach is one command. The transcript rides along as fallback text and the decider publishes it only when the target message is missing or blank, which also fixes the case where a remembered-but-never-projected message id produced an empty message that mobile rendered as nothing.
  • User-requested synthesis can no longer destroy an agent recording, including the race where the recording lands while a user synthesis is already running (the upsert refuses to replace an agent-owned row and serves it instead).
  • Each capability set gets its own MCP endpoint (/mcp, /mcp/preview, /mcp/voice), so tools/list no longer advertises toolkits the credential cannot call and the settings toggle does what its copy says.
  • Client round: written reply forced visible when the audio file is gone (both platforms, including voice-only messages), no "Show written reply" that duplicates the transcript, mobile defers the MP3 download until first play instead of fetching every visible recording, VoiceOver labels say "voice reply", and the toggle is searchable in Settings.
  • voice_reply's description now says when to use it (only when the user asked to hear the answer), and an empty script gets its own error reason.

A third gpt-5.6-sol pass verified the fix commits and found two residuals (the mid-synthesis overwrite race and the mobile voice-only dead-audio case); 512a848 closes both. Accepted limitations stand: staging is memory-only across restarts, and concurrent voice_reply calls resolve last-synthesis-wins.

The thread reducer dropped the speech payload carried on the completion
event, so a recording only appeared after a page reload. Found in the
integrated web pass; the player now shows up the moment the turn ends.
@pandec

pandec commented Aug 24, 2026

Copy link
Copy Markdown
Owner Author

Integrated verification, both surfaces, against a disposable environment with a real Claude session:

Web (paired browser, three voice turns): the agent received voice_reply on the voice-only MCP endpoint (/mcp/voice; all three capability islands answer 401 unauthenticated), called it, and ElevenLabs staged a 147 KB MP3. The recording attached to the correct assistant message with the written text preserved, the player renders as the message's primary content with working "Show written reply" and transcript toggles, the headphones tooltip reads "Hide voice reply", and the audio element plays (9.1 s recording, time advancing). This pass caught one real defect: the client thread reducer dropped the speech payload from the completion event, so the player only appeared after a reload — fixed in ea2d90f (with a reducer test), then verified live: the player now appears the moment the turn ends, no reload.

Mobile (iPhone 17 Pro simulator, dev client + Metro): all recordings render as primary players showing 0:00 / 0:00 until first tap — confirming the deferred MP3 load — then fetch and play to completion on tap. "Show written reply" appears only when the written text differs from the transcript, "View transcript" expands, VoiceOver labels say "Play voice reply", and a fourth turn sent from the device showed its recording live without re-entering the thread.

vp check and repo-wide vp run typecheck green at ea2d90f.

@pandec
pandec merged commit 8183dfd into dev Aug 24, 2026
1 check passed
@pandec
pandec deleted the fork/agent-voice-reply branch August 24, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant