feat(room-io): listen to several participants through one audio input - #6884
Open
Darshak03 wants to merge 1 commit into
Open
feat(room-io): listen to several participants through one audio input#6884Darshak03 wants to merge 1 commit into
Darshak03 wants to merge 1 commit into
Conversation
A single AgentSession could only ever hear the linked participant, so a room
with two humans in it produced a one-sided chat context.
The composition point is an AudioInput rather than RoomIO: `_RoomAudioInput` is
what RoomIO drives, and `_ParticipantAudioInputGroup` holds one unchanged
`_ParticipantAudioInputStream` per participant. `AudioInputOptions.participants`
selects how they are combined:
- "linked" (default): unchanged, a single participant stream.
- "mix": summed by `rtc.AudioMixer`, keeps overlapping speech.
- "pick": only the participant the server reports as speaking, preceded by
their held pre-roll and separated by a silence gap.
RoomIO keeps owning the participant lifecycle and routes it through
`add_participant`/`remove_participant`, which do nothing for an input that only
listens to the linked participant. `participant_identity` and `set_participant`
still choose the linked participant, which now only drives the outputs.
Mixing an idle stream would pace the mix below real time and warn on every
block, so a participant is registered with the mixer only while a live, unmuted
track is being read: `_ParticipantInputStream` now reports when that changes,
including a track that stops delivering without being unpublished.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A single AgentSession could only ever hear the linked participant, so a room with two humans in it produced a one-sided chat context.
The composition point is an AudioInput rather than RoomIO:
_RoomAudioInputis what RoomIO drives, and_ParticipantAudioInputGroupholds one unchanged_ParticipantAudioInputStreamper participant.AudioInputOptions.participantsselects how they are combined:rtc.AudioMixer, keeps overlapping speech.their held pre-roll and separated by a silence gap.
RoomIO keeps owning the participant lifecycle and routes it through
add_participant/remove_participant, which do nothing for an input that only listens to the linked participant.participant_identityandset_participantstill choose the linked participant, which now only drives the outputs.Mixing an idle stream would pace the mix below real time and warn on every block, so a participant is registered with the mixer only while a live, unmuted track is being read:
_ParticipantInputStreamnow reports when that changes, including a track that stops delivering without being unpublished.