WIP: multi-slot WOV arbiter with VAD gate and D0I3/S0iX support - #11107
Draft
lgirdwood wants to merge 20 commits into
Draft
WIP: multi-slot WOV arbiter with VAD gate and D0I3/S0iX support#11107lgirdwood wants to merge 20 commits into
lgirdwood wants to merge 20 commits into
Conversation
Allow multiple downstream WOV detector pipelines to share one KPB instance by extending the KPB to support a sel_sink drain path alongside the existing dedicated host_sink path. Key changes: - Add sel_sink field to kpb_data to track the downstream WOV detector sink; kpb_set_sink() assigns it from the component bind call. - kpb_init_draining: when host_sink is NULL (multi-KPB WOV topology with no dedicated PCM capture), redirect the pre-roll drain through sel_sink so that history reaches the wov_arbiter and onward to the host copier. Initialise host_period_size from sel_sink stream geometry when not already set. Skip pausing the selector component when sel_sink is the active drain path. - kpb_init_draining: cap drain_req to the actual buffered amount instead of aborting when less history is available than requested (partial pre-roll is better than none). - kpb_reset: add immediate-reset path when host_sink==NULL in BUFFERING/DRAINING state; the LL scheduler is gone after STOP so the async-EBUSY path can never complete. - kpb_init_draining: guard against NULL host_sink to prevent NULL deref on WOV-only instances. - Fix fallback sink assignment in kpb_copy: use && instead of || so the fallback only fires when BOTH sel_sink and host_sink are NULL. - Raise KPB_MAX_BUFF_TIME to 6000 ms; clear HOST_WAKEUP_TIME (no extra delay needed with direct sel_sink routing). - kpb.conf: expose host_sink_index parameter for topology binding. - Reduce log noise: demote two comp_err to comp_dbg in the RUN copy path. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add a lightweight VAD gate component that sits between the DAI copier and the mixin fan-out in the WOV capture pipeline. It measures per-frame signal energy and suppresses propagation of near-silence frames to the downstream KPB/detector branches, preventing unnecessary wakeups and reducing idle CPU load. The gate passes frames when the mean absolute sample value exceeds a configurable threshold (VAD_THRESHOLD, default 0 = bypass for testing). Suppressed frames are replaced with PPL_STATUS_PATH_STOP so the LL scheduler can idle the downstream pipelines. Kconfig: CONFIG_COMP_VAD_GATE (depends on IPC_MAJOR_4). Topology: vad-gate.conf widget class for topology2. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add two AMS message UUIDs and their payload structures for the multi-slot WOV arbitration protocol: AMS_WOV_DETECT_MSG_UUID — sent from a keyword detector to the wov_arbiter when a keyword is confirmed. Payload: wov_detect_payload carrying the slot_id (0..N-1) of the detector that fired. AMS_WOV_CTRL_MSG_UUID — sent from the wov_arbiter to all registered keyword detectors. Payload: wov_ctrl_payload carrying a cmd field (WOV_CTRL_CMD_PAUSE or WOV_CTRL_CMD_RESUME) and the active_slot (WOV_ARB_NO_ACTIVE=0xff on RESUME to address all slots). Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the wov_arbiter audio component. It receives audio from up to WOV_ARB_MAX_SLOTS (3) KPB host-sink pins and routes exactly one slot's audio to the single host-copier downstream of it. Before any keyword fires the arbiter fills the host sink with silence (memset zero). When any detector sends an AMS_WOV_DETECT message the arbiter activates that slot (first-wins), broadcasts AMS_WOV_CTRL PAUSE to all other detectors, and starts forwarding KPB pre-roll + live audio to the host. On stream STOP or PAUSE the arbiter resets to NO_ACTIVE and broadcasts RESUME so all detectors restart listening. IPC4 debug parameter param_id=1 (IPC4_WOV_ARB_SET_ACTIVE_SLOT) allows forcing a slot active without AMS, useful for lab testing. The IBS/OBS is set to 32 bytes (matching the 1 ms DMIC LL period). The arbiter fills exactly sink_free bytes on every copy to avoid host copier underruns during the pre-detection silence phase. Kconfig: CONFIG_COMP_WOV_ARBITER (auto-selects CONFIG_AMS). UUID: 4a5b6c7d-8e9f-4a1b-2c3d-4e5f60718293. Topology: wov-arbiter.conf widget class for topology2. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Enable CONFIG_COMP_WOV_ARBITER, CONFIG_COMP_VAD_GATE, CONFIG_COMP_KPB, CONFIG_SAMPLES, and CONFIG_SAMPLE_KEYPHRASE on the intel_adsp_cavs25 board (TigerLake, CAVS2.5) to include the multi-slot WOV pipeline in default firmware builds for that platform. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Extend detect_test (the sample keyword detector) to participate in the
multi-slot WOV arbitration system coordinated by wov_arbiter.
Slot assignment:
A static wov_slot_id is derived from the pipeline ID at component
creation time (pipeline 101/1 → slot 0, 102/3 → slot 1, 103/4 → slot 2).
The slot ID can be overridden via IPC4 LARGE_CONFIG_SET param_id=4
(IPC4_DETECT_TEST_SET_WOV_SLOT).
DP thread batching:
Each slot runs its own Zephyr k_thread at K_PRIO_PREEMPT(12) with a
4096-byte stack. The LL copy path accumulates 320-frame (20 ms at
16 kHz) S16_LE samples into a double-buffer and gives a semaphore when
a batch is ready. The DP thread wakes, runs the detection algorithm
on the completed buffer, and signals the LL thread to switch sides.
Slot 2 is additionally pinned to DSP Core 1 via k_thread_cpu_pin().
Threads are started in prepare() and stopped in reset() + free().
AMS integration:
On detection, detect_test_notify() sends:
1. SOF_IPC4_NOTIFY_PHRASE_DETECTED IPC4 notification to the host
(word_id = wov_slot_id).
2. AMS_KPD_MSG to the KPB to initiate pre-roll drain on host_sink.
3. AMS_WOV_DETECT_MSG to the wov_arbiter.
The component registers as a consumer of AMS_WOV_CTRL_MSG and responds
to PAUSE (stops detecting) and RESUME (resets cd->detected, resumes).
Rotating auto-trigger (test mode):
When CONFIG_AMS is enabled a per-slot auto-trigger fires after the
first 320-frame DP batch if this slot matches the current session target
(kd_test_session % KD_MAX_SLOTS). kd_test_session is incremented in
reset() when slot 0 resets, so each arecord session cycles through
slots 0 → 1 → 2 → 0 without a firmware reload.
Audio pass-through:
detect_test.copy() forwards source audio to the downstream
wov-arbiter sink so the real-time data path remains live both before
and after detection.
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add a standalone three-slot WOV topology (dmic-wov-multi.conf /
dmic-wov-multi-manifest.conf) driven by a single DMIC input with:
Pipeline 100 (Core 0): DAI Copier → vad_gate → mixin (1→3 fan-out)
Pipeline 101 (Core 0): mixout → kpb → detect_test (slot 0, Male)
Pipeline 102 (Core 0): mixout → kpb → detect_test (slot 1, Female)
Pipeline 103 (Core 1): mixout → kpb → detect_test (slot 2, Child)
Pipeline 104 (Core 0): wov_arbiter → host-copier (PCM 11, hw:0,11)
Each KPB has two output pins: sel_sink (to the detector) and host_sink
(to the arbiter). When a detector fires, it switches KPB to drain via
host_sink; the arbiter routes that slot's audio to the host copier.
Also included:
sof-hda-generic-wov-manifest.conf — build entry point for the WOV
topology variant (includes dmic-wov-multi.conf via INCLUDE_WOV).
Bug fixes to shared topology2 infrastructure:
wov.conf: rename wov_init → wov_init_$index so multiple wov widget
instances do not share a control name.
dai-kpb-be.conf: add dai_type/copier_type/stream_name fields to the
dai-copier widget; fix channel config for 4-channel DMIC; source
route now uses $DMIC_WOV_DAI_PIPELINE_SRC for flexibility.
wov-detect.conf: add num_input/output_audio_formats=1 to the micsel
widget (mandatory attribute missing from the original class).
sof-hda-generic.conf: add WOV widget class includes (dai-kpb-be,
wov-detect, vad-gate, kpb, wov, pin-binding helpers) so the generic
HDA topology can instantiate WOV pipelines when INCLUDE_WOV is set;
remove duplicate vad-gate.conf include.
tgl.toml.h: restore CONFIG_SAMPLE_KEYPHRASE guard on detect_test.toml;
add CONFIG_COMP_VAD_GATE and CONFIG_COMP_WOV_ARBITER guards on the
new vad_gate/wov_arbiter toml includes.
Compile with:
alsatplg -I tools/topology/topology2 -p \
-c tools/topology/topology2/dmic-wov-multi-manifest.conf \
-o sof-hda-generic-wov.tplg
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Document the multi-slot WOV subsystem comprehensively:
Architecture:
- Component graph mermaid diagram showing all 5 pipelines (100-104),
their component chains, and the AMS message flows.
- Key design points table (format, pre-roll duration, slot count,
arbitration policy, core affinity, PCM device).
Signal processing flow:
- LL thread flowchart: HW DMA → DAI → VAD gate → mixin → mixout →
KPB → detect_test accumulation → k_sem_give.
- DP thread flowchart: semaphore wake → algorithm → notify or
auto-trigger (test mode).
- KPB drain sequence: AMS → sel_sink switch → ring drain → live audio.
Arbiter state machine:
- stateDiagram-v2 with Idle/Active transitions and first-wins guard.
- Routing logic table (silence vs active slot vs idle slot output).
AMS inter-module messaging:
- Message catalog with UUID constants, directions, payload structs.
- Full detect-to-drain sequenceDiagram from DMIC capture to host close.
Firmware API reference:
- IPC4 LARGE_CONFIG_SET parameter tables for detect_test and wov_arbiter.
- AMS producer/consumer registration code examples.
- detect_test_notify() three-step contract.
- DP thread ping-pong buffer field semantics.
Linux host API:
- ALSA PCM capture (hw:0,11) via arecord and libasound.
- snd_ctl voice detection notification polling in userspace.
- Capture timing model (silence → pre-roll → live audio transition).
- IPC4 slot override via sof-ctl.
Adding a new WOV algorithm:
- Approach A: modify detect_test.c directly (prototype path).
- Approach B: new native SOF module with full skeleton (AMS reg,
IPC4 new/prepare/copy/notify, Kconfig, CMakeLists, UUID registration).
- Approach C: IADK/LLEXT loadable binary for third-party IP.
Topology: build and deploy guide:
- Source layout, alsatplg compile command, generic deploy instructions.
- Firmware build and deploy (generic build dir, generic DUT SCP).
- Topology config reference table. Adding a fourth slot.
Build system: Kconfig minimum set, module UUID catalog.
Testing and verification:
- Three-slot rotating auto-trigger test with Python RMS verification.
- Real-audio frequency sweep test.
- Firmware trace (mtrace) reading procedure and expected log events.
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Expose the new dmic-wov-multi topology through two non-invasive hooks in
dmic-generic.conf, without disturbing existing single-slot WOV users:
INCLUDE_WOV "multi": includes dmic-wov-multi.conf, selecting the
3-slot KPB/arbiter topology. The existing "true" value continues
to select the original dmic-wov.conf (single-slot, unchanged).
DMIC1_ENABLE "wov-multi": includes dmic-wov-multi.conf as an
alternative DMIC1 pipeline mode, alongside the existing
"passthrough" and "mfcc" values.
Build the multi-slot WOV topology variant with:
alsatplg ... -DINCLUDE_WOV=multi
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Single shared KPB ----------------- Move the KPB from pipelines 101/102/103 into pipeline 100, between the VAD Gate and the Mixin fan-out. All three WOV detector paths share one 192 KB history ring buffer instead of three, saving ~384 KB SRAM. Routes change: before: vad-gate.100 → mixin.100 → mixout.1NN → kpb.1NN → wov.1NN after: vad-gate.100 → kpb.100 → mixin.100 → mixout.1NN → wov.1NN Also fix two topology attribute errors that prevented builds: <hda.conf> → <dmic.conf> (wrong DAI conf include) default_hw_conf_id → default_hw_config_id (attribute name typo) add num_pdm_active 1 to DMIC DAI object wov_trigger_id bytes kcontrol ------------------------------ Add a volatile read-only bytes kcontrol to the wov-arbiter widget. access = [tlv_read tlv_callback], extops.get = 260 (SOF_TPLG_KCTL_BYTES_VOLATILE_RO), max = 64. The kernel issues GET_LARGE_CONFIG (param_id=2) on every userspace read. Firmware returns active_slot (0/1/2 after trigger, 0xff when idle) as a uint32_t wrapped in a 36-byte SOF4 ABI blob. Host flow: arecord hw:0,11 (opens pipeline) → detect_test fires → read numid=11 kcontrol → ioctl(SNDRV_CTL_IOCTL_TLV_READ) → bytes[40] = winning slot index
Add a per-slot boolean switch kcontrol (wov_mute_<N>) to each wov widget. When set to 0 (off/muted), test_keyword_copy drains the source buffer and returns without running detect_func, keeping the LL scheduler running without consuming DSP cycles for detection. When set to 1 (on/active), normal detection resumes. New muted field in struct comp_data; SOF_IPC4_SWITCH_CONTROL_PARAM_ID case in test_keyword_set_large_config maps value=0 to muted=true. Verified on spider TGL: muted=off suppresses auto-trigger (active_slot=0xff after 12s), muted=on restores detection (active_slot=1 in ~8s).
Add Object.Control.mixer.1 (max 1, get/put 259) to the wov class. Include controls/mixer.conf for the Class.Control.mixer definition. Use wov_mute_$index as the name so each wov instance gets a distinct kcontrol rather than being deduplicated by ALSA.
Document the new wov_mute_1NN boolean switch kcontrols added in the previous commit: - Add kcontrol numid reference table (all 7 WOV controls with types) - Add wov_mute API subsection in Firmware API Reference (wire format, IPC4 param_id=200, default semantics, amixer example) - Update wov_init numids (8/10/12) to reflect new interleaved layout - Add wov.conf and mixer.conf to topology source layout diagram - Fix ALSA_CONFIG_DIR note (must be absolute path; use $(pwd)) - Rename output tplg to sof-tgl-dmic-wov-multi.tplg to match actual filename loaded by kernel - Add mute test procedure with read_wov_trigger.py helper script - Update all test examples to arm wov_mute controls before recording
Escalate DSP clock to HPRO on voice onset and de-escalate to WOVCRO (38.4 MHz) after the silence hangover expires. During silence only the lightweight VAD energy IIR runs; all downstream KPB and WOV detector components are gated by PPL_STATUS_PATH_STOP. vad_gate_prepare() drops to WOVCRO immediately so the pipeline starts at low power, overriding the HPRO set by pipeline_cps_rebalance() at START. Clock transitions use adsp_clock_set_cpu_freq() via <adsp_clk.h> (CONFIG_SOF_ZEPHYR_NO_SOF_CLOCK=y bypasses the SOF clock layer). Kconfig: raise VAD_GATE_DEFAULT_HANGOVER_FRAMES default from 30 to 200 (200 x 10 ms frames = 2 s continuous silence before clock de-escalation). Threshold remains 0 (pass-through) by default; set via the bytes kcontrol at runtime to activate energy-based gating. Verified on spider TGL/CAVS2.5: WOV detection unchanged (active_slot=255 before trigger, active_slot=0 after auto-trigger), firmware boots clean.
Expose ipc4_vad_gate_config via a LARGE_CONFIG_SET bytes kcontrol so the energy threshold, onset count, and hangover count can be adjusted at runtime without rebuilding firmware. Default blob encodes threshold=0 (pass-through / always-HPRO) so the gate is transparent on first boot. Set a non-zero threshold via amixer or set_vad_threshold.py to enable WOVCRO/HPRO clock switching. Measured DMIC noise floor on TGL/CAVS2.5 (spider): ~42-74M energy units. Silence-blocking threshold for lab use: 100000000 (100M). Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Mark the WOV host-copier stream as D0I3-compatible so the kernel keeps
the capture pipeline active during S0iX system sleep. When the WOV
trigger fires, the firmware sends SOF_IPC4_NOTIFY_PHRASE_DETECTED which
wakes the host from D0I3.
Two placement sites:
- Object.Widget.host-copier.1: parsed by IPC4 kernel (sof_ipc4_widget_setup_pcm
in ipc4-topology.c, see thesofproject/linux#5878)
- Object.PCM.pcm: parsed by IPC3 kernel (topology.c)
Also adds DefineAttribute entries for capture_compatible_d0i3 and
playback_compatible_d0i3 to host-copier.conf so IPC4 alsatplg can
encode the token into the widget TLV tuple set.
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…roll drain Mark the WOV host PCM (pcm ID 11) as compress="true" so the ALSA driver exposes it as comprC0D11 instead of pcmC0D11. With the ALSA compress framework the HDA DMA is not rate-locked to the capture sample clock. The KPB drain EDF task can fill fragments as fast as the downstream host-copier accepts them, delivering the full pre-roll buffer to the host in a burst rather than at realtime rate. Once the pre-roll drains, KPB transitions to KPB_STATE_HOST_COPY and the live stream continues on the same compress device at realtime rate. No firmware source changes are needed; the same ipc4_hda_host_input_class gateway handles both PCM and compress capture. Verified on spider (TGL/CAVS2.5): comprC0D11 appears after driver reload.
Document all architectural changes made since the initial multi-slot WOV documentation: - Host PCM changed from pcmC0D11 (regular PCM) to comprC0D11 (ALSA compress). Explain burst pre-roll delivery: KPB draining EDF task fills HDA fragments without a rate limiter during drain, then settles to realtime rate. - Capture API: replace arecord with snd_compr (crecord / C API flow). Document /dev/snd/comprC0D11, SNDRV_COMPRESS_SET_PARAMS, and poll/read loop. - Kernel notification handler: sof_ipc4_phrase_detected() in ipc4-compress.c calls snd_sof_compr_fragment_elapsed() to wake blocked reads on trigger. Userspace no longer needs to poll an ALSA control-change event. - Add VAD Gate Runtime Configuration section: WOVCRO/HPRO clock transitions, set_vad_threshold.py usage, threshold/onset/hangover/shift field reference, TGL/spider lab calibration (threshold=100M, ambient floor 42-74M). - Add D0i3/S0iX Support section: capture_compatible_d0i3 = 1 on host-copier and PCM widget, ipc4_pcm_ops.d0i3_supported_in_s0ix. - Fix topology build command: ALSA_CONFIG_DIR must be a temp dir with alsa.conf and symlinks to topology2/include and topology2/platform, not the topology2 directory itself. - Update all kcontrol numids: vad_gate_cfg_100 now occupies numid=8 (ahead of wov_init_101=9, wov_mute_101=10, ..., wov_trigger_id=15) following topology rebuild. Add note to confirm with amixer -c 0 controls after each rebuild. - Update modprobe.d example to reflect verified snd_sof options for TGL/spider.
…ology The KPB pre-roll window is controlled by CONFIG_KPB_MAX_BUFF_TIME (default 6000 ms on CAVS2.5+/TGL, 2100 ms on other platforms). The topology kpb.conf class exposes no history-depth attribute — the buffer is sized at prepare() time from the compiled-in value. Add a callout in the Topology Configuration Reference table and an extended explanation in the Kconfig section with the byte-size formula and how to override via west build -- -DCONFIG_KPB_MAX_BUFF_TIME=<ms>.
Add a bytes kcontrol kpb_cfg_<N> to each KPB widget that lets the topology manifest set the ring-buffer pre-roll duration at load time, without a firmware rebuild. Firmware (IPC4 path): - ipc4/kpb.h: add KP_BUF_CFG_BUFF_TIME_MS = 2 to the param-ID enum - kpb.c: add buff_time_ms field to comp_data; add kpb_get_buff_time_ms() helper (returns topology value when set, falls back to CONFIG_KPB_MAX_BUFF_TIME); wire the helper into kpb_params(), kpb_prepare() buffer allocation, drain-request cap; handle the new LARGE_CONFIG_SET case in kpb_set_large_config() Topology: - kpb.conf: add Object.Control.bytes kpb_cfg_$index with IncludeByKey blobs for 6000/4000/2100 ms; blob is a 36-byte sof_abi_hdr + u32 ms - dmic-wov-multi-manifest.conf: Define KPB_BUFF_TIME_MS 6000 for TGL The kernel sof_ipc4_widget_kcontrol_setup() path sends the blob as a LARGE_CONFIG_SET on pipeline open with no userspace intervention. Omitting KPB_BUFF_TIME_MS from the manifest leaves buff_time_ms = 0, and kpb_get_buff_time_ms() falls back to CONFIG_KPB_MAX_BUFF_TIME. Verified on spider (TGL/CAVS2.5): kpb_cfg_101/102/103 kcontrols appear after modprobe; topology file size increased from 22001 to 22829 bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multi-slot WOV arbiter with VAD gate WOVCRO/HPRO clock scaling and D0I3 support.