Openvino whisper support - #21556
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21556
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ❌ 1 New Failure, 1 Cancelled JobAs of commit a83d689 with merge base e8feb9e ( NEW FAILURE - The following job has failed:
CANCELLED JOB - The following job was cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds an OpenVINO-compatible Whisper example by exporting Whisper into three ExecuTorch programs (encoder / cross-KV / decoder-with-cache) and providing scripts + docs to export and run inference.
Changes:
- Introduces Whisper model split modules (encoder, cross-attention KV projection, decoder with self-attention cache).
- Adds export and runtime scripts to generate and run
encoder.pte,cross_kv.pte,decoder.pteplusmetadata.json. - Adds example README and Python dependencies for the Whisper OpenVINO example.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/openvino/whisper/whisper_model.py | Implements the 3-way Whisper split and cached attention modules for export. |
| examples/openvino/whisper/export_whisper.py | Exports and lowers the split modules to the OpenVINO backend and writes metadata. |
| examples/openvino/whisper/run_whisper.py | Loads the exported programs and runs end-to-end greedy decoding. |
| examples/openvino/whisper/requirements.txt | Adds Python dependencies needed to export/run the example. |
| examples/openvino/whisper/README.md | Documents setup, export, and inference steps for the example. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| features = processor( | ||
| audio_array, | ||
| sampling_rate=sampling_rate, | ||
| return_tensors="pt", | ||
| ).input_features | ||
| max_frames = 3000 | ||
| if features.shape[2] > max_frames: | ||
| features = features[:, :, :max_frames].contiguous() | ||
| return features |
| def __init__(self, max_context_length, n_heads, head_dim, dtype): | ||
| super().__init__() | ||
| self.max_context_length = max_context_length | ||
| shape = (1, n_heads, max_context_length, head_dim) | ||
| self.register_buffer("k_cache", torch.zeros(shape, dtype=dtype)) | ||
| self.register_buffer("v_cache", torch.zeros(shape, dtype=dtype)) | ||
|
|
||
| def update(self, cache_position, k_val, v_val): | ||
| # cache_position: [S] (long), k_val/v_val: [B, H, S, D]. | ||
| self.k_cache.index_copy_(2, cache_position, k_val) | ||
| self.v_cache.index_copy_(2, cache_position, v_val) | ||
| return self.k_cache, self.v_cache |
| # Additive mask: 0 for valid positions [0..step], -inf for future positions. | ||
| base_mask = torch.full((1, 1, 1, max_cache), float("-inf")) | ||
| tokens = [start_token] | ||
| decode_start = time.perf_counter() | ||
| for step in range(args.max_new_tokens): | ||
| ids = torch.tensor([[tokens[-1]]], dtype=torch.long) | ||
| pos = torch.tensor([step], dtype=torch.long) | ||
| mask = base_mask.clone() | ||
| mask[..., : step + 1] = 0.0 | ||
| logits = dec_method.execute([ids, pos, mask, *cross_k, *cross_v])[0] |
| for i, (k_proj, v_proj) in enumerate(zip(self.k_projs, self.v_projs)): | ||
| H, D = self.num_heads_list[i], self.head_dim_list[i] | ||
| k = k_proj(encoder_hidden_states).view(B, T_enc, H, D).transpose(1, 2) | ||
| v = v_proj(encoder_hidden_states).view(B, T_enc, H, D).transpose(1, 2) | ||
| k_list.append(k.contiguous()) | ||
| v_list.append(v.contiguous()) | ||
| return tuple(k_list), tuple(v_list) |
| transformers | ||
| soundfile | ||
| librosa | ||
| datasets |
This PR needs a
|
Summary
Enabled whisper support for Executorch OV Backend
Test plan
By exporting and running inference from the README.md file.