Skip to content

perf(mtp): reduce vocab communication with fixed candidates - #1534

Open
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:exact-vocab-parallel-greedy
Open

perf(mtp): reduce vocab communication with fixed candidates#1534
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:exact-vocab-parallel-greedy

Conversation

@sufubao

@sufubao sufubao commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

用小规模候选 all-gather 替代完整词表通信,重点优化连续 MTP draft。输出布局固定,通信次数和 CUDA Graph 变体数量不增加。

新增 --vocab_parallel_sampling

模式 Draft Target
off 原路径 原路径
draft(默认) 精确 top-1 原路径
both 精确 top-1 全局 top-128 近似采样

动态 MTP 保留全词表归一化的 top-1 概率。支持 Llama 标准输出路径;特殊 draft 输出头回退,both 遇到不支持的 target 输出头报错。

both 的 top-p 和 logprob 基于候选集归一化;不兼容的 penalty、约束和完整词表输出需求在入口拒绝,详细限制见中英文参数文档。

验证:TP=1/2/4 数值与 Graph 检查、相关采样/入口回归及格式检查通过。性能实验均经 exp 记录;服务收益依赖负载,动态 MTP 单请求仍有回退,target 质量仅完成小规模冒烟,尚未确认生产收益。

@sufubao sufubao changed the title perf: avoid full-vocab all-gather for draft greedy sampling perf: share vocab-parallel top-k logits for target and draft Sep 2, 2026
@sufubao sufubao closed this Sep 10, 2026
@sufubao sufubao changed the title perf: share vocab-parallel top-k logits for target and draft perf(mtp): reduce vocab communication with fixed candidates Sep 11, 2026
@sufubao sufubao reopened this Sep 11, 2026
@sufubao
sufubao force-pushed the exact-vocab-parallel-greedy branch from 297bbf4 to 0162c60 Compare September 11, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant