perf(mtp): reduce vocab communication with fixed candidates - #1534
Open
sufubao wants to merge 2 commits into
Open
perf(mtp): reduce vocab communication with fixed candidates#1534sufubao wants to merge 2 commits into
sufubao wants to merge 2 commits into
Conversation
sufubao
force-pushed
the
exact-vocab-parallel-greedy
branch
from
September 11, 2026 07:51
297bbf4 to
0162c60
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
用小规模候选 all-gather 替代完整词表通信,重点优化连续 MTP draft。输出布局固定,通信次数和 CUDA Graph 变体数量不增加。
新增
--vocab_parallel_sampling:offdraft(默认)both动态 MTP 保留全词表归一化的 top-1 概率。支持 Llama 标准输出路径;特殊 draft 输出头回退,
both遇到不支持的 target 输出头报错。both的 top-p 和 logprob 基于候选集归一化;不兼容的 penalty、约束和完整词表输出需求在入口拒绝,详细限制见中英文参数文档。验证:TP=1/2/4 数值与 Graph 检查、相关采样/入口回归及格式检查通过。性能实验均经
exp记录;服务收益依赖负载,动态 MTP 单请求仍有回退,target 质量仅完成小规模冒烟,尚未确认生产收益。