Skip to content

[Klaud Cold] kimik2.5-fp4-b300-vllm: nightly image, TP/DEP/TEP sweep / Kimi K2.5 NVFP4 B300 nightly 镜像，TP/DEP/TEP 扫描 #2513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
functionstackx merged 4 commits into from
Aug 7, 2026
+42 −7
Merged

[Klaud Cold] kimik2.5-fp4-b300-vllm: nightly image, TP/DEP/TEP sweep / Kimi K2.5 NVFP4 B300 nightly 镜像，TP/DEP/TEP 扫描 #2513

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f8a7c64
kimik2.5 fp4 b300: nightly image, TP/DEP/TEP sweep
xinli-sw Aug 7, 2026
3b6a21f
restore kimik2.5 fp4 MODEL_PATH routing to /data/models
xinli-sw Aug 7, 2026
7355453
Merge origin/main; fix changelog pr-link 2442 -> 2513 / 合并 origin/mai…
Ankur-singh Aug 7, 2026
a9b7787
Merge remote-tracking branch 'origin/main' into pr-2513-reuse-30295
functionstackx Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b300.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,21 @@ check_env_vars \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
GMU=0.90
PREFILL_SCHEDULE_ARGS=()
if [ "${DP_ATTENTION:-false}" = "true" ]; then
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
GMU=0.85
PREFILL_SCHEDULE_ARGS=(--prefill-schedule-interval 4)
fi

EP_ARGS=()
if [ "${EP_SIZE:-1}" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
fi


# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
# Either way, MODEL_PATH is what the server is launched with.
Expand All @@ -37,6 +52,9 @@ nvidia-smi

export TORCH_CUDA_ARCH_LIST="10.0"
export PYTHONNOUSERSITE=1
export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=""
export VLLM_RPC_TIMEOUT=600000

SERVER_LOG=/workspace/server.log

Expand All @@ -49,13 +67,20 @@ start_gpu_monitor

set -x
vllm serve $MODEL_PATH --served-model-name $MODEL --host 0.0.0.0 --port $PORT \
--tensor-parallel-size $TP \
--gpu-memory-utilization 0.90 \
"${PARALLEL_ARGS[@]}" \
"${EP_ARGS[@]}" \
"${PREFILL_SCHEDULE_ARGS[@]}" \
--gpu-memory-utilization "$GMU" \
--max-model-len $MAX_MODEL_LEN \
--max-num-seqs $CONC \
--reasoning-parser kimi_k2 \
--tool-call-parser kimi_k2 \
--compilation_config.pass_config.fuse_allreduce_rms true \
--kv-cache-dtype fp8 \
--max-cudagraph-capture-size "$((CONC * 2))" \
--stream-interval 32 \
--attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}' \
--linear-backend flashinfer_cutlass \
--no-enable-prefix-caching \
--trust-remote-code > $SERVER_LOG 2>&1 &

Comment on lines 67 to 86

@claude claude Bot Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The header comment (lines 3-5) still says this script 'reuses the existing Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available,' but this PR adds substantial B300-specific tuning (DP-attention/EP parallel arms, --attention-config, --linear-backend flashinfer_cutlass, CONC-scaled cudagraph capture, new VLLM env vars) that has no counterpart in kimik2.5_fp4_b200.sh. Please update or remove the comment (and the matching NOTE above the kimik2.5-fp4-b300-vllm entry in configs/nvidia-master.yaml) so future readers don'''t assume this is an unmodified B200 clone.

Extended reasoning...

The top-of-file comment in benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_b300.sh (lines 3-5) reads:

# NOTE: At the time of submission, https://docs.vllm.ai/projects/recipes/en/latest/moonshotai/Kimi-K2.5.html
# does not have a B300-specific recipe, so this script reuses the existing
# Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available.

That comment predates this PR and was accurate when the B300 script really was a byte-for-byte copy of the B200 recipe. This PR is precisely the 'B300-specific tuning' the comment says is still pending, so its own premise is now false.

Diff against kimik2.5_fp4_b200.sh: I compared the two scripts directly. The B200 script has a fixed --tensor-parallel-size $TP, no DP-attention/EP arm selection, no --attention-config, no --linear-backend, no --prefill-schedule-interval, a fixed --max-cudagraph-capture-size 2048, --stream-interval 20, and none of VLLM_USE_V2_MODEL_RUNNER, VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS, or VLLM_RPC_TIMEOUT. This PR adds all of the following to the B300 script that have no B200 counterpart:

  • PARALLEL_ARGS/GMU/PREFILL_SCHEDULE_ARGS logic that switches between plain TP and DP-attention (TP=1, DP=$TP) with a lower GMU (0.85) and --prefill-schedule-interval 4 (lines 19-25)
  • EP_ARGS toggling --enable-expert-parallel when EP_SIZE > 1 (lines 27-30)
  • Three new env vars: VLLM_USE_V2_MODEL_RUNNER=0, VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS="", VLLM_RPC_TIMEOUT=600000 (lines 55-57)
  • --kv-cache-dtype fp8, --max-cudagraph-capture-size $((CONC * 2)), --stream-interval 32, --attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}', --linear-backend flashinfer_cutlass (lines 79-83)

Step-by-step proof:

  1. Open kimik2.5_fp4_b300.sh and read lines 3-5: a maintainer unfamiliar with the file's history would conclude the launch command is identical to kimik2.5_fp4_b200.sh.
  2. Open kimik2.5_fp4_b200.sh and grep for attention-config, linear-backend, PARALLEL_ARGS, EP_ARGS, VLLM_RPC_TIMEOUT — none exist.
  3. Diff the two vllm serve invocations: the B300 one has 5+ flags/env-vars absent from B200, plus different values for --max-cudagraph-capture-size and --stream-interval.
  4. Therefore the comment's claim ('reuses ... as-is') is now contradicted by the very file it sits in, and the qualifier 'until B300-specific tuning is available' describes work this PR just completed.

The identical stale wording also appears in the NOTE directly above the kimik2.5-fp4-b300-vllm block in configs/nvidia-master.yaml (around line 1436), so both spots should be updated together.

Impact: none functionally — the sweep script runs correctly regardless of the comment's wording. This is purely a documentation-accuracy issue: a future engineer reading the comment could be misled into thinking any B200-recipe change should be mirrored here without modification, or that this script has no independent tuning worth preserving/reviewing on its own. Fix is trivial: replace the comment with something like 'B300-specific tuning: DP-attention/EP parallel-arm selection, attention-config/linear-backend overrides, and cudagraph/env-var tuning added below' or simply delete the now-inaccurate 'as-is' framing.

Expand Down
10 changes: 6 additions & 4 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1529,7 +1529,7 @@ kimik2.5-fp4-b200-vllm:
# does not have a B300-specific recipe, so this config reuses the existing
# Kimi-K2.5 FP4 B200 vLLM recipe as-is until B300-specific tuning is available.
kimik2.5-fp4-b300-vllm:
image: vllm/vllm-openai:v0.22.0
image: vllm/vllm-openai:nightly-e2fa28594f7baad142a426b0b6a2cfe2c79201c7
model: nvidia/Kimi-K2.5-NVFP4
model-prefix: kimik2.5
runner: b300
Expand All @@ -1541,9 +1541,11 @@ kimik2.5-fp4-b300-vllm:
- isl: 8192
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 1, conc-end: 4 }
- { tp: 4, ep: 1, conc-start: 1, conc-end: 128 }

- { tp: 8, ep: 1, conc-list: [1] }
- { tp: 4, ep: 1, conc-start: 1, conc-end: 512 }
- { tp: 8, ep: 8, dp-attn: false, conc-list: [1] }
- { tp: 4, ep: 4, dp-attn: false, conc-start: 1, conc-end: 512 }
- { tp: 4, ep: 4, dp-attn: true, conc-start: 128, conc-end: 512 }
Comment on lines 1543 to +1548

@claude claude Bot Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The PR description says a DEP8 arm was added, but the only dp-attn: true entry in the new search-space is { tp: 4, ep: 4, dp-attn: true, ... }, which per this repo's convention (tp = data-parallel size when dp-attn:true, e.g. dsv4-fp4-b200-vllm's tp:8/ep:8 DEP8 arm) is a DEP4 arm, not DEP8. No tp:8/ep:8/dp-attn:true entry exists anywhere in the diff, so the stated sweep coverage (TP8, TP4, TEP8, TEP4, DEP8) doesn't match the 5 arms actually added — worth a quick confirmation on whether the description is stale or a DEP8 arm was meant to be included.

Extended reasoning...

What's mismatched: The PR description states the sweep adds "TP8 (conc-1 only), TP4, TEP8, TEP4, and DEP8 arms," but the actual search-space diff at configs/nvidia-master.yaml:1450-1455 only contains one dp-attn: true entry:

- { tp: 8, ep: 1, conc-list: [1] }                              # TP8
- { tp: 4, ep: 1, conc-start: 1, conc-end: 512 }                 # TP4
- { tp: 8, ep: 8, dp-attn: false, conc-list: [1] }               # TEP8
- { tp: 4, ep: 4, dp-attn: false, conc-start: 1, conc-end: 512 } # TEP4
- { tp: 4, ep: 4, dp-attn: true, conc-start: 128, conc-end: 512 }# labeled DEP8 in the PR body, but tp:4/ep:4

Why this is DEP4, not DEP8: In this repo's convention, when dp-attn: true the benchmark script sets the tensor-parallel size to 1 and the data-parallel size to the tp config value (confirmed in the modified kimik2.5_fp4_b300.sh: PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP") under the DP_ATTENTION=true branch). So the "DEPn" naming refers to that data-parallel degree, i.e. the tp field value. This is exactly how dsv4-fp4-b200-vllm names its own DEP8 arm at configs/nvidia-master.yaml:946: { tp: 8, ep: 8, dp-attn: true, ... }. Applying the same rule here, { tp: 4, ep: 4, dp-attn: true } is a DEP4 arm (4-way data-parallel-attention + EP4), not DEP8. There is no { tp: 8, ep: 8, dp-attn: true } entry anywhere in this diff.

Step-by-step proof:

  1. Reference case: dsv4-fp4-b200-vllm has { tp: 8, ep: 8, dp-attn: true } and is documented/named as its DEP8 arm → convention is DEPn where n = tp (used as --data-parallel-size) when dp-attn: true.
  2. This PR's new arms map to the description's list as: arm1 tp:8,ep:1 = TP8, arm2 tp:4,ep:1 = TP4, arm3 tp:8,ep:8,dp-attn:false = TEP8, arm4 tp:4,ep:4,dp-attn:false = TEP4, arm5 tp:4,ep:4,dp-attn:true = (per convention) DEP4.
  3. Applying the same naming rule from step 1 to arm5 yields DEP4, contradicting the PR description's "DEP8" label.
  4. Grepping the full diff for tp: 8 combined with ep: 8 and dp-attn: true on the same line returns zero matches — confirming no DEP8 arm was actually added.

Why nothing prevents this / why it isn't blocking: All 5 arms are valid, well-formed configs and will run correctly regardless of their label — the mismatch is purely between the PR's free-text description and the committed YAML, not a functional defect. The committed perf-changelog.yaml entry for this PR is generic ("TP/DEP/TEP sweep, TP8 conc-1 only, DEP gmu 0.85") and does not claim DEP8 specifically, so no committed artifact makes the incorrect claim — only the ephemeral PR body does. It's also plausible this is intentional: the new DEP4 arm exactly mirrors the new TEP4 arm (tp:4, ep:4) at the same GPU count, giving a clean apples-to-apples dp-attn on/off comparison at 4 GPUs, rather than an accidental omission of a true DEP8 arm.

Suggested fix: Reconcile the PR description with the config — either correct "DEP8" to "DEP4" in the description, or, if full-node (8-GPU) attention-DP throughput data was actually intended, add a { tp: 8, ep: 8, dp-attn: true, ... } arm to the search-space.

dsr1-fp8-b200-sglang-mtp:
image: lmsysorg/sglang:v0.5.12-cu130
model: deepseek-ai/DeepSeek-R1-0528
Expand Down
6 changes: 6 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5586,3 +5586,9 @@
description:
- "Kimi K2.5 NVFP4 B200 vLLM: nightly image, TP/DEP/TEP sweep, DEP gmu 0.80"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2512

- config-keys:
- kimik2.5-fp4-b300-vllm
description:
- "Kimi K2.5 NVFP4 B300 vLLM: nightly image, TP/DEP/TEP sweep, TP8 conc-1 only, DEP gmu 0.85"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2513
4 changes: 3 additions & 1 deletion runners/launch_b300-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,9 @@ else
# MODEL stays as the HF id for the client (--served-model-name, tokenizer);
# MODEL_PATH is what the server reads weights from.
MODEL_BASENAME="${MODEL##*/}"
if [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
if [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then
export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}"
elif [[ " ${STAGED_MODELS[*]} " == *" ${MODEL_BASENAME} "* ]]; then
export MODEL_PATH="${HF_HUB_CACHE_MOUNT%/}/${MODEL_BASENAME}"
else
export MODEL_PATH="${WRITABLE_MODELS_DIR%/}/${MODEL_BASENAME}"
Expand Down