Skip to content

chore(config): bump DSR1 FP4 B200 SGLang to v0.5.16 / 将 DSR1 FP4 B200 SGLang 更新至 v0.5.16 - #2492

Open
RohitNagraj wants to merge 3 commits into
mainfrom
dsr1-fp4-b200-sglang-v0.5.16
Open

chore(config): bump DSR1 FP4 B200 SGLang to v0.5.16 / 将 DSR1 FP4 B200 SGLang 更新至 v0.5.16#2492
RohitNagraj wants to merge 3 commits into
mainfrom
dsr1-fp4-b200-sglang-v0.5.16

Conversation

@RohitNagraj

Copy link
Copy Markdown
Collaborator

Summary

中文说明

Update the DSR1 FP4 B200 aggregated STP and MTP configurations to lmsysorg/sglang:v0.5.16-cu130.

中文:将 DSR1 FP4 B200 聚合式 STP 和 MTP 配置更新至 lmsysorg/sglang:v0.5.16-cu130。
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

1 similar comment
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

Record the pull-request URL for the DSR1 FP4 B200 SGLang image update.

中文:记录 DSR1 FP4 B200 SGLang 镜像更新的拉取请求链接。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — straightforward version bump PR with one cosmetic nit flagged inline (unresolved XXX placeholder in the pr-link).

Extended reasoning...

Overview

This PR bumps the SGLang image tag for the dsr1-fp4-b200-sglang and dsr1-fp4-b200-sglang-mtp configs from v0.5.12.post1 to v0.5.16-cu130 in configs/nvidia-master.yaml, and adds a corresponding entry to perf-changelog.yaml. No benchmark parameters, concurrency lists, TP/EP settings, or other logic changed — only the container image string on two config keys.

Security risks

None. This only changes a pinned Docker image tag string and appends a changelog entry; no code execution paths, credentials, or permissions are touched.

Level of scrutiny

Low. This is a mechanical dependency/version bump on config-only files, following the exact same pattern as prior image-bump PRs in this repo (e.g. #2468, #2446). The kind of thing that would normally need scrutiny is whether the new image tag actually works with existing config knobs, which is validated by the CI benchmark sweep rather than code review.

Other factors

I verified directly against the git blob at HEAD that the perf-changelog.yaml addition still has pull/XXX instead of pull/2492" as its pr-link — this is a real, confirmed nit and is already flagged as an inline comment. It's cosmetic (doesn't affect nvidia-master.yaml` or benchmark execution) and easy to fix, so it doesn't rise to the level of blocking approval on an otherwise simple, low-risk version bump.

Comment thread perf-changelog.yaml
description:
- "Update the aggregated DSR1 FP4 B200 SGLang STP and MTP configurations to v0.5.16 with CUDA 13"
- "Image: lmsysorg/sglang:v0.5.16-cu130"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2492

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The new perf-changelog.yaml entry for dsr1-fp4-b200-sglang / dsr1-fp4-b200-sglang-mtp has pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX — the placeholder was never replaced with the real PR number (2492). Every other entry in the file (2462, 2446, 2468, 2470) uses the actual PR number, so this one is a dead link that breaks the changelog's traceability convention. Easy one-token fix: change XXX to 2492.

Extended reasoning...

The bug: Line 5441 of perf-changelog.yaml, added by this PR, sets:

pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX

instead of the real PR number, which is 2492 (this PR). I verified this directly against git ground truth rather than relying on the rendered diff text:

$ git show HEAD:perf-changelog.yaml | sed -n 5441p
  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
$ git log -1 --format='%H %s'
94e403d65edcf8a1f66cafc493d20eecc460ba82 chore(config): bump DSR1 FP4 B200 SGLang to v0.5.16

HEAD is this PR's own merge commit, and both the committed git object (git show HEAD:...) and the working-tree file agree the placeholder XXX was left in, not substituted with 2492.

Addressing the refutation: One reviewer argued this is a false positive because the PR-diff text embedded in the review context shows pull/2492, and that the working tree might be at a 'divergent state' per the general caching caveat. I re-checked this explicitly: git show HEAD:perf-changelog.yaml reads the actual blob recorded in the commit object for HEAD — there is no environment drift possible here, since HEAD is the commit this PR represents (subject line 'chore(config): bump DSR1 FP4 B200 SGLang to v0.5.16' matches the PR title exactly, and git log shows no commits after it). So the diff-rendering tool that displays '2492' must be doing some display-time substitution/templating that doesn't reflect the literal bytes that were committed. The authoritative source — the actual git blob that lands in main — contains XXX, not 2492.

Why nothing else catches this: perf-changelog.yaml is a human/tooling-readable changelog, not consumed by the benchmark runner logic, so there's no schema validation, linter, or CI check that would flag a malformed or placeholder PR URL. It's easy for a placeholder like XXX (likely left in a template snippet while drafting the entry before the PR number was known) to slip through untouched.

Impact: Purely cosmetic/documentation — it does not affect configs/nvidia-master.yaml or any benchmark execution, since perf-changelog.yaml is pure metadata. But it does break the file's established traceability convention: every other entry (pull/2462, pull/2446, pull/2468, pull/2470) links to its real PR, so a reader or tool following this pr-link would hit a 404/dead link instead of the actual PR.

Fix: Change line 5441 from pull/XXX to pull/2492, matching the convention used by every other entry in the file.

Preserve the current main changelog and append the PR #2492 entry at the end.

中文:保留当前 main 的完整变更日志,并将 PR #2492 条目追加至文件末尾。
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

@Ankur-singh

Copy link
Copy Markdown
Collaborator

/reuse-sweep-run 30952773184

@Ankur-singh

Copy link
Copy Markdown
Collaborator

As a PR reviewer and CODEOWNER, I have reviewed this and have:

  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. Run Sweep 30952773184
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. Run Sweep 30952773184
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

  • Head reviewed: 5430f5cf8d0a6782b025ed1be08d27383051a522.
  • Scope: this PR changes two lines of configs/nvidia-master.yaml — the pinned SGLang image for dsr1-fp4-b200-sglang and dsr1-fp4-b200-sglang-mtp, from lmsysorg/sglang:v0.5.12.post1 to lmsysorg/sglang:v0.5.16-cu130 — plus a purely additive perf-changelog.yaml entry. Model, runner, topology, concurrency lists and speculative-decoding settings are unchanged.
  • Validation (item 3): run 30952773184 is a sweep pinned to this exact head and concluded success. All 17 single-node 8k1k / lanes executed non-skipped and succeeded across both entries (STP TP4 at c2-c32, TP4/EP4/DPA at c64/c128/c256, and the matching MTP lanes), as did the canary / lane (TP4 c1), collect-results, compare-results, and calc-success-rate.
  • Evals (item 4): the same run's 6 eval lanes and collect-evals all executed and succeeded. agg_eval_all.json has 6 GSM8K rows for the DeepSeek-R1-0528-NVFP4-v2 checkpoint, em_strict 0.9492-0.9606, against the dsr1 override bar of 0.91 in utils/evals/thresholds.yaml.
  • Items 11 / 11a (recipe documentation) — satisfied via the already-published page. This is a single-node SGLang submission (multinode: false, framework: sglang, runner: b200), so the requirement applies. The matching recipe is already published at the SGLang DeepSeek-R1 cookbook, which the checklist accepts in place of an upstream PR link, and which the PR description also links. I fetched that page and confirmed it is the live DeepSeek-R1-0528 recipe. Reviewer judgment: the page documents the serving recipe without pinning an lmsysorg/sglang: image tag, so bumping the container version does not diverge from the published recipe; the model, tensor-parallel layout and speculative-decoding arguments this PR leaves untouched are what that page specifies.
  • Item 5 (spec-decode chat templates) — verified: benchmarks/single_node/fixed_seq_len/dsr1_fp4_b200_mtp.sh passes --use-chat-template to run_benchmark_serving, so the EAGLE/MTP acceptance-length distribution is measured on chat-formatted prompts rather than random tokens.
  • Item 6 (agentic golden AL) — not applicable: both entries declare only the fixed-seq-len 8k1k scenario; the agentic-coding block is commented out in the config, and no simulated-acceptance knob appears in the diff.
  • Item 7 (MODELS.md) — verified: MODELS.md lists DeepSeek-R1-0528 (dsr1) with active scenario "Single-turn 8k1k" and deprecates only Single-turn 1k1k and 1k8k. This submission is Single-turn 8k1k, which remains active.
  • Item 8 (architecture): no --hf-overrides, no model-config edits, no change to any FLOPs-affecting knob — the diff is a container tag change only.
  • Items 9 / 10 (upstream image and engine-first ordering): lmsysorg/sglang:v0.5.16-cu130 comes from the upstream lmsysorg Docker org, and this entry is itself the SGLang submission for DSR1 FP4 on B200; no additional framework is introduced ahead of it.
  • Item 2 (code quality): the two config lines and the changelog entry are consistent with each other, and the perf-changelog.yaml hunk is purely additive with no deletions and no whitespace changes.
  • CODEOWNER scope: evaluated both changed paths against the live .github/CODEOWNERS with last-match-wins. configs/nvidia-master.yaml is the only specifically owned path and lists @ankur-singh; perf-changelog.yaml matches only the catch-all * @InferenceX/core.
  • Operational: the PR is CONFLICTING with main and needs a rebase before merge; that will move the head and require a fresh sign-off.

Signed: Ankur-singh

@Klaud-Cold

Copy link
Copy Markdown
Collaborator

✅✅✅ Verdict: PASS ✅✅✅

✅ Check 0 (CODEOWNER): PASS — @ankur-singh is a listed owner of configs/nvidia-master.yaml; perf-changelog.yaml matches only the catch-all, which a recognized CODEOWNER satisfies.
✅ Check 1 (passing sweep on in-PR commit): PASS — head 5430f5c carries run 30952773184 with 17 single-node 8k1k / lanes + canary and 6 eval / lanes, all success and executed (non-skipped).
✅ Check 2 (evals pass): PASS — agg_eval_all.json from that run has 6 GSM8K rows for DeepSeek-R1-0528-NVFP4-v2, em_strict 0.9492–0.9606 vs the dsr1 bar of 0.91, and every lane ran lmsysorg/sglang:v0.5.16-cu130 — the exact image this PR pins.
✅ Check 3 (recipe merged & complete): PASS — single-node SGLang submission covered by the published SGLang DeepSeek-R1 cookbook (live page, no image tag pinned, so the version bump does not diverge). Major args match: same model nvidia/DeepSeek-R1-0528-FP4-v2, B200 FP4 TP4/EP4 config, fp8_e4m3 KV cache, DP-attention documented for high throughput, and EAGLE/MTP documented (via the referenced DeepSeek-V3 §4.2.3). Informational only: InferenceX harness tuning (--scheduler-recv-interval, --chunked-prefill-size, --disable-piecewise-cuda-graph, SGLANG_RADIX_FORCE_MISS, prefill-delayer) and explicit trtllm_mla/flashinfer_trtllm backend pins that match Blackwell defaults.
✅ Check 4 (reuse command): PASS — /reuse-sweep-run 30952773184 posted by Ankur-singh (COLLABORATOR).
✅ Check 5 (latest checklist): PASS — all current-template items present; the agentic golden-AL item is unchecked with an explicit N/A explanation in the detail section.
✅ Check 6 (upstream image / engine-first): PASS — lmsysorg/sglang:v0.5.16-cu130 is the upstream SGLang org image, and both entries are themselves the SGLang submissions for dsr1 FP4 on B200.
✅ Check 7 (deprecations): PASS — dsr1 Single-turn 8k1k is active per MODELS.md as of 2026-08-05; only 1k1k/1k8k are deprecated for it.
✅ Check 8 (architecture hacks): PASS — the diff is two image-tag lines plus a changelog entry; no --hf-overrides or FLOPs-affecting change.
✅ Check 9 (spec-decode chat template): PASS — benchmarks/single_node/fixed_seq_len/dsr1_fp4_b200_mtp.sh benchmarks with --use-chat-template.
✅ Check 10 (engine patches): PASS — no patching introduced; the pinned image runs as shipped.
➖ Check 11 (agentic golden AL): N/A — no agentic spec-decode changes; the agentic-coding scenario is commented out and no simulated-acceptance knobs appear.

Note: this verdict is anchored to head 5430f5c. The PR is currently conflicting with main; a rebase will move the head and require a fresh sign-off and re-verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants