From ac3e582abbdc03d46c679180a8b7e77084ebcabe Mon Sep 17 00:00:00 2001 From: Zhiyu Cheng Date: Wed, 5 Aug 2026 14:25:22 -0700 Subject: [PATCH 1/4] docs: rebuild the unified HF deployment support matrix from the deploy test suite (NVBug 6550792) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The deployment support matrix listed 9 model families and no VLMs, while tests/examples/hf_ptq/test_deploy.py already deploy-tests 81 checkpoints across TRT-LLM, vLLM, and SGLang — including Qwen2.5-VL, Qwen3-VL, Phi-4-multimodal, and Nemotron Omni. QA could not use the doc to scope testing, and users could not tell what is actually supported. The matrix also existed in three places that had drifted apart: only the .rst listed Qwen3-VL, only the README listed Qwen3.5 MoE, and the agent skill reference had neither. - Rebuild the matrix in docs/source/deployment/3_unified_hf.rst from test_deploy.py, split into language / vision-language / speculative decoding / diffusion tables, with a three-state legend so a cell distinguishes "validated in CI" from "expected but untested". - Replace the duplicate tables in examples/hf_ptq/README.md and the deployment skill reference with links, leaving one copy to maintain. - Fix stale prose in the deployment tabs that still claimed FP8-only support on vLLM v0.6.5 and a source build of SGLang main from Jan 2025. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Zhiyu Cheng --- docs/source/deployment/3_unified_hf.rst | 162 ++++++++++++++---- examples/hf_ptq/README.md | 35 ++-- .../deployment/references/support-matrix.md | 35 ++-- 3 files changed, 150 insertions(+), 82 deletions(-) diff --git a/docs/source/deployment/3_unified_hf.rst b/docs/source/deployment/3_unified_hf.rst index 59a2782c5e7..64e9dfa34f1 100644 --- a/docs/source/deployment/3_unified_hf.rst +++ b/docs/source/deployment/3_unified_hf.rst @@ -51,48 +51,132 @@ The unified HF export API supports the following quantization formats: 5. INT4_AWQ - 4-bit integer with AWQ optimization 6. W4A8_AWQ - 4-bit weights and 8-bit activations with AWQ optimization -Framework-Specific Support +Minimum Framework Versions -------------------------- -TensorRT-LLM -~~~~~~~~~~~~ +=============== ================= +Framework Minimum version +=============== ================= +TensorRT-LLM v0.17.0 +vLLM v0.10.1 +SGLang v0.4.10 +=============== ================= + +.. _unified-hf-support-matrix: + +Model Support Matrix +-------------------- + +Legend: + +* ``Y`` — covered by the release deployment test suite + (`tests/examples/hf_ptq/test_deploy.py `_), + which loads the exported checkpoint in the framework and runs generation. +* ``~`` — documented as working previously but not in the current test suite; expected to work, unvalidated. +* ``-`` — not currently covered. It may still work; see `Models not listed here`_. + +Language models +~~~~~~~~~~~~~~~ + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Llama 3.1, 3.3 FP8, NVFP4 Y Y Y +Llama 4 Scout, Maverick FP8 Y Y Y +Llama 4 Scout NVFP4 Y Y Y +Llama Nemotron Super 49B v1, v1.5 FP8 Y Y Y +Llama Nemotron Ultra 253B v1 FP8 Y Y Y +Nemotron 3 Nano 30B-A3B FP8, NVFP4 Y Y Y +Nemotron 3 Super 120B-A12B FP8, NVFP4 Y Y Y +Nemotron 3 Ultra 550B-A55B NVFP4 Y Y Y +DeepSeek R1, R1-0528 NVFP4 Y Y Y +DeepSeek V3, V3.1, V3.2 NVFP4 Y Y Y +DeepSeek V4 Flash NVFP4 Y Y Y +DeepSeek V4 Pro NVFP4 \- Y Y +Qwen 3 (8B, 14B, 32B) FP8, NVFP4 Y Y Y +Qwen 3 MoE 235B-A22B FP8, NVFP4 Y Y Y +Qwen 3 MoE 30B-A3B NVFP4 Y Y Y +Qwen 3 Coder 480B-A35B NVFP4 Y Y Y +Qwen 3-Next 80B-A3B NVFP4 Y Y Y +Qwen 3.5 397B-A17B NVFP4 Y Y Y +Qwen 3.5 122B-A10B, Qwen 3.6 35B-A3B NVFP4 \- Y \- +Qwen 2.5 FP8 ~ ~ ~ +Qwen 2.5 NVFP4 ~ ~ \- +QwQ-32B FP8 ~ ~ ~ +QwQ-32B NVFP4 ~ ~ \- +Phi-4 reasoning-plus FP8, NVFP4 Y Y Y +Gemma 4 31B NVFP4 Y Y Y +Gemma 4 26B-A4B NVFP4 \- Y \- +GLM-4.7, GLM-5, GLM-5.2 NVFP4 Y Y Y +GLM-5.1 NVFP4 \- Y Y +Kimi K2-Thinking, K2.5 NVFP4 Y Y Y +Kimi K2.6 NVFP4 \- Y \- +MiniMax M2.5, M3 NVFP4 Y Y Y +Mixtral 8x7B FP8 ~ ~ ~ +Mixtral 8x7B NVFP4 ~ \- \- +============================================ ============== ============ ====== ======== + +Vision-language and multimodal models +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For VLMs, modelopt quantizes the language model only; the vision encoder is kept in high precision. +The exported checkpoint therefore relies on the serving framework's own multimodal support for that +architecture — see the +`TensorRT-LLM multimodal support matrix `_. + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Qwen 2.5-VL 7B FP8, NVFP4 Y Y Y +Qwen 3-VL 235B-A22B NVFP4 Y Y Y +Phi-4-multimodal FP8, NVFP4 Y Y Y +Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 Y Y Y +============================================ ============== ============ ====== ======== + +Speculative decoding drafters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Drafters are deployed on top of their base checkpoint. vLLM is not currently covered for these. + +============================================================ ============ ============ ====== ======== +Drafter Quant format TensorRT-LLM vLLM SGLang +============================================================ ============ ============ ====== ======== +EAGLE3 for Llama 3.3 70B, Llama 4 Maverick FP8 Y \- Y +EAGLE3 for Qwen 3 235B-A22B (incl. Thinking-2507, FP4) BF16, NVFP4 Y \- Y +EAGLE3 for Qwen 3 30B-A3B-Thinking-2507 BF16 Y \- Y +EAGLE3 for Kimi K2-Thinking, K2.5, K2.6 NVFP4 Y \- Y +EAGLE3 for gpt-oss-120b BF16 Y \- Y +Medusa for Llama 3.1 8B FP8 Y \- Y +============================================================ ============ ============ ====== ======== + +Diffusion models +~~~~~~~~~~~~~~~~ + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Wan 2.2 T2V A14B FP8, NVFP4 Y \- Y +DiffusionGemma 26B-A4B NVFP4 Y Y Y +============================================ ============== ============ ====== ======== -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Qwen 3-VL (FP8, NVFP4) - * Deepseek R1/V3 (NVFP4) - * Mixtral 8x7B (FP8, NVFP4) - * Medusa (FP8) - * Eagle (FP8) - -Requirements: TensorRT-LLM v0.17.0 or later - -vLLM -~~~~ - -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Mixtral 8x7B (FP8) - * Deepseek R1/V3 (NVFP4) - -Requirements: vLLM v0.10.1 or later - -SGLang -~~~~~~ +.. note:: + NVFP4 inference requires Blackwell GPUs. Hopper can produce an NVFP4 checkpoint but cannot serve + it. On B300/GB300 (``sm_103``) use a CUDA-13 build of the serving framework; CUDA-12 builds lack + the ``sm_103`` FP4 kernels. -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Deepseek R1/V3 (NVFP4) +Models not listed here +~~~~~~~~~~~~~~~~~~~~~~ -Requirements: SGLang v0.4.10 or later +This matrix records the combinations modelopt validates. It is not an exhaustive list of what will +run: vLLM, SGLang, and TensorRT-LLM load unified HF checkpoints generically, so a model built from +standard ``nn.Linear`` layers with an ``hf_quant_config.json`` will often deploy without any modelopt +change. Check the serving framework's own model support list first, then try it. -Note: While other models and quantization formats may work, they have not been thoroughly tested and validated. +The exact checkpoints behind every ``Y`` above, including tensor-parallel size and minimum SM +version, are listed in +`tests/examples/hf_ptq/test_deploy.py `__; +most are published under the +`NVIDIA Hugging Face organization `_. Deployment with Selected Inference Frameworks @@ -136,7 +220,8 @@ Deployment with Selected Inference Frameworks Follow `vLLM installation instructions. `_ - Currently we support fp8 quantized models (without fp8 kv cache) for vLLM deployment, you need v0.6.5 or later version of vLLM. + FP8 and NVFP4 quantized models are supported; you need v0.10.1 or later version of vLLM. Pass + ``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: @@ -171,7 +256,8 @@ Deployment with Selected Inference Frameworks Follow the `SGLang installation instructions. `_ - Currently we support fp8 quantized models (without fp8 kv cache) for SGLang deployment, you need to use the main branch of SGLang (since Jan 6, 2025) and build it from source. + FP8 and NVFP4 quantized models are supported; you need v0.4.10 or later version of SGLang. Pass + ``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: diff --git a/examples/hf_ptq/README.md b/examples/hf_ptq/README.md index dc852408c7a..2c308b6a88c 100755 --- a/examples/hf_ptq/README.md +++ b/examples/hf_ptq/README.md @@ -587,27 +587,20 @@ print(llm_fp8.generate(["What's the age of the earth? "])) ### Unified HF Checkpoint Deployment Model Support Matrix -| Model | Quant format | TRT-LLM | vLLM | SGLang | -| :---: | :---: | :---: | :---: | :---: | -| LLAMA 3.x | FP8 | ✅ | ✅ | ✅ | -| LLAMA 3.x | FP4 | ✅ | ✅ | ✅ | -| LLAMA 4 | FP8 | ✅ | - | ✅ | -| LLAMA 4 | FP4 | ✅ | - | - | -| DS-R1 | FP8 | ✅ | ✅ | ✅ | -| DS-R1 | FP4 | ✅ | ✅ | ✅ | -| DS-V3 | FP8 | ✅ | ✅ | ✅ | -| DS-V3 | FP4 | ✅ | ✅ | ✅ | -| QWen3 | FP8 | ✅ | ✅ | ✅ | -| QWen3 | FP4 | ✅ | ✅ | - | -| QWen3 MoE | FP8 | ✅ | ✅ | ✅ | -| QWen3 MoE | FP4 | ✅ | - | - | -| QWen3.5 MoE | FP4 | - | - | ✅ | -| QWen2.5 | FP8 | ✅ | ✅ | ✅ | -| QWen2.5 | FP4 | ✅ | ✅ | - | -| QwQ-32B | FP8 | ✅ | ✅ | ✅ | -| QwQ-32B | FP4 | ✅ | ✅ | - | -| Mixtral 8x7B | FP8 | ✅ | ✅ | ✅ | -| Mixtral 8x7B | FP4 | ✅ | - | - | +The deployment support matrix — which model families and quantization formats are validated on +TRT-LLM, vLLM, and SGLang, including vision-language models, speculative decoding drafters, and +diffusion models — lives in the documentation so there is a single copy to keep current: + +**[Unified HF Checkpoint → Model Support Matrix](https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf.html#model-support-matrix)** + +Every validated entry there is backed by [`tests/examples/hf_ptq/test_deploy.py`](../../tests/examples/hf_ptq/test_deploy.py), +which loads the exported checkpoint in each framework and runs generation. That file is also the +place to look for the exact checkpoint, tensor-parallel size, and minimum SM version behind each +entry. + +> *Note: the matrix records what modelopt validates, not the full set of what will run. vLLM, SGLang, +> and TRT-LLM load unified HF checkpoints generically, so unlisted models frequently deploy without +> any modelopt change — check the serving framework's own model support list and try it.* ### (Legacy) TensorRT-LLM Checkpoints diff --git a/plugins/modelopt/skills/deployment/references/support-matrix.md b/plugins/modelopt/skills/deployment/references/support-matrix.md index 265f1b58560..bbef86efe24 100644 --- a/plugins/modelopt/skills/deployment/references/support-matrix.md +++ b/plugins/modelopt/skills/deployment/references/support-matrix.md @@ -2,26 +2,14 @@ ## Unified HF Checkpoint — Framework Compatibility -| Model | Quant Format | TRT-LLM | vLLM | SGLang | -|-------|-------------|---------|------|--------| -| Llama 3.x | FP8 | yes | yes | yes | -| Llama 3.x | FP4 | yes | yes | yes | -| Llama 4 | FP8 | yes | — | yes | -| Llama 4 | FP4 | yes | — | — | -| DeepSeek R1 | FP8 | yes | yes | yes | -| DeepSeek R1 | FP4 | yes | yes | yes | -| DeepSeek V3 | FP8 | yes | yes | yes | -| DeepSeek V3 | FP4 | yes | yes | yes | -| Qwen 3 | FP8 | yes | yes | yes | -| Qwen 3 | FP4 | yes | yes | — | -| Qwen 3 MoE | FP8 | yes | yes | yes | -| Qwen 3 MoE | FP4 | yes | — | — | -| Qwen 2.5 | FP8 | yes | yes | yes | -| Qwen 2.5 | FP4 | yes | yes | — | -| QwQ-32B | FP8 | yes | yes | yes | -| QwQ-32B | FP4 | yes | yes | — | -| Mixtral 8x7B | FP8 | yes | yes | yes | -| Mixtral 8x7B | FP4 | yes | — | — | +**Do not maintain a copy of the matrix here.** The single source of truth is +`docs/source/deployment/3_unified_hf.rst` ("Model Support Matrix"), and every validated entry in it +is backed by `tests/examples/hf_ptq/test_deploy.py`. + +To answer "is model X supported on framework Y", read one of those two files — `test_deploy.py` is +the more precise answer, since it also carries the exact checkpoint, tensor-parallel size, and +minimum SM version per entry. It covers language models, VLMs (Qwen2.5-VL, Qwen3-VL, +Phi-4-multimodal, Nemotron Omni), EAGLE3/Medusa drafters, and diffusion models. ## Supported Quantization Formats @@ -50,9 +38,9 @@ | SGLang | `quantization="modelopt"` | `quantization="modelopt_fp4"` | | TRT-LLM | auto-detected from checkpoint | auto-detected from checkpoint | -## Models not in this list +## Models not in the matrix -This matrix covers officially validated combinations. For unlisted models: +The matrix covers officially validated combinations, not the full set of what will run. For unlisted models: 1. **Check the framework's own docs** — vLLM and SGLang support many HuggingFace models natively. Use WebSearch to check `vllm supported models` or `sglang supported models`. 2. **Try it** — if the model uses standard `nn.Linear` layers and has `hf_quant_config.json`, vLLM/SGLang will likely work with `--quantization modelopt`. @@ -64,4 +52,5 @@ This matrix covers officially validated combinations. For unlisted models: - **B300/GB300 are `sm_103`** and need a **CUDA-13** serving image — from v0.20.0 the unsuffixed tag is CUDA-13 (`-cu129` opts back to CUDA 12); `cu12` images lack the `sm_103` FP4 kernel and serve NVFP4 as gibberish or error out. See the CUDA-13 note in the deployment `SKILL.md`. - **Verify the GPU with `nvidia-smi`** before choosing the image — cluster GPU labels can be stale. - INT4_AWQ and W4A8_AWQ are only supported by TRT-LLM (not vLLM or SGLang). -- Source: `examples/hf_ptq/README.md` and `docs/source/deployment/3_unified_hf.rst` +- For VLMs, only the language model is quantized; the vision encoder stays in high precision, so multimodal serving depends on the framework's own support for that architecture. +- Source: `docs/source/deployment/3_unified_hf.rst` and `tests/examples/hf_ptq/test_deploy.py` From d2727e8c26768f0d7efbb0628f400d1c09c47738 Mon Sep 17 00:00:00 2001 From: Zhiyu Cheng Date: Thu, 6 Aug 2026 22:51:59 -0700 Subject: [PATCH 2/4] =?UTF-8?q?docs:=20address=20review=20=E2=80=94=20TRT-?= =?UTF-8?q?LLM=20v1.2.0=20floor,=20checkmarks,=20conditional=20EAGLE=20cov?= =?UTF-8?q?erage?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Raise the documented TensorRT-LLM minimum from v0.17.0 to v1.2.0. 0.17 is stale: CI runs 1.3.0rc20 and the deploy tests use llmapi APIs (EagleDecodingConfig, CudaGraphConfig) that do not exist in 0.17. Also fix the same stale claim in the hf_ptq and diffusers support-matrix footnotes. - Use the repo's usual checkmarks in the matrix instead of Y/~. - Note that EAGLE3 drafter coverage is conditional on MODELOPT_LOCAL_EAGLE_MODEL; those tests skip when it is unset, so the drafter table's checkmarks are weaker than the other tables'. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Zhiyu Cheng --- docs/source/deployment/3_unified_hf.rst | 106 +++++++++++++----------- examples/diffusers/README.md | 2 +- examples/hf_ptq/README.md | 2 +- 3 files changed, 59 insertions(+), 51 deletions(-) diff --git a/docs/source/deployment/3_unified_hf.rst b/docs/source/deployment/3_unified_hf.rst index 64e9dfa34f1..a564530666a 100644 --- a/docs/source/deployment/3_unified_hf.rst +++ b/docs/source/deployment/3_unified_hf.rst @@ -57,11 +57,15 @@ Minimum Framework Versions =============== ================= Framework Minimum version =============== ================= -TensorRT-LLM v0.17.0 +TensorRT-LLM v1.2.0 vLLM v0.10.1 SGLang v0.4.10 =============== ================= +These are the oldest versions expected to load a unified HF checkpoint. The matrix below is +validated against the versions CI currently runs, which are newer — see the container tags in +``.github/workflows/``. + .. _unified-hf-support-matrix: Model Support Matrix @@ -69,10 +73,10 @@ Model Support Matrix Legend: -* ``Y`` — covered by the release deployment test suite +* ✅ — covered by the release deployment test suite (`tests/examples/hf_ptq/test_deploy.py `_), which loads the exported checkpoint in the framework and runs generation. -* ``~`` — documented as working previously but not in the current test suite; expected to work, unvalidated. +* ⚠ — documented as working previously but not in the current test suite; expected to work, unvalidated. * ``-`` — not currently covered. It may still work; see `Models not listed here`_. Language models @@ -81,39 +85,39 @@ Language models ============================================ ============== ============ ====== ======== Model Quant format TensorRT-LLM vLLM SGLang ============================================ ============== ============ ====== ======== -Llama 3.1, 3.3 FP8, NVFP4 Y Y Y -Llama 4 Scout, Maverick FP8 Y Y Y -Llama 4 Scout NVFP4 Y Y Y -Llama Nemotron Super 49B v1, v1.5 FP8 Y Y Y -Llama Nemotron Ultra 253B v1 FP8 Y Y Y -Nemotron 3 Nano 30B-A3B FP8, NVFP4 Y Y Y -Nemotron 3 Super 120B-A12B FP8, NVFP4 Y Y Y -Nemotron 3 Ultra 550B-A55B NVFP4 Y Y Y -DeepSeek R1, R1-0528 NVFP4 Y Y Y -DeepSeek V3, V3.1, V3.2 NVFP4 Y Y Y -DeepSeek V4 Flash NVFP4 Y Y Y -DeepSeek V4 Pro NVFP4 \- Y Y -Qwen 3 (8B, 14B, 32B) FP8, NVFP4 Y Y Y -Qwen 3 MoE 235B-A22B FP8, NVFP4 Y Y Y -Qwen 3 MoE 30B-A3B NVFP4 Y Y Y -Qwen 3 Coder 480B-A35B NVFP4 Y Y Y -Qwen 3-Next 80B-A3B NVFP4 Y Y Y -Qwen 3.5 397B-A17B NVFP4 Y Y Y -Qwen 3.5 122B-A10B, Qwen 3.6 35B-A3B NVFP4 \- Y \- -Qwen 2.5 FP8 ~ ~ ~ -Qwen 2.5 NVFP4 ~ ~ \- -QwQ-32B FP8 ~ ~ ~ -QwQ-32B NVFP4 ~ ~ \- -Phi-4 reasoning-plus FP8, NVFP4 Y Y Y -Gemma 4 31B NVFP4 Y Y Y -Gemma 4 26B-A4B NVFP4 \- Y \- -GLM-4.7, GLM-5, GLM-5.2 NVFP4 Y Y Y -GLM-5.1 NVFP4 \- Y Y -Kimi K2-Thinking, K2.5 NVFP4 Y Y Y -Kimi K2.6 NVFP4 \- Y \- -MiniMax M2.5, M3 NVFP4 Y Y Y -Mixtral 8x7B FP8 ~ ~ ~ -Mixtral 8x7B NVFP4 ~ \- \- +Llama 3.1, 3.3 FP8, NVFP4 ✅ ✅ ✅ +Llama 4 Scout, Maverick FP8 ✅ ✅ ✅ +Llama 4 Scout NVFP4 ✅ ✅ ✅ +Llama Nemotron Super 49B v1, v1.5 FP8 ✅ ✅ ✅ +Llama Nemotron Ultra 253B v1 FP8 ✅ ✅ ✅ +Nemotron 3 Nano 30B-A3B FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Super 120B-A12B FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Ultra 550B-A55B NVFP4 ✅ ✅ ✅ +DeepSeek R1, R1-0528 NVFP4 ✅ ✅ ✅ +DeepSeek V3, V3.1, V3.2 NVFP4 ✅ ✅ ✅ +DeepSeek V4 Flash NVFP4 ✅ ✅ ✅ +DeepSeek V4 Pro NVFP4 \- ✅ ✅ +Qwen 3 (8B, 14B, 32B) FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 MoE 235B-A22B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 MoE 30B-A3B NVFP4 ✅ ✅ ✅ +Qwen 3 Coder 480B-A35B NVFP4 ✅ ✅ ✅ +Qwen 3-Next 80B-A3B NVFP4 ✅ ✅ ✅ +Qwen 3.5 397B-A17B NVFP4 ✅ ✅ ✅ +Qwen 3.5 122B-A10B, Qwen 3.6 35B-A3B NVFP4 \- ✅ \- +Qwen 2.5 FP8 ⚠ ⚠ ⚠ +Qwen 2.5 NVFP4 ⚠ ⚠ \- +QwQ-32B FP8 ⚠ ⚠ ⚠ +QwQ-32B NVFP4 ⚠ ⚠ \- +Phi-4 reasoning-plus FP8, NVFP4 ✅ ✅ ✅ +Gemma 4 31B NVFP4 ✅ ✅ ✅ +Gemma 4 26B-A4B NVFP4 \- ✅ \- +GLM-4.7, GLM-5, GLM-5.2 NVFP4 ✅ ✅ ✅ +GLM-5.1 NVFP4 \- ✅ ✅ +Kimi K2-Thinking, K2.5 NVFP4 ✅ ✅ ✅ +Kimi K2.6 NVFP4 \- ✅ \- +MiniMax M2.5, M3 NVFP4 ✅ ✅ ✅ +Mixtral 8x7B FP8 ⚠ ⚠ ⚠ +Mixtral 8x7B NVFP4 ⚠ \- \- ============================================ ============== ============ ====== ======== Vision-language and multimodal models @@ -127,10 +131,10 @@ architecture — see the ============================================ ============== ============ ====== ======== Model Quant format TensorRT-LLM vLLM SGLang ============================================ ============== ============ ====== ======== -Qwen 2.5-VL 7B FP8, NVFP4 Y Y Y -Qwen 3-VL 235B-A22B NVFP4 Y Y Y -Phi-4-multimodal FP8, NVFP4 Y Y Y -Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 Y Y Y +Qwen 2.5-VL 7B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3-VL 235B-A22B NVFP4 ✅ ✅ ✅ +Phi-4-multimodal FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 ✅ ✅ ✅ ============================================ ============== ============ ====== ======== Speculative decoding drafters @@ -138,15 +142,19 @@ Speculative decoding drafters Drafters are deployed on top of their base checkpoint. vLLM is not currently covered for these. +Unlike the tables above, drafter coverage is conditional: the EAGLE3 tests run only when +``MODELOPT_LOCAL_EAGLE_MODEL`` points at a directory containing the drafter, and skip otherwise. +✅ below means "covered when that drafter is available locally". + ============================================================ ============ ============ ====== ======== Drafter Quant format TensorRT-LLM vLLM SGLang ============================================================ ============ ============ ====== ======== -EAGLE3 for Llama 3.3 70B, Llama 4 Maverick FP8 Y \- Y -EAGLE3 for Qwen 3 235B-A22B (incl. Thinking-2507, FP4) BF16, NVFP4 Y \- Y -EAGLE3 for Qwen 3 30B-A3B-Thinking-2507 BF16 Y \- Y -EAGLE3 for Kimi K2-Thinking, K2.5, K2.6 NVFP4 Y \- Y -EAGLE3 for gpt-oss-120b BF16 Y \- Y -Medusa for Llama 3.1 8B FP8 Y \- Y +EAGLE3 for Llama 3.3 70B, Llama 4 Maverick FP8 ✅ \- ✅ +EAGLE3 for Qwen 3 235B-A22B (incl. Thinking-2507, FP4) BF16, NVFP4 ✅ \- ✅ +EAGLE3 for Qwen 3 30B-A3B-Thinking-2507 BF16 ✅ \- ✅ +EAGLE3 for Kimi K2-Thinking, K2.5, K2.6 NVFP4 ✅ \- ✅ +EAGLE3 for gpt-oss-120b BF16 ✅ \- ✅ +Medusa for Llama 3.1 8B FP8 ✅ \- ✅ ============================================================ ============ ============ ====== ======== Diffusion models @@ -155,8 +163,8 @@ Diffusion models ============================================ ============== ============ ====== ======== Model Quant format TensorRT-LLM vLLM SGLang ============================================ ============== ============ ====== ======== -Wan 2.2 T2V A14B FP8, NVFP4 Y \- Y -DiffusionGemma 26B-A4B NVFP4 Y Y Y +Wan 2.2 T2V A14B FP8, NVFP4 ✅ \- ✅ +DiffusionGemma 26B-A4B NVFP4 ✅ ✅ ✅ ============================================ ============== ============ ====== ======== .. note:: @@ -186,7 +194,7 @@ Deployment with Selected Inference Frameworks Follow the `TensorRT-LLM installation instructions. `_ - Currently we support fp8 and nvfp4 quantized models for TensorRT-LLM deployment, you need v0.17.0 or later version of TensorRT-LLM. + FP8 and NVFP4 quantized models are supported; you need v1.2.0 or later version of TensorRT-LLM. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: diff --git a/examples/diffusers/README.md b/examples/diffusers/README.md index a9efb5fc3a3..3c8e5c80876 100644 --- a/examples/diffusers/README.md +++ b/examples/diffusers/README.md @@ -78,7 +78,7 @@ mtq.quantize(model=transformer, config=quant_config, forward_func=forward_pass) > *1.The w4a8_awq is an experimental quantization scheme that may result in a higher accuracy penalty.* -> *2.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later* +> *2.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* > *3.The SVDQuant Perf in TRT might not good as the [Nunchaku: MIT-Nvidia](https://github.com/nunchaku-tech/nunchaku) at this moment.* diff --git a/examples/hf_ptq/README.md b/examples/hf_ptq/README.md index 2c308b6a88c..413b8d8e6b1 100755 --- a/examples/hf_ptq/README.md +++ b/examples/hf_ptq/README.md @@ -129,7 +129,7 @@ Please reference our [framework scripts](#framework-scripts) and our [docs](http > *2.For some models, there is only support for exporting quantized checkpoints.* \ > *3.W4A8_AWQ is only available on some models but not all* \ > *4.For some models, KV cache quantization may result in a higher accuracy penalty.* \ -> *5.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later* \ +> *5.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* \ > *6.Some models currently support export to HF format only.* \ > *7.[PTQ for DeepSeek](../deepseek/README.md)* \ > *8.GLM-4.7 has MTP (Multi-Token Prediction) layers that are automatically loaded and excluded from quantization.* \ From f5948d88d1d9c13e3a834b190c35c96310860442 Mon Sep 17 00:00:00 2001 From: Zhiyu Cheng Date: Wed, 12 Aug 2026 12:27:47 -0700 Subject: [PATCH 3/4] docs: drop the Phi series from the deployment matrix main removed Phi-3-vision / Phi-4-multimodal from the PTQ support matrix in #2115 (NVBug 6563509), and the Phi-4 series is being deprecated, so drop the Phi-4-multimodal and Phi-4 reasoning-plus rows. Leaving them would re-introduce the contradiction this PR exists to remove. The Phi rows in the hf_ptq PTQ support matrix are untouched; deprecating those is a separate change. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Zhiyu Cheng --- docs/source/deployment/3_unified_hf.rst | 2 -- plugins/modelopt/skills/deployment/references/support-matrix.md | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/source/deployment/3_unified_hf.rst b/docs/source/deployment/3_unified_hf.rst index a564530666a..3e220a07f0a 100644 --- a/docs/source/deployment/3_unified_hf.rst +++ b/docs/source/deployment/3_unified_hf.rst @@ -108,7 +108,6 @@ Qwen 2.5 FP8 ⚠ Qwen 2.5 NVFP4 ⚠ ⚠ \- QwQ-32B FP8 ⚠ ⚠ ⚠ QwQ-32B NVFP4 ⚠ ⚠ \- -Phi-4 reasoning-plus FP8, NVFP4 ✅ ✅ ✅ Gemma 4 31B NVFP4 ✅ ✅ ✅ Gemma 4 26B-A4B NVFP4 \- ✅ \- GLM-4.7, GLM-5, GLM-5.2 NVFP4 ✅ ✅ ✅ @@ -133,7 +132,6 @@ Model Quant format TensorRT-LLM vLLM ============================================ ============== ============ ====== ======== Qwen 2.5-VL 7B FP8, NVFP4 ✅ ✅ ✅ Qwen 3-VL 235B-A22B NVFP4 ✅ ✅ ✅ -Phi-4-multimodal FP8, NVFP4 ✅ ✅ ✅ Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 ✅ ✅ ✅ ============================================ ============== ============ ====== ======== diff --git a/plugins/modelopt/skills/deployment/references/support-matrix.md b/plugins/modelopt/skills/deployment/references/support-matrix.md index bbef86efe24..24438f0eaf8 100644 --- a/plugins/modelopt/skills/deployment/references/support-matrix.md +++ b/plugins/modelopt/skills/deployment/references/support-matrix.md @@ -9,7 +9,7 @@ is backed by `tests/examples/hf_ptq/test_deploy.py`. To answer "is model X supported on framework Y", read one of those two files — `test_deploy.py` is the more precise answer, since it also carries the exact checkpoint, tensor-parallel size, and minimum SM version per entry. It covers language models, VLMs (Qwen2.5-VL, Qwen3-VL, -Phi-4-multimodal, Nemotron Omni), EAGLE3/Medusa drafters, and diffusion models. +Nemotron Omni), EAGLE3/Medusa drafters, and diffusion models. ## Supported Quantization Formats From ee7f5b4b000fc887d98e17fe09d9592fe9919376 Mon Sep 17 00:00:00 2001 From: Zhiyu Cheng Date: Wed, 12 Aug 2026 12:47:59 -0700 Subject: [PATCH 4/4] docs: reframe the matrix as declared test scope, not CI-validated support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review found that ✅ claimed more than the suite substantiates. Verified each point against the code: - test_deploy.py is marked `release` and collects only under --run-release, which no workflow passes, so the cases do not run on PR CI. Legend now says 'declared in the release deployment suite' and the section states both that limit and the load-and-generate scope. - Every backend sends the same plain-text COMMON_PROMPTS, so no image or audio reaches a processor. The VLM table is now labelled text-only smoke coverage. - The harness builds a speculative config only when the model ID contains 'eagle', so the Medusa case runs ordinary generation -> downgraded to ⚠. - Wan 2.2 runs the autoregressive text helper and asserts on text, never a video path -> downgraded to ⚠. - Kimi K2.6 Eagle3 is declared in test_kimi without the MODELOPT_LOCAL_EAGLE_MODEL gate and does cover vLLM; split from the gated rows and the blanket 'vLLM not covered' claim dropped. - Restored rows the rebuild silently dropped rather than downgraded: DeepSeek R1/V3 FP8 and Llama 4 Maverick NVFP4, both now ⚠. - Split Qwen 3 8B/14B from 32B, which has no FP8 entry. - Fixed a leftover 'every Y above' from the pre-review legend, and qualified the TRT-LLM v1.2.0 floor as the oldest version stated rather than the oldest that works. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Zhiyu Cheng --- docs/source/deployment/3_unified_hf.rst | 71 ++++++++++++++----- examples/hf_ptq/README.md | 14 ++-- .../deployment/references/support-matrix.md | 12 ++-- 3 files changed, 71 insertions(+), 26 deletions(-) diff --git a/docs/source/deployment/3_unified_hf.rst b/docs/source/deployment/3_unified_hf.rst index 3e220a07f0a..ccef639d00e 100644 --- a/docs/source/deployment/3_unified_hf.rst +++ b/docs/source/deployment/3_unified_hf.rst @@ -62,22 +62,38 @@ vLLM v0.10.1 SGLang v0.4.10 =============== ================= -These are the oldest versions expected to load a unified HF checkpoint. The matrix below is -validated against the versions CI currently runs, which are newer — see the container tags in -``.github/workflows/``. +These are the oldest versions expected to load a unified HF checkpoint. The deployment suite itself +targets newer ones — TensorRT-LLM containers in ``.github/workflows/`` are on the 1.3.x line. Older +TensorRT-LLM releases may still serve FP8 checkpoints; that is simply not exercised, so v1.2.0 is +the oldest version stated here rather than the oldest that works. .. _unified-hf-support-matrix: Model Support Matrix -------------------- +What this matrix is based on +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Entries are drawn from the release deployment suite, +`tests/examples/hf_ptq/test_deploy.py `_. +For each entry it loads the exported checkpoint in the framework and generates from four short text +prompts, asserting that each returns non-empty output. + +Two limits are worth stating plainly, because they bound what any ✅ below can mean: + +* **These are declared cases, not PR-gated coverage.** The suite is marked ``release`` and collects + only when pytest is given ``--run-release``, which no workflow in ``.github/workflows/`` currently + passes. A green check on a pull request does not mean these cases ran. +* **Each case is a load-and-generate smoke check on the text path.** It does not verify accuracy, + image or audio inputs, diffusion output, or that speculative decoding actually engages. + Legend: -* ✅ — covered by the release deployment test suite - (`tests/examples/hf_ptq/test_deploy.py `_), - which loads the exported checkpoint in the framework and runs generation. -* ⚠ — documented as working previously but not in the current test suite; expected to work, unvalidated. -* ``-`` — not currently covered. It may still work; see `Models not listed here`_. +* ✅ — declared in the release deployment suite, subject to the two limits above. +* ⚠ — expected to work, but not a suite entry: either carried over from earlier documentation, or + present as a case that does not exercise the feature the row names. +* ``-`` — not in the suite. It may still work; see `Models not listed here`_. Language models ~~~~~~~~~~~~~~~ @@ -88,16 +104,19 @@ Model Quant format TensorRT-LLM vLLM Llama 3.1, 3.3 FP8, NVFP4 ✅ ✅ ✅ Llama 4 Scout, Maverick FP8 ✅ ✅ ✅ Llama 4 Scout NVFP4 ✅ ✅ ✅ +Llama 4 Maverick NVFP4 ⚠ \- \- Llama Nemotron Super 49B v1, v1.5 FP8 ✅ ✅ ✅ Llama Nemotron Ultra 253B v1 FP8 ✅ ✅ ✅ Nemotron 3 Nano 30B-A3B FP8, NVFP4 ✅ ✅ ✅ Nemotron 3 Super 120B-A12B FP8, NVFP4 ✅ ✅ ✅ Nemotron 3 Ultra 550B-A55B NVFP4 ✅ ✅ ✅ DeepSeek R1, R1-0528 NVFP4 ✅ ✅ ✅ +DeepSeek R1, V3 FP8 ⚠ ⚠ ⚠ DeepSeek V3, V3.1, V3.2 NVFP4 ✅ ✅ ✅ DeepSeek V4 Flash NVFP4 ✅ ✅ ✅ DeepSeek V4 Pro NVFP4 \- ✅ ✅ -Qwen 3 (8B, 14B, 32B) FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 8B, 14B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 32B NVFP4 ✅ ✅ ✅ Qwen 3 MoE 235B-A22B FP8, NVFP4 ✅ ✅ ✅ Qwen 3 MoE 30B-A3B NVFP4 ✅ ✅ ✅ Qwen 3 Coder 480B-A35B NVFP4 ✅ ✅ ✅ @@ -127,6 +146,12 @@ The exported checkpoint therefore relies on the serving framework's own multimod architecture — see the `TensorRT-LLM multimodal support matrix `_. +.. important:: + ✅ in this table is **text-only smoke coverage**. The suite sends the same plain-text prompts it + uses for language models, so no image or audio input reaches the processor or vision encoder. + These entries show that the quantized checkpoint loads and that its language path generates — + they do not demonstrate multimodal serving. + ============================================ ============== ============ ====== ======== Model Quant format TensorRT-LLM vLLM SGLang ============================================ ============== ============ ====== ======== @@ -138,11 +163,18 @@ Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 ✅ Speculative decoding drafters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Drafters are deployed on top of their base checkpoint. vLLM is not currently covered for these. +Drafters are deployed on top of their base checkpoint. + +Two caveats specific to this table: -Unlike the tables above, drafter coverage is conditional: the EAGLE3 tests run only when -``MODELOPT_LOCAL_EAGLE_MODEL`` points at a directory containing the drafter, and skip otherwise. -✅ below means "covered when that drafter is available locally". +* **Most entries are doubly conditional.** Beyond the ``--run-release`` gate, the drafter cases in + ``test_eagle`` also require ``MODELOPT_LOCAL_EAGLE_MODEL`` to point at a directory containing the + drafter, and skip otherwise. The exception is EAGLE3 for Kimi K2.6, which is declared in + ``test_kimi`` without that gate — which is also why it is the one row with vLLM coverage. +* **Medusa is marked ⚠ because the case does not exercise Medusa.** The shared harness builds a + speculative-decoding configuration only when the model ID contains ``eagle``, so the Medusa entry + performs ordinary generation. It shows the checkpoint loads and serves; it does not validate + Medusa decoding. ============================================================ ============ ============ ====== ======== Drafter Quant format TensorRT-LLM vLLM SGLang @@ -150,9 +182,10 @@ Drafter Quant format Tens EAGLE3 for Llama 3.3 70B, Llama 4 Maverick FP8 ✅ \- ✅ EAGLE3 for Qwen 3 235B-A22B (incl. Thinking-2507, FP4) BF16, NVFP4 ✅ \- ✅ EAGLE3 for Qwen 3 30B-A3B-Thinking-2507 BF16 ✅ \- ✅ -EAGLE3 for Kimi K2-Thinking, K2.5, K2.6 NVFP4 ✅ \- ✅ +EAGLE3 for Kimi K2-Thinking, K2.5 NVFP4 ✅ \- ✅ +EAGLE3 for Kimi K2.6 NVFP4 ✅ ✅ ✅ EAGLE3 for gpt-oss-120b BF16 ✅ \- ✅ -Medusa for Llama 3.1 8B FP8 ✅ \- ✅ +Medusa for Llama 3.1 8B FP8 ⚠ \- ⚠ ============================================================ ============ ============ ====== ======== Diffusion models @@ -161,10 +194,14 @@ Diffusion models ============================================ ============== ============ ====== ======== Model Quant format TensorRT-LLM vLLM SGLang ============================================ ============== ============ ====== ======== -Wan 2.2 T2V A14B FP8, NVFP4 ✅ \- ✅ +Wan 2.2 T2V A14B FP8, NVFP4 ⚠ \- ⚠ DiffusionGemma 26B-A4B NVFP4 ✅ ✅ ✅ ============================================ ============== ============ ====== ======== +Wan 2.2 is marked ⚠ because its cases run through the same autoregressive text helper as the +language models and assert on generated text. They never call a diffusion or video serving API, so +they do not substantiate text-to-video deployment. + .. note:: NVFP4 inference requires Blackwell GPUs. Hopper can produce an NVFP4 checkpoint but cannot serve it. On B300/GB300 (``sm_103``) use a CUDA-13 build of the serving framework; CUDA-12 builds lack @@ -178,7 +215,7 @@ run: vLLM, SGLang, and TensorRT-LLM load unified HF checkpoints generically, so standard ``nn.Linear`` layers with an ``hf_quant_config.json`` will often deploy without any modelopt change. Check the serving framework's own model support list first, then try it. -The exact checkpoints behind every ``Y`` above, including tensor-parallel size and minimum SM +The exact checkpoints behind every ✅ above, including tensor-parallel size and minimum SM version, are listed in `tests/examples/hf_ptq/test_deploy.py `__; most are published under the diff --git a/examples/hf_ptq/README.md b/examples/hf_ptq/README.md index 413b8d8e6b1..72b2e2348ab 100755 --- a/examples/hf_ptq/README.md +++ b/examples/hf_ptq/README.md @@ -587,16 +587,20 @@ print(llm_fp8.generate(["What's the age of the earth? "])) ### Unified HF Checkpoint Deployment Model Support Matrix -The deployment support matrix — which model families and quantization formats are validated on +The deployment support matrix — which model families and quantization formats are covered on TRT-LLM, vLLM, and SGLang, including vision-language models, speculative decoding drafters, and diffusion models — lives in the documentation so there is a single copy to keep current: **[Unified HF Checkpoint → Model Support Matrix](https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf.html#model-support-matrix)** -Every validated entry there is backed by [`tests/examples/hf_ptq/test_deploy.py`](../../tests/examples/hf_ptq/test_deploy.py), -which loads the exported checkpoint in each framework and runs generation. That file is also the -place to look for the exact checkpoint, tensor-parallel size, and minimum SM version behind each -entry. +Each entry there is drawn from [`tests/examples/hf_ptq/test_deploy.py`](../../tests/examples/hf_ptq/test_deploy.py), +which loads the exported checkpoint in each framework and generates from short text prompts. That +file is also the place to look for the exact checkpoint, tensor-parallel size, and minimum SM +version behind each entry. + +> *Note: those cases are marked `release` and run out-of-band — no workflow currently passes +> `--run-release` — and each is a load-and-generate smoke check on the text path. Read the legend in +> the docs before treating an entry as verified support.* > *Note: the matrix records what modelopt validates, not the full set of what will run. vLLM, SGLang, > and TRT-LLM load unified HF checkpoints generically, so unlisted models frequently deploy without diff --git a/plugins/modelopt/skills/deployment/references/support-matrix.md b/plugins/modelopt/skills/deployment/references/support-matrix.md index 24438f0eaf8..e07d1b115a5 100644 --- a/plugins/modelopt/skills/deployment/references/support-matrix.md +++ b/plugins/modelopt/skills/deployment/references/support-matrix.md @@ -3,8 +3,12 @@ ## Unified HF Checkpoint — Framework Compatibility **Do not maintain a copy of the matrix here.** The single source of truth is -`docs/source/deployment/3_unified_hf.rst` ("Model Support Matrix"), and every validated entry in it -is backed by `tests/examples/hf_ptq/test_deploy.py`. +`docs/source/deployment/3_unified_hf.rst` ("Model Support Matrix"), and every entry in it is drawn +from `tests/examples/hf_ptq/test_deploy.py`. + +Read that doc's legend before reporting a model as supported: the cases are marked `release` and do +not run on PR CI, and each is a load-and-generate smoke check on the text path — so an entry is +declared coverage, not proof the combination serves correctly. To answer "is model X supported on framework Y", read one of those two files — `test_deploy.py` is the more precise answer, since it also carries the exact checkpoint, tensor-parallel size, and @@ -40,11 +44,11 @@ Nemotron Omni), EAGLE3/Medusa drafters, and diffusion models. ## Models not in the matrix -The matrix covers officially validated combinations, not the full set of what will run. For unlisted models: +The matrix covers the combinations modelopt tracks, not the full set of what will run. For unlisted models: 1. **Check the framework's own docs** — vLLM and SGLang support many HuggingFace models natively. Use WebSearch to check `vllm supported models` or `sglang supported models`. 2. **Try it** — if the model uses standard `nn.Linear` layers and has `hf_quant_config.json`, vLLM/SGLang will likely work with `--quantization modelopt`. -3. **Ask the user** — if unsure, ask: "This model isn't in the validated support matrix. Would you like to try deploying it anyway?" +3. **Ask the user** — if unsure, ask: "This model isn't in the support matrix. Would you like to try deploying it anyway?" ## Notes