diff --git a/docs/source/deployment/3_unified_hf.rst b/docs/source/deployment/3_unified_hf.rst index 59a2782c5e7..ccef639d00e 100644 --- a/docs/source/deployment/3_unified_hf.rst +++ b/docs/source/deployment/3_unified_hf.rst @@ -51,48 +51,175 @@ The unified HF export API supports the following quantization formats: 5. INT4_AWQ - 4-bit integer with AWQ optimization 6. W4A8_AWQ - 4-bit weights and 8-bit activations with AWQ optimization -Framework-Specific Support +Minimum Framework Versions -------------------------- -TensorRT-LLM -~~~~~~~~~~~~ +=============== ================= +Framework Minimum version +=============== ================= +TensorRT-LLM v1.2.0 +vLLM v0.10.1 +SGLang v0.4.10 +=============== ================= + +These are the oldest versions expected to load a unified HF checkpoint. The deployment suite itself +targets newer ones — TensorRT-LLM containers in ``.github/workflows/`` are on the 1.3.x line. Older +TensorRT-LLM releases may still serve FP8 checkpoints; that is simply not exercised, so v1.2.0 is +the oldest version stated here rather than the oldest that works. + +.. _unified-hf-support-matrix: + +Model Support Matrix +-------------------- + +What this matrix is based on +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Entries are drawn from the release deployment suite, +`tests/examples/hf_ptq/test_deploy.py `_. +For each entry it loads the exported checkpoint in the framework and generates from four short text +prompts, asserting that each returns non-empty output. + +Two limits are worth stating plainly, because they bound what any ✅ below can mean: + +* **These are declared cases, not PR-gated coverage.** The suite is marked ``release`` and collects + only when pytest is given ``--run-release``, which no workflow in ``.github/workflows/`` currently + passes. A green check on a pull request does not mean these cases ran. +* **Each case is a load-and-generate smoke check on the text path.** It does not verify accuracy, + image or audio inputs, diffusion output, or that speculative decoding actually engages. + +Legend: + +* ✅ — declared in the release deployment suite, subject to the two limits above. +* ⚠ — expected to work, but not a suite entry: either carried over from earlier documentation, or + present as a case that does not exercise the feature the row names. +* ``-`` — not in the suite. It may still work; see `Models not listed here`_. + +Language models +~~~~~~~~~~~~~~~ + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Llama 3.1, 3.3 FP8, NVFP4 ✅ ✅ ✅ +Llama 4 Scout, Maverick FP8 ✅ ✅ ✅ +Llama 4 Scout NVFP4 ✅ ✅ ✅ +Llama 4 Maverick NVFP4 ⚠ \- \- +Llama Nemotron Super 49B v1, v1.5 FP8 ✅ ✅ ✅ +Llama Nemotron Ultra 253B v1 FP8 ✅ ✅ ✅ +Nemotron 3 Nano 30B-A3B FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Super 120B-A12B FP8, NVFP4 ✅ ✅ ✅ +Nemotron 3 Ultra 550B-A55B NVFP4 ✅ ✅ ✅ +DeepSeek R1, R1-0528 NVFP4 ✅ ✅ ✅ +DeepSeek R1, V3 FP8 ⚠ ⚠ ⚠ +DeepSeek V3, V3.1, V3.2 NVFP4 ✅ ✅ ✅ +DeepSeek V4 Flash NVFP4 ✅ ✅ ✅ +DeepSeek V4 Pro NVFP4 \- ✅ ✅ +Qwen 3 8B, 14B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 32B NVFP4 ✅ ✅ ✅ +Qwen 3 MoE 235B-A22B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3 MoE 30B-A3B NVFP4 ✅ ✅ ✅ +Qwen 3 Coder 480B-A35B NVFP4 ✅ ✅ ✅ +Qwen 3-Next 80B-A3B NVFP4 ✅ ✅ ✅ +Qwen 3.5 397B-A17B NVFP4 ✅ ✅ ✅ +Qwen 3.5 122B-A10B, Qwen 3.6 35B-A3B NVFP4 \- ✅ \- +Qwen 2.5 FP8 ⚠ ⚠ ⚠ +Qwen 2.5 NVFP4 ⚠ ⚠ \- +QwQ-32B FP8 ⚠ ⚠ ⚠ +QwQ-32B NVFP4 ⚠ ⚠ \- +Gemma 4 31B NVFP4 ✅ ✅ ✅ +Gemma 4 26B-A4B NVFP4 \- ✅ \- +GLM-4.7, GLM-5, GLM-5.2 NVFP4 ✅ ✅ ✅ +GLM-5.1 NVFP4 \- ✅ ✅ +Kimi K2-Thinking, K2.5 NVFP4 ✅ ✅ ✅ +Kimi K2.6 NVFP4 \- ✅ \- +MiniMax M2.5, M3 NVFP4 ✅ ✅ ✅ +Mixtral 8x7B FP8 ⚠ ⚠ ⚠ +Mixtral 8x7B NVFP4 ⚠ \- \- +============================================ ============== ============ ====== ======== + +Vision-language and multimodal models +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For VLMs, modelopt quantizes the language model only; the vision encoder is kept in high precision. +The exported checkpoint therefore relies on the serving framework's own multimodal support for that +architecture — see the +`TensorRT-LLM multimodal support matrix `_. + +.. important:: + ✅ in this table is **text-only smoke coverage**. The suite sends the same plain-text prompts it + uses for language models, so no image or audio input reaches the processor or vision encoder. + These entries show that the quantized checkpoint loads and that its language path generates — + they do not demonstrate multimodal serving. + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Qwen 2.5-VL 7B FP8, NVFP4 ✅ ✅ ✅ +Qwen 3-VL 235B-A22B NVFP4 ✅ ✅ ✅ +Nemotron 3 Nano Omni 30B-A3B FP8, NVFP4 ✅ ✅ ✅ +============================================ ============== ============ ====== ======== + +Speculative decoding drafters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Drafters are deployed on top of their base checkpoint. + +Two caveats specific to this table: + +* **Most entries are doubly conditional.** Beyond the ``--run-release`` gate, the drafter cases in + ``test_eagle`` also require ``MODELOPT_LOCAL_EAGLE_MODEL`` to point at a directory containing the + drafter, and skip otherwise. The exception is EAGLE3 for Kimi K2.6, which is declared in + ``test_kimi`` without that gate — which is also why it is the one row with vLLM coverage. +* **Medusa is marked ⚠ because the case does not exercise Medusa.** The shared harness builds a + speculative-decoding configuration only when the model ID contains ``eagle``, so the Medusa entry + performs ordinary generation. It shows the checkpoint loads and serves; it does not validate + Medusa decoding. + +============================================================ ============ ============ ====== ======== +Drafter Quant format TensorRT-LLM vLLM SGLang +============================================================ ============ ============ ====== ======== +EAGLE3 for Llama 3.3 70B, Llama 4 Maverick FP8 ✅ \- ✅ +EAGLE3 for Qwen 3 235B-A22B (incl. Thinking-2507, FP4) BF16, NVFP4 ✅ \- ✅ +EAGLE3 for Qwen 3 30B-A3B-Thinking-2507 BF16 ✅ \- ✅ +EAGLE3 for Kimi K2-Thinking, K2.5 NVFP4 ✅ \- ✅ +EAGLE3 for Kimi K2.6 NVFP4 ✅ ✅ ✅ +EAGLE3 for gpt-oss-120b BF16 ✅ \- ✅ +Medusa for Llama 3.1 8B FP8 ⚠ \- ⚠ +============================================================ ============ ============ ====== ======== + +Diffusion models +~~~~~~~~~~~~~~~~ + +============================================ ============== ============ ====== ======== +Model Quant format TensorRT-LLM vLLM SGLang +============================================ ============== ============ ====== ======== +Wan 2.2 T2V A14B FP8, NVFP4 ⚠ \- ⚠ +DiffusionGemma 26B-A4B NVFP4 ✅ ✅ ✅ +============================================ ============== ============ ====== ======== + +Wan 2.2 is marked ⚠ because its cases run through the same autoregressive text helper as the +language models and assert on generated text. They never call a diffusion or video serving API, so +they do not substantiate text-to-video deployment. -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Qwen 3-VL (FP8, NVFP4) - * Deepseek R1/V3 (NVFP4) - * Mixtral 8x7B (FP8, NVFP4) - * Medusa (FP8) - * Eagle (FP8) - -Requirements: TensorRT-LLM v0.17.0 or later - -vLLM -~~~~ - -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Mixtral 8x7B (FP8) - * Deepseek R1/V3 (NVFP4) - -Requirements: vLLM v0.10.1 or later - -SGLang -~~~~~~ +.. note:: + NVFP4 inference requires Blackwell GPUs. Hopper can produce an NVFP4 checkpoint but cannot serve + it. On B300/GB300 (``sm_103``) use a CUDA-13 build of the serving framework; CUDA-12 builds lack + the ``sm_103`` FP4 kernels. -Models: - * Llama 4, 3.x (FP8, NVFP4) - * Qwen 3, 2.5 (FP8, NVFP4) - * Qwen 3 MoE (FP8, NVFP4) - * Deepseek R1/V3 (NVFP4) +Models not listed here +~~~~~~~~~~~~~~~~~~~~~~ -Requirements: SGLang v0.4.10 or later +This matrix records the combinations modelopt validates. It is not an exhaustive list of what will +run: vLLM, SGLang, and TensorRT-LLM load unified HF checkpoints generically, so a model built from +standard ``nn.Linear`` layers with an ``hf_quant_config.json`` will often deploy without any modelopt +change. Check the serving framework's own model support list first, then try it. -Note: While other models and quantization formats may work, they have not been thoroughly tested and validated. +The exact checkpoints behind every ✅ above, including tensor-parallel size and minimum SM +version, are listed in +`tests/examples/hf_ptq/test_deploy.py `__; +most are published under the +`NVIDIA Hugging Face organization `_. Deployment with Selected Inference Frameworks @@ -102,7 +229,7 @@ Deployment with Selected Inference Frameworks Follow the `TensorRT-LLM installation instructions. `_ - Currently we support fp8 and nvfp4 quantized models for TensorRT-LLM deployment, you need v0.17.0 or later version of TensorRT-LLM. + FP8 and NVFP4 quantized models are supported; you need v1.2.0 or later version of TensorRT-LLM. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: @@ -136,7 +263,8 @@ Deployment with Selected Inference Frameworks Follow `vLLM installation instructions. `_ - Currently we support fp8 quantized models (without fp8 kv cache) for vLLM deployment, you need v0.6.5 or later version of vLLM. + FP8 and NVFP4 quantized models are supported; you need v0.10.1 or later version of vLLM. Pass + ``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: @@ -171,7 +299,8 @@ Deployment with Selected Inference Frameworks Follow the `SGLang installation instructions. `_ - Currently we support fp8 quantized models (without fp8 kv cache) for SGLang deployment, you need to use the main branch of SGLang (since Jan 6, 2025) and build it from source. + FP8 and NVFP4 quantized models are supported; you need v0.4.10 or later version of SGLang. Pass + ``quantization="modelopt"`` for FP8 and ``quantization="modelopt_fp4"`` for NVFP4. To run modelopt quantized model from Huggingface model hub, e.g., `nvidia/Llama-3.1-8B-Instruct-FP8`_, refer to the sample code below: diff --git a/examples/diffusers/README.md b/examples/diffusers/README.md index a9efb5fc3a3..3c8e5c80876 100644 --- a/examples/diffusers/README.md +++ b/examples/diffusers/README.md @@ -78,7 +78,7 @@ mtq.quantize(model=transformer, config=quant_config, forward_func=forward_pass) > *1.The w4a8_awq is an experimental quantization scheme that may result in a higher accuracy penalty.* -> *2.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later* +> *2.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* > *3.The SVDQuant Perf in TRT might not good as the [Nunchaku: MIT-Nvidia](https://github.com/nunchaku-tech/nunchaku) at this moment.* diff --git a/examples/hf_ptq/README.md b/examples/hf_ptq/README.md index dc852408c7a..72b2e2348ab 100755 --- a/examples/hf_ptq/README.md +++ b/examples/hf_ptq/README.md @@ -129,7 +129,7 @@ Please reference our [framework scripts](#framework-scripts) and our [docs](http > *2.For some models, there is only support for exporting quantized checkpoints.* \ > *3.W4A8_AWQ is only available on some models but not all* \ > *4.For some models, KV cache quantization may result in a higher accuracy penalty.* \ -> *5.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v0.17 or later* \ +> *5.A selective set of the popular models are internally tested. The actual model support list may be longer. NVFP4 inference requires Blackwell GPUs and TensorRT-LLM v1.2 or later* \ > *6.Some models currently support export to HF format only.* \ > *7.[PTQ for DeepSeek](../deepseek/README.md)* \ > *8.GLM-4.7 has MTP (Multi-Token Prediction) layers that are automatically loaded and excluded from quantization.* \ @@ -587,27 +587,24 @@ print(llm_fp8.generate(["What's the age of the earth? "])) ### Unified HF Checkpoint Deployment Model Support Matrix -| Model | Quant format | TRT-LLM | vLLM | SGLang | -| :---: | :---: | :---: | :---: | :---: | -| LLAMA 3.x | FP8 | ✅ | ✅ | ✅ | -| LLAMA 3.x | FP4 | ✅ | ✅ | ✅ | -| LLAMA 4 | FP8 | ✅ | - | ✅ | -| LLAMA 4 | FP4 | ✅ | - | - | -| DS-R1 | FP8 | ✅ | ✅ | ✅ | -| DS-R1 | FP4 | ✅ | ✅ | ✅ | -| DS-V3 | FP8 | ✅ | ✅ | ✅ | -| DS-V3 | FP4 | ✅ | ✅ | ✅ | -| QWen3 | FP8 | ✅ | ✅ | ✅ | -| QWen3 | FP4 | ✅ | ✅ | - | -| QWen3 MoE | FP8 | ✅ | ✅ | ✅ | -| QWen3 MoE | FP4 | ✅ | - | - | -| QWen3.5 MoE | FP4 | - | - | ✅ | -| QWen2.5 | FP8 | ✅ | ✅ | ✅ | -| QWen2.5 | FP4 | ✅ | ✅ | - | -| QwQ-32B | FP8 | ✅ | ✅ | ✅ | -| QwQ-32B | FP4 | ✅ | ✅ | - | -| Mixtral 8x7B | FP8 | ✅ | ✅ | ✅ | -| Mixtral 8x7B | FP4 | ✅ | - | - | +The deployment support matrix — which model families and quantization formats are covered on +TRT-LLM, vLLM, and SGLang, including vision-language models, speculative decoding drafters, and +diffusion models — lives in the documentation so there is a single copy to keep current: + +**[Unified HF Checkpoint → Model Support Matrix](https://nvidia.github.io/Model-Optimizer/deployment/3_unified_hf.html#model-support-matrix)** + +Each entry there is drawn from [`tests/examples/hf_ptq/test_deploy.py`](../../tests/examples/hf_ptq/test_deploy.py), +which loads the exported checkpoint in each framework and generates from short text prompts. That +file is also the place to look for the exact checkpoint, tensor-parallel size, and minimum SM +version behind each entry. + +> *Note: those cases are marked `release` and run out-of-band — no workflow currently passes +> `--run-release` — and each is a load-and-generate smoke check on the text path. Read the legend in +> the docs before treating an entry as verified support.* + +> *Note: the matrix records what modelopt validates, not the full set of what will run. vLLM, SGLang, +> and TRT-LLM load unified HF checkpoints generically, so unlisted models frequently deploy without +> any modelopt change — check the serving framework's own model support list and try it.* ### (Legacy) TensorRT-LLM Checkpoints diff --git a/plugins/modelopt/skills/deployment/references/support-matrix.md b/plugins/modelopt/skills/deployment/references/support-matrix.md index 265f1b58560..e07d1b115a5 100644 --- a/plugins/modelopt/skills/deployment/references/support-matrix.md +++ b/plugins/modelopt/skills/deployment/references/support-matrix.md @@ -2,26 +2,18 @@ ## Unified HF Checkpoint — Framework Compatibility -| Model | Quant Format | TRT-LLM | vLLM | SGLang | -|-------|-------------|---------|------|--------| -| Llama 3.x | FP8 | yes | yes | yes | -| Llama 3.x | FP4 | yes | yes | yes | -| Llama 4 | FP8 | yes | — | yes | -| Llama 4 | FP4 | yes | — | — | -| DeepSeek R1 | FP8 | yes | yes | yes | -| DeepSeek R1 | FP4 | yes | yes | yes | -| DeepSeek V3 | FP8 | yes | yes | yes | -| DeepSeek V3 | FP4 | yes | yes | yes | -| Qwen 3 | FP8 | yes | yes | yes | -| Qwen 3 | FP4 | yes | yes | — | -| Qwen 3 MoE | FP8 | yes | yes | yes | -| Qwen 3 MoE | FP4 | yes | — | — | -| Qwen 2.5 | FP8 | yes | yes | yes | -| Qwen 2.5 | FP4 | yes | yes | — | -| QwQ-32B | FP8 | yes | yes | yes | -| QwQ-32B | FP4 | yes | yes | — | -| Mixtral 8x7B | FP8 | yes | yes | yes | -| Mixtral 8x7B | FP4 | yes | — | — | +**Do not maintain a copy of the matrix here.** The single source of truth is +`docs/source/deployment/3_unified_hf.rst` ("Model Support Matrix"), and every entry in it is drawn +from `tests/examples/hf_ptq/test_deploy.py`. + +Read that doc's legend before reporting a model as supported: the cases are marked `release` and do +not run on PR CI, and each is a load-and-generate smoke check on the text path — so an entry is +declared coverage, not proof the combination serves correctly. + +To answer "is model X supported on framework Y", read one of those two files — `test_deploy.py` is +the more precise answer, since it also carries the exact checkpoint, tensor-parallel size, and +minimum SM version per entry. It covers language models, VLMs (Qwen2.5-VL, Qwen3-VL, +Nemotron Omni), EAGLE3/Medusa drafters, and diffusion models. ## Supported Quantization Formats @@ -50,13 +42,13 @@ | SGLang | `quantization="modelopt"` | `quantization="modelopt_fp4"` | | TRT-LLM | auto-detected from checkpoint | auto-detected from checkpoint | -## Models not in this list +## Models not in the matrix -This matrix covers officially validated combinations. For unlisted models: +The matrix covers the combinations modelopt tracks, not the full set of what will run. For unlisted models: 1. **Check the framework's own docs** — vLLM and SGLang support many HuggingFace models natively. Use WebSearch to check `vllm supported models` or `sglang supported models`. 2. **Try it** — if the model uses standard `nn.Linear` layers and has `hf_quant_config.json`, vLLM/SGLang will likely work with `--quantization modelopt`. -3. **Ask the user** — if unsure, ask: "This model isn't in the validated support matrix. Would you like to try deploying it anyway?" +3. **Ask the user** — if unsure, ask: "This model isn't in the support matrix. Would you like to try deploying it anyway?" ## Notes @@ -64,4 +56,5 @@ This matrix covers officially validated combinations. For unlisted models: - **B300/GB300 are `sm_103`** and need a **CUDA-13** serving image — from v0.20.0 the unsuffixed tag is CUDA-13 (`-cu129` opts back to CUDA 12); `cu12` images lack the `sm_103` FP4 kernel and serve NVFP4 as gibberish or error out. See the CUDA-13 note in the deployment `SKILL.md`. - **Verify the GPU with `nvidia-smi`** before choosing the image — cluster GPU labels can be stale. - INT4_AWQ and W4A8_AWQ are only supported by TRT-LLM (not vLLM or SGLang). -- Source: `examples/hf_ptq/README.md` and `docs/source/deployment/3_unified_hf.rst` +- For VLMs, only the language model is quantized; the vision encoder stays in high precision, so multimodal serving depends on the framework's own support for that architecture. +- Source: `docs/source/deployment/3_unified_hf.rst` and `tests/examples/hf_ptq/test_deploy.py`