feat(nvidia): support model type: granitemoehybrid

[WHoutstanding]

Summary

• Add support for the GraniteMoeHybrid architecture, a decoder-only hybrid language model that combines Mamba-2, self-attention, and Mixture-of-Experts (MoE) blocks. GraniteMoeHybrid uses a configuration-driven hybrid decoder stack: each decoder layer selects either a Mamba-2 state-space mixer or a causal self-attention mixer. Supported models include:

ibm-granite/granite-4.0-micro
ibm-granite/granite-4.0-h-micro
ibm-granite/granite-4.0-h-tiny
ibm-granite/granite-4.0-h-small

• Supported backends include: Default Attention、FlashAttention、PagedAttention
• Add Tensor Parallelism (TP) and Pipeline Parallelism (PP) support for GraniteMoeHybrid, enabling distributed inference across multiple devices.
• Support Inference Service.

Motivation

Add new model type.

Type of Change

• feat — new feature / new model
• fix — bug fix
• perf — performance improvement (no behavioral change)
• refactor — code restructuring without behavior change
• test — adding or fixing tests only
• docs — documentation only
• build / ci — build system or CI configuration
• chore — tooling, formatting, or other non-code changes
• Breaking change

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

1. Test environment:
   device: A single node with NVIDIA RTX 4090 * 4
   environment variables: export NCCL_CUMEM_HOST_ENABLE=0

2. Correctness validation
   Comparing inference results against the Hugging Face implementation.The comparison uses greedy decoding with top_k = 1, ensuring that each decoding step always selects the token with the highest probability and removes randomness introduced by sampling.

3. Test commands:

#!/usr/bin/env bash

# model: 1. ibm-granite/granite-4.0-micro
#        2. ibm-granite/granite-4.0-h-micro
#        3. ibm-granite/granite-4.0-h-tiny
#        4. ibm-granite/granite-4.0-h-small


test_file_path="test/models/granitemoehybrid/test_infer.py"
model_path="/data/huggingface_home/hub/models--ibm-granite--granite-4.0-h-micro/snapshots/7e9213d4d1c2d31a19422f0572b16c8f34daaa11/"
DEVICE="nvidia"
tp_size=2
pp_size=2
master_port=29500
master_addr="10.254.0.3"

## 单次推理测试（batch_size=1）

### Cache Type
#### StaticKVCache
python "$test_file_path" --model "$model_path" --device "$DEVICE" --attn=default --disable-prefix-caching

#### PagedKVCache
##### --attn=paged-attn
python "$test_file_path" --model "$model_path" --device "$DEVICE"  --enable-paged-attn --attn=paged-attn --disable-prefix-caching
##### --attn=flash-attn
python "$test_file_path" --model "$model_path" --device "$DEVICE"  --enable-paged-attn --attn=flash-attn --disable-prefix-caching

### 分布式推理
#### 张量并行
##### StaticKVCache
python "$test_file_path" --model "$model_path" --device "$DEVICE" --attn=default  --tp "$tp_size" --disable-prefix-caching
##### PagedKVCache && --attn=paged-attn
python "$test_file_path" --model "$model_path" --device "$DEVICE"  --enable-paged-attn --attn=paged-attn --tp "$tp_size" --disable-prefix-caching
##### PagedKVCache && --attn=flash-attn
python "$test_file_path" --model "$model_path" --device "$DEVICE"  --enable-paged-attn --attn=flash-attn --tp "$tp_size" --disable-prefix-caching

### 流水线并行 && 张量并行
##### StaticKVCache
# Terminal 1: stage 0 / coordinator (--node-rank=0)
CUDA_VISIBLE_DEVICES=0,1 python "$test_file_path"  --model "$model_path" --device "$DEVICE" --attn=default --tp "$tp_size" --pp "$pp_size" --node-rank=0 --master-addr "$master_addr" --master-port "$master_port" --disable-prefix-caching

# Terminal 2: stage 1 / worker (--node-rank=1)
CUDA_VISIBLE_DEVICES=2,3 python "$test_file_path"  --model "$model_path" --device "$DEVICE" --attn=default --tp "$tp_size" --pp "$pp_size" --node-rank=1 --master-addr "$master_addr" --master-port "$master_port" --disable-prefix-caching


##### PagedKVCache && --attn=paged-attn
# Terminal 1: stage 0 / coordinator (--node-rank=0)
CUDA_VISIBLE_DEVICES=0,1 python "$test_file_path"  --model "$model_path" --device "$DEVICE" --enable-paged-attn --attn=paged-attn --tp "$tp_size" --pp "$pp_size" --node-rank=0 --master-addr "$master_addr" --master-port "$master_port" --disable-prefix-caching

# Terminal 2: stage 1 / worker (--node-rank=1)
CUDA_VISIBLE_DEVICES=2,3 python "$test_file_path"  --model "$model_path" --device "$DEVICE" --enable-paged-attn --attn=paged-attn --tp "$tp_size" --pp "$pp_size" --node-rank=1 --master-addr "$master_addr" --master-port "$master_port" --disable-prefix-caching

##### PagedKVCache && --attn=flash-attn
# Terminal 1: stage 0 / coordinator (--node-rank=0)
CUDA_VISIBLE_DEVICES=0,1 python "$test_file_path"  --model "$model_path" --device "$DEVICE" --enable-paged-attn --attn=flash-attn --tp "$tp_size" --pp "$pp_size" --node-rank=0 --master-addr "$master_addr" --master-port "$master_port" --disable-prefix-caching

# Terminal 2: stage 1 / worker (--node-rank=1)
CUDA_VISIBLE_DEVICES=2,3 python "$test_file_path"  --model "$model_path" --device "$DEVICE" --enable-paged-attn --attn=flash-attn --tp "$tp_size" --pp "$pp_size" --node-rank=1 --master-addr "$master_addr" --master-port "$master_port" --disable-prefix-caching

4. Test resultss:
   All tests passed.

Benchmark / Performance Impact

Notes for Reviewers

CI / ChatOps

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat(nvidia): …, fix(cuda/gemm): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from main — the branch is rebased cleanly on top of the current main.
• No fixup! / squash! / wip commits remain.
• Existing PR/branch/commit that followed the legacy issue format.

Scope and Design

• Changes are minimal — nothing unrelated to the stated motivation was added (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene (applies to all languages)

• The code is self-explanatory; comments were added only where the why is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages are wrapped in backticks (e.g. the `seqlens_k` tensor) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• No raw new/delete; RAII / smart pointers / existing allocators are used.
• Changed files are formatted by scripts/format.py.
• No changes/reference to csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• Code is PEP 8 compliant.
• Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• Changed files are formatted by scripts/format.py.
• No changes/reference to python/infinilm/auto_config.py.

Testing

• For any platform that could not be tested, an explicit reason is given in the table and a reviewer with access has been tagged.
• Passed single request test (examples/test_infer.py), or specify the reason for skipping.
• Passed offline performance test (examples/bench.py), or specify the reason for skipping.
• Passed sanity test (test/bench/test_benchmark.py), or specify the reason for skipping.
• Passed service test (python/infinilm/server/inference_server.py + scripts/test_perf.py), or specify the reason for skipping.

Build, CI, and Tooling

• The project builds cleanly from a fresh directory on at least one affected platform.
• CI has been triggered manually (Actions → CI on this branch), or /retest was requested.

Documentation

• README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• Any user-visible breaking change is called out explicitly under "Motivation" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.