F #867: gpu-benchmark - bare-metal vs vLLM appliance vs Kubernetes GPU/LLM validation - #54
Merged
Merged
Conversation
…ments
end-to-end: hardware health, per-GPU compute/PCIe/DCGM diagnostics, and a
real LLM inference benchmark (vLLM + GuideLLM) run individually on every GPU,
with GPU-to-GPU comparison and a client-ready report section (SVG charts,
per-GPU tables, SLO verdicts). Opt-in via `validation.run_gpu_benchmark: true`.
Per cluster: discovery → health → LLM benchmark → aggregate → report.
1. **Discovery** — finds hosts with NVIDIA GPUs and the vLLM appliance
(ONEAI Marketplace); skips cleanly (WARN) when there are none.
2. **GPU health (per host)** — boots a probe VM with all GPUs attached:
nvidia-smi inventory (VRAM, ECC, throttle, PCIe width, clocks, temps),
per-GPU cuBLAS SGEMM GFLOP/s, pinned-memory PCIe H2D/D2H bandwidth,
and NVIDIA DCGM diagnostics (`dcgmi diag`, configurable level).
3. **LLM benchmark (per host × model)** — one appliance VM, then **each GPU
is benchmarked separately** (`CUDA_VISIBLE_DEVICES` pinning, TP=1) with a
fixed GuideLLM sweep (512/256 tokens, configurable window): TTFT/ITL/TPOT
p95, sustained throughput, power/thermals sampled under load
(tokens-per-watt), provisioning-time SLA (VM create → model serving), and
a 6-test functional API conformance suite.
4. **Verdicts** — three independent gates, none hard-fail by default:
- **GPU-to-GPU delta**: an outlier card (throughput/latency beyond
`llm_variance_pct` of the best card) flags the host. Guarded against
noise: no flags when the sample is too small (`compare_min_samples`)
or throughput is below an absolute floor (`compare_min_throughput_req_s`).
- **Reference SLOs**: per-(GPU model, LLM) calibrated gates
(`gpu_benchmark.reference`) override the global chat profile, so a T4
is never judged by GB200 numbers. Null gates render as
"not evaluated (calibration)" — never a fake green.
- **Hardware floors**: ECC errors, throttling, PCIe width/bandwidth,
DCGM failures.
New GPU section in the cloud verification report (HTML + PDF): KPI cards,
per-GPU hardware & health table, cuBLAS spread bar chart, throughput/TTFT
vs load line charts (hand-rolled SVG — weasyprint-safe), per-GPU LLM metric
tables with SLO verdicts, load-sweep tables with ok/incomplete/error counts,
power/efficiency lines, functional suite results, GPU-to-GPU delta table.
Report filenames are namespaced per inventory
(`cloud_verification_report_<inventory>.html`) so parallel validations do
not overwrite each other.
```yaml
validation:
run_gpu_benchmark: true
gpu_benchmark:
models: ["Qwen/Qwen2.5-3B-Instruct"]
gpu_count: all # or N to limit
guidellm_max_duration_s: 40 # raise for slow GPUs / big models
hf_token: "{{ lookup('env', 'HF_TOKEN') | default('', true) }}"
reference:
'NVIDIA L40S':
'Qwen/Qwen2.5-3B-Instruct': {ttft_p95_ms: 200, min_throughput_req_s: 4.5}
See roles/gpu-benchmark/README.md for the calibration procedure and
inventory/reference/group_vars/all.yml for a documented example.
Signed-off-by: rpetrescu-one <rpetrescu@opennebula.io>
… 3-way report, self-healing deploy and measurement-honesty guards Signed-off-by: rpetrescu-one <rpetrescu@opennebula.io>
Signed-off-by: rpetrescu-one <rpetrescu@opennebula.io>
Signed-off-by: rpetrescu-one <rpetrescu@opennebula.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the gpu-benchmark role: an opt-in (validation.run_gpu_benchmark: true) end-to-end validation of GPU-accelerated OpenNebula deployments that measures the same hardware in up to three environments with ONE shared load profile and reports the honest virtualization overhead of each:
Per environment: hardware health (nvidia-smi inventory, cuBLAS SGEMM, PCIe H2D/D2H bandwidth, DCGM diagnostics), LLM inference benchmark (vLLM + GuideLLM sweep, per GPU, TP=1), power/tokens-per-watt, provisioning time, and API conformance suite. The report gets a client-ready section: KPI cards, per-GPU tables and load-sweep charts, GPU-to-GPU comparison, SLO verdicts, and an overhead table with one column per environment (Bare-metal | vLLM appliance | Kubernetes | Overhead % (vm / k8s) | Status naming the breaching environment).