Skip to content

build: auto-fetch prebuilt wheels in install_deps.sh (Hopper), fall back to source - #322

Draft
Andrewxu313 wants to merge 8 commits into
mainfrom
infra/install-deps-autowheel
Draft

build: auto-fetch prebuilt wheels in install_deps.sh (Hopper), fall back to source#322
Andrewxu313 wants to merge 8 commits into
mainfrom
infra/install-deps-autowheel

Conversation

@Andrewxu313

@Andrewxu313 Andrewxu313 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Description

Make the canonical source install fast by default: install_deps.sh now tries matching pre-built wheels (flash-attn 3, FlashMLA, DeepGEMM, batchgen_kernels) from the public GitHub release for the detected (python, GPU-arch) and installs them (~2 min, no compile), and transparently falls back to a full source build if the complete set isn't available for that environment.

  • New try_download_wheels() (Hopper): GitHub-API asset lookup → filter by python-ABI + BUILD_ARCH → curl into $INSTALL_DIR/prebuilt_wheels → all-or-nothing (a partial/missing set falls back to source).
  • install_batchgen_kernels now skips compilation when batchgen_kernels is already importable (e.g. from a wheel) — matches the existing FA3/FlashMLA/DeepGEMM guards.
  • New flags --from-source (force compile) and --release-tag TAG / BATCHGEN_RELEASE_TAG. Existing local --wheel-dir path unchanged.

⚠️ Not yet run on hardware — verified with bash -n + a unit test of the asset filter; needs one Hopper-node run to confirm end-to-end.
⚠️ Companion required to activate the fast path: the release CI must publish the complete wheel set (FA3 + FlashMLA + DeepGEMM + batchgen_kernels) on one release tag. They are currently split across tags, so against today's releases this safely falls back to source until the wheels are co-located.

Motivation

pip-from-PyPI and a per-config wheel matrix are out of scope (see release discussion); the canonical distribution is source install via install_deps.sh. This removes the manual "download wheels from GitHub Releases" step and the 40–60 min compile for users whose env matches the reference build, while keeping source-build as the universal fallback — the same UX model as pip install flash-attn.

Type of Change

  • model — add/extend model support (models/** + registration seam + model kernels only)
  • kernel — add/optimize a compute kernel (batchgen_kernels/** + in-tree kernel dirs)
  • core — change scheduling/serving/runtime scaffolding (the only type that may)
  • fix — narrow bug fix (+ a regression test)
  • infra — build / CI / packaging / scripts / Docker
  • docs — documentation only

File changes

File Δ Note
scripts/install_deps.sh mod auto-download matching prebuilt wheels (Hopper) with source fallback; --from-source/--release-tag; skip batchgen_kernels recompile if already installed

Checklist

@Andrewxu313
build: auto-fetch prebuilt wheels in install_deps.sh (Hopper), fall b…
b24f33d 
…ack to source

install_deps.sh now tries matching pre-built wheels (flash-attn 3, FlashMLA,
DeepGEMM, batchgen_kernels) from the public GitHub release for the detected
Python/GPU-arch and installs them (~2 min, no compile); if the complete set is
not available it transparently falls back to building from source. Adds
--from-source and --release-tag, skips batchgen_kernels recompile when a wheel is
already present, and keeps the local --wheel-dir path intact.
@Andrewxu313 Andrewxu313 mentioned this pull request Sep 11, 2026
Draft
9 tasks
@github-actions github-actions Bot added the ci:run Trigger build + GPU regression on H20 label Sep 12, 2026
@Andrewxu313
build: make install_deps.sh source build robust (drop bc dep, auto-fi…
2f12a0a 
…nd nvcc, warn on CUDA-major mismatch)

check_prerequisites used 'bc' for the Python version check, which breaks on
minimal images without bc (e.g. fresh Ubuntu 22.04 GPU pods) — replaced with a
pure-python comparison. Also auto-add a CUDA toolkit under /usr/local/cuda* to
PATH (preferring 12.x) when nvcc isn't found, and warn loudly when the CUDA
major != 12 (reference is 12.8/torch2.9+cu128) so a mismatch fails clearly
upfront instead of cryptically mid-build. Surfaced validating source install on
a fresh Hopper H20 node.
@Andrewxu313
fix(install): modifier-only flags no longer silently skip the install
a85af1a 
--from-source (and --skip-gpu-check/--wheel-dir/--release-tag/--keep-build)
passed alone left INSTALL_ALL=0 with every target flag 0, so the script built
nothing yet printed 'Installation complete!'. Default to a full install unless
a target flag explicitly narrows it.
@Andrewxu313
build(install): honour CUDA_HOME for nvcc discovery
6f91f35 
Lets users point at a CUDA 12.8 toolkit provisioned to a non-standard prefix,
or override a mismatched system nvcc (e.g. a CUDA 13 fleet image), which the
prior /usr/local-only scan could not reach. Guidance now names CUDA_HOME.
@Andrewxu313
build(install): provision UCX >=1.14 for the core_engine distributed-…
57c6e6d 
…weight daemon

The core_engine JIT (first server launch) compiles Weights_Storage/distributed_weight_daemon.cpp,
which uses the UCP memory-handle API (UCP_OP_ATTR_FIELD_MEMH) added in UCX 1.14. A fresh Ubuntu
node has no UCX, and apt only offers 1.12 (too old) -> the JIT fails with 'ucp/api/ucp.h: No such
file' or the memh compile error, so the server never starts. Build UCX 1.17 from source to
/usr/local when a suitable UCX is absent; skip if UCX >=1.14 already present.
@Andrewxu313
build(kernels): align batchgen_kernels requires-python to >=3.11 (mat…
73efe05 
…ch main package)

Prevents a user landing on Python 3.10, which the main batchgen package rejects and
for which no cp310 prebuilt wheels exist.
@Andrewxu313
build: provide UCX via the libucx-cu12 pip wheel instead of a source …
466a5fe 
…build

The core_engine distributed-weight daemon needs UCX >=1.14 (UCP memh API). Declare
libucx-cu12 (manylinux wheel, ships headers+libs, portable, no compiler/OS-pkg) in
requirements.txt; point the core_engine JIT include/link at the wheel and bake an
rpath so libucp/libuct/libucs/libucm resolve at runtime; preload via libucx.load_library()
in engine_loader as a belt-and-suspenders. Removes install_deps.sh's from-source UCX
build (install_ucx_dev). Falls back to a system UCX if the wheel is absent.
@Andrewxu313
serve: remove three fresh-machine sharp edges (host-kv-cache-size, lo…
e532861 
…cal --model, core_engine warm)

- server_args: fail fast with a clear message when --host-kv-cache-size is omitted
  (was a cryptic 'NoneType * int' crash deep in the worker at startup).
- worker_manager: if --model is a local directory, use it directly as cache_dir
  (skip snapshot_download, which rejected a filesystem path as an HF repo id).
- install_deps: warm the core_engine JIT at install time so the first server launch
  needs no CUDA_HOME / nvcc on PATH (CPU-only compile; non-fatal on failure).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ci:run Trigger build + GPU regression on H20

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Andrewxu313