Ship the CUDA delegate as its own library by shoumikhin · Pull Request #21531 · pytorch/executorch

shoumikhin · 2026-07-31T22:22:05Z

The CUDA delegate runs models on an NVIDIA GPU. Today it is compiled
into whichever component links it, so a C++ application cannot use it without building from
source, and the compatibility layer it needs ends up duplicated across several shipped
libraries.

before:
  the CUDA delegate and its shim are baked into whatever links them
  your C++ app cannot use the GPU without building from source

after:
  executorch/lib/libexecutorch_cuda_backend.so.1     the delegate
  executorch/lib/libexecutorch_extension_cuda.so.1   stream handling it needs
  executorch/backends/cuda/libaoti_cuda_shims.so     the compatibility layer

How you use it

find_package(executorch REQUIRED)
target_link_libraries(my_app PRIVATE executorch::runtime executorch::cuda_backend)

Your program then looks the same as a CPU one. Activation tensors can point at GPU memory and
the delegate handles the rest:

#include <executorch/extension/module/module.h>
#include <executorch/extension/tensor/tensor.h>
using namespace executorch::extension;

Module module("model_cuda.pte");
auto result = module.forward(make_tensor_ptr({1, 3, 224, 224}, gpu_data));

The CUDA runtime itself is not bundled in the wheel. It comes from the environment, the same way
PyTorch does it.

Tested

Built on Linux x86_64 with an H100-class GPU and on aarch64 with a Jetson device, installed into
a clean environment with no source checkout reachable, then ran a real model on the GPU and
compared its output against the CPU result. Confirmed by symbol inspection that exactly one
shipped library defines the delegate and one defines the shim, where before several did.

[ghstack-poisoned]

shoumikhin · 2026-07-31T22:22:06Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2026-07-31T22:22:08Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21531

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 331 Pending

As of commit 5e2b7b6 with merge base 6bbb75a ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 964e8af ghstack-comment-id: 5147959548 Pull-Request: #21531

[ghstack-poisoned]

The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: e5585aa ghstack-comment-id: 5147959548 Pull-Request: #21531

[ghstack-poisoned]

The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 475d170 ghstack-comment-id: 5147959548 Pull-Request: #21531

[ghstack-poisoned]

The CUDA delegate is compiled into whichever component links it, so a C++ application cannot use it without building from source, and the shim layer it depends on ends up duplicated across several shipped libraries. Build the delegate as a shared library and ship it in the wheel, so a process has one copy and both the Python bindings and a C++ application can link the same one. The CUDA runtime itself is not bundled; it continues to come from the environment. Two things were needed to make that actually reduce duplication: The platform helper resolved the ExecuTorch core statically, which would give the delegate its own copy of the backend registry. It now resolves the runtime from the shared runtime library instead. The shim library force-links its shim objects so their symbols survive, which is correct, but it did so as a public link option. That propagated to every consumer, so each one embedded another copy of the same shim code. Making it private keeps the symbols in the library that owns them while consumers resolve against it, which takes a representative shim symbol from three definitions down to one. Registration still happens through a static initializer, and the delegate is retained on the link line so that initializer runs even though no symbol from it is referenced directly. This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the Linux wheel changes; every other build keeps linking static libraries exactly as before. Test plan: The wheel smoke test now asserts that exactly one shipped library defines the CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel, and XNNPACK assertions. Because the delegate is only present in wheels built with CUDA, that assertion skips cleanly when it is absent rather than failing. All three behaviors were confirmed: it passes on a wheel built with this change, it fails on a wheel built before it (correctly reporting three definers by name), and it skips on a wheel built without CUDA. The delegate's own methods are weak symbols, so the assertion checks a strong symbol from the shim layer instead. Built the wheel with CUDA enabled from a clean checkout and verified against a fresh virtual environment with a normal dependency-resolving install: - The wheel ships the delegate as its own versioned library next to the runtime, the thread pool, the CPU kernels, and the XNNPACK delegate. - `nm -DC` across every shipped shared object shows exactly one definition of a representative delegate symbol, down from three. - Only the runtime library defines the backend registry, so the delegate does not introduce a second one. - The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the CUDA delegate loaded. - With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and no new shared object is produced, so every build that does not opt in is unaffected. Note for anyone building with CUDA: the build needs an explicit `CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an integer dot-product intrinsic used by one of the kernels does not compile. That is independent of this change. ghstack-source-id: 5fb1483 ghstack-comment-id: 5147959548 Pull-Request: #21531

[ghstack-poisoned]

Update

1ac4281

[ghstack-poisoned]

shoumikhin requested review from kirklandsign and larryliu0820 as code owners July 31, 2026 22:22

shoumikhin temporarily deployed to cadence July 31, 2026 22:22 — with GitHub Actions Inactive

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 31, 2026

shoumikhin temporarily deployed to cadence July 31, 2026 22:56 — with GitHub Actions Inactive

Update

9415cc5

[ghstack-poisoned]

shoumikhin had a problem deploying to cadence August 1, 2026 00:12 — with GitHub Actions Error

Update

448aec8

[ghstack-poisoned]

shoumikhin temporarily deployed to cadence August 1, 2026 00:28 — with GitHub Actions Inactive

shoumikhin had a problem deploying to cadence August 1, 2026 00:54 — with GitHub Actions Error

Update

c8cb1fc

[ghstack-poisoned]

shoumikhin temporarily deployed to cadence August 1, 2026 01:11 — with GitHub Actions Inactive

shoumikhin temporarily deployed to cadence August 1, 2026 01:42 — with GitHub Actions Inactive

Update

d843d61

[ghstack-poisoned]

shoumikhin added the ciflow/periodic label Aug 1, 2026

shoumikhin added 17 commits August 2, 2026 23:58

Update

c67a766

[ghstack-poisoned]

Update

7196f9e

[ghstack-poisoned]

Update

db6565f

[ghstack-poisoned]

Update

59e538d

[ghstack-poisoned]

Update

57dc7e0

[ghstack-poisoned]

Update

66ca854

[ghstack-poisoned]

Update

ca35116

[ghstack-poisoned]

Update

bbe319d

[ghstack-poisoned]

Update

d19116a

[ghstack-poisoned]

Update

a19256c

[ghstack-poisoned]

Update

9829655

[ghstack-poisoned]

Update

5f4725c

[ghstack-poisoned]

Update

787bcf7

[ghstack-poisoned]

Update

bea599c

[ghstack-poisoned]

Update

f1d7443

[ghstack-poisoned]

Update

41b1113

[ghstack-poisoned]

Update

58f5e85

[ghstack-poisoned]

shoumikhin added 8 commits August 4, 2026 00:28

Update

400a3d0

[ghstack-poisoned]

Update

d8c29dd

[ghstack-poisoned]

Update

a3ea1a7

[ghstack-poisoned]

Update

fb2b031

[ghstack-poisoned]

Update

4312835

[ghstack-poisoned]

Update

91e0d8f

[ghstack-poisoned]

Update

e01fdd6

[ghstack-poisoned]

Update

5e2b7b6

[ghstack-poisoned]

Navigation Menu

Uh oh!

Ship the CUDA delegate as its own library - #21531

Ship the CUDA delegate as its own library#21531
shoumikhin wants to merge 96 commits into
gh/shoumikhin/81/headfrom
gh/shoumikhin/83/head

shoumikhin commented Jul 31, 2026 •

edited

Loading

Uh oh!

shoumikhin commented Jul 31, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shoumikhin commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How you use it

Tested

Uh oh!

shoumikhin commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21531

⏳ No Failures, 331 Pending

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shoumikhin commented Jul 31, 2026 •

edited

Loading

shoumikhin commented Jul 31, 2026 •

edited

Loading

pytorch-bot Bot commented Jul 31, 2026 •

edited

Loading