Ship only the TensorRT delegate in the ExecuTorch runtime wheel

[shoumikhin]

What this does

The torch-tensorrt-executorch-runtime wheel now ships one thing: the TensorRT backend, as a prebuilt shared library. Nothing else.

The goal is that it looks like one of the backends ExecuTorch already ships in its own wheel, such as the CUDA backend, only built and distributed separately. Same file naming, same layout, same linking rules. A Python user registers it by importing the package. A C++ app links it through CMake.

The problem

Two copies of the same runtime in one process

The old wheel carried its own libexecutorch.so and its own copy of ExecuTorch's Python extension, then took over executorch.extension.pybindings at import time.

That works only while both copies match. Two C++ runtimes in one process can also pull in two different libstdc++ versions, which breaks in ways that are hard to read. It also meant the wheel had to get in first, before anything imported ExecuTorch, or the wrong copy won.

An inference API that already existed

The wheel shipped a runtime.py with a Program class and a load() function. ExecuTorch already exports Runtime, Program and Method. Most of runtime.py was the same code written again, down to the line that keeps the file buffer alive so the program is not freed.

One part was worse than redundant. Program.run() copied a CUDA input to CPU every time. If you exported a program that keeps its inputs on the GPU, that copy quietly undid it, and you could not get a device-resident input through the wrapper at all.

No way to use it from C++

The shared library was reachable only from Python. ExecuTorch ships each of its own backends as a prebuilt library plus a CMake package, so a C++ app writes find_package and links a target. This wheel shipped no CMake package, so the only way to get the delegate into a C++ program was to build this repository from source.

The fix

The wheel is a backend library and its loader

torch_tensorrt_executorch_runtime/
  lib/libexecutorch_backend_tensorrt.so                     the backend
  __init__.py                                               loads it on import
  lib/cmake/torchtrt_executorch/*.cmake                     lets C++ link it

This mirrors what ExecuTorch does with its own backends:

executorch/                                    torch_tensorrt_executorch_runtime/
  lib/libexecutorch_backend_cuda.so              lib/libexecutorch_backend_tensorrt.so
  lib/cmake/executorch/executorch-config.cmake   lib/cmake/torchtrt_executorch/torchtrt_executorch-config.cmake

lib/cmake/<name>/ is where find_package looks under a prefix, and it is where
ExecuTorch's own package resolves from, so a consumer points CMAKE_PREFIX_PATH at the
two package roots and both are found the same way.

The library also matches theirs where it counts: its SONAME is its own filename, it has DT_RUNPATH and no DT_RPATH, it has no PyInit_ because it is not a Python extension, and it imports register_backend rather than defining it. It links libexecutorch.so from the installed executorch wheel instead of bringing its own, so there is one runtime in the process.

One thing differs on purpose. ExecuTorch's backend sits in the same wheel as libexecutorch.so, so its search path reaches it with one ../. This one sits in a different wheel, so it has to climb out to site-packages and back down, which takes two:

$ORIGIN
$ORIGIN/../../executorch/lib
$ORIGIN/../../tensorrt_libs
$ORIGIN/../../nvidia/cu13/lib

Python: importing the package registers the backend

There is no API to call:

import torch_tensorrt_executorch_runtime  # noqa: F401
from executorch.runtime import Runtime

program = Runtime.get().load_program("model.pte")
outputs = program.load_method("forward").execute((tensor,))

ExecuTorch's own backends register because they are linked into its Python extension, so loading that extension pulls them in and their static initializers run. A backend in a separate wheel cannot join that link, and ExecuTorch has no discovery hook for out-of-tree backends, so this package performs the equivalent step itself at import time.

The library exports no PyInit_, so a plain import cannot load it; something has to dlopen it. That is all register() does, and it runs once on import.

If the load fails, the import raises with the real cause, for example a CPU-only executorch wheel or an ABI mismatch. Failing loudly is on purpose: this wheel exists only to register the backend, so a load it cannot finish leaves nothing useful behind, and ExecuTorch's later "backend not available" cannot name the cause. Set TORCH_TENSORRT_SKIP_DELEGATE_REGISTRATION=1 to import without the side effect, for tooling that only wants the metadata.

C++: link it the way you link an ExecuTorch backend

find_package(executorch REQUIRED COMPONENTS backend_cuda)
find_package(torchtrt_executorch REQUIRED)
target_link_libraries(my_app PRIVATE
  executorch::runtime executorch::backend_cuda executorch::extension_cuda
  executorch::kernels_optimized torchtrt::executorch_backend)

Point CMake at both wheels, since they are separate packages, and use CMake 3.28 or newer because the backend_cuda component requires it:

cmake -DCMAKE_PREFIX_PATH="<site-packages>/executorch;<site-packages>/torch_tensorrt_executorch_runtime" ...

There is nothing to include. The backend has no public header: it registers itself from a static initializer inside the shared library, and everything after that is ExecuTorch's own runtime API. The CMake target links the library with --no-as-needed, so the dependency survives even if the consumer never names a symbol from it.

Loading a .pte is ExecuTorch's job

torch_tensorrt.load(path, format="executorch") is removed, along with the format argument. This matches how the other save formats already work: Torch-TensorRT saves the file, and the framework that owns the runtime loads it.

Format	Saved by	Loaded by
.pt2 (AOTInductor)	torch_tensorrt.save	torch._inductor.aoti_load_package
.pte (ExecuTorch)	torch_tensorrt.save	executorch.runtime.Runtime

output_format="executorch" on torch_tensorrt.save is unchanged. Only the load side moved.

Device-resident inputs now work

With the copy in Program.run() gone, nothing in the Python layer touches your tensors, so a program exported to keep its inputs and outputs on the GPU keeps them there. Two settings are needed for that export, not one:

ExecutorchBackendConfig(
    propagate_device_config=PropagateDeviceConfig(
        skip_h2d_for_method_inputs=True,
        skip_d2h_for_method_outputs=True,
    ),
    enable_non_cpu_memory_planning=True,
    memory_planning_pass=MemoryPlanningPass(
        alloc_graph_input=False, alloc_graph_output=False
    ),
)

Skipping the copy is not enough on its own. Memory planning allocates graph inputs and outputs by default, so the runtime would still reserve its own buffer and fill it from your memory with a host copy, which puts the copy back.

examples/torchtrt_executorch_example/export_device_resident.py exports such a program and checks the result rather than trusting the flags: it reads the operator table of the saved file and fails if either boundary copy operator is still there, and it checks that every method input and output is recorded as a CUDA tensor.

Breaking changes

These names are gone from the runtime package:

• runtime.py, including Program and load(). Use executorch.runtime.Runtime.
• get_runtime(). Import the package, then use Runtime.get().
• activate(), which is now register() and is called for you on import.
• torch_tensorrt.load(..., format="executorch"). The format argument no longer exists; passing a value raises TypeError naming the replacement. Passing None, the old default, still works.
• torch_tensorrt_executorch_runtime._portable_lib and .data_loader, because the wheel no longer ships them.

The delegate also now needs a CUDA build of the executorch wheel at runtime, not only at build time, because it links a library that only the CUDA wheels ship. With a CPU-only executorch installed, the import fails with a message that says so.

Test plan

Run on Linux x86_64 with CUDA 13 and an NVIDIA H100, against the wheel this change builds in CI:

• Importing the package registers TensorRTBackend, with nothing else called.
• Python, four combinations, all pass: TensorRT-only and coalesced TensorRT plus CUDA, each with CPU inputs and with device-resident inputs. Shapes and devices are read from the program rather than hardcoded.
• C++, linking the wheel through CMake with no source checkout: the backend registers and a TensorRT-delegated program runs.
• As a control, the CPU-boundary program was checked to contain et_copy::_h2d_copy and et_copy::_d2h_copy, the two operators the device-resident program is asserted not to have. Without that check the assertion could pass because those operators never appear.

Not covered: running a device-resident program from C++. That program requires the caller to own both the input and the output buffers in device memory, and the C++ example here supplies host buffers.

CI builds the wheel, checks its contents and its search paths, runs the C++ reference runner against three saved programs, and runs the Python runner. Unit tests: 24 in test_python_runtime.py, plus the pin and updater suites.