Skip to content

feat: apply_runtime_settings — permanent runtime config for AOT-loaded engines - #4570

Open
tp5uiuc wants to merge 11 commits into
mainfrom
feat/apply-runtime-settings
Open

feat: apply_runtime_settings — permanent runtime config for AOT-loaded engines#4570
tp5uiuc wants to merge 11 commits into
mainfrom
feat/apply-runtime-settings

Conversation

@tp5uiuc

@tp5uiuc tp5uiuc commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds torch_tensorrt.runtime.apply_runtime_settings(target, settings), a
public API to permanently configure runtime settings on engines loaded via
torch_tensorrt.load() (AOT-loaded artifacts with no TorchTensorRTModule
wrapper).

What — A new entry point that applies a RuntimeSettings object to every
TRT engine reachable from an nn.Module, a list of modules, or an
ExportedProgram. Returns the number of engines updated.

Why — The existing runtime_config and runtime_cache context managers
auto-restore on exit and anchor to TorchTensorRTModule; they cannot reach
engines produced by the AOT save/load path. There was no supported way to
attach a RuntimeCache (or any other setting) to a bare loaded artifact.

How

  • py/torch_tensorrt/runtime/_runtime_config.py: add _is_trt_engine,
    _iter_trt_engines, _send_settings_to_engine, and apply_runtime_settings;
    update both context managers to use _iter_trt_engines and raise TypeError
    on module-less engines (naming apply_runtime_settings in the message).
  • py/torch_tensorrt/dynamo/runtime/_TorchTensorRTModule.py: reduce
    _send_to_engine to a one-line delegate to _send_settings_to_engine.
  • py/torch_tensorrt/runtime/__init__.py: export apply_runtime_settings.
  • docsrc/py_api/runtime.rst + docsrc/user_guide/runtime_performance/runtime_settings.rst:
    add API entry and AOT section with example and caller-ownership rules.
  • tests/py/dynamo/runtime/test_005_apply_runtime_settings.py: new test file
    covering type errors, module-owned engines, module-less engines, mixed targets,
    CM raises, and warm-load byte-count verification.
  • tests/py/dynamo/runtime/test_{000,001,004}_*.py,
    tests/py/dynamo/models/test_*_models.py: replace local _apply_runtime_settings
    helpers with the public API.

Key constraints applied: string runtime_cache paths are rejected for
module-less engines (caller owns the handle and must call cache.load() before
applying); ExportedProgram supported via ep.constants; apply is
validate-before-mutate (all-or-nothing across a mixed target list).

Testing

Full tests/py/dynamo/runtime/ suite and the three model test files pass on
both the Python-only and C++ runtime builds. New test file adds 11 tests; all
green.

Note: this PR stacks on #4482 (fix/python-runtime-cache-on-cpp-build);
those commits appear in the diff until #4482 merges.

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes

@meta-cla meta-cla Bot added the cla signed label Aug 24, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation component: tests Issues re: Tests component: core Issues re: The core compiler component: api [Python] Issues re: Python API component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Aug 24, 2026
@github-actions
github-actions Bot requested a review from narendasan August 24, 2026 20:36
tp5uiuc and others added 6 commits August 28, 2026 10:40
Free _send_settings_to_engine(engine, rs) in _runtime_config.py;
TorchTensorRTModule._send_to_engine becomes a one-line delegate.
Pure refactor, zero behaviour change.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add apply_runtime_settings(target, settings) -> int that walks TRT
engines via get_attr constants (AOT-loaded) and TorchTensorRTModule
(in-process). Accepts nn.Module, ExportedProgram, or a sequence.
Raises RuntimeError on zero engines; raises TypeError when
runtime_cache is a path string and any engine is module-less (no
owner to persist the handle). Returns engine count.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t restore

runtime_config and runtime_cache now use _iter_trt_engines for
discovery. On encountering a module-less engine (e.g. from an
AOT-loaded artifact) they raise TypeError naming
apply_runtime_settings, before mutating any engine state. Prior
behaviour was a silent no-op (runtime_config) or an unhelpful
module-only error (runtime_cache).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ngines

New test_005_apply_runtime_settings.py: module-less engine cache
round-trip, path-string rejection, cuda_graph_strategy dispatch,
ExportedProgram reaching shared engine objects, mixed-target atomic
failure, and CM raise with apply_runtime_settings in the message.
All AOT tests are shown failing on base by ImportError (function does
not exist) plus assertions base cannot satisfy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…c API

Replace the local _apply_runtime_settings helper in 7 test files
(31 call sites) with the public apply_runtime_settings. The local
bodies were semantically identical (walk named_modules, set
mod.runtime_settings = rs); the public entry point is a strict
superset. Last commit so reviewers can skip the mechanical diff.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…le-less engines

The module path auto-calls RuntimeCache.load via _resolve_runtime_cache,
warming the implicit handle from disk before first use.  There is no
equivalent hook on the module-less path -- the caller owns the handle
and must warm it.  This asymmetry was intentional but undocumented.

- apply_runtime_settings docstring: state that caller owns .load() as
  well as .save() on the module-less path, and explain why the module
  path is different (_resolve_runtime_cache warm-load).
- runtime_settings.rst: add ownership paragraph to the AOT subsection.
- test_005_apply_runtime_settings.py: add test_warm_load_bytes_transferred
  to TestApplyRuntimeSettingsModuleLess; writes a cache file, warms a new
  RuntimeCache via load_from_stream, attaches via apply_runtime_settings,
  and asserts load_from_stream returned > 0 bytes (non-vacuous).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tp5uiuc
tp5uiuc force-pushed the feat/apply-runtime-settings branch from e506f69 to 7dd89af Compare August 28, 2026 17:42
@tp5uiuc
tp5uiuc marked this pull request as ready for review August 28, 2026 17:43
@tp5uiuc
tp5uiuc marked this pull request as draft August 28, 2026 17:44
tp5uiuc and others added 5 commits August 28, 2026 11:03
_send_to_engine called _send_settings_to_engine verbatim; the call site
now imports and calls _send_settings_to_engine directly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n API

Direct assignment (mod.runtime_settings = rs) was listed as a separate
"way" to apply settings; it is now an implementation detail that
apply_runtime_settings calls internally for module-owned engines.

- runtime_settings.rst: collapse four ways -> three ways; remove the
  Direct assignment subsection; broaden the apply_runtime_settings
  section to cover both compiled and AOT-loaded targets; update the
  Advanced, Best practices, and Quick reference sections to use
  apply_runtime_settings throughout; replace "Setter is per-module"
  footgun with a positive note about automatic subgraph walking.
- _runtime_config.py module docstring: four concepts -> three; remove
  the Programmatic bullet from the usage list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…er caller-owned cache

The section now flows: compiled module example -> context managers ->
caller-owned RuntimeCache -> AOT/ExportedProgram. The ExportedProgram
and module-less ownership rules are deferred until after the caller-owned
RuntimeCache concept is introduced, which they build on.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_iter_trt_engines

- test_005_apply_runtime_settings.py removed; all tests moved into
  test_004_runtime_settings.py where they belong alongside the rest of
  the RuntimeSettings test suite. Added _compile_with_inputs() and
  _save_load() helpers for the save/load path.
- _iter_trt_engines: add _visit_one compositor to eliminate the
  duplicated EP/Module dispatch in the sequence branch; replace
  hasattr(__iter__) guard with isinstance(Iterable) + positive
  scalar-type check; remove redundant list() conversion on self._targets.
- Import Iterable from typing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The list was materialized only to test truthiness and format a count
into the error message. any() short-circuits on the first None owner
without a second pass over the already-drained engines list. Also
removes the redundant list() wrapping of self._targets in
_RuntimeCacheContextManager.__enter__.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tp5uiuc
tp5uiuc marked this pull request as ready for review August 29, 2026 02:08
@tp5uiuc tp5uiuc self-assigned this Aug 29, 2026
@tp5uiuc
tp5uiuc requested a review from cehongwang August 29, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend: TensorRT-RTX ci: full cla signed component: api [Python] Issues re: Python API component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime component: tests Issues re: Tests documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant