fix: engines default to no runtime cache; the module owns the implicit one by tp5uiuc · Pull Request #4482 · pytorch/TensorRT

tp5uiuc · 2026-08-13T00:42:11Z

Description

What

An engine built without a TorchTensorRTModule came up with a path-string
runtime_cache and attached a cache nothing owned, freeing it out from under
the engine. Engines now default to no cache; the module owns the implicit one.

Why

RuntimeSettings.runtime_cache defaults to a path string, and
TorchTensorRTModule resolves it into a RuntimeCache it owns, so the compile
path is fine. An engine reached without a module -- built from packed engine
info, or loaded as a graph constant from a saved ExportedProgram -- never
passes through that resolver, so TRTRuntimeConfig wrapped the string in a
local, handed the live IRuntimeCache to the engine's IRuntimeConfig, and let
it be collected on return. The next createExecutionContext read freed memory.

The C++ runtime has no such path: RuntimeSettings::runtime_cache is an
intrusive_ptr defaulting to nullptr, with no string form to mis-own.

The module's post-load resets have to move with the engine, or the two disagree:
a module holding a stale path string lets a later runtime_config(...) block --
a call that need not mention caching -- resolve it and leave an autosaving handle
installed at the shared default path on exit.

How

_TRTEngine.py: engines construct TRTRuntimeConfig with
runtime_cache=None, matching the C++ default. RuntimeSettings()'s own
default is unchanged -- that one belongs to the module, which pushes the
resolved handle down in setup_engine.
_TorchTensorRTModule.py: the set_extra_state / __setstate__ resets drop
to runtime_cache=None so the module agrees with the engine it rebuilds; an
empty-string runtime_cache normalizes to None so no string reaches an
engine.
_runtime_config.py: _apply_settings accepts only None or a
RuntimeCache and raises TypeError on a string; TRTRuntimeConfig's own
no-settings default follows suit. The docstring already claimed raw strings
were not accepted here.
Tests: module-less engines default to no cache, execute without one, accept an
explicitly attached RuntimeCache, and reject strings; the compile path still
builds and persists its implicit cache; the post-load resets leave no cache,
and a cuda-graph-only context manager over a loaded module installs none on
enter and leaves none on exit.

Testing

TensorRT-RTX on A100, tests/py/dynamo/runtime/ plus
models/test_runtime_cache_models.py:

Build	before	after
Python runtime	4 failed, 170 passed, 60 skipped	174 passed, 60 skipped
C++ runtime	4 failed, 171 passed, 59 skipped	175 passed, 59 skipped

The four failures before are the new tests themselves; each was confirmed
failing on the parent commit and passing here, on both builds. There are no
other failures on either build.

models/test_export_serde.py, test_cross_runtime_serde.py and
test_fallback_data_dependent_ops.py were run before and after on both builds
because the post-load resets sit on the torch.load / load_state_dict paths:
identical results either side, with the only failures (test_save_load_aoti,
test_save_load_ts) pre-existing and unrelated.

Save/load deployment was checked on both runtimes: the loaded engine runs with
no cache attached, and an explicitly attached RuntimeCache persists.

Cost / Gotchas

An engine used without a module now gets no runtime cache instead of an implicit
one, and neither do modules restored by torch.load / load_state_dict. No
configuration loses working behaviour: that path raised on the Python runtime,
already attached nothing on the C++ runtime, and never initializes a runtime
config on standard TensorRT. Callers wanting a cache there attach a
RuntimeCache explicitly.

Supersedes #4541 -- the TestPythonRuntimeAliasedIO failures it skipped are
fixed here rather than skipped.

Followups

runtime_config() / runtime_cache() locate engines via named_modules(), so
neither reaches an engine loaded as a graph constant. Attaching settings to a
saved-and-loaded program currently requires the engine API directly.

Restoring implicit caching to load_state_dict -- by re-applying the setter
after engine construction in set_extra_state, as setup_engine does -- is
deliberately left out; it changes C++-runtime behaviour too and wants its own
testing.

Type of change

Bug fix (non-breaking change which fixes an issue)

Checklist:

My code follows the style guidelines of this project (You can use the linters)
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas and hacks
I have made corresponding changes to the documentation
I have added tests to verify my fix or my feature
New and existing unit tests pass locally with my changes
I have added the relevant labels to my PR in so that relevant reviewers are notified

A TRTEngine built without a module came up with the default RuntimeSettings, whose runtime_cache is a path string. Nothing owned the wrapper that string implies, so attaching it handed the live IRuntimeCache to the engine's IRuntimeConfig and then let it be collected -- a use-after-free at the next createExecutionContext. Engines now default to runtime_cache=None, matching the cpp side, where RuntimeSettings::runtime_cache is an intrusive_ptr defaulting to nullptr and no string form exists. The implicit cache belongs to the module, which resolves its path string to a RuntimeCache and pushes it down in setup_engine; that path is unchanged. TorchTensorRTModule resets its own RuntimeSettings on the post-load paths (set_extra_state, __setstate__) and those resets move to runtime_cache=None too, so the module and the engine it rebuilds agree. Leaving them at the string default would let a later runtime_config(...) block resolve the stale path and install an autosaving handle at the shared default location on exit -- switching caching on via a call that never mentioned it. An engine reached without a module -- built from packed engine info, or loaded as a graph constant from a saved ExportedProgram -- now runs with no cache instead of a dangling one, and a caller can attach a RuntimeCache explicitly. No configuration loses working behaviour: on the Python runtime this path raised, on the cpp runtime it already attached nothing, and on standard TensorRT the runtime config is never initialized.

_apply_settings had three arms, and the str one built a RuntimeCache it did not outlive. Its own docstring already claimed raw strings were not accepted here; the code twenty lines below accepted them. Engines now take only something that owns what it points at -- the Python equivalent of the cpp intrusive_ptr<RuntimeCacheHandle>. A str raises TypeError naming the module as the place path strings are resolved. The class's own default follows: a TRTRuntimeConfig built with no settings would otherwise start from the string form this commit exists to abolish. TorchTensorRTModule._resolve_runtime_cache normalizes an empty-string runtime_cache to None rather than passing it through, so no str can reach an engine from the module. Also corrects the RuntimeSettings.runtime_cache docstring, which promised the engine owned the implicit handle and saved it on __del__, and a reference to a method renamed some time ago.

TestEngineOwnsNoCache pins the contract at the engine: a module-less engine defaults to no cache, executes without one, accepts an explicitly attached RuntimeCache, and raises TypeError on a path string. TestModuleStillOwnsImplicitCache guards the other direction -- the compile path must keep building, attaching and persisting its implicit cache -- and covers the empty-string normalization. TestPostLoadOwnsNoCache pins the same contract on the reset paths, where module and engine could otherwise disagree: the config's own default, and what torch.load / load_state_dict leave behind. Its last test is the one that matters -- a cuda-graph-only context manager over a loaded module must not install a cache on enter or leave one installed on exit, because re-applying a path string through the setter creates a handle rather than restoring one.

cehongwang

Overall OK

cehongwang · 2026-08-25T21:23:44Z

-                logger.warning(f"Failed to warm-load runtime cache from {rc!r}: {e}")
-            cache = wrapped.ensure_cache(self._live)
-            self._live.set_runtime_cache(cache)
+            self._live.set_runtime_cache(rc.ensure_cache(self._live))
        else:
            raise TypeError(


We probably want to destroy the self._live here because the initialization was not successful?

Good catch, done in cb95af3 by resetting the live and propagating the exception up the call stack. Thanks Adrian!

ensure_initialized assigns self._live before _apply_settings runs, so an exception out of the apply left a half-configured IRuntimeConfig in place -- strategies set, runtime cache never attached. The early-return guard at the top then made the next call a no-op, so a caller who caught the error and retried proceeded silently against those partial settings; the error was only ever raised once. Reset self._live and re-raise instead, so a retry re-attempts initialization and fails the same way. Measured before the change: first execute raised TypeError, second returned normally with _live still populated.

tp5uiuc · 2026-08-28T02:39:50Z

[by Claude Code] CI failures on this PR are pre-existing, not introduced here.

Failure	Lanes	Why it isn't from this PR
`test_bitwise_{and,or}_aten` (4)	standard-converters	Same 4 fail on #4601 (unrelated)
`test_aten_lowering_passes::TestComplexSubgraph`	rtx + standard lowering	Same on #4601
`test_decompositions::test_lowering_baddbmm`	rtx-lowering	Decomposition path, untouched here
`test_model_refit::complex_buffer` (3)	rtx + standard models	Refit's `load_state_dict` calls are guarded to non-TRT submodules (`_refit.py:353,363`), so they never reach `set_extra_state`
`test_export_serde::test_save_load_aoti`	rtx + standard models	Measured identical before/after locally, both runtimes
`ts-api::test_fallback_mobilenet_v2`	ts-api	TorchScript path, untouched
`gate` (2)	—	Roll-up only: "a Windows channel failed"

The base commit (57908060a1) has 89 failing checks of its own, so this branch
inherits a red base.

Verified locally on TensorRT-RTX/A100 — tests/py/dynamo/runtime/ plus
models/test_runtime_cache_models.py: 175 passed, 60 skipped, 0 failed on both
the Python and C++ runtimes. The serde suites (test_export_serde,
test_cross_runtime_serde, test_fallback_data_dependent_ops) were run before and
after on both runtimes, since this touches the torch.load / load_state_dict
paths — identical results either side.

lanluo-nvidia · 2026-08-28T08:59:49Z

@tp5uiuc
whether this fix will fix the following error in ci:
https://github.com/pytorch/TensorRT/actions/runs/33145264851/job/98790133431

…t one (#4482)

meta-cla Bot added the cla signed label Aug 13, 2026

github-actions Bot added component: tests Issues re: Tests component: api [Python] Issues re: Python API labels Aug 13, 2026

github-actions Bot requested a review from narendasan August 13, 2026 00:42

tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from c3c4cf7 to 03513c2 Compare August 13, 2026 01:36

tp5uiuc changed the title ~~fix: python engine crashed attaching a runtime cache on builds with the C++ runtime~~ fix: path-string runtime cache freed itself out from under the engine Aug 13, 2026

tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from 03513c2 to 5522dcd Compare August 13, 2026 01:48

tp5uiuc added this to the v2.15.0 milestone Aug 18, 2026

tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch 2 times, most recently from 3a8a16e to f42577c Compare August 21, 2026 00:50

tp5uiuc self-assigned this Aug 21, 2026

tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from f42577c to c31831c Compare August 21, 2026 07:51

github-actions Bot added component: core Issues re: The core compiler component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Aug 21, 2026

tp5uiuc changed the title ~~fix: path-string runtime cache freed itself out from under the engine~~ fix: engines default to no runtime cache; the module owns the implicit one Aug 21, 2026

tp5uiuc mentioned this pull request Aug 21, 2026

tests: skip TestPythonRuntimeAliasedIO on TensorRT-RTX with the C++ runtime #4541

Closed

7 tasks

tp5uiuc marked this pull request as ready for review August 21, 2026 08:03

tp5uiuc added 3 commits August 24, 2026 12:36

tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from c31831c to 7fffe7c Compare August 24, 2026 20:32

tp5uiuc mentioned this pull request Aug 24, 2026

feat: apply_runtime_settings — permanent runtime config for AOT-loaded engines #4570

Open

6 tasks

narendasan requested a review from cehongwang August 25, 2026 17:30

cehongwang approved these changes Aug 25, 2026

View reviewed changes

tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from 509cfbd to cb95af3 Compare August 26, 2026 02:16

lanluo-nvidia added the ci: full label Aug 27, 2026

github-actions Bot requested a review from zewenli98 August 27, 2026 20:13

lanluo-nvidia merged commit 6d26eab into pytorch:main Aug 28, 2026
178 of 194 checks passed

lanluo-nvidia pushed a commit that referenced this pull request Aug 28, 2026

fix: engines default to no runtime cache; the module owns the implici…

67636f5

…t one (#4482)

Navigation Menu

Uh oh!

fix: engines default to no runtime cache; the module owns the implicit one - #4482

fix: engines default to no runtime cache; the module owns the implicit one#4482
lanluo-nvidia merged 4 commits into
pytorch:mainfrom
tp5uiuc:fix/python-runtime-cache-on-cpp-build

tp5uiuc commented Aug 13, 2026 •

edited

Loading

Uh oh!

cehongwang left a comment

Uh oh!

cehongwang Aug 25, 2026

Uh oh!

tp5uiuc Aug 26, 2026

Uh oh!

tp5uiuc commented Aug 28, 2026

Uh oh!

lanluo-nvidia commented Aug 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

tp5uiuc commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What

Why

How

Testing

Cost / Gotchas

Followups

Type of change

Checklist:

Uh oh!

cehongwang left a comment

Choose a reason for hiding this comment

Uh oh!

cehongwang Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

tp5uiuc Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

tp5uiuc commented Aug 28, 2026

Uh oh!

lanluo-nvidia commented Aug 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tp5uiuc commented Aug 13, 2026 •

edited

Loading