feat: apply_runtime_settings — permanent runtime config for AOT-loaded engines - #4570
Open
tp5uiuc wants to merge 11 commits into
Open
feat: apply_runtime_settings — permanent runtime config for AOT-loaded engines#4570tp5uiuc wants to merge 11 commits into
tp5uiuc wants to merge 11 commits into
Conversation
Free _send_settings_to_engine(engine, rs) in _runtime_config.py; TorchTensorRTModule._send_to_engine becomes a one-line delegate. Pure refactor, zero behaviour change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add apply_runtime_settings(target, settings) -> int that walks TRT engines via get_attr constants (AOT-loaded) and TorchTensorRTModule (in-process). Accepts nn.Module, ExportedProgram, or a sequence. Raises RuntimeError on zero engines; raises TypeError when runtime_cache is a path string and any engine is module-less (no owner to persist the handle). Returns engine count. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t restore runtime_config and runtime_cache now use _iter_trt_engines for discovery. On encountering a module-less engine (e.g. from an AOT-loaded artifact) they raise TypeError naming apply_runtime_settings, before mutating any engine state. Prior behaviour was a silent no-op (runtime_config) or an unhelpful module-only error (runtime_cache). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ngines New test_005_apply_runtime_settings.py: module-less engine cache round-trip, path-string rejection, cuda_graph_strategy dispatch, ExportedProgram reaching shared engine objects, mixed-target atomic failure, and CM raise with apply_runtime_settings in the message. All AOT tests are shown failing on base by ImportError (function does not exist) plus assertions base cannot satisfy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…c API Replace the local _apply_runtime_settings helper in 7 test files (31 call sites) with the public apply_runtime_settings. The local bodies were semantically identical (walk named_modules, set mod.runtime_settings = rs); the public entry point is a strict superset. Last commit so reviewers can skip the mechanical diff. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…le-less engines The module path auto-calls RuntimeCache.load via _resolve_runtime_cache, warming the implicit handle from disk before first use. There is no equivalent hook on the module-less path -- the caller owns the handle and must warm it. This asymmetry was intentional but undocumented. - apply_runtime_settings docstring: state that caller owns .load() as well as .save() on the module-less path, and explain why the module path is different (_resolve_runtime_cache warm-load). - runtime_settings.rst: add ownership paragraph to the AOT subsection. - test_005_apply_runtime_settings.py: add test_warm_load_bytes_transferred to TestApplyRuntimeSettingsModuleLess; writes a cache file, warms a new RuntimeCache via load_from_stream, attaches via apply_runtime_settings, and asserts load_from_stream returned > 0 bytes (non-vacuous). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tp5uiuc
force-pushed
the
feat/apply-runtime-settings
branch
from
August 28, 2026 17:42
e506f69 to
7dd89af
Compare
tp5uiuc
marked this pull request as ready for review
August 28, 2026 17:43
tp5uiuc
marked this pull request as draft
August 28, 2026 17:44
_send_to_engine called _send_settings_to_engine verbatim; the call site now imports and calls _send_settings_to_engine directly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n API Direct assignment (mod.runtime_settings = rs) was listed as a separate "way" to apply settings; it is now an implementation detail that apply_runtime_settings calls internally for module-owned engines. - runtime_settings.rst: collapse four ways -> three ways; remove the Direct assignment subsection; broaden the apply_runtime_settings section to cover both compiled and AOT-loaded targets; update the Advanced, Best practices, and Quick reference sections to use apply_runtime_settings throughout; replace "Setter is per-module" footgun with a positive note about automatic subgraph walking. - _runtime_config.py module docstring: four concepts -> three; remove the Programmatic bullet from the usage list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…er caller-owned cache The section now flows: compiled module example -> context managers -> caller-owned RuntimeCache -> AOT/ExportedProgram. The ExportedProgram and module-less ownership rules are deferred until after the caller-owned RuntimeCache concept is introduced, which they build on. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_iter_trt_engines - test_005_apply_runtime_settings.py removed; all tests moved into test_004_runtime_settings.py where they belong alongside the rest of the RuntimeSettings test suite. Added _compile_with_inputs() and _save_load() helpers for the save/load path. - _iter_trt_engines: add _visit_one compositor to eliminate the duplicated EP/Module dispatch in the sequence branch; replace hasattr(__iter__) guard with isinstance(Iterable) + positive scalar-type check; remove redundant list() conversion on self._targets. - Import Iterable from typing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The list was materialized only to test truthiness and format a count into the error message. any() short-circuits on the first None owner without a second pass over the already-drained engines list. Also removes the redundant list() wrapping of self._targets in _RuntimeCacheContextManager.__enter__. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
torch_tensorrt.runtime.apply_runtime_settings(target, settings), apublic API to permanently configure runtime settings on engines loaded via
torch_tensorrt.load()(AOT-loaded artifacts with noTorchTensorRTModulewrapper).
What — A new entry point that applies a
RuntimeSettingsobject to everyTRT engine reachable from an
nn.Module, a list of modules, or anExportedProgram. Returns the number of engines updated.Why — The existing
runtime_configandruntime_cachecontext managersauto-restore on exit and anchor to
TorchTensorRTModule; they cannot reachengines produced by the AOT save/load path. There was no supported way to
attach a
RuntimeCache(or any other setting) to a bare loaded artifact.How
py/torch_tensorrt/runtime/_runtime_config.py: add_is_trt_engine,_iter_trt_engines,_send_settings_to_engine, andapply_runtime_settings;update both context managers to use
_iter_trt_enginesand raiseTypeErroron module-less engines (naming
apply_runtime_settingsin the message).py/torch_tensorrt/dynamo/runtime/_TorchTensorRTModule.py: reduce_send_to_engineto a one-line delegate to_send_settings_to_engine.py/torch_tensorrt/runtime/__init__.py: exportapply_runtime_settings.docsrc/py_api/runtime.rst+docsrc/user_guide/runtime_performance/runtime_settings.rst:add API entry and AOT section with example and caller-ownership rules.
tests/py/dynamo/runtime/test_005_apply_runtime_settings.py: new test filecovering type errors, module-owned engines, module-less engines, mixed targets,
CM raises, and warm-load byte-count verification.
tests/py/dynamo/runtime/test_{000,001,004}_*.py,tests/py/dynamo/models/test_*_models.py: replace local_apply_runtime_settingshelpers with the public API.
Key constraints applied: string
runtime_cachepaths are rejected formodule-less engines (caller owns the handle and must call
cache.load()beforeapplying);
ExportedProgramsupported viaep.constants; apply isvalidate-before-mutate (all-or-nothing across a mixed target list).
Testing
Full
tests/py/dynamo/runtime/suite and the three model test files pass onboth the Python-only and C++ runtime builds. New test file adds 11 tests; all
green.
Note: this PR stacks on #4482 (
fix/python-runtime-cache-on-cpp-build);those commits appear in the diff until #4482 merges.
Type of change
Checklist: