Skip to content

XNNPACK: Add optional performance stage to pytest/tester #21549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mansnils wants to merge 1 commit into
base: main
Choose a base branch
from
Open

XNNPACK: Add optional performance stage to pytest/tester #21549

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions backends/xnnpack/test/models/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# XNNPACK Model Tests

## Optional Pytest Perf Stage

Model tests can opt in to an optional latency check after correctness
comparison:

```python
.run_method_and_compare_outputs(xnnpack_perf=True)
```

The perf stage is skipped by default. Enable it only for runs where local
latency comparison is useful.

### Commands

Run the normal correctness test with no perf stage:

```bash
python3 -m pytest -q \
backends/xnnpack/test/models/resnet.py::TestResNet18::test_fp32_resnet18
```

Run correctness plus perf comparison against the recorded result:

```bash
EXECUTORCH_XNNPACK_PYTEST_PERF=1 \
python3 -m pytest -q \
backends/xnnpack/test/models/resnet.py::TestResNet18::test_fp32_resnet18
```

Record or update the local perf result:

```bash
EXECUTORCH_XNNPACK_PYTEST_PERF=1 \
EXECUTORCH_XNNPACK_PYTEST_PERF_UPDATE=1 \
python3 -m pytest -q \
backends/xnnpack/test/models/resnet.py::TestResNet18::test_fp32_resnet18
```

Use more samples for a less noisy result:

```bash
EXECUTORCH_XNNPACK_PYTEST_PERF=1 \
EXECUTORCH_XNNPACK_PYTEST_PERF_RUNS=50 \
EXECUTORCH_XNNPACK_PYTEST_PERF_WARMUP_RUNS=10 \
python3 -m pytest -q \
backends/xnnpack/test/models/resnet.py::TestResNet18::test_fp32_resnet18
```

### Environment Variables

| Variable | Default | Meaning |
| ---------------------------------------------- | ------: | ------------------------------------------------ |
| `EXECUTORCH_XNNPACK_PYTEST_PERF` | off | Enables the optional perf stage. |
| `EXECUTORCH_XNNPACK_PYTEST_PERF_UPDATE` | off | Records the current result instead of comparing. |
| `EXECUTORCH_XNNPACK_PYTEST_PERF_RUNS` | `10` | Number of timed runs. |
| `EXECUTORCH_XNNPACK_PYTEST_PERF_WARMUP_RUNS` | `2` | Number of warmup runs. |
| `EXECUTORCH_XNNPACK_PYTEST_PERF_THRESHOLD_PCT` | `10.0` | Allowed slowdown before comparison fails. |

Boolean variables accept values such as `1`, `true`, `yes`, and `on`.

The perf stage forces the ExecuTorch threadpool to one thread while timing,
then restores the previous thread count after the run.

### Results

Perf results are keyed by the pytest test name and host/runtime identity. For
example, the ResNet test writes:

```text
backends/xnnpack/test/models/resnet_pytest_perf_results.json
backends/xnnpack/test/models/resnet_pytest_perf_results.md
```

The JSON file is the complete machine-readable record used for comparisons.
It includes samples, host details, runtime details, linked-symbol hints, and
the recorded latency metrics.

The Markdown file is a compact human-readable summary for review. It includes
the host, processor, expected hardware path, thread count, mean latency,
median latency, coefficient of variation, and recorded time.

`Expected HW path` is deduced from host ML features and linked runtime symbols.
It is not proof of the exact runtime microkernel selected for every operator.
12 changes: 9 additions & 3 deletions backends/xnnpack/test/models/resnet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Copyright (c) Meta Platforms, Inc. and affiliates.
# All rights reserved.
# Copyright 2026 Arm Limited and/or its affiliates.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree.
Expand Down Expand Up @@ -39,7 +40,7 @@ def forward(self, x):
)
return self.model(x)

def _test_exported_resnet(self, tester):
def _test_exported_resnet(self, tester, xnnpack_perf=False):
(
tester.export()
.to_edge_transform_and_lower()
Expand All @@ -52,11 +53,16 @@ def _test_exported_resnet(self, tester):
.check(["torch.ops.higher_order.executorch_call_delegate"])
.to_executorch()
.serialize()
.run_method_and_compare_outputs()
.run_method_and_compare_outputs(
xnnpack_perf=xnnpack_perf,
)
)

def test_fp32_resnet18(self):
self._test_exported_resnet(Tester(torchvision.models.resnet18(), self.inputs))
self._test_exported_resnet(
Tester(torchvision.models.resnet18(), self.inputs),
xnnpack_perf=True,
)

@unittest.skip("T187799178: Debugging Numerical Issues with Calibration")
def _test_qs8_resnet18(self):
Expand Down
242 changes: 242 additions & 0 deletions backends/xnnpack/test/models/resnet_pytest_perf_results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
{
"entries": {
"xnnpack.models.resnet.test_fp32_resnet18": {
"darwin-arm64-apple-m3-pro-mac15-6-nosme2-pybinding-buffer-threads1-_portable_lib.cpython-313-darwin.so": {
"host": {
"cpu_id": "apple-m3-pro-mac15-6",
"cpu_model": "Apple M3 Pro",
"linux_features": [],
"machine": "arm64",
"macos_features": {
"hw.optional.arm.FEAT_BF16": "1",
"hw.optional.arm.FEAT_DotProd": "1",
"hw.optional.arm.FEAT_FP16": "1",
"hw.optional.arm.FEAT_I8MM": "1",
"hw.optional.arm.FEAT_SME": "0",
"hw.optional.arm.FEAT_SME2": "0"
},
"macos_model": "Mac15,6",
"ml_features": [
"fp",
"asimd",
"bf16",
"asimddp",
"fphp",
"i8mm"
],
"processor": "arm",
"release": "25.5.0",
"sme2_available": false,
"system": "Darwin"
},
"kernel_path": {
"backend": "xnnpack",
"basis": [
"host.ml_features",
"runtime.linked_symbols"
],
"confidence": "expected_not_proven",
"expected_hardware_path": "kleidiai+neon",
"host_ml_features": [
"fp",
"asimd",
"bf16",
"asimddp",
"fphp",
"i8mm"
],
"runtime_symbol_hints": {
"avx2_symbols_present": false,
"avx512_symbols_present": false,
"avx_symbols_present": false,
"kleidiai_symbols_present": true,
"neon_symbols_present": true,
"neonsme2_symbols_present": true,
"sme2_symbols_present": true,
"sme_symbols_present": true,
"sse_symbols_present": false,
"xnnpack_symbols_present": true
}
},
"metrics": {
"cv_pct": 0.246957433434771,
"load_mode": "buffer",
"max_ms": 41.886459,
"mean_ms": 41.6864669,
"median_ms": 41.6786465,
"min_ms": 41.502291,
"p90_ms": 41.7885084,
"samples_ms": [
41.502291,
41.560417,
41.886459,
41.729917,
41.675084,
41.651167,
41.777625,
41.682209,
41.65375,
41.74575
],
"thread_count_observed": 1,
"thread_count_requested": 1,
"timed_runs": 10,
"warmup_runs": 2
},
"recorded_at_utc": "2026-06-25T09:24:37.854788+00:00",
"test_id": "xnnpack.models.resnet.test_fp32_resnet18",
"threshold_pct": 10.0,
"timing_runtime": {
"avx2_symbols_present": false,
"avx512_symbols_present": false,
"avx_symbols_present": false,
"kind": "pybinding",
"kleidiai_symbols_present": true,
"native_mtime_utc": "2026-06-24T13:58:43+00:00",
"native_path": "extension/pybindings/_portable_lib.cpython-313-darwin.so",
"neon_symbols_present": true,
"neonsme2_symbols_present": true,
"pte_load_mode": "buffer",
"runtime_key": "darwin-arm64-apple-m3-pro-mac15-6-nosme2-pybinding-buffer-threads1-_portable_lib.cpython-313-darwin.so",
"sme2_symbols_present": true,
"sme_symbols_present": true,
"sse_symbols_present": false,
"thread_count": 1,
"xnnpack_symbols_present": true
}
},
"linux-aarch64-aarch64-0x41-0x000-nosme2-pybinding-buffer-threads1-_portable_lib.cpython-312-aarch64-linux-gnu.so": {
"host": {
"cpu_id": "aarch64-0x41-0x000",
"cpu_model": "aarch64",
"linux_features": [
"fp",
"asimd",
"evtstrm",
"aes",
"pmull",
"sha1",
"sha2",
"crc32",
"atomics",
"fphp",
"asimdhp",
"cpuid",
"asimdrdm",
"jscvt",
"fcma",
"lrcpc",
"dcpop",
"sha3",
"asimddp",
"sha512",
"asimdfhm",
"dit",
"uscat",
"ilrcpc",
"flagm",
"sb",
"paca",
"pacg",
"dcpodp",
"flagm2",
"frint",
"bf16",
"afp"
],
"machine": "aarch64",
"macos_features": {},
"macos_model": null,
"ml_features": [
"fp",
"asimd",
"asimddp",
"asimdhp",
"bf16",
"fphp"
],
"processor": "aarch64",
"release": "6.8.0-117-generic",
"sme2_available": false,
"system": "Linux"
},
"kernel_path": {
"backend": "xnnpack",
"basis": [
"host.ml_features",
"runtime.linked_symbols"
],
"confidence": "expected_not_proven",
"expected_hardware_path": "kleidiai+neon",
"host_ml_features": [
"fp",
"asimd",
"asimddp",
"asimdhp",
"bf16",
"fphp"
],
"runtime_symbol_hints": {
"avx2_symbols_present": false,
"avx512_symbols_present": false,
"avx_symbols_present": false,
"kleidiai_symbols_present": true,
"neon_symbols_present": true,
"neonsme2_symbols_present": true,
"sme2_symbols_present": true,
"sme_symbols_present": true,
"sse_symbols_present": false,
"xnnpack_symbols_present": true
}
},
"metrics": {
"cv_pct": 1.1082648754607403,
"load_mode": "buffer",
"max_ms": 43.026209,
"mean_ms": 41.900096,
"median_ms": 41.9061045,
"min_ms": 41.338542,
"p90_ms": 42.3321956,
"samples_ms": [
41.604084,
41.338542,
41.700709,
41.923334,
42.255083,
43.026209,
41.34325,
41.987333,
41.888875,
41.933541
],
"thread_count_observed": 1,
"thread_count_requested": 1,
"timed_runs": 10,
"warmup_runs": 2
},
"recorded_at_utc": "2026-06-26T13:41:33.202559+00:00",
"test_id": "xnnpack.models.resnet.test_fp32_resnet18",
"threshold_pct": 10.0,
"timing_runtime": {
"avx2_symbols_present": false,
"avx512_symbols_present": false,
"avx_symbols_present": false,
"kind": "pybinding",
"kleidiai_symbols_present": true,
"native_mtime_utc": "2026-06-12T13:39:04+00:00",
"native_path": "extension/pybindings/_portable_lib.cpython-312-aarch64-linux-gnu.so",
"neon_symbols_present": true,
"neonsme2_symbols_present": true,
"pte_load_mode": "buffer",
"runtime_key": "linux-aarch64-aarch64-0x41-0x000-nosme2-pybinding-buffer-threads1-_portable_lib.cpython-312-aarch64-linux-gnu.so",
"sme2_symbols_present": true,
"sme_symbols_present": true,
"sse_symbols_present": false,
"thread_count": 1,
"xnnpack_symbols_present": true
}
}
}
},
"version": 1
}
8 changes: 8 additions & 0 deletions backends/xnnpack/test/models/resnet_pytest_perf_results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# XNNPACK Pytest Perf Results

Generated from `resnet_pytest_perf_results.json`.

| Test | Host | Processor | ML features | Expected HW path | Threads | Mean ms | Median ms | CV % | Recorded |
| ---------------------------------------- | -------------------- | ------------ | --------------------------------------- | ---------------- | ------: | ------: | --------: | ----: | -------------------- |
| xnnpack.models.resnet.test_fp32_resnet18 | Darwin arm64 Mac15,6 | Apple M3 Pro | fp, asimd, bf16, asimddp, fphp, i8mm | kleidiai+neon | 1 | 41.686 | 41.679 | 0.247 | 2026-06-25 09:24 UTC |
| xnnpack.models.resnet.test_fp32_resnet18 | Linux aarch64 | aarch64 | fp, asimd, asimddp, asimdhp, bf16, fphp | kleidiai+neon | 1 | 41.900 | 41.906 | 1.108 | 2026-06-26 13:41 UTC |
1 change: 1 addition & 0 deletions backends/xnnpack/test/tester/BUCK
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ fbcode_target(_kind = runtime.python_library,
name = "tester",
srcs = [
"__init__.py",
"performance.py",
"tester.py",
],
base_module = "executorch.backends.xnnpack.test.tester",
Expand Down
Loading
Loading