XNNPACK: Add optional performance stage to pytest/tester

[mansnils]

Force the ExecuTorch threadpool to one thread while timing, then restore the previous thread count after the run. This reduces noise and keeps results easier to compare across repeated local measurements.

Store the complete measurement record in a json file for regression checks. Also write a short md file with only the key review fields.

Deduce expected HW path based on host features.

Use Resnet as first example with json result data.

cc @GregoryComer @digantdesai @cbilgin @JakeStevens @freddan80 @per @zingo @oscarandersson8218 @Sebastian-Larsson @robell @rascani

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21549

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ You can merge normally! (9 Unrelated Failures), 8 Unclassified Failures

As of commit 4994261 with merge base adaaa8b ():

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• trunk / test-custom-ops-macos (cmake) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID CDF2:EE54C:85F7A2:1D11A72:6A70C56F and timestamp 2026-08-03 16:44:31 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
• trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID D61D:601A9:E51B1:30038E:6A70C615 and timestamp 2026-08-03 16:47:17 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
• trunk / test-models-macos-cpu (llama, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID C1FD:141B9C:6C4FB:16BB8D:6A70C5A4 and timestamp 2026-08-03 16:45:24 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID D279:C97B0:8111C7:1BD9390:6A70C529 and timestamp 2026-08-03 16:43:21 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
• trunk / test-models-macos-cpu (llama3_2_vision_encoder, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID C2BE:255A6A:86472E:1CF9BA4:6A70C5DA and timestamp 2026-08-03 16:46:18 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
• trunk / test-models-macos-cpu (mobilebert, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID F551:3FC6DF:851888:1CCA563:6A70C56A and timestamp 2026-08-03 16:44:26 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
• trunk / test-models-macos-cpu (resnet50, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID EABA:91B04:316E3:A6970:6A70C528 and timestamp 2026-08-03 16:43:20 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting
• trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]API rate limit exceeded for installation. If you reach out to GitHub Support for help, please include the request ID D32D:DD07C:1EFCD8:67E75E:6A70C51F and timestamp 2026-08-03 16:43:11 UTC. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service) - https://docs.github.com/en/rest/using-the-rest-api/getting-started-with-the-rest-api#rate-limiting

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-coreml (dl3) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-coreml (emformer_join) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-coreml (ic3) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-coreml (mv2) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-coreml (vit) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-mps / macos-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.