ci: complete comparative benchmark contract for #309

.github/workflows/tests.yml

@@ -69,11 +69,45 @@ jobs:
         run: ./tests/full_validate.sh --gate style
       - name: Validate docs, schemas, contracts, and generated files
         run: ./tests/full_validate.sh --gate contracts
-      - name: Enforce benchmark budget
-        run: ./tests/full_validate.sh --gate benchmark
       - name: Run dependency and static security checks
         run: ./tests/full_validate.sh --gate security
 
+  benchmarks:
+    name: Benchmark (${{ matrix.profile }})
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - profile: unix
+            os: ubuntu-latest
+          - profile: macos
+            os: macos-latest
+          - profile: windows
+            os: windows-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.13"
+      - name: Install declared benchmark comparators
+        run: python -m pip install ".[typer,benchmark]"
+      - name: Run calibrated comparative benchmark
+        env:
+          BASE_CLI_BENCHMARK_PLATFORM: ${{ matrix.profile }}
+          SOURCE_REVISION: ${{ github.sha }}
+        run: python scripts/benchmark_runtime.py --check --iterations 31 --output benchmark-results.json
+      - name: Retain dated machine-readable benchmark evidence
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: base-cli-benchmark-${{ matrix.profile }}-${{ github.run_id }}
+          path: benchmark-results.json
+          if-no-files-found: ignore
+          retention-days: 90
+
   linux-distributions:
     name: Validate (${{ matrix.name }})
     runs-on: ubuntu-latest
@@ -136,4 +170,13 @@ jobs:
           $drive = $env:GITHUB_WORKSPACE.Substring(0, 1).ToLowerInvariant()
           $path = $env:GITHUB_WORKSPACE.Substring(2).Replace('\', '/')
           $linuxWorkspace = "/mnt/$drive$path"
-          wsl --distribution Ubuntu --user root -- bash -lc "set -eu; cd '$linuxWorkspace'; sed -i 's/\r$//' tests/full_validate.sh tests/validate.sh; apt-get update -qq; apt-get install -y -qq nodejs npm python3-venv python3.14-venv; python3 -m venv /tmp/base-cli-venv; . /tmp/base-cli-venv/bin/activate; python -m pip install '.[dev,typer]'; bash tests/full_validate.sh --gate runtime"
+          $revision = $env:GITHUB_SHA
+          wsl --distribution Ubuntu --user root -- bash -lc "set -eu; cd '$linuxWorkspace'; sed -i 's/\r$//' tests/full_validate.sh tests/validate.sh; apt-get update -qq; apt-get install -y -qq nodejs npm python3-venv python3.14-venv; python3 -m venv /tmp/base-cli-venv; . /tmp/base-cli-venv/bin/activate; python -m pip install '.[dev,typer,benchmark]'; bash tests/full_validate.sh --gate runtime; BASE_CLI_BENCHMARK_PLATFORM=wsl SOURCE_REVISION='$revision' python scripts/benchmark_runtime.py --check --iterations 31 --output base-cli-benchmark-wsl.json"
+      - name: Retain WSL benchmark evidence
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: base-cli-benchmark-wsl-${{ github.run_id }}
+          path: base-cli-benchmark-wsl.json
+          if-no-files-found: ignore
+          retention-days: 90

CHANGELOG.md

@@ -11,6 +11,11 @@ and versions are tracked in the repo-root `VERSION` file.
 
 - Continue compatibility hardening and adoption work for the next release.
 
+### Added
+
+- Publish versioned, comparative CLI benchmark reports with lifecycle and
+  feature scenarios, platform-specific regression gates, and retained CI evidence.
+
 ### Fixed
 
 - Preserve explicit application identities losslessly while using

docs/performance.md

@@ -1,42 +1,93 @@
 # Performance and adversarial-regression contract
 
 `base-cli` treats startup and filesystem behavior as part of its public
-quality contract. The checked benchmark is intentionally small and runs from
-the source checkout:
+quality contract. The benchmark is a comparative regression check, not a claim
+that a lifecycle framework should outpace bare parsers. Its scenarios separate
+interpreter/import cost, parser dispatch, the base-cli lifecycle, optional
+features, and persistence.
+
+Install the complete local validation set, including every comparator:
 
 ```bash
-python scripts/benchmark_runtime.py --check
+python -m pip install '.[dev,typer,quality,benchmark]'
+python scripts/benchmark_runtime.py --check --iterations 31 --output benchmark-results.json
 ```
 
-It records fresh-process import time and the cost of an isolated production
-invocation through `base_cli.testing.invoke`. The comparison mode measures
-equivalent no-op commands for base-cli, Click, Typer, and (when installed)
-Cyclopts. Install the optional benchmark extra to include Cyclopts:
+The benchmark is also part of the local aggregate:
 
 ```bash
-python -m pip install 'base-cli[benchmark]'
+./tests/full_validate.sh --gate benchmark
 ```
 
-The CI quality job checks the base-cli sample p95 against these budgets. The
-benchmark records the selected platform profile in both text and JSON output;
-set `BASE_CLI_BENCHMARK_PLATFORM` when a runner's filesystem or virtualization
-boundary is not represented by the host operating system. Supported profiles
-are `unix`, `macos`, `windows`, and `wsl`.
-
-| Measurement | Budget |
-| --- | ---: |
-| Fresh `import base_cli` (native Unix/macOS) | 750 ms |
-| Fresh `import base_cli` (native Windows) | 1,000 ms |
-| Fresh `import base_cli` (WSL2 on a Windows-mounted checkout) | 1,000 ms |
-| Isolated invocation and runtime filesystem setup | 1,500 ms |
-
-The benchmark reports the median, p95, and maximum for seven samples. Pass
-`--json` for a stable machine-readable result suitable for archiving or CI
-comparison. These
-budgets are intentionally broad enough for hosted runners while still
-detecting accidental quadratic startup work, unbounded metadata scans, or
-unexpected dependency imports. A performance improvement should preserve the
-same lifecycle and persistence assertions covered by the adversarial tests.
+## Scenario contract
+
+The comparative set is Click, Typer, Cyclopts, and base-cli. Each framework
+registers an equivalent zero-argument no-op command. Fresh-process
+measurements include Python startup, framework import, command construction,
+and dispatch through the framework's normal entry point. Warm parser samples
+reuse command objects; Click and Typer use Click's `CliRunner`, Cyclopts uses
+its `App` call, and base-cli reports both a shared Click-runner lifecycle
+sample and an end-to-end `base_cli.testing.invoke()` sample. The runner shape
+for each value is recorded here so comparisons do not imply identical
+mechanisms where framework APIs differ.
+
+Base-cli-only feature samples cover:
+
+- successful and failed JSON envelopes;
+- debug diagnostics on the user stream;
+- nested-command dispatch;
+- persistence disabled versus enabled, with the same log event in both cases.
+
+These feature costs are reported separately from parser comparisons. JSON
+success/error and nested dispatch use the public `App`/lifecycle API; persistence
+samples differ only in whether file logging is enabled. Measurements are
+in-process, warm, and use isolated temporary homes.
+
+## CI budgets and evidence
+
+CI collects 31 samples per scenario on Python 3.13 for each supported
+benchmark profile: native Unix, macOS, Windows, and WSL2. `--check` fails if a
+required comparator or scenario is missing, a p95 exceeds its profile budget,
+or the measured base-cli lifecycle increment over Click exceeds its separate
+profile budget. The lifecycle-to-Click ratio remains visible for interpretation,
+but is not itself gated because Click's sub-millisecond baseline makes ratios
+highly sensitive to timer granularity. Warm budgets apply to all non-persistence
+base-cli feature scenarios; file persistence has a separate platform budget
+because runner filesystems vary materially. Percentile gates catch practical
+regressions while keeping noisy single maxima visible without making one
+scheduler outlier block a change.
+
+| Budget (p95) | Unix | macOS | Windows | WSL2 |
+| --- | ---: | ---: | ---: | ---: |
+| Cold import, including interpreter startup | 750 ms | 750 ms | 1,000 ms | 1,000 ms |
+| Cold no-op invocation, including startup and dispatch | 2,000 ms | 2,000 ms | 4,000 ms | 4,000 ms |
+| Base-cli lifecycle increment over Click warm dispatch | 5 ms | 5 ms | 15 ms | 15 ms |
+| Warm invocation and non-persistence feature scenarios | 50 ms | 50 ms | 100 ms | 100 ms |
+| File-persistence-enabled scenario | 50 ms | 50 ms | 250 ms | 50 ms |
+
+An initial 31-sample local calibration on macOS (Python 3.14.6, Apple Silicon)
+measured approximately 101 ms for base-cli cold import, 0.56 ms for warm
+lifecycle dispatch, and 15.9 ms p95 for file-persisted logging. These are
+development-host measurements, not adoption claims or release comparisons.
+The first hosted 31-sample baseline measured file-persistence p95 at 22 ms on
+Ubuntu, 21 ms on macOS, 158 ms on Windows, and 27 ms on WSL2. Windows also had
+a high median absolute deviation (26 ms), so persistence has its own Windows
+budget instead of weakening other warm-scenario gates. These measurements are
+CI calibration evidence, not adoption claims or release comparisons; review
+subsequent retained artifacts before tightening platform budgets.
+
+Each report is versioned as `base-cli.benchmark` schema version 1 and contains
+the package version, source revision, UTC timestamp, platform profile, Python
+version/ABI, OS release, architecture, CPU count, sample count, medians, p95,
+maximum, median absolute deviation, and parser/lifecycle comparison values.
+The Tests workflow retains a distinct JSON artifact for each platform profile
+for 90 days. Download the artifact from the corresponding `Benchmark (...)`
+or `Validate (WSL)` Actions job to compare dated runs.
+
+The profile can be selected explicitly with
+`BASE_CLI_BENCHMARK_PLATFORM` when a runner's filesystem or virtualization
+boundary is not represented by its host OS. Supported values are `unix`,
+`macos`, `windows`, and `wsl`.
 
 ## Retention recovery work bounds
 

docs/testing.md

@@ -7,11 +7,11 @@ documentation/schema/contract validation, benchmark budgets, and security.
 Bandit and pip-audit are required; a missing tool is an error rather than a
 skipped check.
 
-Run it from a clean checkout after installing the development and quality
-extras:
+Run it from a clean checkout after installing the development, quality, and
+benchmark extras (the latter installs every declared framework comparator):
 
 ```bash
-python -m pip install '.[dev,typer,quality]'
+python -m pip install '.[dev,typer,quality,benchmark]'
 ./tests/full_validate.sh
 ```
 
@@ -30,15 +30,17 @@ validation gate. Individual gates can be selected for focused local work:
 ```
 
 The Tests workflow runs the runtime suite across the OS/Python matrix and on
-the supported Linux distributions/WSL. Its single quality job runs the
-platform-independent coverage, typing, style, contract, benchmark, and
-security gates once, with each group visible as a named Actions step. The
-workflow validates feature branches through pull requests rather than
-launching a second full run on every feature-branch push; direct pushes to
-`main` and version tags remain validated. The
-Package workflow focuses on release-boundary checks: building and validating
-the wheel/sdist, checksums/SBOM, and clean installed-wheel smoke tests. It
-does not repeat the source test, typing, lint, documentation, benchmark, or
+the supported Linux distributions/WSL. Its quality job runs platform-
+independent coverage, typing, style, contract, and security gates once, with
+each group visible as a named Actions step. A separate comparative benchmark
+matrix measures Click, Typer, Cyclopts, and base-cli on Unix, macOS, Windows,
+and WSL; each job publishes an Actions summary and retains its versioned JSON
+report as a dated artifact. The workflow validates feature branches through
+pull requests rather than launching a second full run on every feature-branch
+push; direct pushes to `main` and version tags remain validated. The Package
+workflow focuses on release-boundary checks: building and validating the
+wheel/sdist, checksums/SBOM, and clean installed-wheel smoke tests. It does
+not repeat the source test, typing, lint, documentation, benchmark, or
 security suites. `./tests/full_validate.sh` remains the one-command local
 aggregate of all source gates.