fix(bench): the fixture was 74% compiler startup, and the CI workflow never parsed

—— 三个「看起来在测,其实没在测」

**fixture 几乎不含编译。** 实测单个 TU 0.23s,其中 0.17s 是 g++ 启动;
而 `weight` 这个旋钮推不动它 —— 它只产生 O(weight²) 次同一个平凡 constexpr 递归的
实例化(weight=40 也才几百次),编译器微秒级做完。真实对照(gcc 16.1,x86_64):

    空模块 .................................. 0.17s
    旧 fixture 单元 weight=6 ................ 0.23s   ← 74% 是启动
    旧 fixture 单元 weight=40 ............... 0.28s   ← 6.7× 的旋钮只买到 20%
    带真实 global module fragment 的单元 ..... 0.97s
    mcpp 自己的单元(57k 行 / 139 个)........ 0.57s

只有 units 是线性的(0.088s/个)。**这套东西过去主要在测 g++ 启动。**
工作负载改成真实 C++ 的成本来源:标准库头 + 按**不同类型**实例化
(共用类型的话编译器只实例化一次,后面全免费 —— 这正是旧旋钮失效的原因)。
现在是 `0.38s + 0.066s × weight`,并且有实测扫描钉住:20 units 下
weight 0/4/12 = 4.7s/18.0s/31.4s。默认 weight=4 让单元成本落在 0.64s,
和真实工程同一量级。

**`.github/workflows/bench.yml` 从提交那天起就不是合法 YAML** ——
`run: "$BENCH" --list` 被读成一个带引号的标量后面跟垃圾。这个 workflow
一次都没能启动过,而且**没有任何东西会说** :GitHub 仍把坏 workflow 列为 active,
`workflow_dispatch`-only 的 workflow 不会被 push 触发,也没有测试看过它。
新增 e2e 232 逐个 parse `.github/workflows/*.yml`,并要求每个都声明了 jobs
(能 parse 但没有 jobs 是同一类「静默的什么都不做」)。两侧都验过:
修好的文件通过,坏形态必失败。

**尺寸必须有名字。** 自由三元组 (units, fanin, weight) 无法在两个人之间比较。
加 `--preset smoke|standard|large`,并让**默认形状就等于 standard** ——
否则「没带参数」和「--preset standard」会是两个不同的东西。

bench/README 补成一份真正的规范:§1a 工作负载必须真的是工作负载(新旋钮必须
附实测扫描,否则默认认定为惰性)、§1b 命名尺寸、§4a **有效性规则**
(R1 分辨率:落在本引擎 noop 2× 以内的单元测的是进程启动不是构建;
R2 离散度:极差/中位数 > 20% 只支持数量级结论)、§4b 明确不做的事、
§4c 采纳了哪些既有实践(SPEC 的全披露与禁止针对性调优、hyperfine 的
预热与离散度报告)以及**这不是什么**(没有审计、单机、跨机器只比表内比值)。

同时修掉一处被自己实测推翻的旧论断:注释里写「GCC 和 Clang 的 BMI 都携带函数体」,
实测 GCC 16.1 **不**携带导出非模板函数的函数体 —— 所以 modules-impl 变体量的是
两种决策规则(比 BMI 内容 vs 信 mtime)的差别,不是编译器限制。

e2e 230 的相对路径检查改成从二进制自身目录运行:relpath 在 Windows 跨盘符
直接抛 `path is on mount 'D:'`,在 macOS 上 `mktemp -d` 给 /var/… 而真实 cwd 是
/private/var/…(深一层)会让 `..` 少一级 —— 这两个 CI 红都是测试自己的缺陷。

Author: speak-agent

.github/workflows/bench.yml

@@ -31,10 +31,14 @@ on:
         # ever runs — `touch-leaf` was defined, documented and advertised, and
         # had never appeared in a single result file.
         default: 'cold,noop,touch-hub,touch-leaf,edit-body,edit-comment'
+      preset:
+        description: 'named fixture size: smoke | standard | large (overridden by units/fanin/weight below)'
+        required: false
+        default: 'standard'
       units:
-        description: 'fixture translation units'
+        description: 'fixture translation units (0 = use the preset)'
         required: false
-        default: '40'
+        default: '0'
       fanin:
         description: 'dependencies per unit (controls graph depth)'
         required: false
@@ -122,19 +126,26 @@ jobs:
 
       - name: Report engine availability
         shell: bash
-        run: "$BENCH" --list
+        run: |
+          "$BENCH" --list
 
       - name: Run benchmark
         shell: bash
         run: |
           set -euo pipefail
+          # The preset names the size; units/fanin override it only when set to a
+          # positive number. Passing raw numbers unconditionally would make every
+          # run's size an accident of this file rather than a named, comparable
+          # workload — and --preset must come first so the overrides still win.
+          args=( --preset "${{ inputs.preset }}" )
+          [ "${{ inputs.units }}" -gt 0 ] 2>/dev/null && args+=( --units "${{ inputs.units }}" )
+          [ "${{ inputs.fanin }}" -gt 0 ] 2>/dev/null && args+=( --fanin "${{ inputs.fanin }}" )
           "$BENCH" \
             --engines   '${{ inputs.engines }}' \
             --variants  '${{ inputs.variants }}' \
             --scenarios '${{ inputs.scenarios }}' \
             --profile   '${{ inputs.profile }}' \
-            --units     '${{ inputs.units }}' \
-            --fanin     '${{ inputs.fanin }}' \
+            "${args[@]}" \
             --runs      '${{ inputs.runs }}' \
             --work      "$RUNNER_TEMP/bench-work" \
             --out       "bench-${{ matrix.name }}.json"

bench/README.md

@@ -41,12 +41,63 @@ And, orthogonally, **the source form**: the same project emitted three ways.
 | `modules` | `unit_k.cppm` declares **and** defines | what most module code looks like |
 | `modules-impl` | `unit_k.cppm` declares, `unit_k_impl.cpp` defines | does splitting implementation out of the interface stop edit cascades? |
 
-`modules-impl` exists because of a measured result: on **both** GCC 16.1 and
-Clang 22.1 a module interface unit's BMI carries function bodies, so editing any
-body changes the BMI and cascades to every importer. No compiler flag fixes it
-(`-fmodules-reduced-bmi` was measured and does not). Moving bodies into
-implementation units is the only available fix, and this variant is how that
-claim gets a number instead of an argument.
+`modules-impl` gives the "move bodies out of interface units" advice a number.
+What that number is turns out to depend on the compiler, and an earlier version
+of this paragraph asserted the **opposite** of the measurement:
+
+* GCC 16.1 does **not** put the body of an exported non-template function into
+  the BMI. Editing such a body changes the object file and leaves the BMI
+  byte-identical apart from its embedded timestamps.
+* So the cascade other engines pay for that edit is avoidable, and the engines
+  split by their *decision rule*: compare the BMI's **content** (mcpp, 0.3 s) or
+  trust its **mtime** (cmake and xmake, ~10 s).
+* Templates and inline functions in an interface unit **do** change the BMI. The
+  advice survives; its justification is narrower than it was written to be.
+
+Establishing this needs a control — compile the *same* source twice and diff the
+BMIs. The differing bytes land at the same offsets either way, inside
+`buildtime:`/`localtime:`. Without that control the timestamp reads as a content
+change and the conclusion inverts.
+
+### 1a. The workload must actually be the workload
+
+A size knob that does not move the cost is worse than no knob: it makes a
+benchmark look tunable while it measures something else. The first version of
+this fixture failed exactly there.
+
+| | cost per unit (gcc 16.1, x86_64) |
+|---|---|
+| empty module | 0.17 s |
+| **old fixture unit, `weight 6`** | **0.23 s** — 74% of it compiler startup |
+| old fixture unit, `weight 40` | 0.28 s — a 6.7x knob bought 20% |
+| one unit with a realistic global module fragment | 0.97 s |
+| **mcpp's own units** (57k lines / 139 units) | **0.57 s** |
+
+The old `weight` emitted O(weight²) instantiations of one trivial `constexpr`
+recursion — a few hundred at weight 40, which a compiler does in microseconds.
+Unit *count* scaled cost linearly at 0.088 s each; `weight` did not scale it at
+all. **The suite was largely measuring `g++` starting up.**
+
+The workload is now built from what actually costs time in real C++: standard
+library headers, plus instantiation over **distinct types** so blocks cannot
+share instantiations. Cost is `0.38 s + 0.066 s × weight`, and the knob is
+verified to move: at 20 units, `weight` 0 / 4 / 12 gives 4.7 s / 18.0 s / 31.4 s
+cold.
+
+**Rule.** Any future knob must come with a measured sweep showing it changes
+cost, in this file. A knob without one is assumed inert.
+
+### 1b. Named sizes
+
+A benchmark whose size is a free-form triple of numbers cannot be compared
+between two people. `--preset` names it, and the default shape **is** `standard`
+so that "no flags" and `--preset standard` cannot mean different things.
+
+| preset | units | fan-in | weight | mcpp cold (gcc, modules) |
+|---|---|---|---|---|
+| `smoke` | 4 | 2 | 1 | ~2 s — CI and the e2e test, not for publication |
+| `standard` | 20 | 3 | 4 | ~18 s — what published results use |
+| `large` | 60 | 3 | 6 | minutes |
 
 ---
 
@@ -163,6 +214,64 @@ Two details that are easy to get wrong and change the answer:
   the log lives under the WORK root, never inside the measured tree, so a
   `--project` run cannot drop scratch into someone's repository.
 
+### 4a. Validity rules — when a cell must NOT be compared
+
+Every result file carries `median_s`, `min_s`, `max_s` and every raw `sample`.
+Two rules decide whether a number means anything, and both are computable from
+those fields alone — no trust in the harness required.
+
+**R1 — resolution.** Each engine's `noop` row for the same variant is its floor:
+what it costs to ask "is anything out of date?" before any work happens. A cell
+within **2x of its own engine's `noop`** is measuring process startup and
+bookkeeping, not building, and must not be read as a build comparison.
+
+> This is why the `headers` rows read the way they do at small sizes. With the
+> old fixture, `cmake` `noop` was 0.33 s and `cmake` `edit-body` was 0.79 s —
+> 2.4x, right at the edge. The three fastest engines sat inside a 0.15 s band
+> that is *entirely* startup. Those cells were never a ranking.
+
+**R2 — dispersion.** If `(max_s − min_s) / median_s > 0.20`, the cell is noisy
+and only order-of-magnitude claims survive it. Report it, do not silently
+re-run: a cell that needs re-running to look stable is a cell whose number
+depends on the machine's mood.
+
+Neither rule is applied automatically. Automatic suppression hides data; the
+rules are stated so a reader applies them, and so a table that violates them is
+visibly wrong rather than quietly wrong.
+
+### 4b. What this suite deliberately does not do
+
+* **No CPU pinning, no governor forcing, no `nice`.** Developers do not build
+  that way. The cost is variance, which R2 exposes rather than hides.
+* **No cache dropping.** A cold page cache is not a situation anyone builds in,
+  and it adds variance unrelated to the engine.
+* **No engine-specific tuning.** Each engine gets the same standard, the same
+  sources, the same compiler binary, the same optimisation level, and whatever
+  its own documentation says is the normal way to build. Tuning one engine and
+  not the others is how build-system benchmarks usually go wrong.
+* **No confidence intervals.** Run counts are small by necessity; a computed
+  interval would imply rigour that is not there. Medians with min/max and the
+  raw samples are what the data supports.
+
+### 4c. Practices this follows, and what it is not
+
+Adopted, with the source of the practice:
+
+| practice | from | here |
+|---|---|---|
+| full disclosure — host, tool versions, exact command, all flags | SPEC's run rules | §11 + every engine's version recorded by the engine itself |
+| no benchmark-specific tuning | SPEC's run rules | §4b |
+| warm-up run excluded from the timing | hyperfine, Google Benchmark | one untimed seed build per cell |
+| report dispersion, not just a central value | hyperfine | `min_s`/`max_s`/`samples` + R2 |
+| distinguish "cannot run" from "ran and failed" | — | `unavailable` vs `failed`, both requiring a reason |
+| a versioned, machine-readable result format | — | `protocol_version` |
+
+**What this is not.** It is not an audited or certified benchmark, there is no
+reviewing body, and the numbers are single-host. Reproducing a published table
+requires the same preset, the same engine versions and a comparable machine —
+all of which the result file states, which is the point. Treat cross-machine
+comparison of absolute seconds as invalid; compare **ratios within one table**.
+
 ---
 
 ## 5. Declared asymmetries

bench/src/fixture/generate.cppm

@@ -13,11 +13,21 @@
 //   modules-impl unit_k.cppm declares, unit_k_impl.cpp defines (interface and
 //                                                              implementation split)
 //
-// modules-impl exists because of a measured result: with GCC and Clang alike, a
-// module interface unit's BMI carries function bodies, so editing ANY body
-// cascades to every importer. Moving bodies into implementation units is the
-// only fix available (no compiler flag does it — `-fmodules-reduced-bmi` was
-// measured and does not). This variant is how that claim gets a number.
+// modules-impl exists to give the "move bodies out of interface units" advice a
+// number. What that number IS turns out to depend on the compiler, and an
+// earlier version of this comment asserted the opposite of the measurement:
+//
+//   * GCC 16.1 does NOT put the body of an exported non-template function into
+//     the BMI. Editing such a body changes the object file and leaves the BMI
+//     byte-identical apart from its embedded timestamps, so an engine that
+//     compares BMI CONTENT correctly rebuilds one unit and stops.
+//   * An engine that decides from the BMI's mtime cascades anyway, which is why
+//     cmake and xmake pay ~10 s for that edit where mcpp pays 0.3 s.
+//
+// So this variant measures the difference between the two decision rules, not a
+// compiler limitation. Templates and inline functions in an interface unit are a
+// different story and DO change the BMI — the advice survives, its justification
+// is narrower than it was written to be.
 export module bench.fixture.generate;
 
 import std;
@@ -26,9 +36,16 @@ import bench.protocol;
 export namespace bench::fixture {
 
 struct Shape {
-    int units{40};   // how many translation units
+    // These defaults ARE the `standard` preset, deliberately: if "no flags" and
+    // "--preset standard" produced different fixtures, two people comparing
+    // results would have no way to tell which they each ran.
+    int units{20};   // how many translation units
     int fanin{3};    // how many earlier units each one depends on → graph depth
-    int weight{6};   // template instantiations per unit → per-unit compile cost
+    // Distinct template-instantiation blocks per unit. Calibrated, not guessed:
+    // each block costs ~0.066 s on top of a 0.38 s floor, so 4 puts a unit at
+    // ~0.64 s — the same order as a real project's units (mcpp's are 0.57 s).
+    // See detail::support_header() for the measurements behind those numbers.
+    int weight{4};
 };
 
 // Which files a scenario should perturb. The generator knows the shape, so it
@@ -50,42 +67,71 @@ inline std::vector<int> deps_of(int k, const Shape& s) {
 }
 
 // Body shared by all three variants, so the WORK is identical and only the
-// packaging differs. Templates rather than plain statements: they cost real
-// front-end time, which is what a modules benchmark is actually about.
+// packaging differs. `weight` blocks, each a DISTINCT template instantiation —
+// see support_header() for why distinctness is the whole point, and for what
+// one block costs.
 inline std::string function_body(int k, const Shape& s) {
     std::string b;
     b += "    long long acc = " + std::to_string(k) + ";\n";
-    for (int w = 0; w < s.weight; ++w) {
-        b += std::format(
-            "    acc += ::bench_fixture::mix<{}>(std::tuple<int, long, double>{{{}, {}L, {}.0}});\n",
-            w, k + w, k * 2 + w, w + 1);
-    }
+    for (int w = 0; w < s.weight; ++w)
+        b += std::format("    acc += ::bench_fixture::work<{}>({});\n", w, k + w);
     for (int d : deps_of(k, s))
         b += std::format("    acc += {}_value();\n", unit_name(d));
     b += "    return static_cast<int>(acc & 0x7fffffff);\n";
     return b;
 }
 
-// The template the bodies instantiate. Header form for the headers variant,
-// global-module-fragment form for the module variants — same code either way.
+// The template the bodies instantiate. Included by EVERY generated unit — in the
+// global module fragment for the module variants, directly for the header
+// variant — so all three pay the same per-translation-unit cost and differ only
+// in how they share declarations.
+//
+// CALIBRATION, and why this is not the workload it started as. The first version
+// measured almost no compilation: a unit cost 0.23 s of which 0.17 s was the
+// compiler starting up — 74% process startup — and the `weight` knob barely
+// moved it, because it emitted O(weight^2) instantiations of a single trivial
+// constexpr recursion (a few hundred at weight 40, which a compiler does in
+// microseconds). Measured on gcc 16.1.0, x86_64:
+//
+//     empty module ................................. 0.17 s
+//     old fixture unit, weight 6 ................... 0.23 s
+//     one unit with a realistic global module fragment 0.97 s
+//     mcpp's own units (57k lines / 139 units) ..... 0.57 s
+//
+// So the workload is now built from what actually costs time in real C++:
+// standard library headers, plus instantiation over DISTINCT types so the
+// instantiations cannot be shared between blocks. Cost is 0.38 s + 0.066 s per
+// weight unit, which puts the default weight at the same order as a real
+// project's units instead of two orders below it.
 inline std::string support_header() {
     return R"(#pragma once
-#include <tuple>
-#include <utility>
+#include <algorithm>
+#include <map>
+#include <numeric>
+#include <string>
+#include <vector>
 
 namespace bench_fixture {
 
-// A small, deliberately template-heavy helper: each instantiation costs the
-// front end real work, which is what makes per-unit compile time non-trivial
-// enough to measure. Nothing here is meant to be fast at runtime.
-template <int N, typename Tuple>
-constexpr long long mix(Tuple t) {
-    if constexpr (N <= 0) {
-        return static_cast<long long>(std::get<0>(t));
-    } else {
-        constexpr std::size_t idx = N % std::tuple_size_v<Tuple>;
-        return static_cast<long long>(std::get<idx>(t)) + mix<N - 1>(t);
-    }
+// The tag makes every `work<N>` a distinct instantiation of map, vector, string
+// and sort. Without it the compiler instantiates one set and every later block
+// is free — which is precisely why the previous knob did nothing.
+template <int Tag>
+struct Key {
+    int v;
+    friend bool operator<(const Key& a, const Key& b) { return a.v < b.v; }
+};
+
+template <int Tag>
+long long work(int seed) {
+    std::map<Key<Tag>, std::vector<std::string>> m;
+    for (int i = 0; i < 4; ++i)
+        m[Key<Tag>{seed + i}].push_back(std::to_string(seed * i));
+    std::vector<int> v;
+    v.reserve(8);
+    for (const auto& [k, strs] : m) v.push_back(static_cast<int>(k.v + strs.size()));
+    std::sort(v.begin(), v.end());
+    return std::accumulate(v.begin(), v.end(), 0LL);
 }
 
 }  // namespace bench_fixture

bench/src/main.cpp

@@ -2,7 +2,7 @@
 //
 //   bench [--engines a,b] [--variants headers,modules,modules-impl]
 //         [--scenarios cold,noop,...] [--profile release|debug]
-//         [--compiler default|gcc|clang] [--units N] [--fanin N] [--weight N]
+//         [--compiler default|gcc|clang] [--preset NAME] [--units N] [--fanin N] [--weight N]
 //         [--jobs N] [--runs N] [--work DIR] [--out FILE] [--list]
 //
 // Writes a protocol-versioned JSON report to --out (default bench-report.json)
@@ -61,9 +61,14 @@ void usage() {
     std::println("  --scenarios LIST   cold,noop,touch-hub,touch-leaf,edit-body,edit-comment");
     std::println("  --profile NAME     release | debug                         (default: release)");
     std::println("  --compiler NAME    default | gcc | clang                   (default: default)");
-    std::println("  --units N          fixture translation units               (default: 40)");
+    std::println("  --preset NAME      smoke | standard | large  — a NAMED size, so two runs on");
+    std::println("                     two machines compare. standard is the default shape.");
+    std::println("                       smoke     4 units  / fan-in 2 / weight 1   (~2s, CI)");
+    std::println("                       standard 20 units  / fan-in 3 / weight 4   (~18s cold, mcpp)");
+    std::println("                       large    60 units  / fan-in 3 / weight 6");
+    std::println("  --units N          fixture translation units               (default: 20)");
     std::println("  --fanin N          dependencies per unit                   (default: 3)");
-    std::println("  --weight N         template instantiations per unit        (default: 6)");
+    std::println("  --weight N         distinct template blocks per unit       (default: 4)");
     std::println("  --jobs N           parallelism handed to each engine       (default: engine's)");
     std::println("  --runs N           repetitions per cell                    (default: per scenario)");
     std::println("  --work DIR         scratch directory                       (default: bench-work)");
@@ -105,6 +110,18 @@ std::expected<Options, std::string> parse(int argc, char** argv) {
         else if (a == "--compiler")  { auto v = value(a); if (!v) return std::unexpected(v.error()); o.compiler = *v; }
         else if (a == "--work")      { auto v = value(a); if (!v) return std::unexpected(v.error()); o.work = *v; }
         else if (a == "--out")       { auto v = value(a); if (!v) return std::unexpected(v.error()); o.out = *v; }
+        // Presets come FIRST so an explicit --units/--weight after one still
+        // wins. A benchmark whose size is a free-form pair of numbers cannot be
+        // compared between two people who ran it; a named size can.
+        else if (a == "--preset") {
+            if (i + 1 >= argc) return std::unexpected(std::string("--preset needs a value"));
+            const std::string_view v = argv[++i];
+            if      (v == "smoke")    o.shape = {4, 2, 1};
+            else if (v == "standard") o.shape = {20, 3, 4};
+            else if (v == "large")    o.shape = {60, 3, 6};
+            else return std::unexpected(std::format(
+                "unknown preset '{}' (smoke | standard | large)", v));
+        }
         else if (a == "--units")     { if (auto e = take_int(a, o.shape.units))  return std::unexpected(*e); }
         else if (a == "--fanin")     { if (auto e = take_int(a, o.shape.fanin))  return std::unexpected(*e); }
         else if (a == "--weight")    { if (auto e = take_int(a, o.shape.weight)) return std::unexpected(*e); }