Skip to content

233_bench_matrix: matrix.json's hub does not exist in the pinned tree, and the check only runs where submodules are initialised #599

Description

@Sunrisepeak

Found while running the full e2e suite locally (332 passed, 1 failed, 35 skipped). Unrelated to the change that surfaced it.

What fails

FAIL: bench/matrix.json
  linux/gcc/mcpp-2026.8.11.3:   hub='modules/platform/src/platform.cppm' does not exist in the pinned tree
  linux/clang/mcpp-2026.8.11.3: hub='modules/platform/src/platform.cppm' does not exist in the pinned tree
  windows/clang/mcpp-2026.8.11.3: hub='modules/platform/src/platform.cppm' does not exist in the pinned tree

bench/projects/mcpp/mcpp-2026.8.11.3 is checked out at a749e9f (= v2026.8.11.3), and modules/platform/src/platform.cppm does not exist in that tree — the file lives at src/platform.cppm there. The path in matrix.json was written for the CURRENT layout after the module move.

The test's own message says what that costs: "every scenario that perturbs it would be reported skipped and the job would still pass".

Why CI is green

The check skips a workload whose tree has no mcpp.toml, which it reads as "submodule not initialised":

if not os.path.isfile(os.path.join(tree, "mcpp.toml")):
    ...
    uninit.add(c["project"])
    break

Only the bench workflow checks submodules out, so on every other job the three cells are skipped and the test passes. On a developer machine with submodules initialised, it fails — which is where it was found.

What a fix has to decide

Not just which path to write. The pinned tree is a historical mcpp, so a hub path that is correct today is wrong for it by construction, and the next module move breaks it again. The options are a per-project hub (the pinned tree's own path), or a check that resolves the hub against each tree rather than against one string.

Whichever is chosen, the uninit branch is worth revisiting in the same pass: a check that is silent on every CI job except one is a check whose failures are found by hand.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions