Skip to content

fix(ci): remove the three sources of flakiness in the weekly example runs - #1482

Open
marc0olo wants to merge 4 commits into
masterfrom
fix/flaky-example-tests
Open

marc0olo wants to merge 4 commits into
masterfrom
fix/flaky-example-tests

Conversation

@marc0olo

@marc0olo marc0olo commented Sep 14, 2026

Copy link
Copy Markdown
Member

Three unrelated causes behind the weekly-run failures, each reproduced before fixing.

evm_block_explorer (not flaky — reliably broken)

test.sh asserted on Ethereum mainnet block 1. PublicNode routes requests across regional node pools and some of them only retain history from shortly before the Merge, so the call comes back as pruned history unavailable: requested 1, earliest available 15500000 depending on which pool serves the outcall. Both the Motoko and Rust jobs failed on it, and re-running does not help from a region whose pool is pruned.

Now asserts on block 20000000 (post-Merge, ~4.5M blocks above that boundary). Verified locally: all three assertions pass. The READMEs note the limitation so users who query early blocks in the frontend know why it fails.

unit_testable_rust_canister (flaky)

test_get_proposal_info and test_get_proposal_titles both call get_governance_wasm(), which downloaded to a single shared path with no synchronisation. libtest runs them on separate threads, so std::fs::write truncating the file under the peer's std::fs::read yields a partial gzip — surfacing as the opaque CanisterInvalidWasm: unsupported canister module format in the failed run. curl's exit status was also never checked, so a failed transfer wrote an empty file that the exists() check would then treat as a valid cached artifact.

Each artifact is now prepared exactly once per test binary behind a LazyLock, the download is status-checked with retries, and it lands via a temp file + rename so a partial transfer is never cached.

parallel_calls / pub-sub — Motoko jobs only (flaky, upstream bug)

Both are the only Motoko examples with two canisters in one mops project. icp-cli runs one mops build <canister> per canister concurrently, and on a cold cache both invocations download and extract the moc toolchain into the same two shared locations — .mops/_tmp/motoko-Linux-x86_64-<ver>.tar.gz and ~/.cache/mops/moc/<ver>. Whichever build loses reads a half-written archive or execs a half-extracted moc, and the failure surfaces as Build failed for canister <name> (exit code: undefined) with no diagnostics — or, when the timing lands differently, as the ZlibError: zlib: unexpected end of file visible in run 32750148987.

That matches which canister fails being random across runs (caller, callee, publisher, subscriber have all appeared) and only ever in these two projects.

Reproduced locally with icp build on motoko/pub-sub, 3/3 failures on a cold moc cache, and still present on the latest mops (3.2.0). It is a mops bug, filed as caffeinelabs/mops#818 with the offending code path and a minimal reproduction. mops install up front makes the download happen once, which is what this adds to _run-example.yml; 4/4 clean afterwards. Guarded on mops.toml existing and mops being on PATH, so it is a no-op for Rust examples, and to be removed once the upstream fix ships.

🤖 Generated with Claude Code

…runs

- evm_block_explorer: query a post-Merge block. PublicNode routes across
  regional pools and some only retain history from block ~15500000, so the
  block 1 assertions failed with `pruned history unavailable` depending on
  which pool served the outcall.
- unit_testable_rust_canister: prepare each test WASM once per test binary.
  Two tests downloaded the NNS governance WASM concurrently to the same path,
  and `fs::write` truncating under the peer's `fs::read` produced a partial
  gzip that PocketIC rejected as `CanisterInvalidWasm`. curl's exit status was
  also unchecked, so a failed transfer was cached as a valid artifact.
- _run-example.yml: warm the mops toolchain cache. Projects with more than one
  Motoko canister run concurrent `mops build` invocations that race on the
  shared moc download and extraction, failing the build with no diagnostics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
marc0olo and others added 2 commits September 14, 2026 15:12
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-unique

Keeps the atomic-rename guarantee if the tests are ever run per-process
(cargo nextest) or in two checkouts at once, rather than relying on
libtest running tests as threads within one process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Truncated gzip cache entries must be fully validated before reuse.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR reduces weekly CI flakiness across EVM, Rust, and Motoko examples.

Changes:

  • Uses retained Ethereum block 20,000,000 and documents pruning limitations.
  • Synchronizes and hardens Rust WASM downloads.
  • Pre-warms Motoko toolchain caches in CI.
File summaries
File Reviewed changes
rust/unit_testable_rust_canister/backend/tests/integration_tests.rs Synchronizes artifact preparation and adds reliable downloads. Moderate finding: truncated gzip cache entries are not fully validated.
rust/evm_block_explorer/test.sh Tests a retained Ethereum block.
rust/evm_block_explorer/README.md Documents PublicNode history limitations.
motoko/evm_block_explorer/test.sh Tests a retained Ethereum block.
motoko/evm_block_explorer/README.md Documents PublicNode history limitations.
.github/workflows/_run-example.yml Pre-warms the Motoko toolchain cache.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rust/unit_testable_rust_canister/backend/tests/integration_tests.rs Outdated
The magic-prefix check accepted a truncated archive left behind by an
earlier interrupted download, so a corrupt cache entry survived and kept
failing as CanisterInvalidWasm. Decoding the stream covers the trailing
CRC and length, and a corrupt entry is now replaced rather than reused.
flate2 is already built for the test profile via pocket-ic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The reviewed changes address the identified CI flakiness without unresolved blocking issues.

Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants