fix: strip the shipped payload and mirror assets concurrently - #302
Merged
Conversation
….29.1) The GitCode mirror leg of `publish-ecosystem` has failed four releases in a row — 0.0.94 / 0.0.97 / 0.0.105 / 2026.7.28.2 — each time on the biggest tarballs, each time costing ~5 minutes of manual `gtc release upload`. The per-asset 180s cap was tuned three times without anyone measuring what it was capping. Probe PR #301 measured it: GitHub US runner -> file.gitcode.com 0.012 MB/s <- the failing path same runner <- file.gitcode.com 3.87 MB/s same runner -> github.com 16 MB/s mainland-CN host -> file.gitcode.com 1.84 MB/s file.gitcode.com is a single Huawei Cloud origin in Beijing, so it is the inbound-to-CN direction that is shaped — not the host (the same runner downloads from it at 3.87 MB/s), not the client (curl and urllib measure identically), and not the runner's egress (16 MB/s to GitHub). The rate also swings ~4.6x run to run, so no fixed per-asset cap can be both safe and useful. At 0.012 MB/s a 34.8MB asset needs ~45 minutes; 180s never had a chance. Two fixes, both measured rather than guessed. S1 — stop shipping 30MB of debug info. Of the 34.8MB linux-x86_64 tarball, almost none was mcpp: the vendored `registry/bin/xlings` shipped at 97.3MB with full debug info (86.9MB on aarch64) against a 4.3MB mcpp. Stripping both takes the tarball to 4.62MB — 7.5x — and the stripped binaries were verified working (`--version`, `new`, `build`, `run`). Two defects made this possible: * the vendored xlings was never stripped at any of the four injection sites, only `cp`'d; * the x86_64 `strip` that DID exist ran before `mcpp pack`, which rebuilds the binary and overwrote it — which is why every release up to and including 2026.7.28.2 shipped an unstripped bin/mcpp while aarch64 (which stages by hand) shipped a stripped one. tools/slim_linux_payload.sh now strips both binaries on the STAGED payload and ASSERTS the result, so a strip that silently stops working fails the release instead of quietly shipping a fat tarball again. The aarch64 leg's best-effort `|| true` strip is now a hard requirement. macOS and Windows are deliberately left alone: their payloads are already 6.1MB / 4.2MB, and stripping a Mach-O invalidates its ad-hoc signature. S2 — mirror a host's assets concurrently, and bound the LEG instead of each asset. The shaping is per-connection: 1/4/8 concurrent 1MB uploads took 76s/80s/93s wall (6.6x aggregate at N=8). Assets within a leg now upload in parallel under MIRROR_MAX_PARALLEL, and MIRROR_UPLOAD_TIMEOUT is replaced by MIRROR_LEG_DEADLINE_GH/GTC. The v0.0.90 rule still holds — nothing is killed and retried inside a round, because the presigned OBS PUT has no multipart/resume (confirmed in #301); exhausting the budget ends the leg and the completeness gate fails loudly as before. Together the gitcode leg goes from ~45 min plus a manual step to ~6 min unattended, and every user's download shrinks 7.5x. Verified locally: slim on the real 2026.7.28.2 payload (101.6MB -> 13.9MB of binaries, 34.81MB -> 4.62MB tarball, stripped mcpp builds and runs an `import std` project); an isolated harness for the concurrent launcher (bounded concurrency, ordered log replay, correct per-asset success/failure); a full real mirror_res.sh run against the existing 2026.7.28.2 tag (all skipped, gate 16/16); and a real parallel-upload run on a throwaway `0.0.0-mirrortest` tag against BOTH hosts (gate 8/8, exit 0), whose artifacts were deleted afterwards with the real release verified intact. NB: release.yml is not exercised by PR CI — it runs on tag/dispatch only — so S1 lands its first real proof at the next release, where the new assertions fail loudly rather than silently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements S1 + S2 from the measurement in #301.
Why
The GitCode mirror leg of
publish-ecosystemhas failed four releases in a row — 0.0.94, 0.0.97, 0.0.105, 2026.7.28.2 — always on the biggest tarballs, each costing ~5 min of manualgtc release upload. The 180s per-asset cap was tuned three times without anyone measuring what it was capping. #301 measured it:file.gitcode.comgitcode.comgithub.comfile.gitcode.comfile.gitcode.comis a single Huawei Cloud origin in Beijing, so it is the inbound-to-CN direction that is shaped — not the host, not the client (curl ≡ urllib), not the runner's egress. The rate also swings ~4.6× run to run, so no fixed per-asset cap can be both safe and useful. At 0.012 MB/s a 34.8 MB asset needs ~45 minutes; 180s never had a chance.S1 — stop shipping 30 MB of debug info
Almost none of the 34.8 MB was mcpp:
registry/bin/xlingswith debug_info, not stripped)bin/mcppnot stripped)Two defects made that possible:
xlingswas never stripped at any of the four injection sites — onlycp'd;stripthat did exist ran beforemcpp pack, which rebuilds the binary and overwrote it. That is why every release up to and including 2026.7.28.2 shipped an unstrippedbin/mcpp, while aarch64 (which stages by hand) shipped a stripped one.tools/slim_linux_payload.shnow strips both binaries on the staged payload and asserts the result, so a strip that silently stops working fails the release instead of quietly shipping a fat tarball again. The aarch64 leg's best-effort|| truestrip becomes a hard requirement.macOS and Windows are deliberately untouched: their payloads are already 6.1 MB / 4.2 MB, and stripping a Mach-O invalidates its ad-hoc signature — not worth the risk for ~2 MB when 100% of the problem is the two Linux tarballs.
S2 — concurrent per-asset upload, leg deadline instead of per-asset cap
The shaping is per-connection, so concurrency scales almost linearly (measured): 1/4/8 concurrent 1 MB uploads → 76 s / 80 s / 93 s wall, i.e. 0.013 / 0.050 / 0.086 MB/s aggregate.
MIRROR_MAX_PARALLEL(default 8);MIRROR_UPLOAD_TIMEOUT→MIRROR_LEG_DEADLINE_GH(600s) /MIRROR_LEG_DEADLINE_GTC(2400s).The v0.0.90 rule still holds — nothing is killed and retried inside a round, because the presigned OBS PUT has no multipart/resume (confirmed in #301). Exhausting the budget ends the leg and the completeness gate fails loudly, exactly as before.
Net: the gitcode leg goes from ~45 min + a manual step to ~6 min unattended, and every user's download shrinks 7.5×.
Verification
slim_linux_payload.shon the real 2026.7.28.2 payload: 101.6 MB → 13.9 MB of binaries, tarball 34.81 → 4.62 MB; the stripped mcpp reports its version and builds+runs animport stdproject.mirror_res.shrun against the existing2026.7.28.2tag — all assets skipped as already-serving, gate 16/16 × 200, exit 0.0.0.0-mirrortesttag: 4 concurrent uploads, gitcode leg 6 s, gate 8/8, exit 0. Artifacts deleted afterwards; the real 2026.7.28.2 release verified intact (200 36498305).mcpp build+mcpp test(37 binaries) green locally.release.ymlis not exercised by PR CI — it runs on tag/dispatch only — so S1 gets its first real proof at the next release. That is precisely why the new step asserts instead of hoping.