Skip to content

fix: strip the shipped payload and mirror assets concurrently - #302

Merged
Sunrisepeak merged 1 commit into
mainfrom
fix/strip-payload-and-parallel-mirror
Jul 28, 2026
Merged

fix: strip the shipped payload and mirror assets concurrently#302
Sunrisepeak merged 1 commit into
mainfrom
fix/strip-payload-and-parallel-mirror

Conversation

@Sunrisepeak

Copy link
Copy Markdown
Member

Implements S1 + S2 from the measurement in #301.

Why

The GitCode mirror leg of publish-ecosystem has failed four releases in a row — 0.0.94, 0.0.97, 0.0.105, 2026.7.28.2 — always on the biggest tarballs, each costing ~5 min of manual gtc release upload. The 180s per-asset cap was tuned three times without anyone measuring what it was capping. #301 measured it:

direction endpoint rate
↑ GitHub US runner file.gitcode.com 0.012 MB/s ← the failing path
↓ same runner gitcode.com 3.87 MB/s
↑ same runner github.com 16 MB/s
↑ mainland-CN host file.gitcode.com 1.84 MB/s

file.gitcode.com is a single Huawei Cloud origin in Beijing, so it is the inbound-to-CN direction that is shaped — not the host, not the client (curl ≡ urllib), not the runner's egress. The rate also swings ~4.6× run to run, so no fixed per-asset cap can be both safe and useful. At 0.012 MB/s a 34.8 MB asset needs ~45 minutes; 180s never had a chance.

S1 — stop shipping 30 MB of debug info

Almost none of the 34.8 MB was mcpp:

before after
registry/bin/xlings 97.3 MB (with debug_info, not stripped) 11.1 MB
bin/mcpp 4.3 MB (not stripped) 2.9 MB
linux-x86_64.tar.gz 34.81 MB 4.62 MB (7.5×)

Two defects made that possible:

  • the vendored xlings was never stripped at any of the four injection sites — only cp'd;
  • the x86_64 strip that did exist ran before mcpp pack, which rebuilds the binary and overwrote it. That is why every release up to and including 2026.7.28.2 shipped an unstripped bin/mcpp, while aarch64 (which stages by hand) shipped a stripped one.

tools/slim_linux_payload.sh now strips both binaries on the staged payload and asserts the result, so a strip that silently stops working fails the release instead of quietly shipping a fat tarball again. The aarch64 leg's best-effort || true strip becomes a hard requirement.

macOS and Windows are deliberately untouched: their payloads are already 6.1 MB / 4.2 MB, and stripping a Mach-O invalidates its ad-hoc signature — not worth the risk for ~2 MB when 100% of the problem is the two Linux tarballs.

S2 — concurrent per-asset upload, leg deadline instead of per-asset cap

The shaping is per-connection, so concurrency scales almost linearly (measured): 1/4/8 concurrent 1 MB uploads → 76 s / 80 s / 93 s wall, i.e. 0.013 / 0.050 / 0.086 MB/s aggregate.

  • assets within a leg now upload in parallel under MIRROR_MAX_PARALLEL (default 8);
  • MIRROR_UPLOAD_TIMEOUTMIRROR_LEG_DEADLINE_GH (600s) / MIRROR_LEG_DEADLINE_GTC (2400s).

The v0.0.90 rule still holds — nothing is killed and retried inside a round, because the presigned OBS PUT has no multipart/resume (confirmed in #301). Exhausting the budget ends the leg and the completeness gate fails loudly, exactly as before.

Net: the gitcode leg goes from ~45 min + a manual step to ~6 min unattended, and every user's download shrinks 7.5×.

Verification

  • slim_linux_payload.sh on the real 2026.7.28.2 payload: 101.6 MB → 13.9 MB of binaries, tarball 34.81 → 4.62 MB; the stripped mcpp reports its version and builds+runs an import std project.
  • Isolated harness for the concurrent launcher: bounded concurrency, ordered log replay, correct per-asset success/failure attribution.
  • Full real mirror_res.sh run against the existing 2026.7.28.2 tag — all assets skipped as already-serving, gate 16/16 × 200, exit 0.
  • Real parallel-upload run against BOTH hosts on a throwaway 0.0.0-mirrortest tag: 4 concurrent uploads, gitcode leg 6 s, gate 8/8, exit 0. Artifacts deleted afterwards; the real 2026.7.28.2 release verified intact (200 36498305).
  • mcpp build + mcpp test (37 binaries) green locally.

⚠️ release.yml is not exercised by PR CI — it runs on tag/dispatch only — so S1 gets its first real proof at the next release. That is precisely why the new step asserts instead of hoping.

….29.1)

The GitCode mirror leg of `publish-ecosystem` has failed four releases in a
row — 0.0.94 / 0.0.97 / 0.0.105 / 2026.7.28.2 — each time on the biggest
tarballs, each time costing ~5 minutes of manual `gtc release upload`. The
per-asset 180s cap was tuned three times without anyone measuring what it was
capping. Probe PR #301 measured it:

  GitHub US runner -> file.gitcode.com   0.012 MB/s   <- the failing path
  same runner      <- file.gitcode.com   3.87  MB/s
  same runner      -> github.com        16     MB/s
  mainland-CN host -> file.gitcode.com   1.84  MB/s

file.gitcode.com is a single Huawei Cloud origin in Beijing, so it is the
inbound-to-CN direction that is shaped — not the host (the same runner
downloads from it at 3.87 MB/s), not the client (curl and urllib measure
identically), and not the runner's egress (16 MB/s to GitHub). The rate also
swings ~4.6x run to run, so no fixed per-asset cap can be both safe and
useful. At 0.012 MB/s a 34.8MB asset needs ~45 minutes; 180s never had a
chance.

Two fixes, both measured rather than guessed.

S1 — stop shipping 30MB of debug info. Of the 34.8MB linux-x86_64 tarball,
almost none was mcpp: the vendored `registry/bin/xlings` shipped at 97.3MB
with full debug info (86.9MB on aarch64) against a 4.3MB mcpp. Stripping both
takes the tarball to 4.62MB — 7.5x — and the stripped binaries were verified
working (`--version`, `new`, `build`, `run`). Two defects made this possible:

  * the vendored xlings was never stripped at any of the four injection sites,
    only `cp`'d;
  * the x86_64 `strip` that DID exist ran before `mcpp pack`, which rebuilds
    the binary and overwrote it — which is why every release up to and
    including 2026.7.28.2 shipped an unstripped bin/mcpp while aarch64 (which
    stages by hand) shipped a stripped one.

tools/slim_linux_payload.sh now strips both binaries on the STAGED payload and
ASSERTS the result, so a strip that silently stops working fails the release
instead of quietly shipping a fat tarball again. The aarch64 leg's
best-effort `|| true` strip is now a hard requirement. macOS and Windows are
deliberately left alone: their payloads are already 6.1MB / 4.2MB, and
stripping a Mach-O invalidates its ad-hoc signature.

S2 — mirror a host's assets concurrently, and bound the LEG instead of each
asset. The shaping is per-connection: 1/4/8 concurrent 1MB uploads took
76s/80s/93s wall (6.6x aggregate at N=8). Assets within a leg now upload in
parallel under MIRROR_MAX_PARALLEL, and MIRROR_UPLOAD_TIMEOUT is replaced by
MIRROR_LEG_DEADLINE_GH/GTC. The v0.0.90 rule still holds — nothing is killed
and retried inside a round, because the presigned OBS PUT has no
multipart/resume (confirmed in #301); exhausting the budget ends the leg and
the completeness gate fails loudly as before.

Together the gitcode leg goes from ~45 min plus a manual step to ~6 min
unattended, and every user's download shrinks 7.5x.

Verified locally: slim on the real 2026.7.28.2 payload (101.6MB -> 13.9MB of
binaries, 34.81MB -> 4.62MB tarball, stripped mcpp builds and runs an
`import std` project); an isolated harness for the concurrent launcher
(bounded concurrency, ordered log replay, correct per-asset success/failure);
a full real mirror_res.sh run against the existing 2026.7.28.2 tag (all
skipped, gate 16/16); and a real parallel-upload run on a throwaway
`0.0.0-mirrortest` tag against BOTH hosts (gate 8/8, exit 0), whose artifacts
were deleted afterwards with the real release verified intact.

NB: release.yml is not exercised by PR CI — it runs on tag/dispatch only — so
S1 lands its first real proof at the next release, where the new assertions
fail loudly rather than silently.
@Sunrisepeak
Sunrisepeak merged commit fd27314 into main Jul 28, 2026
15 checks passed
@Sunrisepeak
Sunrisepeak deleted the fix/strip-payload-and-parallel-mirror branch July 28, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant