From 3c53c18de247ce1fb46f01fc4dc421073c6b5184 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 07:51:46 +0800 Subject: [PATCH 1/8] fix(pack): the build machine does not travel, and packages ship stripped (#460) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `kind = "shared"` package kept the DT_RUNPATH the link gave it — a list of absolute paths into the BUILD MACHINE's store — so on any other machine the consumer died with `libstdc++.so.6: cannot open shared object file`. The issue's suggested fix does not work, and that is the whole design. Measured on a real package with the build machine's store made unreachable: stale absolute DT_RUNPATH consumer's DT_RPATH inherited? no rc=127 no tag at all YES ok DT_RUNPATH = $ORIGIN no rc=127 DT_RUNPATH = "" (what --set-rpath '' writes) no rc=127 An object carrying ANY DT_RUNPATH makes the loader skip the whole inherited DT_RPATH chain for that object's dependencies. So the criterion is "there is no tag", and removing it is the right answer rather than a compromise: the consumer's own DT_RPATH is the same closure — payload, package dir, SubOS farm — resolved on the machine that will actually run it. New `mcpp.pack.relocate` edits PT_DYNAMIC in process (delete the slot, shift the tail, pad with DT_NULL; same file length) instead of shelling out to patchelf. Library packs are cross-target by construction and have no host gate, so `sandbox_patchelf` resolves to nothing on a macOS or Windows host — and the application packer's shape for that is `if (!patchelf.empty())`, i.e. silently do nothing. ELF32 and big-endian are covered by unit tests; no CI job produces one and `--target` can. Mach-O LC_RPATH is read and reported, not yet rewritten. Also refuses `mcpp pack ` for a Mach-O artifact. That path resolves the dependency closure with `LD_TRACE_LOADED_OBJECTS=1 ''`, which is glibc's variable — dyld ignores it and RUNS THE PROGRAM, then parses its stdout as a dependency table and reports `Packed`. Keyed on the format, not the host, like the `_WIN32` refusal beside it. Never noticed because the e2e harness grants the `pack` capability only where elf+patchelf exist, i.e. Linux. And the third silent one: the SONAME alias' copy fallback read `leg.artifact` rather than the staged file. Byte-identical while nothing modified the staging copy; with relocate and strip in place it would ship an unprocessed library under the exact name the loader asks for, on the machines where create_symlink fails. Packaging now builds release and strips what it ships. Only the profile FALLBACK changes (dev -> release); `--profile` and `[build] default-profile` still win, so pack never produces flags `mcpp build` would not. Stripping follows dh_strip's division, and the archive row is measured: `--strip-all` on a `.a` removes the archive symbol index and the consumer's link fails with `archive has no index; run ranlib to add one`, while `--strip-debug` links and runs. Shared libraries get `--strip-unneeded` (keeps .dynsym), executables `--strip-all`, and bundled third-party .so files nothing at all. New `--profile` / `--no-strip` / `--debug-symbols DIR` and `[pack] strip` / `[pack] debug_symbols`; `--debug-symbols` separates rather than discards and adds a .gnu_debuglink. e2e 264 puts the defect BACK with patchelf and requires the consumer to FAIL before restoring it: 251 consumed the package on the machine that built it, so it was green throughout this bug's life. The guard reads the DYNAMIC ENTRIES, never the file's bytes — the dead string stays in .dynstr (patchelf leaves the identical residue; .dynstr is tail-merged and deleting it cannot be shown safe), and a byte-pattern check would also have flipped to green for an unrelated reason the day stripping landed. --- ...6-08-20-issue460-shared-library-runpath.md | 605 ++++++++++++++++++ ...26-08-20-pack-and-consumer-model-review.md | 416 ++++++++++++ CHANGELOG.md | 88 +++ docs/02-pack-and-release.md | 75 ++- docs/12-binary-distribution.md | 68 +- docs/zh/02-pack-and-release.md | 64 +- docs/zh/12-binary-distribution.md | 60 +- mcpp.toml | 2 +- src/build/prepare.cppm | 21 +- src/cli.cppm | 9 + src/cli/cmd_publish.cppm | 10 + src/manifest/toml.cppm | 4 + src/manifest/types.cppm | 14 + src/pack/library.cppm | 126 +++- src/pack/library_pipeline.cppm | 23 + src/pack/pack.cppm | 143 ++++- src/pack/pipeline.cppm | 29 +- src/pack/relocate.cppm | 377 +++++++++++ src/pack/strip.cppm | 228 +++++++ src/toolchain/clang.cppm | 8 - src/toolchain/registry.cppm | 77 ++- src/version.cppm | 2 +- .../215_pack_has_no_build_machine_paths.sh | 58 +- tests/e2e/264_pack_library_is_relocatable.sh | 155 +++++ tests/e2e/265_pack_strips_but_stays_usable.sh | 174 +++++ tests/e2e/266_pack_refuses_a_macho_program.sh | 94 +++ tests/e2e/_elf_tag.sh | 82 +++ tests/unit/test_pack_relocate.cpp | 329 ++++++++++ 28 files changed, 3209 insertions(+), 132 deletions(-) create mode 100644 .agents/docs/2026-08-20-issue460-shared-library-runpath.md create mode 100644 .agents/docs/2026-08-20-pack-and-consumer-model-review.md create mode 100644 src/pack/relocate.cppm create mode 100644 src/pack/strip.cppm create mode 100755 tests/e2e/264_pack_library_is_relocatable.sh create mode 100755 tests/e2e/265_pack_strips_but_stays_usable.sh create mode 100755 tests/e2e/266_pack_refuses_a_macho_program.sh create mode 100644 tests/e2e/_elf_tag.sh create mode 100644 tests/unit/test_pack_relocate.cpp diff --git a/.agents/docs/2026-08-20-issue460-shared-library-runpath.md b/.agents/docs/2026-08-20-issue460-shared-library-runpath.md new file mode 100644 index 00000000..2e7d1881 --- /dev/null +++ b/.agents/docs/2026-08-20-issue460-shared-library-runpath.md @@ -0,0 +1,605 @@ +# `mcpp pack` 的 `kind = "shared"` 产物带走了构建机:#460 的实测、根因与优化方案 + +> 2026-08-20 · issue #460 · 复现于本仓库 HEAD 构建出的 `mcpp 2026.8.19.4`(与报告的 2026.8.18.3 同形) +> 状态:**P0-1 / P0-2 / P0-3 + P2-1 已实施,发布于 2026.8.20.1**;P1-1 / P1-3 已批准未实施 + +--- + +## 0. 一句话结论 + +报告说的现象是对的,但**它给出的期望行为有一半是错的**。 + +> 期望:打包出的 `.so` 应该是可重定位的——没有 `RUNPATH`(**或为 `$ORIGIN`**) + +实测:`$ORIGIN` 修不好这个 bug,空字符串也修不好。因为在 ELF 装载器里, +**DT_RUNPATH 只要*存在*,消费方可执行文件的 DT_RPATH 就不再被继承**——不管 +DT_RUNPATH 里写的是什么。库自己的 `libstdc++.so.6` 从此无解。 + +所以判据不是「路径是可重定位的」,而是「**这条 tag 不存在**」。 + +| 打包出的 `.so` 上的状态 | 消费方 exe 的 DT_RPATH 是否被继承 | 另一台机器上的结果 | +|---|---|---| +| `DT_RUNPATH = <构建机 store 路径>`(**今天的行为**) | 否 | `libstdc++.so.6: cannot open shared object file`,rc=127 | +| **无 tag**(`patchelf --remove-rpath`) | **是** | **`ok=42`** | +| `DT_RUNPATH = $ORIGIN`(报告建议的另一半) | 否 | 同样 rc=127 | +| `DT_RUNPATH = ""`(`--set-rpath ''`,最容易写出来的「修复」) | 否 | 同样 rc=127 | +| `DT_RPATH = <构建机 store 路径>`(`--force-rpath`) | 是 | `ok=42`(但违反 loader 契约,见 §5.1) | + +四行都是在**真实的 `mcpp pack` 产物 + 真实的 mcpp 消费方**上跑出来的,复现命令见附录 A。 + +--- + +## 1. 现状:两个 packer,一条契约,只有一个实现了它 + +`mcpp pack` 有两条互不相干的实现路径,由 `[targets.].kind` 分流 +(`src/cli/cmd_publish.cppm::cmd_pack` → `mcpp::pack::route_pack_target`): + +| | 应用打包 | **库打包** | +|---|---|---| +| 入口 | `mcpp::pack::build_and_pack` (`src/pack/pipeline.cppm`) | `mcpp::pack::build_and_pack_library` (`src/pack/library_pipeline.cppm:111`) | +| 落地 | `run_pack` (`src/pack/pack.cppm:~954`) | `run_library_pack` (`src/pack/library.cppm:197`) | +| 问「运行时需要什么」 | 是(`ldd` 闭包 + 捆绑) | 否(问的是「消费方要编什么、能链什么」) | +| **ELF 重定位** | **有**(§1.1) | **无**(§1.2) | +| 宿主/格式拒绝 | 有(`_WIN32` 上产 ELF 直接拒绝并给理由) | 无 | + +两个文件都写了很长的注释解释自己在小心什么,但**「产物不能带走构建机」这一条只写在 +应用侧**。 + +### 1.1 应用侧已经做对了,而且写了理由 + +`src/pack/pack.cppm:1028-1096`: + +```cpp +auto patchelf = sandbox_patchelf(cfg); +if (!patchelf.empty()) { + const char* rpath = toBundle.empty() ? "" : "$ORIGIN/../lib"; + set_search_path(bundledBinary, rpath, loader::Form::Executable, patchelf); + + // EVERY BUNDLED LIBRARY, not just the executable. + // ... + // /xim-x-glibc/2.44/lib64 : /xim-x-gcc/16.1.0/lib64 + // : /compat-x-glx-runtime/…/lib : $ORIGIN + // Those directories do not exist on the target ... + for (auto const& dep : toBundle) { ... set_search_path(staged, "$ORIGIN", ...); } +``` + +注释里那段 store 路径,和 #460 贴出来的 `readelf -d` 输出**是同一段**。也就是说: +这个缺陷的形状在 2026-08-11 就已经被完整地描述过一次,并且在应用侧修好了。 + +### 1.2 库侧一行 ELF 处理都没有 + +`src/pack/library.cppm:247-337`,每条 leg 做的事: + +```cpp +auto dst = plan.stagingRoot / "lib" / leg.triple / name; +copy_into(leg.artifact, dst); // 255 —— 只是 copy_file +... +if (leg.shared && !leg.soname.empty()) { create_symlink(name, alias); } // 287-294 +... +.digest = file_digest(dst), // 333 +``` + +全文 grep 佐证: + +``` +$ grep -n "patchelf\|set_search_path\|set_rpath" src/pack/library.cppm src/pack/library_pipeline.cppm +(无输出) +``` + +`.so` 就这样带着链接期的 RUNPATH 原封不动进了 tarball。 + +### 1.3 那条 RUNPATH 是谁写进去的 + +不是 bug,是设计——**dev 构建需要它**: + +- `src/toolchain/linkmodel.cppm:123-134` —— payload/sysroot 模式下,每个 `libDirs` + 条目发一对 `-L -Wl,-rpath,`(glibc payload、gcc payload)。 +- `src/build/flags.cppm:624-637` —— SubOS farm 作为 `ldRuntimeFallback` 挂在最后。 + +于是本机构建出的 `.so` 拿到: + +``` +DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 + : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 + : /registry/subos/default/lib +``` + +`SharedLibrary` 形态拿 **DT_RUNPATH** 而不是 DT_RPATH,是 `src/build/loader_contract.cppm:49-56` +明确规定的,而且是实测出来的(给库强上 DT_RPATH 会让 `eglInitialize` 挂掉, +openxlings/xlings#593)。**这条契约本身没有问题**,问题在于打包时没人把它取下来。 + +### 1.4 消费方本来是能自己解决的 + +`src/pack/manifest_emit.cppm:273-280` 为 shared 包发: + +```toml +[runtime] +runtime_search_dirs = ["lib/x86_64-linux-gnu"] +``` + +这条经 `linkIntent.runtimeSearchDirs` → `src/build/flags.cppm:362` 变成消费方的 +`-Wl,-rpath`。实测消费方 exe 拿到的是: + +``` +0x0f (RPATH) [/…/xim-x-glibc/2.44/lib64 ← Payload + : /…/xim-x-gcc/16.1.0/lib64 ← Payload + : /lib/x86_64-linux-gnu ← Package(来自 runtime_search_dirs) + : /registry/subos/default/lib] ← SubosFarm +``` + +**这正是打包 `.so` 需要的那个闭包,只不过是在「将要运行它的那台机器上」解析的。** +所以正确答案不是给 `.so` 换一个可重定位的路径,而是**把这条 tag 删掉,让它落到 +消费方的 DT_RPATH 上**——那份地址是对的、是本机的、而且是 mcpp 自己算的。 + +> 注意实测里的一个细节:消费方 exe 的 `DT_NEEDED` 里**没有 `libstdc++.so.6`** +> (它自己的代码没有拉到 libstdc++ 符号)。所以进程里通往 libstdc++ 的**唯一**路径 +> 就是 `libmathkit.so.1` 自己的解析——而那条路被它自己的 DT_RUNPATH 堵死了。 + +--- + +## 2. 实测 + +全部在本机跑完,mcpp 用 `target/x86_64-linux-gnu/5adc2f74a17360b2/bin/mcpp`(2026.8.19.4)。 + +### 2.1 复现(与 issue 逐字一致) + +fixture 就是 `tests/e2e/251_pack_library_shared.sh` 里的 `mathkit`: + +``` +$ mcpp pack mathkit-shared + Packed …/target/dist/mathkit-0.1.0-x86_64-linux-gnu-gcc16-libstdcxx16-c++23.tar.gz + +$ readelf -d /lib/x86_64-linux-gnu/libmathkit-shared.so + (NEEDED) [libstdc++.so.6] + (NEEDED) [libm.so.6] + (NEEDED) [libgcc_s.so.1] + (NEEDED) [libc.so.6] + (SONAME) [libmathkit.so.1] + (RUNPATH) [/home/speak/.mcpp/registry/data/xpkgs/xim-x-glibc/2.44/lib64: + /home/speak/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/lib64: + /home/speak/.mcpp/registry/subos/default/lib] +``` + +### 2.2 模拟「另一台机器」的四态实验 + +把包里 `.so` 的 RUNPATH 改写成**同形但不存在**的路径 +(`/other-machine/.mcpp/registry/…`),消费方 exe 不重新链接,直接跑: + +| 状态 | `.so` 的 tag | rc | 输出 | +|---|---|---|---| +| S1 今天的行为 | `RUNPATH=[/other-machine/…]` | **127** | `error while loading shared libraries: libstdc++.so.6: cannot open shared object file` | +| S2 `--remove-rpath` | 无 | **0** | `ok=42` | +| S3 `--set-rpath '$ORIGIN'` | `RUNPATH=[$ORIGIN]` | **127** | 同 S1 | +| S4 `--set-rpath ''` | `RUNPATH=[]` | **127** | 同 S1 | + +S1 的输出**与 #460 报告的错误信息一字不差**。 + +### 2.3 隔离出机制(合成最小例,不依赖 mcpp) + +三个文件:`libdep.so` 在 `deps/`;`libfoo.so` NEEDED `libdep.so`;`app` 的 +`DT_RPATH` 含 `deps/`(`--disable-new-dtags`)。 + +| `libfoo.so` 上的状态 | 结果 | +|---|---| +| `DT_RUNPATH=/nonexistent/…` | `libdep.so: cannot open shared object file`,rc=127 | +| 无 tag | `ok=42` | +| `DT_RUNPATH=$ORIGIN` | rc=127 | +| **`DT_RPATH=/nonexistent/…`**(`--force-rpath`) | **`ok=42`** | + +最后一行是关键对照:**同样是失效路径,写成 DT_RPATH 就不阻断继承,写成 DT_RUNPATH 就阻断**。 +所以这不是「路径找不到」,是「tag 的种类关掉了整条继承链」。 + +### 2.4 `LD_LIBRARY_PATH` 能盖住它,但 `mcpp run` 这次没盖住 + +| 启动方式 | 结果 | +|---|---| +| 裸执行 | rc=127 | +| `LD_LIBRARY_PATH=` 显式 | `ok=42` | +| `mcpp run` | **rc=127** | + +装载器的顺序是 DT_RPATH → `LD_LIBRARY_PATH` → DT_RUNPATH,所以 `LD_LIBRARY_PATH` +**不受** DT_RUNPATH 抑制。但 `compute_run_env` 只把 `plan.runtimeLibraryDirs` +(依赖产物目录)放进去,不含工具链 payload,所以 `mcpp run` 忠实地复现了失败。 + +⇒ **守卫可以用 `mcpp run`,只要让构建机路径不可达。**(见 §5.3) + +--- + +## 3. 根因 + +### 3.1 装载器物理 + +glibc 解析 `libfoo.so` 的 DT_NEEDED 时: + +1. 沿 loader 链走 DT_RPATH(`libfoo.so` → 加载它的 exe → …) +2. `LD_LIBRARY_PATH` +3. **`libfoo.so` 自己的** DT_RUNPATH(只有自己的,不继承) +4. `ld.so.cache` +5. 默认目录 + +而 `elf_get_dynamic_info` 在 **每个对象** 上执行「若有 DT_RUNPATH 则丢弃 DT_RPATH」。 +实测(§2.3 最后一行)说明:对携带 DT_RUNPATH 的对象,**第 1 步整条链都不再生效**。 + +于是:一个 tag 的存在,把消费方精心算好的四段 DT_RPATH 全部作废,只留下构建机上 +那三个不存在的目录 + ld.so.cache。ld.so.cache 里没有 mcpp payload 的 libstdc++ +(它就不该在),于是 127。 + +### 3.2 三层遮蔽,所以这个缺陷是外部用户报上来的 + +**遮蔽一:e2e 251 全程在同一台机器上。** +`tests/e2e/251_pack_library_shared.sh:97-118` 用 `path = ""` 依赖 + `mcpp run`, +包和构建机 store 都在原地。RUNPATH 指向的目录**真的存在**,于是永远绿。 +文件开头自己写的 "the process starts and the loader resolves the SONAME" 是真的—— +只是它解析的是构建机那份。 + +**遮蔽二:`215_pack_has_no_build_machine_paths.sh` 的清扫到不了库包。** +这个测试的标题就是「打包产物不得含构建机路径」,清扫逻辑(`$MCPP_HOME` 出现在任何 +ELF 的 RPATH/RUNPATH 里就失败)**正好能抓住 #460**。但它只跑 `mcpp pack`(应用), +从没跑过 `mcpp pack `。一个覆盖面窄于自身声明的检查,对它看不见的东西 +报「clean」。 + +**遮蔽三:mcpp 自己的运行期闭包模型表达不了这条规则。** +`src/platform/elf_runtime.cppm:325-330`: + +```cpp +std::vector dirs; +for (auto const& raw : requester.runpaths) + append_unique_path(dirs, expand_origin(raw, requester.artifact)); +for (auto const& dir : additionalSearchDirs) append_unique_path(dirs, dir); +for (auto const& dir : binding.libraryDirs) append_unique_path(dirs, dir); +``` + +它把「requester 自己的搜索路径」和「上层传下来的搜索路径」做**并集**,即把继承链 +建模成恒为开、且可叠加。真实规则是:requester 带 DT_RUNPATH ⇒ 后两组**全部不适用**。 + +`ElfRuntimeFacts` 里**已经有** `searchPathTag`(`elf_runtime.cppm:52`,`loader_contract` +就靠它判违规),`resolve_needed` 只是没读它。所以这不是「信息不够」,是「同一份信息 +两处解释,其中一处没跟上」。 + +这个模块自己的注释写着: + +> When the model and the artifact disagree about which loader runs, the model +> wins the report and the artifact wins reality. + +——正是这一条,只不过这次分歧不在 loader 上,在 tag 上。 + +### 3.3 政策模块有一个不存在的调用方 + +`src/platform/runtime_search.cppm:107-116`: + +```cpp +// Is this directory part of THIS machine's private state? +// +// `mcpp pack` asks this to decide what may not be baked into a distributable +// artifact. ... +bool is_machine_local(Origin origin); +``` + +实际调用方: + +``` +src/build/prepare.cppm:6696 写 resolution.json 的 machine_local 字段 +src/doctor.cppm:742 打印 [machine-local] 标记 +tests/unit/test_runtime_search.cpp +``` + +**没有任何 packer 调用它。** 两个 packer 各自硬编码了自己的答案:应用侧无条件改写成 +`$ORIGIN`,库侧什么都不做。注释描述的那个架构(策略在一处、打包器去问)从未接线。 + +### 3.4 同一类缺陷在 Mach-O 上解决过一次,ELF 侧漏了 + +`tests/e2e/259_shared_library_macho.sh` 的开头: + +> a library built in `/private/var/folders/…/target/…/bin` records that path, and +> the moment it is packed and extracted somewhere else, every consumer of it fails +> at load time — **on the publisher's machine it works perfectly**. +> +> ⚠️ AND WHY IT ASSERTS THE PATH IS *GONE*. Checking that the consumer runs in +> place proves nothing … so the library is packed, **the producer's whole build +> tree is DELETED**, and only then is the consumer built and run. + +Mach-O 侧的修法是**链接期**的(`ninja_backend.cppm:247` 无条件发 +`-Wl,-install_name,@rpath/`),守卫是**删掉构建树**。 + +ELF 侧:同样的缺陷类,既没有对应的修法,守卫(251)也恰恰是 259 明确点名「证明不了 +任何事」的那种——原地跑。 + +--- + +## 4. 影响范围 + +| 轴 | 结论 | 证据 | +|---|---|---| +| `kind = "shared"` + ELF | **受影响**,产物在另一台机器上不可用 | 实测 §2 | +| `kind = "lib"`(静态) | 不受影响 —— `.a` 是归档,没有动态段 | 格式事实 | +| PE(`.dll`) | 不受影响 —— PE 没有 rpath;DLL 靠 exe 同目录 / PATH 解析 | 读码 | +| Mach-O(`.dylib`) | **需核验**。`install_name` 已是 `@rpath/`(259 已钉),但 `LC_RPATH` 是否会带上 payload 目录取决于该机是否用 payload clang(`linkmodel.cppm:206-209` 会发 `-Wl,-rpath,`)。判据:`otool -l \| grep -A2 LC_RPATH` | 读码 | +| 已发布的二进制库包 | **同样受影响,且不会因为 mcpp 升级而自愈**。注意机制:`mcpp publish` 走的是**源码**发布(`publish/pipeline.cppm:109` 用 `git archive` 打源码 tarball),**不**调 `build_and_pack_library`;二进制库包的发布是作者手动上传 `mcpp pack` 的产物。所以受影响的是**已经躺在 release / 索引里的那些 tarball**,里面就是本节讨论的 `.so` | 读码 | +| 交叉打包 | `run_library_pack` **没有** `pack.cppm` 那样的宿主/格式拒绝,所以 ELF leg 可以从 Windows / macOS 宿主产出(mcpp 支持 Windows 宿主产 Linux ELF,PR#339);而 `sandbox_patchelf` 在那些宿主上根本不存在 | 读码 | +| 应用打包 | **不受影响**,已修(§1.1)。被捆绑的 `.so` 拿 `$ORIGIN`,其未捆绑的系统依赖走 ld.so.cache;`--mode self-contained` 走 wrapper 的 `--library-path`(等价 `LD_LIBRARY_PATH`,不受 DT_RUNPATH 抑制) | 读码 + §2.4 | + +--- + +## 5. 优化方案 + +按优先级排。P0 三条是一个整体:**不带守卫的修复会以同样的方式再次隐身**。 + +### P0-1 · 库包必须做 ELF 重定位:**删掉 tag**,而不是改写它 + +在 `run_library_pack` 的 leg 循环里,`copy_into(leg.artifact, dst)`(`library.cppm:255`) +之后、`file_digest(dst)`(`:333`)之前,插入一步 relocate: + +``` +对 staging 里的每个 ELF 产物: + 读 PT_DYNAMIC + 若无 DT_RPATH / DT_RUNPATH → 什么都不做(不是错误) + 否则 → 删除这两个 tag 的全部条目 +``` + +**为什么是「删除」而不是「改写成 `$ORIGIN`」**:§2.2 的 S3/S4 已经实测否掉了改写。 +更根本的理由在 §1.4:消费方的 DT_RPATH 是同一个闭包在正确的机器上解析的结果, +包括 payload、包目录、SubOS farm 三段。留下任何非空 DT_RUNPATH 都会把它们全部作废。 + +**顺序上的两个坑(都必须一起改)**: + +1. **digest 必须覆盖重定位后的字节。** `file_digest(dst)` 在 `:333`,relocate 必须 + 排在它前面,否则 manifest 记录的 digest 与包里的文件不一致。 +2. **soname 别名的 copy 回退拷的是原始产物。** `library.cppm:293` 是 + `copy_into(leg.artifact, alias)` —— 拷的是 `leg.artifact` 不是 `dst`。今天两者 + 逐字节相同所以看不出来;relocate 落地后,**符号链接创建失败的机器(Windows、 + 部分网络文件系统)会拿到一份未重定位的别名**,而 SONAME 别名恰恰是装载器真正 + 打开的那个名字。改成从 `dst` 拷。 + +**判据(四条都要,缺一条这个修复就会假绿)**: + +- (a) `readelf -d <包里的 .so>` 里**没有** `RPATH` 也没有 `RUNPATH` 行 —— 判据是 tag + 缺席,不是路径为空(S4); +- (b) SONAME 别名(符号链接或副本)与主文件的 (a) 结论相同; +- (c) 包里任何 ELF 的任何字符串都不含 `$MCPP_HOME`(复用 215 的清扫谓词); +- (d) **消费方在构建机状态不可达时仍能启动**(§5.3 的双向探针)。 + +### P0-2 · 机制:进程内改写 PT_DYNAMIC,不要调 patchelf + +**已写原型并实测通过**(附录 B)。做法:`Elf64_Dyn` 数组按 16 字节槽压缩——删掉目标 +槽、后面的整体前移一槽、末尾补 `DT_NULL`。**文件尺寸不变,没有任何偏移需要修**。 + +``` +removed 1 slot(s), 27 slots rewritten in place +size before/after: 17976 / 17976 +readelf -h: ok; readelf -d: RUNPATH 行消失; 消费方: ok=42 +``` + +为什么不用 `sandbox_patchelf`: + +- **交叉打包会静默跳过。** 库打包支持 `--target` 列表(fat 包),且没有宿主拒绝。 + 从 macOS/Windows 宿主产 ELF leg 时 `sandbox_patchelf()` 返回空——今天应用侧对此 + 的处理是 `if (!patchelf.empty())`,即**安静地不做**。在库侧照抄这个形状,等于把 + #460 保留给一半的宿主。 +- **多一个外部工具就多一个「装了没装」的轴**,而这一步的正确性是包能不能用的前提。 +- mcpp 已经有完整的 ELF 读取器(`mcpp.platform.elf_runtime`),缺的只是一个写侧, + 而这个写侧要做的事**只有删槽**——不需要 patchelf 那些搬 segment 的能力。 + +**边界(必须在实现里显式处理,不能默默跳过)**:ELF32 / 大端(RISC-V/ARM 32 位交叉 +leg 会遇到)、`Both`(同时有 DT_RPATH 和 DT_RUNPATH)、非 ELF 输入(`.a`/`.dll`/`.dylib` +直接返回「不适用」)、只读文件权限。 + +**patchelf 缺席时怎么办**:不适用了——进程内实现没有缺席这一说。这也顺带删掉了 +「库打包依赖 sandbox 里装了 patchelf」这条不成文前提。 + +> 建议把这个能力放在新模块 `mcpp.pack.relocate`(或 `mcpp.platform.elf_write`), +> 由**两个 packer 共用**,并让它按 `mcpp::platform::search::is_machine_local` 做决策—— +> 这样 §3.3 里那句注释就第一次变成真的。应用侧现有的 `set_search_path` 保持不变 +> (它要**写入** `$ORIGIN`,是另一个动作),但「哪些条目不许带走」应该由同一个谓词回答。 + +### P0-3 · 守卫:两侧钉,且必须让构建机状态不可达 + +新增 e2e(建议 `264_pack_library_is_relocatable.sh`),`# requires: elf python3`: + +1. `mcpp pack mathkit-shared`; +2. **静态判据**:包里每个 ELF 都断言 §P0-1 的 (a)(b)(c)——复用 215 的 `read_tag` + python 片段(建议抽到 `tests/e2e/_read_elf_tag.sh`,两个测试共用一份实现); +3. **动态判据,双向**: + - 构建消费方 → `mcpp run` → 断言 `ok=42`; + - **故意打回缺陷**:把包里 `.so` 的 RUNPATH 设成 `/nonexistent/…`,再跑 → 断言 + **失败**且信息里有 `cannot open shared object file`; + - 还原 → 再跑 → 断言恢复 `ok=42`。 + +第三步的中间那半是**这个测试存在的理由**:只钉「修好之后能跑」区分不了「守卫生效」 +和「根本没有这道门」——修复前后它都会绿(§3.2 遮蔽一)。反向那半只在缺陷真的能被 +观测到时才会红。 + +> 打回缺陷这一步需要一个能写 RUNPATH 的工具。若不想让测试依赖 patchelf,可以让 +> `mcpp` 暴露一个只在测试里用的内部子命令,或者直接用 python 就地把 `DT_NULL` 槽 +> 改回 `DT_RUNPATH`——反向操作和 P0-2 的正向操作是同一段代码。 + +另外两处: + +- **把 215 的清扫扩到库包**(或在新测试里复用它的谓词)。它的标题声称的范围本来就 + 包含库包。 +- **251 保留原样**,它钉的是「两个名字都在 + 消费方能起来」,不该被改成移植性测试。 + +### P1-1 · 让运行期闭包模型能表达这条规则 —— **已批准(2026-08-20)** + +`resolve_needed`(`elf_runtime.cppm:310`)按 `requester.searchPathTag` 分支: + +``` +requester 带 DT_RUNPATH(含 Both,因为装载器按 RUNPATH 处理) + → dirs = requester.runpaths (+ 非 hermetic 时的 host 默认目录) +否则 + → dirs = requester.runpaths + additionalSearchDirs + binding.libraryDirs (+ host 默认) +``` + +**收益**:mcpp 能在消费方 `build` 阶段就报出「这个预编译 `.so` 的 DT_RUNPATH 会挡住 +你的 RPATH」,而不是等到用户在另一台机器上拿到 127。对**第三方/手写的** +`[[runtime.artifacts]]` 预编译 `.so`(#433 那条线),这是唯一的防线——那些 `.so` +不是 mcpp 产的,P0-1 管不到。 + +**风险(必须先量再定)**:模型收紧会把今天判 `Pass` 的一些产物变成 `Unresolvable`, +而 `Unresolvable` 是 blocking 的。farm 里那些自带 RUNPATH 的第三方库,其依赖在模型里 +将只能从它们自己的 RUNPATH + host 默认目录找——hermetic binding 下没有 ld.so.cache +这一层,可能出现「真实能跑但模型说找不到」。 + +**建议的落法**:先把新规则作为**诊断**(`Inconclusive` + 明确文案)接进去,跑全量 +e2e 与几个真实工程做对照,确认没有新的红,再决定是否升级为 blocking。不要一步到位。 + +### P1-2 · `is_machine_local` 收敛为唯一策略点 + +让 P0-2 的 relocate 走 `is_machine_local`,并让应用侧的 `set_search_path` 决策也从 +它派生。这样「什么不许带走」只有一处定义;`resolution.json` / `doctor` 报的 +`[machine-local]` 与打包器实际剥掉的东西,第一次成为同一个答案。 + +### P1-3 · 打包 shared 目标时,校验它自己的 DT_NEEDED 闭包(读码,未实测) + +`run_library_pack` 只发 `targetName` 这一个产物。如果这个 `shared` 目标链接了同工程的 +另一个 `shared` 目标,那个 `.so` 不在包里,`check_prebuilt` 也只校验**声明过的** +artifact 存在(`prebuilt.cppm:82-91`),不看 `.so` 自己的 DT_NEEDED。 + +删掉 RUNPATH 之后,packer 手里正好有了做这件事所需的一切:staging 里的 `.so` 用 +`inspect_elf_runtime` 读一遍,每个 NEEDED 若既不是系统库、又不在包里、又不在 +`[dependencies]` 里 → 拒绝或警告(带文件名)。这与 `library.cppm:303` 对「没有 +archiver 就不许打包」的既有立场一致:**这类缺陷正是这个 feature 存在的理由**。 + +### P2-1 · 发布出去的是 debug、未 strip、且含发布者绝对源码路径的产物(实测)—— **已批准:默认 release + strip,并给可配置开关(2026-08-20)。设计见 `2026-08-20-pack-and-consumer-model-review.md` §6.1** + +`build_and_pack_library` 里 `BuildOverrides ov` 只设了 `target_triple`,没有 profile。 +实测产物: + +``` +$ file /lib/…/libmathkit-shared.so +ELF 64-bit LSB shared object, …, with debug_info, not stripped + +$ strings -a … | grep +…/b460/mathkit/src/mathkit.cppm +…/b460/mathkit/target/x86_64-linux-gnu/e16a674b43e1ee6f +…/b460/mathkit/src +``` + +这不是运行期缺陷(DWARF 路径找不到只影响调试),但它是: + +- **第二条构建机泄漏**——如果 §P0-1 的判据 (c) 收紧到「全文不含构建机路径」而不是 + 「RPATH/RUNPATH 不含」,这一条会直接把测试打红。**所以 (c) 必须明确写成只查动态 + 段**,否则守卫会因为一个不同的问题而红,读的人会以为 relocate 没生效。 +- 体积与发布质量问题(参考 2026.7.29.1:未 strip 让镜像上传从 34.81MB 降到 4.62MB + 才是真因)。 + +**这是策略决定,不是 bug。已拍板(2026-08-20):默认 release + strip,并提供可选/可配置 +开关。** 具体的键、CLI 形状、与 `dropObjects`/digest 的交互次序,见 +`2026-08-20-pack-and-consumer-model-review.md` §6.1。 + +### P2-2 · `mcpp doctor` 侧的事后检查 + +对已安装的 mcpp-pack 包(`is_distribution_package`)扫一遍其 `[[runtime.artifacts]]` +指向的 ELF,发现带 DT_RPATH/DT_RUNPATH 就报出来。这能覆盖**已经发布出去的**那批 +tarball(§4 最后一行)——它们不会因为 mcpp 升级而自动变好,只能靠重新打包;在那之前 +至少要让使用者能看见原因,而不是拿到一句 `cannot open shared object file`。 + +--- + +## 6. 明确不建议做的 + +| 方案 | 为什么不 | +|---|---| +| 把 `.so` 的 RUNPATH 改写成 `$ORIGIN` | **实测无效**(S3)。任何非空 DT_RUNPATH 都会关掉继承链。 | +| `--set-rpath ''` / 任何「只清空字符串」的工具 | **实测无效**(S4)。tag 还在。判据必须是 tag 缺席。 | +| 给库强制 DT_RPATH(`--force-rpath`) | 实测**能跑**(§2.3),但违反 `loader_contract` 的实测结论:库上的 DT_RPATH 有传递性,会把自己的搜索路径压进其下每一次查找,`eglInitialize` 因此挂过(xlings#593)。用一个已知会炸的机制换另一个。 | +| 链接期就不给 `shared` 目标发 RUNPATH | 会同时打断 dev 流(工程内 `.so` 的自解析、`ldd` 可读性)和被第三方宿主 `dlopen` 的场景。**dev 产物和 dist 产物的要求本就不同**——差异应该发生在打包这一步,这也正是应用侧的做法。 | +| 把 libstdc++ 捆进库包 | 与 `dist::Role::SharedLibrary` 的契约直接冲突:一个自带静态 libstdc++ 的 `.so` 导出过 777 个 GLOBAL std 符号,劫持了 exe 自己的 `-static-libstdc++`(#336 / 2026.8.11.3)。 | +| 只改文档、把 `patchelf --remove-rpath` 写成已知规避 | 报告里给的规避**是对的**,但它要求每个使用者都知道这件事;而失败发生在下游用户的机器上,发布者永远看不到。 | + +--- + +## 7. Review 对照表 + +| # | 主张 | 证据等级 | +|---|---|---| +| 1 | 库打包路径对 ELF 零处理 | **实测 + grep** | +| 2 | 产物 RUNPATH 与 issue 逐字一致 | **实测** | +| 3 | 非空 DT_RUNPATH 关掉消费方 DT_RPATH 的继承 | **实测**(真实产物 + 合成对照,4+4 态) | +| 4 | `$ORIGIN` 与空串都修不好 | **实测**(S3/S4) | +| 5 | 删 tag 后消费方在同一条件下能跑 | **实测**(S2) | +| 6 | 进程内删 `Elf64_Dyn` 槽可行、尺寸不变、产物有效 | **实测**(原型,附录 B) | +| 7 | 251 因为「同机」而永远绿;215 覆盖不到库包 | **读码 + 实测** | +| 8 | `resolve_needed` 把继承链建模成并集 | 读码(`elf_runtime.cppm:325-330`) | +| 9 | `is_machine_local` 没有 packer 调用方 | **grep** | +| 10 | 已发布的二进制库包同样受影响(但 `mcpp publish` 本身发的是源码,不是这条路径) | 读码(`publish/pipeline.cppm:109`) | +| 11 | soname 别名的 copy 回退拷的是未重定位的源 | 读码(`library.cppm:293`) | +| 12 | 库包产物是 debug、未 strip、含发布者源码路径 | **实测** | +| 13 | 打包 shared 目标不校验其 DT_NEEDED 闭包 | 读码,**未实测** | +| 14 | Mach-O 的 `LC_RPATH` 是否泄漏 | **未核验**,需在 macOS 上跑 | + +--- + +## 附录 A · 复现步骤 + +```bash +# 1. 造 fixture(与 tests/e2e/251 相同) +mkdir -p mathkit/src && cd mathkit +cat > src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit-shared] +kind = "shared" +soname = "libmathkit.so.1" +EOF + +# 2. 打包并看 RUNPATH +mcpp pack mathkit-shared +PKG=$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*') +readelf -d "$PKG/lib/x86_64-linux-gnu/libmathkit-shared.so" | grep -E 'RUNPATH|NEEDED' + +# 3. 造消费方(path 依赖),构建 +# app/mcpp.toml 里 [dependencies] mathkit = { path = "" } +# exe 会拿到含 payload + 包目录 + farm 的 DT_RPATH + +# 4. 模拟另一台机器:把包里 .so 的 RUNPATH 指向不存在的同形路径 +SO="$PKG/lib/x86_64-linux-gnu/libmathkit-shared.so"; cp "$SO" "$SO.orig" +patchelf --set-rpath '/other-machine/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/lib64' "$SO" +./app/target/*/*/bin/app # → libstdc++.so.6: cannot open shared object file + +# 5. 三个对照 +patchelf --remove-rpath "$SO"; ./app/target/*/*/bin/app # ok=42 +cp "$SO.orig" "$SO"; patchelf --set-rpath '$ORIGIN' "$SO"; ./app/target/*/*/bin/app # 127 +cp "$SO.orig" "$SO"; patchelf --set-rpath '' "$SO"; ./app/target/*/*/bin/app # 127 +``` + +## 附录 B · 进程内删 `Elf64_Dyn` 槽的原型 + +```python +# 删除 DT_RPATH(15) / DT_RUNPATH(29):按 16 字节槽压缩,末尾补 DT_NULL。 +# 文件尺寸不变,不动任何 segment/section 偏移。 +phoff, = struct.unpack_from(' 2026-08-20 · 承接 `2026-08-20-issue460-shared-library-runpath.md` +> 覆盖:架构完备性 / 跨平台 / 兼容性 / 易用性 +> 状态:**B0 / B1 / B3 已实施,发布于 2026.8.20.1**(§7 的前三批); +> B2 / B4 / B5 / B6 仍是设计。已批准的两项(#460 P1-1 = B4,P2-1 = B3)见 §6.1 与 §7。 + +--- + +## 0. 总评 + +**消费侧是这套设计里完成度最高的一半,生产侧缺的是同一根骨架。** + +- **消费侧**几乎没有结构性缺口。「包就是一个普通 mcpp 包」这个决定(零新 section、零新 key) + 是整个设计里最好的一步:它同时买到了老客户端可用、多种依赖形态(path/git/index)复用、 + 以及「不需要为分发再写一条解析路径」。ABI tag 的 don't-care 规则、interface digest、 + 中立链接通道与 GNU 拼写双写——每一条都有实测背书。真缺口只有两个(§3.2 / §3.3)。 + +- **生产侧**的三个缺口是**同一个形状**:应用侧做了一半、库侧一点没做,而中间**没有共同的 + 骨架**去强制两边一致。#460 只是这个形状最先炸出来的那一个: + + | 缺的东西 | 应用侧 | 库侧 | + |---|---|---| + | 二进制格式抽象(ELF/PE/Mach-O) | 有一半(`binfmt` 只服务 PE 闭包;ELF 靠跑二进制;Mach-O 没有) | 完全没有(格式无关地 `copy_file`) | + | 「什么不许带走」的策略点 | 硬编码成 `$ORIGIN` | 没有(#460) | + | profile / strip 轴 | 没有 | 没有 | + +- **跨平台的缺口不是「少支持一个平台」,而是「在做不到的平台上不拒绝」。** Windows 宿主 + 有一段写得很好的拒绝;macOS 没有,而 macOS 走的那条路会**执行用户的程序**(§4.1)。 + 这是本次 review 里优先级最高的一条,高于 #460。 + +--- + +## 1. 缺口全景(轴 × 两个 packer) + +| 轴 | 应用打包 `build_and_pack` | 库打包 `build_and_pack_library` | 判断 | +|---|---|---|---| +| 路由输入 | `[targets.].kind`,一处解析(`route.cppm`) | 同 | ✅ 好 | +| 格式判定 | `binfmt::identify` + `plan.targetIsPe` | 无 | ⚠️ 不对称 | +| 依赖闭包 | ELF=跑二进制;PE=读导入表;**Mach-O=无** | **无** | ⚠️ §3.4 / §4.1 | +| 可分发性重定位 | ELF 有(patchelf) | **无** | ❌ #460 | +| mode 轴(system/vendored/self-contained/static) | 四档 | 无(`--mode` 被 warning 忽略) | ✅ 合理(§6.2) | +| target 轴 | 恰好一个 | 多 leg(fat) | ✅ 合理,原因写在代码里 | +| ABI 轴 | n/a | **只有 triple**(`cfg_predicate_for` 只发 arch/os/env) | ⚠️ §3.2 | +| profile / strip | 无 | 无 | ❌ P2-1(已批准) | +| 宿主门 | Windows 拒绝 + 说理由;**macOS 不拒绝** | 无(不跑产物,但需要 relocate) | ❌ §4.1 / §4.3 | +| 归档确定性 | zip 确定(排序 + 无时间戳);tar **不确定** | 同 | ⚠️ §5.3 | +| 产物自洽性校验 | `ldd` 闭包顺带校验 | **无** | ⚠️ §3.4 | +| 老客户端兼容 | n/a | 静态 + 真实两半(e2e 252) | ✅ 很好 | + +--- + +## 2. 生产侧:三个缺口是同一个形状 + +### 2.1 缺一个「二进制格式」抽象层 + +今天格式知识散在四处,各自用不同的方式回答同一个问题: + +| 谁 | 怎么知道格式 | 怎么读 | +|---|---|---| +| `pack.cppm::run` | `plan.targetIsPe`(布尔) | PE 读导入表 / 否则跑二进制 | +| `pack.cppm::ldd_parse` | 不判定,假设 glibc | `LD_TRACE_LOADED_OBJECTS=1` | +| `binfmt.cppm` | `identify()` → `{Elf, Pe, MachO, Unknown}` | 有完整三态,**但只被 PE 路径用** | +| `library.cppm` | 不判定 | 不读 | +| `elf_runtime.cppm` | ELF only | 真正的解析器 | + +`binfmt::Format` 已经是三态了,`elf_runtime` 已经是一个像样的 ELF 读取器了——缺的是把 +「一个可分发二进制」抽象出来的那一层: + +``` +DistributableImage + ├ format() Elf | MachO | Pe | NotAnImage + ├ dependencies() DT_NEEDED / LC_LOAD_DYLIB / import table + ├ search_paths() DT_RPATH+RUNPATH / LC_RPATH / (PE: 无) + ├ strip_machine_local_paths() #460 的动作,按格式实现,PE 上是 no-op + └ id_name() SONAME / install_name / (PE: 无) +``` + +有了它,#460 的修复是「库 packer 多调一个方法」,而不是「再写一遍 patchelf 调用」; +Mach-O 的闭包也有了落点(§4.1);`is_machine_local` 有了唯一的调用方(§2.2)。 + +**这不是重构洁癖**:今天这四个答题者已经给出过两次不一致的答案(#460 是第一次: +应用侧剥、库侧不剥;§4.1 是第二次:PE 有宿主门、Mach-O 没有)。 + +### 2.2 缺一个「什么不许带走」的策略点 + +`runtime_search.cppm::is_machine_local` 的注释写着「`mcpp pack` asks this」,而 +**没有任何 packer 调用它**(grep 只有 `prepare.cppm` 写 `resolution.json`、`doctor` 打印、 +单测)。两个 packer 各自硬编码。 + +**修法**:relocate 一步按 `Origin` 逐条判定,而不是「全删」或「全换成 `$ORIGIN`」。 +注意 #460 实测的约束——ELF 上**过滤后若结果非空,仍然会阻断继承链**,所以库包这一档的 +正确答案仍然是删空 tag;但**判定过程**应该走 `is_machine_local`,这样: + +- 「这条被剥掉了,因为它是 machine-local」可以打印/记录,而不是静默; +- 一条**不是** machine-local 却仍被剥掉的条目(例如 `$ORIGIN`)成为一个可以被诊断的事件 + ——它意味着这个 `.so` 依赖包里没有的兄弟库(§3.4); +- `resolution.json` 里报的 `[machine-local]` 与打包器实际剥掉的东西第一次是同一个答案。 + +### 2.3 缺 profile / strip 轴 + +`BuildOverrides` 里没有 profile,`PackConfig` 里只有 +`default_mode / include / exclude / also_skip / force_bundle`。**两个 packer 发出去的都是 +dev 构建**(#460 实测:未 strip、含发布者绝对源码路径)。已批准修复,设计见 §6.1。 + +--- + +## 3. 消费侧:完备的部分与两个真缺口 + +### 3.1 做对了的(不需要动) + +- **包 = 普通 mcpp 包**。`[[runtime.artifacts]].provenance = "mcpp-pack …"` 作为标记, + 没有新 section、没有新 key ⇒ 老 mcpp 能构建、不能校验,而不是**加载失败**。这一条避开了 + 「新键让整份 manifest 加载失败,已发布包永远无法采用」的老坑。 +- **双写链接通道**:GNU 拼写的 `ldflags` + 方言中立的 `[target..runtime]`, + `merge_conditional_config` 里「中立形式**替换**库引用、但保留 `-Wl,-Bdynamic`」—— + 这个「只替换它能表达的部分」的规则是对的,而且是 e2e 257 逼出来的。 +- **ABI tag 的 don't-care 规则**:C surface 发短 tag,`standard` 按下界比较而不是相等。 + 形状即声明,不需要 flag。 +- **根目录拒绝**:`prepare.cppm:757` 拒绝把分发包本身当源码树构建,并给出可照抄的修法。 +- **不重建**:`plan.cppm:1472` 跳过分发包的 shared target,理由写清楚了(否则 relink 会 + 产出一个缺少所有实现单元的库)。 +- **诊断质量**:`check_prebuilt` 的「最接近的拒绝」+ 逐维度 need/got,是能直接照做的。 + +### 3.2 缺口 A:leg 选择轴只有 triple,而 ABI tag 有四维 + +`cfg_predicate_for` 只发 `arch / os / env`。于是: + +- 一个 fat 包**无法**同时携带同一 triple 的 gcc16 与 clang22 两条 leg——两个 + `[target.'cfg(all(arch="x86_64", os="linux", env="gnu"))'.build]` 会同时匹配, + `-L`/`-l` 各来一份。 +- 实际形态因此是「一个 ABI 一个包」:`dirName` 在单 leg 时带 abiTag,所以两次 pack 产出 + 两个不同的目录/tarball。但**包名与版本相同** ⇒ 索引里只能有一个 `latest`。 +- 结果:`check_prebuilt` 会给出一句很好的拒绝(「pin `[toolchain]` 到发布者用的那个」), + 但**生态层面没有出路**——gcc 用户和 clang 用户不能共用一个包名+版本。 + +**这是设计上的已知边界还是缺陷,需要你定调。** 两条可能的方向: + +- **A1(小)**:承认边界,把它写进 docs/12 的 limitations 表(现在没有这一行), + 并让 `check_prebuilt` 的拒绝里提示「同一版本可能存在其他 ABI 的包」。 +- **A2(大)**:给索引/解析加一条 ABI 轴——包身份从 `(ns, name, version)` 变成 + `(ns, name, version, abi)`,或在一个包里允许 `[target.'abi(...)']` 谓词。 + 代价很大(触及 `SPEC-001` 包身份与索引 schema),**不建议现在做**,但值得记下来: + 这是「二进制生态」真正要长大时必然撞上的墙。 + +### 3.3 缺口 B:库产物的 digest 记录了,但从来没人校验 + +`manifest_emit` 为每条 leg 写 `digest = "fnv1a:…"`,`check_prebuilt` 只校验 +`role == "interface"` 的那一条。于是: + +- 「interface 与二进制是成对产生的、不可分别替换」这个论点**只钉了一半**; +- 一个被替换/截断的 `.so` 只会在链接或运行时报错,而不是在门口被指出来; +- 而记录一个从不被检查的字段,读代码的人会以为它被检查了(这正是 §2.2 的同一种病)。 + +**修法很小**:`check_prebuilt` 的第 1 步(「artifact 在不在」)顺手把 digest 也比了。 +成本是每次 prepare 多读几个 `.so`/`.a`——可以只在 artifact 的 mtime/size 变化时算, +或者干脆接受(fnv1a 很快)。 + +### 3.4 半个缺口:`.so` 自己的依赖闭包没人看(#460 P1-3) + +`run_library_pack` 只发 `targetName` 一个产物;`check_prebuilt` 只校验**声明过的** +artifact 存在。所以一个 `shared` 目标若链接了同工程的另一个 `shared` 目标,包里缺的那个 +`.so` 不会被任何一侧发现。 + +`#460` 的修复正好把所需的一切放到 packer 手里(它已经要读 ELF 了),所以这两件事应该 +**一起做**:剥路径时顺便把 `DT_NEEDED` 读出来,凡是既不是系统库、又不在包里、又不在 +`[dependencies]` 里的,拒绝或警告并点名文件。 + +--- + +## 4. 跨平台 + +### 4.1 ⚠️ 最高优先级:macOS 宿主上 `mcpp pack`(应用)会**执行用户的程序** + +**读码结论,需在 macOS 上核验。** 路径: + +``` +run(plan, cfg) + → plan.targetIsPe ? run_pe // 否 + → #if defined(_WIN32) 拒绝 // macOS 上不成立 + → #else … ldd_parse(bundledBinary) // ← 走到这里 + cmd = "LD_TRACE_LOADED_OBJECTS=1 '' 2>&1" +``` + +`LD_TRACE_LOADED_OBJECTS` 是 **glibc ld.so 的**变量。dyld 不认它(它的对应物是 +`DYLD_PRINT_LIBRARIES`),所以这条命令在 macOS 上就是**把用户的程序跑起来**。之后: + +- 程序退出码为 0 ⇒ 它的 stdout 被当作 ldd 输出解析 ⇒ 一条依赖都解析不出来 ⇒ + `toBundle` 为空 ⇒ `sandbox_patchelf` 在 macOS 上也不存在 ⇒ `if (!patchelf.empty())` + 整段跳过 ⇒ 产出一个只含二进制和 wrapper 的 tarball,并打印 **`Packed`**; +- 程序退出码非 0 ⇒ 报 `ldd failed on : command exited with N`, + 一个既不提 macOS 也不提 dyld 的错误; +- 程序是交互式/长驻的 ⇒ `mcpp pack` 挂住; +- 程序有副作用(写文件、发网络请求)⇒ 打包这个动作把它做了一遍。 + +`docs/02` 把 macOS dylib 列在「Planned Support」,所以**不支持是已知的**; +问题是**代码不拒绝**。而 Windows 那条完全同源的路径写了一段很好的拒绝: + +> The dependency closure for that format is resolved by running the artifact under +> the target's own dynamic linker, which this machine has no way to do. + +macOS 的情况**更糟**而不是更轻:Windows 宿主根本跑不动那个 ELF,macOS 宿主**跑得动** +那个 Mach-O,只是不会产生 trace。 + +**建议(P0,独立于 #460)**: + +1. 立刻加一道拒绝——按 `binfmt::identify(builtBinary).format == MachO` 判定(而不是按宿主 + `__APPLE__`,理由与 `run()` 现有注释一致:这从来不是宿主的问题,是格式的问题), + 文案照 Windows 那段的形状写清楚为什么; +2. 判据必须**双向**:macOS 上跑一次断言「拒绝且信息提到 Mach-O」;同时保留一条断言 + 「Linux 上同一命令仍然成功」——只钉拒绝分不清「门生效」和「pack 整个坏了」; +3. 真正的支持另开:Mach-O 的闭包是 `otool -L` / 解析 `LC_LOAD_DYLIB`,重定位是 + `install_name_tool -change` + `LC_RPATH`。这是 §2.1 那个抽象层的第一个真实客户。 + +> 同一段代码还有一个较小的隐患:即使在 Linux 上,`ldd_parse` 也是**执行产物**。 +> 对交叉产物(比如 `--target aarch64-linux-gnu`)这同样跑不起来。今天靠什么挡住? +> 值得核验一遍——`binfmt::Ident` 里已经有「这台机器能不能跑这个文件」的语义, +> 说明这个问题被想过,但 `run()` 的分支只用了 `targetIsPe`。 + +### 4.2 三种格式的重定位机制不在同一层 + +| 格式 | 「不带走构建机」靠什么 | 在哪一层 | 守卫 | +|---|---|---|---| +| Mach-O | `-Wl,-install_name,@rpath/` | **链接期**,无条件 | e2e 259,**删掉生产者构建树**后才断言 | +| ELF | 什么都没有(#460) | —— | 无(251 原地跑,永远绿) | +| PE | 不需要(无 rpath);DLL 由消费方 deploy 到 `bin/` | 消费期 | e2e 257 + wine | + +三条各自都合理,但**没有一处写下「这三条是同一个问题的三种答案」**。§2.1 的抽象层就是那个 +写下来的地方;在它落地之前,至少应该在 `loader_contract.cppm` 或新的 relocate 模块的头部 +把这张表写进去——那个文件已经证明了「把规则写一次」在这个仓库里是有效的做法。 + +### 4.3 交叉打包:库侧没有宿主门,而修复所需的工具是 Linux-only + +`run_library_pack` 里没有任何 `_WIN32` / 格式判定,这在今天是**对的**(它不跑产物), +而且是有用的:mcpp 支持 Windows 宿主产 Linux ELF(PR#339),所以一个 Windows CI 可以产 +Linux 的库包。 + +但 #460 的修复会给这条路径引入一个 Linux-only 的依赖(`sandbox_patchelf` 只在 +`/patchelf/*/bin/patchelf` 找)。若照抄应用侧的 `if (!patchelf.empty())`, +**非 Linux 宿主会安静地不剥**——把 #460 原样留给一半的宿主。 + +⇒ 这就是 #460 P0-2 选择「进程内改写 PT_DYNAMIC」的架构理由,不只是省一个依赖。 + +### 4.4 macOS 只能服务一个目标 ⇒ 不能产 fat 包 + +已在 docs/12 的 verification scope 里写明并标注 *impossible*(不是 gap)。✅ 无需处理。 + +--- + +## 5. 兼容性 + +### 5.1 做得好的 + +- **零新 section / 零新 key**,并且 e2e 252 用**两半**钉:静态(生成的 manifest 的 section + 集合是既有词汇的子集,字面列出而不是推导)+ 真实(用 `$MCPP_BOOT` 消费)。 + 第二半在 CI 里因 xvm shim 而跳过——**这一点被诚实地写进了 docs/12**,值得保持。 +- **`cfg()` 而不是裸 triple**:裸 triple 只在 `--target` 时匹配,生成 `cfg()` 才是 + 「在每个 mcpp 上都是同一句话」。这一条是踩过坑的。 + +### 5.2 已发布的包无法追溯修复 + +#460 修好之后,**已经躺在 release / 索引里的 shared 包仍然是坏的**,而且坏在下游用户的 +机器上、发布者永远看不到。所以消费侧的检测(#460 P2-2)不是锦上添花,它是这批包唯一的 +出路:让使用者拿到一句能行动的话,而不是 `cannot open shared object file`。 + +**建议把它提到与 P0 同批**:检测的实现成本极低(读 `[[runtime.artifacts]]` 指向的 ELF, +看有没有 RPATH/RUNPATH),收益是把一个「无解的历史包袱」变成「一句可行动的诊断」。 + +### 5.3 确定性:zip 确定,tar 不确定,而文档只说了前者 + +`docs/02` 写着: + +> The archive is **deterministic**: no timestamps are read, so two packs of the same +> tree are byte-identical and a published checksum means something. + +这句话对 **zip 路径**(PE)成立——`run_pe` 里显式排序、`zip::write` 不读时间戳。 +但 tar 路径是 `tar -czf -C `: + +- 目录遍历顺序来自 readdir,不排序; +- tar 记录每个成员的 mtime / uid / gid / mode; +- gzip 头部默认写入时间戳。 + +⇒ **同一棵树两次打包,Linux/macOS 上得到两个不同的 sha256。** 这与「published checksum +means something」直接冲突,而索引条目正是靠 sha256 认包的。 + +**修法(低成本)**:`tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner` ++ `gzip -n`(或 `--format=ustar`)。注意 BSD tar(macOS 自带)不认 `--sort`, +需要按平台分支或改为自己产 tar——**这正好是「先量再改」的地方:先加一条 e2e 断言 +「两次 pack 的 sha256 相同」,它今天应该是红的。** + +### 5.4 未来加键的规则应该写下来 + +现有设计规避了「新键让老 mcpp 整份 manifest 加载失败」的坑,但那是**这次**的选择, +不是一条被写下来的规则。建议在 `manifest_emit.cppm` 头部把它升格为约束: + +> 这个文件只允许输出在 `<某个版本>` 之前就已被解析的键。要表达新东西,先让它成为 +> 一个**已被解析且被忽略**的键在生态里流通一个发布周期,再开始输出。 + +--- + +## 6. 易用性,以及已批准项的落地形状 + +### 6.1 P2-1 落地:默认 release + strip,可配置 —— **已批准** + +**建议的形状**(两个 packer 共用,不是只给库包): + +```toml +[pack] +profile = "release" # 默认。可写 "dev";未来若有更多 profile 直接沿用同一词汇 +strip = true # 默认。false = 保留符号与 debug 段 +``` + +``` +mcpp pack [--profile dev|release] [--no-strip] [--debug-symbols ] +``` + +**六个必须一起定的点**: + +1. **默认值改变是一次行为变更。** 今天所有人拿到的是 dev 产物;改默认之后同一条命令产出 + 不同的二进制。⇒ 输出里必须说一句(`Packing … (release, stripped)`), + 并写进 CHANGELOG 的 breaking 段。 +2. **次序**:`strip` 必须排在 **relocate 之后、`file_digest(dst)` 之前**。三个动作都改 + 字节,而 digest 是包的凭证。次序写死在一个地方,不要让两个 packer 各排一次。 +3. **strip 用什么**:`llvm-strip`/`strip` 来自**当前 leg 的工具链**(和 `archive_tool` + 同一个来源,`dialect_for(ctx->tc)` 已经有这个抽象),**不能用宿主的 strip** —— + 交叉 leg 上宿主 strip 可能不认目标格式。若解析不到,应该像 `archiver` 那条一样**拒绝** + 而不是静默跳过(`library.cppm:303` 已经确立了这个立场)。 +4. **静态库的 strip 要小心**:`strip` 一个 `.a` 会删掉符号表,库就不可链接了。 + 静态归档只能 `--strip-debug`,不能 `--strip-all`。**这一条必须有单独的 e2e**, + 否则「打包成功、消费方 undefined reference」。 +5. **与 `dropObjects` 的关系**:先删成员再 strip,还是反过来?建议先 `ar d` 后 strip, + 这样 strip 处理的是最终归档。两者都改归档 ⇒ 同样在 digest 之前。 +6. **debug 符号不要直接丢掉**:`--debug-symbols ` 或默认在 `target/dist/` 旁边留一份 + `*.debug`(`objcopy --only-keep-debug` + `--add-gnu-debuglink`),否则用户拿到崩溃栈时 + 没有任何东西可用。这一条可以排在后面做,但**接口要现在留出来**,不然默认 strip 之后 + 再补就是第二次行为变更。 + +**判据**:packed `.so`/`.a` 的 `file` 输出为 `stripped`;`strings` 里不含发布者的绝对 +源码路径;**静态库 strip 后消费方仍能链接并跑出正确结果**(第 4 点); +`--no-strip` 与 `[pack] strip = false` 两条通道各钉一次。 + +> 与 #460 守卫的交互(重要):#460 的「无构建机路径」判据**必须写明只查动态段**。 +> 如果写成「全文不含 `$MCPP_HOME`」,它在 P2-1 落地前会因 DWARF 而红,落地后又会因为 +> strip 掉了而变绿——两次都不是因为 relocate。**一个测试只量一件事。** + +### 6.2 `--mode` 对库目标只是 warning + +``` +--mode is an application-bundle depth and does not apply to the library target 'x' yet; ignoring it +``` + +这个处理是对的(拒绝会让 `mcpp pack --mode static` 在混合工程里变得难用),但那个 +**「yet」**暗示以后会有。值得现在就想清楚:**库包需要 mode 轴吗?** + +我的判断:**不需要 mode,但需要一条「第三方 `.so` 怎么办」的答案。** 今天 docs/12 写着 +「bundling dependencies into the package ❌ declare them instead」,这是一个清楚的立场, +而且和 `[dependencies]` 的传递是自洽的。⇒ 建议把 warning 里的「yet」去掉,改成指向那条 +立场的一句话。措辞暗示的路线图,读的人会当承诺。 + +### 6.3 消费者拿到 127 时的可操作性 + +这是当前整条链路里**唯一一处诊断质量明显低于本仓库水准**的地方,而且正好是终端用户所在的 +位置:`mcpp` 在生产侧和链接侧的每一条错误都能照做,而运行期失败落到的是 ld.so 的 +`cannot open shared object file`——它不提包名、不提 mcpp、不提该做什么。 + +**建议**:`mcpp run` / `mcpp test` 在子进程以 127 退出且 stderr 匹配 +`cannot open shared object file` 时,接管这条信息:用已有的 `resolve_runtime_closure` +跑一遍,把「哪个对象需要它 / 它的搜索路径是什么 / 为什么继承没生效」打出来。 +P1-1(闭包模型学会 RUNPATH 抑制规则)落地后,这段诊断才**说得对**——所以它是 P1-1 的 +自然下游,建议排在同一批。 + +### 6.4 缺一个「验证这个包」的入口 + +今天要回答「我发出去的包能用吗」,只有「换一台机器试试」。建议加 +`mcpp pack --verify `(或 `mcpp doctor --package `): + +- 每个 ELF 的 loader tag 契约 + 有无 machine-local 路径; +- `[[runtime.artifacts]]` 指向的文件都在、digest 都对(§3.3); +- `.so` 的 DT_NEEDED 闭包被包 + `[dependencies]` 覆盖(§3.4); +- interface digest 与 `interface/` 一致。 + +它同时是 §5.2 那批**存量坏包**的检测入口,也是 e2e 守卫可以直接调用的东西—— +守卫用产品自己的检查,比守卫各写一份 python 解析器更不容易腐坏。 + +--- + +## 7. 建议的落地顺序 + +| 批次 | 内容 | 理由 | +|---|---|---| +| **B0** | §4.1 macOS 宿主拒绝(格式判定,不是宿主判定)+ 双向 e2e | 唯一一条「会执行用户程序」的缺陷,且修复极小 | +| **B1** | #460 P0-1/P0-2/P0-3:relocate(进程内 ELF 写)+ digest 次序 + 别名从 `dst` 拷 + 双向守卫 | 用户已报;不带守卫的修复会再次隐身 | +| **B2** | §5.2 / #460 P2-2:消费侧与 `doctor` 检测存量坏包 | 已发布的包唯一的出路,成本极低 | +| **B3** | P2-1 release + strip + 可配置(§6.1 六点)| 已批准;必须排在 B1 之后,否则两个改动在同一个判据上互相干扰 | +| **B4** | P1-1 闭包模型学会 RUNPATH 抑制(先诊断、跑全量 e2e 对照、再考虑升级为 blocking)+ §6.3 的 127 诊断 | 已批准;下游有真实收益,但有把绿判红的风险 | +| **B5** | §2.1 `DistributableImage` 抽象 + §2.2 `is_machine_local` 收敛 + §3.3 digest 校验 + §3.4 闭包校验 | 结构性;做完之后 Mach-O 的真实支持才有落点 | +| **B6** | §5.3 归档确定性(先加红测试)、§6.4 `--verify`、§6.2 措辞 | 收尾 | +| **暂不做** | §3.2 A2(包身份加 ABI 轴) | 触及 SPEC-001 与索引 schema;先按 A1 把边界写进文档 | + +--- + +## 8. 判据清单(review 时对照) + +| # | 主张 | 证据等级 | +|---|---|---| +| 1 | macOS 宿主上应用打包走 glibc 路径并执行产物 | **读码**(`pack.cppm:934` 分支 + `ldd_parse:442`),**需 macOS 核验** | +| 2 | e2e 的 `pack` 能力 = ELF + patchelf ⇒ 应用打包在 macOS 上一条 e2e 都不跑 | **读码**(`tests/e2e/run_all.sh:43-76` 的 `Linux)` / `Darwin)` 分支) | +| 3 | `binfmt::Format` 已是三态但只服务 PE 路径 | 读码 | +| 4 | `is_machine_local` 无 packer 调用方 | **grep** | +| 5 | leg 选择只有 arch/os/env 三维 | 读码(`manifest_emit.cppm:142-161`) | +| 6 | 库 leg 的 digest 记录但从不校验 | 读码(`prebuilt.cppm:137-139` 只匹配 `role == "interface"`) | +| 7 | tar 路径不确定、zip 路径确定,而文档只声明了确定性 | 读码(`pack.cppm:797` vs `:917`)+ docs/02:306 | +| 8 | `PackConfig` 无 profile / strip 键 | 读码(`types.cppm:791-800`) | +| 9 | 静态库不能 `--strip-all`(会删符号表) | 通用知识,**落地前必须实测** | +| 10 | 老客户端兼容用静态+真实两半钉,且 CI 上只跑了静态那半 | 读码 + docs/12 自述 | +| 11 | `--mode` 对库目标 warning 且措辞含 "yet" | 读码(`cmd_publish.cppm:78-82`) | diff --git a/CHANGELOG.md b/CHANGELOG.md index 5345c966..7c68d34b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,94 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.8.20.1] — 2026-08-20 + +### 修复 + +- **`mcpp pack` 打出的 `kind = "shared"` 库带走了构建机,产物在别人机器上起不来 + (#460)。** + + 打包只是把链接产物 `copy_file` 进包里,于是 `.so` 保留了链接期的 + `DT_RUNPATH`——一串指向**构建机** `~/.mcpp/` 的绝对路径。消费者在另一台机器上 + 拿到的是 `libstdc++.so.6: cannot open shared object file`。 + + ⚠️ **issue 里建议的 `$ORIGIN` 修不好它,空串也修不好。** 在真实包 + 真实消费者上 + 实测(把构建机 store 变成不可达): + + | 发货 `.so` 上的状态 | 消费方 `DT_RPATH` 被继承 | 结果 | + |---|---|---| + | 失效绝对路径的 `DT_RUNPATH`(此前的行为) | 否 | rc=127 | + | **没有这条 tag** | **是** | ok | + | `DT_RUNPATH = $ORIGIN` | 否 | rc=127 | + | `DT_RUNPATH = ""` | 否 | rc=127 | + + 关掉继承的是这条 tag 的**存在**而不是内容,所以判据是「tag 不存在」。删掉它也 + 不是妥协:消费方自己的 `DT_RPATH` 是同一个闭包,只不过是在真正要运行它的机器上 + 解析的。 + + 实现是**进程内改写 `PT_DYNAMIC`**(删槽、后移、补 `DT_NULL`,文件尺寸不变), + 不是调 patchelf——库打包支持 `--target` 且没有宿主门,非 Linux 宿主上 + `sandbox_patchelf` 会解析为空,照抄应用侧的 `if (!patchelf.empty())` 等于把这个 + 缺陷留给一半的宿主。新增 `mcpp.pack.relocate`,覆盖 ELF32/64 × 大小端(单测), + Mach-O 只读报告 `LC_RPATH`。 + +- **`mcpp pack` 一个 Mach-O 程序会**执行用户的程序**,然后报告 `Packed`。** + + 非 PE 路径靠 `LD_TRACE_LOADED_OBJECTS=1 ''` 向动态链接器要依赖表,而这个 + 变量是 glibc 的;dyld 不认它,于是那条命令在 macOS 上就是把程序跑起来,程序的 + 输出被当成依赖表解析(解析出零条),然后写出一个只含二进制的包。有副作用的程序 + 会把副作用做一遍,交互式的会把打包器挂住。现在按**产物格式**(而非宿主)拒绝, + 理由与旁边那条 `_WIN32` 拒绝完全同源。macOS 上 `kind = "lib"` / `"shared"` 照常 + 打包——库打包从不运行产物。 + + 为什么此前没人发现:e2e 的 `pack` 能力 = `elf` + `patchelf`,只有 `Linux)` 分支 + 给,所以应用打包在 macOS 上**一条 e2e 都没跑过**。 + +- **共享库包的 SONAME 别名在符号链接失败时会拷到未处理的原始产物。** + + 别名的 copy 回退读的是 `leg.artifact`(构建树里的文件)而不是暂存后的 `dst`。 + 在没有重定位/strip 之前两者逐字节相同,所以看不出来;之后它会在 + `create_symlink` 失败的机器上,用装载器真正要打开的那个名字,发出一份未重定位、 + 未 strip 的库。 + +### 变更 + +- **`mcpp pack` 默认走 release 并 strip 发货产物。** + + 此前两个打包器发的都是 dev 构建:未 strip、带着发布者的绝对源码路径。现在 + profile 的**兜底**从 `dev` 改为 `release`——其余优先级不变(`--profile` > + `[build] default-profile` > 兜底),所以声明过 profile 的工程仍然拿到它声明的 + 那个。 + + 剥什么取决于产物**是什么**,用的是 dh_strip 的分档: + + | 产物 | 参数 | 为什么不能更狠 | + |---|---|---| + | 可执行文件 | `--strip-all` | 没有人链接它 | + | 共享库 | `--strip-unneeded` | 保留 `.dynsym`——那**就是**导出表 | + | 静态归档 | `--strip-debug --enable-deterministic-archives` | ⚠️ `--strip-all` 会删掉归档的**符号索引**,消费方链接时报 `archive has no index; run ranlib to add one`(实测) | + + 被捆绑进 bundle 的第三方 `.so` **不**剥——它们不是 mcpp 构建的。 + + 新增 `--profile` / `--no-strip` / `--debug-symbols ` 与 `[pack] strip`、 + `[pack] debug_symbols`。`--debug-symbols` 是分离而不是丢弃:写出 + `/<产物>.debug` 并给发货产物加 `.gnu_debuglink`。 + + > `[pack] strip` 与 `[profile.].strip` 是两个决定:后者给**链接**加 `-s` + > (碰不到静态归档,也分离不出任何东西),前者管**包里带什么**。 + +### 内部 + +- `mcpp::toolchain::binutils_tool(tc, name)`:四个工具链家族对同一个 binutils 工具 + 的四种拼法,此前只有 `ar` 知道。`archive_tool` 现在由它表达(MSVC 的 `lib.exe` + 仍是特例,因为它不是 binutils 的名字)。 +- `tests/e2e/_elf_tag.sh`:215 与新增的 264 共用同一个 ELF 读取器,两份拷贝会变成 + 「构建机路径」的两个定义。 +- **判据只查动态段,不查文件字节**:重定位删的是条目,字符串留在 `.dynstr` + (`patchelf --remove-rpath` 实测残留完全相同,`.dynstr` 有尾部合并,删不安全)。 + 一个 `grep` 式的判据会把正确重定位的产物报成脏的,而且会在 strip 落地那天因为 + 另一个原因变绿——两次都不是因为重定位。 + ## [2026.8.18.3] — 2026-08-18 ### 新增 diff --git a/docs/02-pack-and-release.md b/docs/02-pack-and-release.md index ff44a63f..caf8166b 100644 --- a/docs/02-pack-and-release.md +++ b/docs/02-pack-and-release.md @@ -133,6 +133,9 @@ mcpp pack --target aarch64-linux-musl # ARM64 equivalent mcpp pack --format dir # output as a directory, no tarball mcpp pack -o myapp.tar.gz # filename only: lands at target/dist/myapp.tar.gz mcpp pack -o /abs/path/myapp.tar.gz # includes a directory: output to the literal path +mcpp pack --profile dev # build with a different profile (default: release) +mcpp pack --no-strip # ship the artifacts as built +mcpp pack --debug-symbols dbg/ # write the separated *.debug files under dbg/ ``` When `-o` is given a bare filename, the output is placed under `target/dist/`; @@ -140,6 +143,47 @@ when it includes a directory (relative or absolute), the literal path is used. For the full set of options, see `mcpp pack --help`. +### What a packed artifact is built with, and what travels inside it + +Two things differ from `mcpp build`, and both exist because a package leaves +this machine: + +**The profile falls back to `release`, not `dev`.** Precedence is unchanged +otherwise — `--profile` beats `[build] default-profile`, which beats the +fallback. Only the last step differs, so a project that states a profile still +gets the one it stated, and `mcpp pack` never produces an artifact built with +flags `mcpp build` would not. + +**Debug information is stripped, and the publisher's paths go with it.** An +unstripped artifact carries DWARF, and DWARF carries the absolute paths of the +producer's source tree and build directory. What is removed depends on what the +artifact *is* — this is dh_strip's division, and the archive row is the one that +matters: + +| artifact | strip flags | why not more | +|---|---|---| +| executable | `--strip-all` | nothing links against it | +| shared library | `--strip-unneeded` | keeps `.dynsym` — that IS the export list | +| static archive | `--strip-debug --enable-deterministic-archives` | `--strip-all` removes the archive **symbol index**, and the consumer's link then fails with `archive has no index; run ranlib to add one` | + +All three also drop `.comment` and `.note`. Section removal is by exact name, so +`.note.gnu.build-id` survives and still pairs with `--add-gnu-debuglink`. + +`--no-strip` (or `[pack] strip = false`) ships the artifacts exactly as built. +`--debug-symbols ` separates the information instead of discarding it: +`/.debug` is written and the shipped artifact gets a +`.gnu_debuglink` pointing at it, which is what a debugger and `debuginfod` +follow. + +> `[pack] strip` is not `[profile.].strip`. The profile key appends `-s` +> to the **link**, which never touches a static archive and cannot separate +> anything; this one governs what the **package** carries. Two different +> decisions, two different names. + +**Bundled libraries are never stripped.** They came out of the store or off the +host, mcpp did not build them, and rewriting somebody else's shared payload for +this bundle's benefit is not the packer's business. + ## Output Layout The tarball contents are wrapped in a single top-level directory whose name @@ -310,6 +354,21 @@ The reverse direction — packing a Linux or macOS artifact *from* Windows — still does not work, and for the original reason: that closure is resolved by the target's own dynamic linker, which a Windows host has no way to run. +#### Packing a Mach-O program is refused — on every host, including macOS + +The same closure step asks the dynamic linker for the dependency list by running +the artifact with `LD_TRACE_LOADED_OBJECTS=1`. That variable is glibc's; dyld +has never heard of it. So on a Mac the command does not trace anything — **it +runs the program**, and whatever the program prints is then parsed as a +dependency table. mcpp refuses instead, and says which mechanism is missing. + +The refusal is keyed on the artifact's **format**, not on the host, for the same +reason the Windows one is: `LD_TRACE_LOADED_OBJECTS` cannot trace a Mach-O from +Linux either. + +A `kind = "lib"` / `"shared"` target packs normally on macOS — a library package +never runs the artifact. This restriction is only for programs. + ## Configuration Packaging behavior is configured via the `[pack]` section in `mcpp.toml`. The @@ -317,9 +376,11 @@ common fields are: ```toml [pack] -default_mode = "static" # override the normal vendored default for bare `mcpp pack` -include = ["share/**", "config/*.toml"] # extra files to bundle -exclude = ["debug/**"] +default_mode = "static" # override the normal vendored default for bare `mcpp pack` +strip = true # default. false ships the artifacts as built +debug_symbols = "dist/debug" # separate the debug info here instead of discarding it +include = ["share/**", "config/*.toml"] # extra files to bundle +exclude = ["debug/**"] # Fine-tune the vendored filtering policy. The configuration key keeps its # established `bundle-project` spelling. @@ -339,7 +400,11 @@ The `static` mode additionally requires a musl toolchain configured under ## Planned Support -macOS dylib, Windows DLL, and distribution formats such as `.deb` / `.rpm` / -AppImage are still on the roadmap. This document evolves alongside the +macOS **program** bundling (the Mach-O dependency closure, via `otool -L` / +`LC_LOAD_DYLIB`, and `install_name_tool` for relocation) is still on the +roadmap; until it lands `mcpp pack ` refuses on that format rather than +producing something that only looks like a bundle. Windows DLL bundling beyond +the current `.zip`, and distribution formats such as `.deb` / `.rpm` / AppImage, +are also on the roadmap. This document evolves alongside the `mcpp pack` implementation; for the latest options, refer to `mcpp pack --help`. diff --git a/docs/12-binary-distribution.md b/docs/12-binary-distribution.md index 6373a778..c54f51da 100644 --- a/docs/12-binary-distribution.md +++ b/docs/12-binary-distribution.md @@ -256,16 +256,77 @@ That is a degradation, not a break, and it is the right direction. But it means **the gate protects new clients only**, which belongs in the release notes of any package published to a mixed-version audience. +## What travels inside a package, and what deliberately does not + +A published package must work on a machine that is not the publisher's. Two +steps enforce that, and both run on every artifact the packer stages. + +### The build machine's loader paths are removed + +A dev build bakes the toolchain's own directories into every shared object: + +```text +DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 + : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 + : /registry/subos/default/lib +``` + +That is correct for a dev build and fatal for a package (issue #460), because +of one rule in the ELF loader: **an object that carries any `DT_RUNPATH` makes +the loader skip the entire inherited `DT_RPATH` chain when resolving that +object's own dependencies.** The consumer's `DT_RPATH` — payload, package +directory, SubOS farm, all computed on the machine that will actually run it — +is therefore not consulted, and the program dies with + +```text +error while loading shared libraries: libstdc++.so.6: cannot open shared object file +``` + +⚠️ **`$ORIGIN` is not the fix.** Measured on a real package with the build +machine's store made unreachable: + +| state on the shipped `.so` | consumer's `DT_RPATH` inherited? | result | +|---|---|---| +| stale absolute `DT_RUNPATH` | no | fails | +| **no tag at all** | **yes** | **runs** | +| `DT_RUNPATH = $ORIGIN` | no | fails | +| `DT_RUNPATH = ""` | no | fails | + +It is the tag's *presence* that disables inheritance, not its contents. So +`mcpp pack` removes the entry rather than rewriting it — and removing it is not +a compromise, it is the right answer: the consumer's own `DT_RPATH` is the same +closure, resolved where it means something. + +The path *string* stays in `.dynstr`, unreferenced. `.dynstr` is tail-merged by +the linker, so a shorter live string can begin inside the dead one and deleting +those bytes cannot be shown safe; `patchelf --remove-rpath` leaves the identical +residue at the identical file size. **A guard for this must therefore read the +dynamic entries, never `grep` the file's bytes** — see +`tests/e2e/_elf_tag.sh`. + +On Mach-O the packer reads `LC_RPATH` and warns when a package would carry one; +rewriting it (`install_name_tool -delete_rpath`) is not automated yet, because +no test in this suite produces a `.dylib` to measure the edit on. + +### Debug information is removed + +See [docs/02](02-pack-and-release.md) for the flags, the per-shape table, and +`--debug-symbols`. The rule that matters for a *library* package: a static +archive is only ever `--strip-debug`ed, because `--strip-all` removes the +archive symbol index and the consumer's link then fails with `archive has no +index; run ranlib to add one`. + ## Current limitations | | status | |---|---| | `kind = "lib"` (static) | ✅ every target, tested on all three | -| `kind = "shared"` on Linux/ELF | ✅ — the package carries both the link name and the SONAME | +| `kind = "shared"` on Linux/ELF | ✅ — the package carries both the link name and the SONAME, and no build-machine loader path | | `kind = "shared"` on PE / MinGW (`*-windows-gnu`) | ✅ — the package carries the `.dll` **and** its import library | -| `kind = "shared"` on Mach-O (`*-macos`) | ✅ — install name is `@rpath/`, so the `.dylib` relocates | +| `kind = "shared"` on Mach-O (`*-macos`) | ✅ — install name is `@rpath/`, so the `.dylib` relocates. `LC_RPATH` is reported, not yet rewritten | | `kind = "shared"` on PE / MSVC (`*-windows-msvc`) | ✅ — mcpp generates the `.def`; see below | | `kind = "shared"` on `*-musl` | ❌ a musl target links statically | +| one package carrying two ABIs for the same triple (gcc **and** clang) | ❌ leg selection is `cfg(arch/os/env)`; publish one package per ABI | | shipping prebuilt BMIs | ❌ not attempted; BMIs are compiler-build-exact | | bundling dependencies into the package | ❌ declare them instead (above) | | consuming a package with **native `cl.exe`** | ✅ — via the neutral link intent; see below | @@ -415,6 +476,9 @@ The e2e suite gates each test on host capabilities, so "the suite is green" and | Mach-O shared library relocating out of its build tree | — | ✅ | — | | MSVC refusing `kind = "shared"` for the export reason | — | — | ✅ | | a released mcpp consuming a package this one produced | local only | local only | local only | +| a packed `.so` carries no build-machine loader path, **and the guard can see the defect when it is put back** | ✅ | — | — | +| a stripped static archive still links; a stripped shared library still loads; `--no-strip` / `[pack] strip` / `--debug-symbols` from both sides | ✅ | — | — | +| the ELF editor on ELF32 and big-endian | unit test | unit test | unit test | *impossible* is not a gap: a macOS host can serve exactly one target (`host_can_serve`, `registry.cppm`), so a package with two legs cannot be produced diff --git a/docs/zh/02-pack-and-release.md b/docs/zh/02-pack-and-release.md index 08700fb0..e2643f68 100644 --- a/docs/zh/02-pack-and-release.md +++ b/docs/zh/02-pack-and-release.md @@ -99,6 +99,9 @@ mcpp pack --target aarch64-linux-musl # ARM64 等价写法 mcpp pack --format dir # 输出为目录,不打包 tarball mcpp pack -o myapp.tar.gz # 仅文件名:落到 target/dist/myapp.tar.gz mcpp pack -o /abs/path/myapp.tar.gz # 含目录:按字面路径输出 +mcpp pack --profile dev # 换一个 profile 构建(默认 release) +mcpp pack --no-strip # 按构建原样发货,不剥符号 +mcpp pack --debug-symbols dbg/ # 把分离出的 *.debug 写到 dbg/ ``` `-o` 接受裸文件名时自动归到 `target/dist/`;含目录(相对或绝对) @@ -106,6 +109,39 @@ mcpp pack -o /abs/path/myapp.tar.gz # 含目录:按字面路径输出 完整选项参见 `mcpp pack --help`。 +### 打包产物用什么构建,里面带什么走 + +与 `mcpp build` 有两点不同,都因为「这个产物要离开本机」: + +**profile 的兜底是 `release` 而不是 `dev`。** 其余优先级不变 —— +`--profile` > `[build] default-profile` > 兜底。只有最后一步不同,所以声明过 +profile 的工程仍然拿到它声明的那个,`mcpp pack` 也不会产出一个 `mcpp build` +产不出来的 flag 组合。 + +**调试信息会被剥掉,发布者的路径随之消失。** 未 strip 的产物带着 DWARF,而 +DWARF 带着发布者源码树与构建目录的绝对路径。剥什么取决于产物**是什么** —— +这是 dh_strip 的分档,而其中归档那一行是要命的: + +| 产物 | strip 参数 | 为什么不能更狠 | +|---|---|---| +| 可执行文件 | `--strip-all` | 没有人链接它 | +| 共享库 | `--strip-unneeded` | 保留 `.dynsym` —— 那**就是**导出表 | +| 静态归档 | `--strip-debug --enable-deterministic-archives` | `--strip-all` 会删掉归档的**符号索引**,消费方链接时报 `archive has no index; run ranlib to add one` | + +三档都会去掉 `.comment` 与 `.note`。段删除按**精确名字**匹配,所以 +`.note.gnu.build-id` 会保留,`--add-gnu-debuglink` 仍然有东西可配对。 + +`--no-strip`(或 `[pack] strip = false`)按构建原样发货。 +`--debug-symbols <目录>` 则是分离而不是丢弃:写出 `<目录>/<产物>.debug`, +并给发货的产物加上指向它的 `.gnu_debuglink` —— 调试器与 `debuginfod` 认这个。 + +> `[pack] strip` 不是 `[profile.].strip`。后者是给**链接**加 `-s`, +> 既碰不到静态归档、也无法分离出任何东西;前者管的是**包里带什么**。 +> 两个不同的决定,两个不同的名字。 + +**被捆绑进来的库永远不 strip。** 它们来自 store 或宿主,不是 mcpp 构建的, +为了这一个 bundle 去改写别人的共享载荷不是打包器该做的事。 + ## 产物布局 tarball 内容包在一个顶层目录里,该目录的名字与 tarball 文件名(去掉 @@ -265,9 +301,11 @@ mcpp pack --target x86_64-windows-gnu # 在 Linux 宿主上 ```toml [pack] -default_mode = "static" # 覆盖裸 `mcpp pack` 的正常 vendored 默认值 -include = ["share/**", "config/*.toml"] # 额外打包的文件 -exclude = ["debug/**"] +default_mode = "static" # 覆盖裸 `mcpp pack` 的正常 vendored 默认值 +strip = true # 默认值。false = 按构建原样发货 +debug_symbols = "dist/debug" # 把调试信息分离到这里,而不是丢弃 +include = ["share/**", "config/*.toml"] # 额外打包的文件 +exclude = ["debug/**"] # 微调 vendored 的过滤策略。配置键保留既有的 `bundle-project` 拼写。 [pack.bundle-project] @@ -282,8 +320,24 @@ force_bundle = ["libfoo.so"] # 即使命中 PEP 600 名单也强制打包 `static` 模式还需在 `[target.]` 中配置 musl 工具链,完整写法 参见 [`examples/03-pack-static`](../../examples/03-pack-static/) 的 `mcpp.toml`。 +### Mach-O 程序会被拒绝 —— 在所有宿主上,包括 macOS + +同一步闭包解析是靠 `LD_TRACE_LOADED_OBJECTS=1` **运行产物**来问动态链接器要 +依赖表的。这个变量属于 glibc 的 ld.so,dyld 从来不认(它的对应物是 +`DYLD_PRINT_LIBRARIES`)。所以在 Mac 上这条命令不会 trace 任何东西 —— +**它会把用户的程序跑起来**,然后把程序的输出当成依赖表解析。mcpp 现在直接拒绝, +并在信息里点名缺的是哪个机制。 + +判定按产物的**格式**而不是宿主,理由与 Windows 那条完全相同: +`LD_TRACE_LOADED_OBJECTS` 在 Linux 上也 trace 不了一个 Mach-O。 + +`kind = "lib"` / `"shared"` 目标在 macOS 上照常打包 —— 库打包从不运行产物。 +这条限制只针对程序。 + ## 待支持 -macOS dylib、Windows DLL,以及 `.deb` / `.rpm` / AppImage 等分发格式 -尚在规划中。本文档随 `mcpp pack` 实现演进,最新选项以 +macOS **程序** bundling(Mach-O 依赖闭包,走 `otool -L` / `LC_LOAD_DYLIB`, +重定位走 `install_name_tool`)仍在规划中;在它落地之前,`mcpp pack <程序>` +会在该格式上拒绝,而不是产出一个只是看起来像 bundle 的东西。当前 `.zip` +之外的 Windows DLL 分发,以及 `.deb` / `.rpm` / AppImage 等格式,同样在规划中。本文档随 `mcpp pack` 实现演进,最新选项以 `mcpp pack --help` 为准。 diff --git a/docs/zh/12-binary-distribution.md b/docs/zh/12-binary-distribution.md index 4a636872..2b5cb5a4 100644 --- a/docs/zh/12-binary-distribution.md +++ b/docs/zh/12-binary-distribution.md @@ -231,16 +231,69 @@ ldflags = ["-Llib/x86_64-linux-musl", "-lmathkit"] 这是**降级**而不是变砖,方向是对的。但它意味着**闸门只保护新客户端**, 面向混合版本用户群发布时,这一条应写进发布说明。 +## 包里带什么走,以及刻意不带什么 + +发布出去的包必须能在**不是发布者的**机器上工作。两个步骤保证这件事, +它们作用在打包器暂存的每一个产物上。 + +### 构建机的 loader 搜索路径会被删掉 + +dev 构建会把工具链自己的目录烙进每一个共享对象: + +```text +DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 + : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 + : /registry/subos/default/lib +``` + +这对 dev 构建是对的,对一个包则是致命的(issue #460),原因是 ELF 装载器的 +一条规则:**一个携带任何 `DT_RUNPATH` 的对象,会让装载器在解析它自己的依赖时 +跳过整条继承来的 `DT_RPATH` 链。** 于是消费方那条 `DT_RPATH` —— 载荷、包目录、 +SubOS farm,全都是在真正要运行它的那台机器上算出来的 —— 根本不被查,程序死在 + +```text +error while loading shared libraries: libstdc++.so.6: cannot open shared object file +``` + +⚠️ **`$ORIGIN` 不是解药。** 在真实的包上、把构建机的 store 变成不可达之后实测: + +| 发货 `.so` 上的状态 | 消费方 `DT_RPATH` 被继承? | 结果 | +|---|---|---| +| 失效的绝对路径 `DT_RUNPATH` | 否 | 失败 | +| **完全没有这条 tag** | **是** | **能跑** | +| `DT_RUNPATH = $ORIGIN` | 否 | 失败 | +| `DT_RUNPATH = ""` | 否 | 失败 | + +关掉继承的是这条 tag 的**存在**,不是它的内容。所以 `mcpp pack` 删掉这个条目而 +不是改写它 —— 而且删掉不是妥协,它就是正确答案:消费方自己的 `DT_RPATH` 是同一个 +闭包,只不过是在它有意义的那台机器上解析的。 + +路径**字符串**会留在 `.dynstr` 里,没有人再指向它。`.dynstr` 被链接器做了尾部合并, +一个更短的活字符串可能从这条死字符串的中间开始,删掉这些字节无法被证明是安全的; +`patchelf --remove-rpath` 留下的残留在尺寸上逐字节相同。**所以这件事的守卫必须读 +动态段的条目,绝不能 `grep` 文件字节** —— 见 `tests/e2e/_elf_tag.sh`。 + +Mach-O 上打包器会读出 `LC_RPATH` 并在包会携带它时告警;自动改写 +(`install_name_tool -delete_rpath`)尚未做,因为这个套件里还没有任何测试能产出 +一个 `.dylib` 来给这次字节编辑做判据。 + +### 调试信息会被剥掉 + +参数、分档表与 `--debug-symbols` 见 [docs/02](02-pack-and-release.md)。 +对**库**包最要紧的一条:静态归档只做 `--strip-debug`,因为 `--strip-all` 会删掉 +归档的符号索引,消费方链接时会报 `archive has no index; run ranlib to add one`。 + ## 当前边界 | | 状态 | |---|---| | `kind = "lib"`(静态) | ✅ 所有 target,三平台都测了 | -| `kind = "shared"` on Linux/ELF | ✅ —— 包里同时带链接名与 SONAME | +| `kind = "shared"` on Linux/ELF | ✅ —— 包里同时带链接名与 SONAME,且不含构建机的 loader 路径 | | `kind = "shared"` on PE / MinGW(`*-windows-gnu`) | ✅ —— 包里同时带 `.dll` **和它的导入库** | -| `kind = "shared"` on Mach-O(`*-macos`) | ✅ —— install name 是 `@rpath/`,`.dylib` 可重定位 | +| `kind = "shared"` on Mach-O(`*-macos`) | ✅ —— install name 是 `@rpath/`,`.dylib` 可重定位。`LC_RPATH` 只报告,尚未改写 | | `kind = "shared"` on PE / MSVC(`*-windows-msvc`) | ✅ —— mcpp 生成 `.def`;见下 | | `kind = "shared"` on `*-musl` | ❌ musl target 是静态链接的 | +| 一个包同时携带同一 triple 的两套 ABI(gcc **与** clang) | ❌ leg 选择是 `cfg(arch/os/env)`;一个 ABI 发一个包 | | 发布预编译 BMI | ❌ 未尝试;BMI 与编译器构建逐位绑定 | | 把依赖打包进去 | ❌ 改为声明依赖(见上) | | 用**原生 `cl.exe`** 消费这种包 | ✅ —— 经方言中立的链接意图;见下 | @@ -364,6 +417,9 @@ e2e 套件按宿主能力给每条测试开门,所以「套件是绿的」和「 | Mach-O 共享库离开构建树仍可加载 | — | ✅ | — | | MSVC 以「导出」为理由拒绝 `kind = "shared"` | — | — | ✅ | | 已发布的 mcpp 消费本版产出的包 | 仅本机 | 仅本机 | 仅本机 | +| 打包出的 `.so` 不含构建机 loader 路径,**且把缺陷放回去时守卫看得见** | ✅ | — | — | +| strip 过的静态归档仍可链接、strip 过的共享库仍可加载,`--no-strip` / `[pack] strip` / `--debug-symbols` 两侧都钉 | ✅ | — | — | +| ELF 编辑器在 ELF32 与大端上的行为 | 单测 | 单测 | 单测 | *不可能* 不是缺口:macOS 宿主只能服务一个 target(`host_can_serve`, `registry.cppm`),那里根本产不出两个 target 的产物的包。 diff --git a/mcpp.toml b/mcpp.toml index 8f51eeeb..dd921dcd 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.19.4" +version = "2026.8.20.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 95c552ef..781e9bfa 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -575,14 +575,23 @@ export CacheMode resolve_cache_mode(const mcpp::manifest::Manifest& m, // place. The rule is pure (manifest + one override string), so both sides can // evaluate it without resolving a toolchain or scanning the module graph. // -// Precedence: --profile/--release/--dev > [build].default-profile > "dev". +// Precedence: --profile/--release/--dev > [build].default-profile > `fallback`. // The global default is "dev" (-O0 -g) per the dominant convention // (Cargo/Meson/CMake/Zig/Bazel/MSBuild all default to debug). +// +// `fallback` exists for ONE caller: `mcpp pack`, where the artifact leaves this +// machine and an unoptimized build with the publisher's absolute source paths +// in it is never what was meant. It changes the LAST step only, so a manifest +// that states `[build] default-profile` still decides — packaging an artifact +// with different flags than `mcpp build` produces would be its own surprise. +// Adding a parameter here rather than a second resolver keeps the precedence +// rule in one function, which is why this function exists at all. export std::string resolve_profile_name(const mcpp::manifest::Manifest& m, - std::string_view override_name) { + std::string_view override_name, + std::string_view fallback = "dev") { if (!override_name.empty()) return std::string(override_name); if (!m.buildConfig.defaultProfile.empty()) return m.buildConfig.defaultProfile; - return "dev"; + return fallback.empty() ? std::string("dev") : std::string(fallback); } // Command-level overrides (--target / --static). @@ -642,6 +651,10 @@ export struct BuildOverrides { bool force_static = false; // --static (or implied by musl target) std::string package_filter; // -p : only build this workspace member std::string profile; // --profile (default "release") + // What `resolve_profile_name` falls back to when neither the command line + // nor `[build] default-profile` says. Empty = "dev", which is every + // interactive command. `mcpp pack` sets "release": see resolve_profile_name. + std::string profile_fallback; std::string features; // --features a,b,c (root package activation) bool strict = false; // --strict: schema warnings become errors std::string capabilities; // --cap blas=openblas,lapack=mkl (provider pins) @@ -1121,7 +1134,7 @@ prepare_build(bool print_fingerprint, // wants its plain `mcpp build` optimized sets // [build].default-profile = "release" (mcpp's own mcpp.toml does this, // so the released binary stays -O2). - pname = resolve_profile_name(*m, overrides.profile); + pname = resolve_profile_name(*m, overrides.profile, overrides.profile_fallback); mcpp::manifest::Profile pr; if (pname == "dev" || pname == "debug") { pr.optLevel = "0"; pr.debug = true; } else if (pname == "dist") { pr.optLevel = "3"; pr.strip = true; } diff --git a/src/cli.cppm b/src/cli.cppm index bd4431c6..5e3818b7 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -447,6 +447,15 @@ int run(int argc, char** argv) { .help("tar (default; .zip for a Windows target) | dir")) .option(cl::Option("output").short_name('o').takes_value() .help("Override output path")) + // Packaging builds RELEASE by default — the artifact leaves this + // machine. `[build] default-profile` still wins when it is set; + // this only replaces the "dev" fallback every other command uses. + .option(cl::Option("profile").takes_value() + .help("Build profile (default: [build] default-profile, else release)")) + .option(cl::Option("no-strip") + .help("Ship the artifacts as built (default: strip debug info)")) + .option(cl::Option("debug-symbols").takes_value().value_name("DIR") + .help("Write the separated *.debug files here (default: discard)")) .action(wrap_rc(cmd_pack))) // ─── emit (one nested subcommand: xpkg) ──────────────────────── diff --git a/src/cli/cmd_publish.cppm b/src/cli/cmd_publish.cppm index d9d6d93a..dab84430 100644 --- a/src/cli/cmd_publish.cppm +++ b/src/cli/cmd_publish.cppm @@ -58,6 +58,16 @@ export int cmd_pack(const mcpplibs::cmdline::ParsedArgs& parsed) { } if (auto v = parsed.value("output")) opts.output = *v; + // ⚠️ `value()`, not `option_or_empty()`, for `profile` — and NOT for + // anything whose name a positional shares. `ParsedArgs::value()` falls back + // to a same-named positional when the option is unset, which is how + // `mcpp run q` once became `--target=q`. `pack`'s positional is `target`, + // so `--target` is read through `option()` above and stays unaffected; + // `profile` and `debug-symbols` have no positional twin. + if (auto v = parsed.value("profile")) opts.profile = *v; + if (parsed.is_flag_set("no-strip")) opts.strip = false; + if (auto v = parsed.value("debug-symbols")) opts.debugSymbols = *v; + // `--target` is repeatable: one leg per triple, which is how a library // package ships for several targets at once. The application path has // always taken exactly one, and still does — packing one executable for diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index cd80af7b..26131662 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -1359,6 +1359,10 @@ std::expected parse_string(std::string_view content, } m.packConfig.defaultMode = s; } + if (auto v = doc->get_bool("pack.strip")) + m.packConfig.strip = *v; + if (auto v = doc->get_string("pack.debug_symbols")) + m.packConfig.debugSymbols = *v; if (auto v = doc->get_string_array("pack.include")) m.packConfig.include = *v; if (auto v = doc->get_string_array("pack.exclude")) diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 5c83888c..ececfd44 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -790,6 +790,20 @@ struct LibConfig { // "bundle-all" — bundle every dynamic dep including libc / libstdc++ struct PackConfig { std::string defaultMode; // empty → "bundle-project" + // ⚠️ THERE IS DELIBERATELY NO `[pack] profile`. Which profile `mcpp pack` + // builds with is `--profile` > `[build] default-profile` > "release" — + // packaging only changes the LAST step (from "dev"), because a fourth + // precedence level would have to be resolved before `prepare_build` runs + // and this manifest is what `prepare_build` produces. + // Strip the SHIPPED artifacts (not a link-time `-s`; see mcpp.pack.strip). + // Tri-state: unset = the default (strip), which is what a published binary + // wants. `false` ships the artifact exactly as built. + std::optional strip; + // Where the separated `*.debug` files go, package-root-relative or + // absolute. Empty = do not separate, which is the default: most publishers + // do not ship a debug package, and writing one by default would double the + // output of every `mcpp pack`. + std::string debugSymbols; std::vector include; // extra files/globs to ship std::vector exclude; // patterns to drop from include // Mode C overrides — let the user expand or contract the PEP 600 diff --git a/src/pack/library.cppm b/src/pack/library.cppm index 4260b3bd..fdf13710 100644 --- a/src/pack/library.cppm +++ b/src/pack/library.cppm @@ -38,9 +38,12 @@ export module mcpp.pack.library; import std; import mcpp.pack.digest; import mcpp.pack.manifest_emit; +import mcpp.pack.relocate; +import mcpp.pack.strip; import mcpp.source_kind; // builtin_extension_table — what needs no declaring import mcpp.pack.zip; import mcpp.platform; +import mcpp.ui; export namespace mcpp::pack { @@ -71,6 +74,11 @@ struct LibraryLeg { // package that no linker can use, so the package carries both and the // emitted manifest points consumers at this one. std::filesystem::path importLibrary; + // Debug-information removal for THIS leg's toolchain, resolved by the + // caller from `mcpp::toolchain::binutils_tool`. Per leg for the same + // reason `archiveTool` is: a fat package's aarch64 leg must not be + // stripped by the x86_64 host's tool. + mcpp::pack::StripTools stripTools; }; struct LibraryPackPlan { @@ -95,6 +103,11 @@ struct LibraryPackPlan { std::vector> dependencies; std::vector extras; // README / LICENSE / [pack].include hits + // Remove debug information from the shipped artifacts (mcpp.pack.resolve_strip). + bool strip = true; + // Where the separated `*.debug` files go, absolute. Empty = do not separate. + std::filesystem::path debugDir; + std::vector legs; }; @@ -273,26 +286,6 @@ run_library_pack(const LibraryPackPlan& plan) return std::unexpected(r.error()); } - // A shared library needs BOTH of its names present. - // - // `-lmathkit-shared` resolves `libmathkit-shared.so` at link time, but - // the object records `SONAME libmathkit.so.1`, and that is the name the - // loader asks for. Ship only the built file and the consumer links, - // then fails to start — mcpp's own runtime-closure check reports - // "libmathkit.so.1 not found on the search path this artifact will - // actually use", which is how this was caught. - // - // A symlink is what a distribution ships; a copy is the fallback for - // filesystems (and archives) that cannot carry one. - if (leg.shared && !leg.soname.empty() && leg.soname != name) { - auto alias = dst.parent_path() / leg.soname; - std::error_code linkEc; - std::filesystem::remove(alias, linkEc); - std::filesystem::create_symlink(name, alias, linkEc); - if (linkEc) - if (auto r = copy_into(leg.artifact, alias); !r) return std::unexpected(r.error()); - } - // Delete the objects of the units published as source. The consumer // compiles those itself; leaving them in the archive means two // definitions of the module initialiser, resolved by link order. @@ -324,6 +317,99 @@ run_library_pack(const LibraryPackPlan& plan) } } + // ── THE ORDER, AND IT IS NOT FREE ───────────────────────────── + // + // Four steps change the artifact's bytes and one records them. They + // are written here, once, because every one of them is a way to ship + // a package whose manifest describes a file that is not the one in the + // archive: + // + // 1. copy (above) + // 2. drop objects (above) — changes the archive's members + // 3. relocate — remove the build machine's loader paths + // 4. strip — remove debug info (and separate it) + // 5. soname alias — a symlink to, or a COPY OF, the FINAL file + // 6. digest — the package's evidence, over the final bytes + // + // 5 after 3–4 is the one that used to be wrong in a way nothing could + // see: the alias' copy fallback read `leg.artifact` (the BUILD tree's + // file), which was byte-identical only because nothing here modified + // anything. With 3 and 4 in place it would ship an unrelocated, + // unstripped library under the exact name the loader asks for — and + // only on the machines where `create_symlink` fails, which is where + // nobody looks. + + // 3. The build machine does not travel. See mcpp.pack.relocate for why + // this removes the tag rather than rewriting it to `$ORIGIN`. + { + auto r = mcpp::pack::relocate::strip_search_paths(dst); + if (!r) return std::unexpected(LibraryPackError{ std::format( + "cannot make '{}' relocatable: {}", dst.string(), r.error()) }); + using O = mcpp::pack::relocate::Outcome; + if (r->outcome == O::Removed) { + std::string what; + for (auto const& p : r->paths) { if (!what.empty()) what += " "; what += p; } + mcpp::ui::status("Relocated", + std::format("{} ({} dropped from the loader search path)", + name, what.empty() ? std::string("build-machine paths") : what)); + } else if (r->outcome == O::Reported && !r->paths.empty()) { + // Mach-O: read, not rewritten. Saying nothing here would let a + // `.dylib` carry the publisher's LC_RPATH into a package while + // the ELF leg beside it is clean. + std::string what; + for (auto const& p : r->paths) { if (!what.empty()) what += ", "; what += p; } + mcpp::ui::warning(std::format( + "{} carries LC_RPATH entries that mcpp does not yet rewrite: {}\n" + " If any of them names a directory on THIS machine, the package is " + "not relocatable. Remove it with `install_name_tool -delete_rpath " + " ` before publishing.", name, what)); + } else if (r->outcome == O::Unanalysed) { + mcpp::ui::warning(std::format( + "{} could not be checked for build-machine loader paths: {}", + name, r->note)); + } + } + + // 4. Debug information does not travel either — unless asked. + if (plan.strip) { + const auto shape = leg.shared ? mcpp::pack::ArtifactShape::SharedLibrary + : mcpp::pack::ArtifactShape::StaticArchive; + auto r = mcpp::pack::strip_artifact(dst, shape, leg.stripTools, plan.debugDir); + if (!r) return std::unexpected(LibraryPackError{ r.error() }); + if (r->outcome == mcpp::pack::StripOutcome::Stripped) { + mcpp::ui::status("Stripped", std::format("{} {} → {} bytes{}", + name, r->before, r->after, + r->debugFile.empty() ? std::string{} + : std::format(" (debug: {})", + r->debugFile.filename().string()))); + } + // The IMPORT LIBRARY is deliberately not stripped: it is an archive + // of linker stubs with no debug information to remove, and dh_strip + // makes the same exclusion. + } + + // 5. A shared library needs BOTH of its names present. + // + // `-lmathkit-shared` resolves `libmathkit-shared.so` at link time, but + // the object records `SONAME libmathkit.so.1`, and that is the name the + // loader asks for. Ship only the built file and the consumer links, + // then fails to start — mcpp's own runtime-closure check reports + // "libmathkit.so.1 not found on the search path this artifact will + // actually use", which is how this was caught. + // + // A symlink is what a distribution ships; a copy is the fallback for + // filesystems (and archives) that cannot carry one — and it copies + // `dst`, never `leg.artifact`. See the order note above. + if (leg.shared && !leg.soname.empty() && leg.soname != name) { + auto alias = dst.parent_path() / leg.soname; + std::error_code linkEc; + std::filesystem::remove(alias, linkEc); + std::filesystem::create_symlink(name, alias, linkEc); + if (linkEc) + if (auto r = copy_into(dst, alias); !r) return std::unexpected(r.error()); + } + + // 6. Evidence, over the bytes that actually ship. docLegs.push_back(PackageLeg{ .triple = leg.triple, .libFile = name, diff --git a/src/pack/library_pipeline.cppm b/src/pack/library_pipeline.cppm index 844adfef..1521af0b 100644 --- a/src/pack/library_pipeline.cppm +++ b/src/pack/library_pipeline.cppm @@ -35,6 +35,8 @@ import mcpp.pack; import mcpp.pack.abi_tag; import mcpp.pack.interface; import mcpp.pack.library; +import mcpp.pack.strip; +import mcpp.toolchain.model; import mcpp.platform; import mcpp.toolchain.dialect; import mcpp.toolchain.registry; @@ -130,6 +132,11 @@ export int build_and_pack_library(const std::string& targetName, for (auto const& want : legs) { mcpp::build::BuildOverrides ov; ov.target_triple = want; + // A packaged artifact leaves this machine, so the fallback is release + // rather than the interactive "dev". `[build] default-profile` still + // decides when the project states one — see resolve_profile_name. + ov.profile = opts.profile; + ov.profile_fallback = "release"; auto ctx = mcpp::build::prepare_build(false, /*includeDevDeps=*/false, {}, ov); if (!ctx) { mcpp::ui::error(ctx.error()); return 2; } @@ -287,6 +294,12 @@ export int build_and_pack_library(const std::string& targetName, declaredPlatforms = ctx->manifest.package.platforms; plan.dependencies = publishable_dependencies(ctx->manifest); plan.extras = extras_of(ctx->manifest, ctx->projectRoot); + // Read from the FIRST leg's manifest, like every other package-wide + // fact here: one package, one answer. A `[pack] strip` that differed + // per target would describe two packages. + plan.strip = mcpp::pack::resolve_strip(opts, ctx->manifest.packConfig); + plan.debugDir = mcpp::pack::resolve_debug_dir( + opts, ctx->manifest.packConfig, ctx->projectRoot); plan.interfaceSources = closure.published; plan.dropObjects = published_object_names(closure); for (auto const& d : ctx->manifest.buildConfig.includeDirs) { @@ -339,6 +352,16 @@ export int build_and_pack_library(const std::string& targetName, .soname = target->soname, .shared = shared, .importLibrary = importLib, + // From THIS leg's toolchain, for the same reason `archiveTool` is: + // a fat package's foreign leg must not be stripped by the host's + // tool. `inBandDebugInfo` is the one bit of "does this even apply" + // — PE/MSVC keeps debug information in a separate `.pdb`. + .stripTools = mcpp::pack::StripTools{ + .strip = mcpp::toolchain::binutils_tool(ctx->tc, "strip"), + .objcopy = mcpp::toolchain::binutils_tool(ctx->tc, "objcopy"), + .inBandDebugInfo = + ctx->tc.compiler != mcpp::toolchain::CompilerId::MSVC, + }, }); mcpp::ui::status("Packed leg", std::format("{} [{}]", triple, tag.str())); } diff --git a/src/pack/pack.cppm b/src/pack/pack.cppm index bd20e0a5..d138310b 100644 --- a/src/pack/pack.cppm +++ b/src/pack/pack.cppm @@ -39,6 +39,8 @@ import mcpp.build.loader_contract; import mcpp.config; import mcpp.pack.binfmt; import mcpp.pack.host_requirements; +import mcpp.pack.relocate; +import mcpp.pack.strip; import mcpp.pack.zip; import mcpp.platform; import mcpp.platform.xlings; @@ -83,8 +85,38 @@ struct Options { // caller resolves the contract; this is the one bit of it that packaging // acts on. bool carryToolchainRuntime = false; + + // ── how the shipped artifact is BUILT and what travels inside it ── + // + // `profile` is the `--profile` override only. The default is not spelled + // here: `mcpp pack` passes `BuildOverrides::profile_fallback = "release"`, + // so `[build] default-profile` still decides when it is set, and the + // precedence rule stays in `resolve_profile_name` where the rest of mcpp + // reads it. + std::string profile; + // Tri-state on purpose. `nullopt` = "nobody said", which is what lets + // `[pack] strip` be consulted at all — a plain `bool` defaulted to true + // would make the manifest key unreachable from the CLI's point of view. + std::optional strip; + // `--debug-symbols `: where the separated `*.debug` files go. Empty = + // do not separate. + std::filesystem::path debugSymbols; }; +// The strip decision for this run: `--strip`/`--no-strip` > `[pack] strip` > +// stripped. +// +// Spelled once because both packers ask it and a distribution that strips its +// libraries but not its programs is a distribution whose rule nobody can state. +bool resolve_strip(const Options& opts, const mcpp::manifest::PackConfig& cfg); + +// Where the separated debug files go, absolute. Empty = do not separate. +// A manifest-relative path is resolved against the project root, like every +// other path a manifest names. +std::filesystem::path resolve_debug_dir(const Options& opts, + const mcpp::manifest::PackConfig& cfg, + const std::filesystem::path& projectRoot); + // Resolved plan — all paths absolute, all decisions baked in. struct Plan { Options opts; @@ -110,6 +142,14 @@ struct Plan { // The search set the PE closure resolves names against, after the // contract has had its say (see make_plan). std::vector searchDirs; + // ── debug information: the RESOLVED decision, not the request ───── + // + // On the Plan rather than in Options because `Options` is what the user + // asked for and this is what that came out as once the manifest and the + // toolchain had their say. The library packer keeps the same split. + bool strip = true; + std::filesystem::path debugDir; // absolute; empty = discard + mcpp::pack::StripTools stripTools; }; struct Error { std::string message; }; @@ -222,6 +262,23 @@ std::string wrapper_dirname_from_archive(const std::filesystem::path& archive) { } // namespace detail +bool resolve_strip(const Options& opts, const mcpp::manifest::PackConfig& cfg) { + if (opts.strip) return *opts.strip; + if (cfg.strip) return *cfg.strip; + return true; +} + +std::filesystem::path resolve_debug_dir(const Options& opts, + const mcpp::manifest::PackConfig& cfg, + const std::filesystem::path& projectRoot) +{ + auto raw = !opts.debugSymbols.empty() + ? opts.debugSymbols + : std::filesystem::path(cfg.debugSymbols); + if (raw.empty()) return {}; + return raw.is_absolute() ? raw : projectRoot / raw; +} + std::expected make_plan(const mcpp::manifest::Manifest& manifest, const mcpp::config::GlobalConfig& /*cfg*/, @@ -551,6 +608,26 @@ set_interpreter(const std::filesystem::path& binary, return {}; } +// Remove the program's debug information — and ONLY the program's. +// +// A bundled `.so` is somebody else's file: it came out of the store or off the +// host, mcpp did not build it, and stripping it would change a shared payload's +// bytes for no gain to this bundle. dh_strip draws the same line (a package +// strips what it built). +// +// Shared with `run_pe` deliberately: a MinGW `.exe` carries DWARF in-band just +// like an ELF one, so "does the bundle ship debug info" must not depend on +// which output family it lands in. +std::expected +strip_program(const Plan& plan, const std::filesystem::path& staged) +{ + if (!plan.strip) return {}; + auto r = mcpp::pack::strip_artifact(staged, mcpp::pack::ArtifactShape::Executable, + plan.stripTools, plan.debugDir); + if (!r) return std::unexpected(Error{r.error()}); + return {}; +} + // Bundle all `deps` into /lib/. We dereference any // symlinks so the bundle is self-contained even if /usr/lib/foo.so → /usr/lib/foo.so.1. std::expected @@ -890,6 +967,8 @@ run_pe(const Plan& plan) } } + if (auto r = strip_program(plan, stagedExe); !r) return r; + if (plan.opts.format != Format::Tar) return {}; std::vector entries; @@ -933,6 +1012,42 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) // the host: `LD_TRACE_LOADED_OBJECTS` cannot trace a PE from Linux either. if (plan.targetIsPe) return detail::run_pe(plan); + // A Mach-O artifact is REFUSED, on every host including macOS. + // + // The closure below asks the dynamic linker for the dependency list by + // running the artifact with `LD_TRACE_LOADED_OBJECTS=1`. That variable + // belongs to glibc's ld.so; dyld has never heard of it (its counterpart is + // `DYLD_PRINT_LIBRARIES`), so on macOS the command does not trace anything + // — IT RUNS THE USER'S PROGRAM. Whatever that program prints is then parsed + // as a dependency table, which yields nothing, and the bundle is written + // and reported as `Packed`. A program with side effects performs them; an + // interactive one hangs the packer. + // + // ASKED OF THE FORMAT, NOT OF THE HOST — the same correction the `_WIN32` + // branch below already carries. `LD_TRACE_LOADED_OBJECTS` cannot trace a + // Mach-O from Linux either, and a macOS host is not the thing that makes + // this impossible. + // + // docs/02 lists macOS bundling under "Planned Support"; until it lands, + // saying so is strictly better than producing an empty bundle that claims + // to be one. + if (mcpp::pack::binfmt::identify(plan.builtBinary).format + == mcpp::pack::binfmt::Format::MachO) { + return std::unexpected(Error{ + "cannot package a Mach-O program yet.\n" + " The dependency closure for that format is resolved by running the " + "artifact under\n" + " the target's own dynamic linker, and the mechanism mcpp uses " + "(LD_TRACE_LOADED_OBJECTS)\n" + " is glibc's — dyld ignores it and simply RUNS the program, which is " + "why this is\n" + " refused rather than attempted.\n" + " A `kind = \"lib\"` / `\"shared\"` target packs normally on macOS " + "(`mcpp pack `);\n" + " for a program, ship the build tree or use a platform bundler until " + "macOS support lands."}); + } + #if defined(_WIN32) // A NON-PE artifact on a Windows host: a cross build to Linux or macOS. // The closure below asks the dynamic linker by running the binary, which @@ -1032,11 +1147,23 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) // empty bundle → clear the original dev-sandbox RUNPATH // (~/.mcpp/registry/... doesn't exist on // a user's target machine) - const char* rpath = toBundle.empty() ? "" : "$ORIGIN/../lib"; - if (auto r = set_search_path(bundledBinary, rpath, - mcpp::build::loader::Form::Executable, - patchelf); !r) + // An EMPTY bundle gets the tag REMOVED, not set to "". + // + // `patchelf --set-rpath ''` leaves the tag present with an empty + // string, and a present-but-empty DT_RUNPATH is not inert: it + // suppresses the inherited DT_RPATH chain exactly like a stale one + // does (measured — see mcpp.pack.relocate). Harmless on an + // executable, which is the top of that chain, but there is no + // reason to write a tag that says nothing, and the library packer + // needs the removal path anyway. + if (toBundle.empty()) { + if (auto r = mcpp::pack::relocate::strip_search_paths(bundledBinary); !r) + return std::unexpected(Error{r.error()}); + } else if (auto r = set_search_path(bundledBinary, "$ORIGIN/../lib", + mcpp::build::loader::Form::Executable, + patchelf); !r) { return std::unexpected(Error{r.error()}); + } // EVERY BUNDLED LIBRARY, not just the executable. // @@ -1123,6 +1250,14 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) return std::unexpected(Error{r.error()}); } + // 4b. Debug information does not travel either. + // + // AFTER every byte-changing step above (patchelf's search path, PT_INTERP) + // and before the archive: strip must see the final image, and the archive + // must see the stripped one. Same ordering rule the library packer states + // at its leg loop. + if (auto r = strip_program(plan, bundledBinary); !r) return r; + // 5. Output. if (plan.opts.format == Format::Tar) { if (auto r = make_tarball(plan.stagingRoot, plan.archivePath); !r) diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index ab21b1da..770143fd 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -18,6 +18,9 @@ import mcpp.build.plan; import mcpp.config; import mcpp.fetcher.progress; import mcpp.pack; +import mcpp.pack.strip; +import mcpp.toolchain.model; +import mcpp.toolchain.registry; import mcpp.ui; namespace mcpp::pack { @@ -47,6 +50,10 @@ export int build_and_pack(Options opts, bool modeFromUser, ov.target_triple = "x86_64-linux-musl"; else ov.target_triple = opts.targetTriple; + // A bundled program leaves this machine: release is the fallback, not dev. + // `[build] default-profile` still decides when the project states one. + ov.profile = opts.profile; + ov.profile_fallback = "release"; auto ctx = mcpp::build::prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, /*extraTargets=*/{}, ov); @@ -76,7 +83,9 @@ export int build_and_pack(Options opts, bool modeFromUser, && ctx->tc.targetTriple.find("-musl") == std::string::npos) { // Need to re-prepare the build with the musl target. mcpp::build::BuildOverrides ov2; - ov2.target_triple = "x86_64-linux-musl"; + ov2.target_triple = "x86_64-linux-musl"; + ov2.profile = opts.profile; + ov2.profile_fallback = "release"; auto ctx2 = mcpp::build::prepare_build(false, false, {}, ov2); if (!ctx2) { mcpp::ui::error(ctx2.error()); return 2; } ctx = std::move(ctx2); @@ -176,9 +185,23 @@ export int build_and_pack(Options opts, bool modeFromUser, ctx->plan.runtimeRequirements); if (!plan) { mcpp::ui::error(plan.error().message); return 1; } - mcpp::ui::info("Packing", std::format("{} v{} ({})", + // The RESOLVED debug-information decision. On the plan, not in Options: + // Options is the request, this is what it came out as once the manifest + // and the toolchain had their say. Tools come from the build's own + // toolchain so a cross bundle is stripped by the cross tool. + plan->strip = mcpp::pack::resolve_strip(opts, ctx->manifest.packConfig); + plan->debugDir = mcpp::pack::resolve_debug_dir(opts, ctx->manifest.packConfig, + ctx->projectRoot); + plan->stripTools = mcpp::pack::StripTools{ + .strip = mcpp::toolchain::binutils_tool(ctx->tc, "strip"), + .objcopy = mcpp::toolchain::binutils_tool(ctx->tc, "objcopy"), + .inBandDebugInfo = ctx->tc.compiler != mcpp::toolchain::CompilerId::MSVC, + }; + + mcpp::ui::info("Packing", std::format("{} v{} ({}{})", plan->packageName, plan->packageVersion, - mcpp::pack::mode_cli_name(plan->opts.mode))); + mcpp::pack::mode_cli_name(plan->opts.mode), + plan->strip ? ", stripped" : "")); auto r = mcpp::pack::run(*plan, *cfg); if (!r) { diff --git a/src/pack/relocate.cppm b/src/pack/relocate.cppm new file mode 100644 index 00000000..6d21cdab --- /dev/null +++ b/src/pack/relocate.cppm @@ -0,0 +1,377 @@ +// mcpp.pack.relocate — removing the BUILD MACHINE from a distributable image. +// +// THE RULE IS "THE TAG IS GONE", NOT "THE PATH IS RELATIVE" +// +// A dev build bakes the toolchain's own directories into every artifact: +// +// DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 +// : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 +// : /registry/subos/default/lib +// +// That is correct for a dev build and wrong for a package, and the obvious fix +// — rewrite it to `$ORIGIN` — DOES NOT WORK. Measured on a real `mcpp pack` +// product consumed by a real mcpp-built program, with the build machine's +// store made unreachable (issue #460): +// +// state on the shipped .so consumer's DT_RPATH inherited? result +// ------------------------------ ----------------------------- ------ +// DT_RUNPATH = no 127 +// no tag at all YES ok +// DT_RUNPATH = $ORIGIN no 127 +// DT_RUNPATH = "" (empty string) no 127 +// DT_RPATH = yes ok +// +// The ELF loader drops the whole inherited DT_RPATH chain for any object that +// carries a DT_RUNPATH — whatever that RUNPATH says. So a non-empty +// replacement is not "less wrong", it is exactly as broken as the original, +// and an EMPTY one (which is what `patchelf --set-rpath ''` writes, and what +// any "clear the string" tool leaves behind) is broken too. +// +// Removing the tag is not merely the least-bad option, it is the RIGHT one: +// the consumer's own DT_RPATH is the same closure — payload, package dir, +// SubOS farm — resolved on the machine that will actually run it. +// +// WHY THIS EDITS BYTES INSTEAD OF SHELLING OUT TO patchelf +// +// A library package is cross-target by construction (`--target` is repeatable, +// and `run_library_pack` has no host gate because it never runs the artifact). +// `sandbox_patchelf` resolves only under a Linux xlings store, so a macOS or +// Windows host producing an ELF leg would find nothing — and the application +// packer's shape for that case is `if (!patchelf.empty())`, i.e. SILENTLY DO +// NOTHING. Copying that shape here would hand half of the hosts the very +// defect this module exists to remove. +// +// The edit needed is also far smaller than what patchelf is built for: delete +// one entry from the `Elf*_Dyn` array by moving the tail up one slot and +// padding the end with DT_NULL. Same file length, no segment moves, no offset +// fixups. +// +// WHAT THIS DELIBERATELY DOES NOT DO +// +// The path STRING stays in `.dynstr`. Nothing points at it any more, so it is +// not reachable by the loader — but `strings` still finds it. That is not +// sloppiness and it is not a regression against the reference tool: measured, +// `patchelf --remove-rpath` leaves the identical residue at the identical file +// size. `.dynstr` is TAIL-MERGED by the linker, so a shorter live string may +// begin inside the dead one, and removing those bytes cannot be shown safe +// without a full reference analysis of `.dynstr`'s three consumers (DT_* +// strings, `.dynsym` names, `.gnu.version_r`). Silently truncating a symbol +// name is a worse outcome than a dead string. +// +// ⚠️ CONSEQUENCE FOR ANY GUARD: "this artifact carries no build-machine path" +// must be asked of the DYNAMIC ENTRIES, never of the file's bytes. A +// byte-pattern check reports a correctly relocated artifact as dirty. + +export module mcpp.pack.relocate; + +import std; +import mcpp.pack.binfmt; + +export namespace mcpp::pack::relocate { + +// What happened to one image. +// +// FIVE-VALUED, and the last two are the point: an image this module could not +// analyse has NOT been shown to be clean, and reporting that as `NothingToDo` +// would make "nothing was wrong" and "nothing was checked" the same answer. +enum class Outcome { + Removed, // an ELF search-path tag was present and is now gone + NothingToDo, // ELF, analysed, carried no DT_RPATH / DT_RUNPATH + NotApplicable, // PE, an archive, or not an image — no such concept + Reported, // Mach-O: entries were read and are reported, bytes untouched + Unanalysed, // recognised as an image, but this module cannot read it +}; + +struct Report { + Outcome outcome = Outcome::NotApplicable; + // What the removed (ELF) or found (Mach-O) entries said, for the log. + std::vector paths; + // Why, when `outcome == Unanalysed`. Empty otherwise. + std::string note; +}; + +std::string_view describe(Outcome o); + +// Remove every loader search-path entry from `image`, in place. +// +// Errors are reserved for "the file is there and something is wrong with it" +// (unreadable, truncated, not writable). A file that simply has no such +// concept — a `.a`, a `.dll`, a `.lib` — is `NotApplicable`, not an error: +// the caller packs every leg through the same step and must not have to know +// the format first. +std::expected strip_search_paths(const std::filesystem::path& image); + +} // namespace mcpp::pack::relocate + +namespace mcpp::pack::relocate { + +namespace { + +// ── endian- and class-aware byte access ───────────────────────────────── +// +// ELF32 and big-endian are not hypothetical for mcpp: a `--target` list can +// name a 32-bit or big-endian triple, and the packer is the same code for all +// of them. Getting this wrong would corrupt an artifact rather than refuse it, +// so the accessors are parameterised rather than assumed. +struct Bytes { + std::string* d = nullptr; + bool little = true; + + std::uint64_t read(std::size_t off, std::size_t width) const { + std::uint64_t v = 0; + for (std::size_t i = 0; i < width; ++i) { + auto byte = static_cast((*d)[off + (little ? i : width - 1 - i)]); + v |= static_cast(byte) << (8 * i); + } + return v; + } + void write(std::size_t off, std::size_t width, std::uint64_t v) { + for (std::size_t i = 0; i < width; ++i) { + auto byte = static_cast((v >> (8 * i)) & 0xFF); + (*d)[off + (little ? i : width - 1 - i)] = byte; + } + } + bool has(std::size_t off, std::size_t len) const { + return d && off <= d->size() && d->size() - off >= len; + } +}; + +std::optional slurp(const std::filesystem::path& p) { + std::ifstream in(p, std::ios::binary); + if (!in) return std::nullopt; + std::ostringstream ss; + ss << in.rdbuf(); + if (!in && !in.eof()) return std::nullopt; + return ss.str(); +} + +bool spit(const std::filesystem::path& p, const std::string& data) { + std::ofstream out(p, std::ios::binary | std::ios::trunc); + if (!out) return false; + out.write(data.data(), static_cast(data.size())); + return static_cast(out); +} + +// Read a NUL-terminated string at `off`, bounded by the file. +std::string cstr_at(const std::string& d, std::size_t off) { + if (off >= d.size()) return {}; + auto end = d.find('\0', off); + return d.substr(off, end == std::string::npos ? std::string::npos : end - off); +} + +constexpr std::uint64_t kDtNull = 0; +constexpr std::uint64_t kDtStrtab = 5; +constexpr std::uint64_t kDtRpath = 15; +constexpr std::uint64_t kDtRunpath = 29; +constexpr std::uint32_t kPtLoad = 1; +constexpr std::uint32_t kPtDynamic = 2; + +std::expected strip_elf(const std::filesystem::path& image, + std::string data) +{ + const bool elf64 = static_cast(data[4]) == 2; + const std::uint8_t dataEnc = static_cast(data[5]); + if (dataEnc != 1 && dataEnc != 2) + return std::unexpected(std::format( + "'{}' declares an unknown ELF data encoding ({})", image.string(), dataEnc)); + + Bytes b{ &data, dataEnc == 1 }; + const std::size_t ptrW = elf64 ? 8 : 4; + + // e_phoff / e_phentsize / e_phnum — the two layouts differ only in offsets. + const std::size_t phoffOff = elf64 ? 0x20 : 0x1C; + const std::size_t phentOff = elf64 ? 0x36 : 0x2A; + const std::size_t phnumOff = elf64 ? 0x38 : 0x2C; + if (!b.has(phnumOff, 2)) return std::unexpected(std::format( + "'{}' is truncated before its ELF header ends", image.string())); + + const auto phoff = b.read(phoffOff, ptrW); + const auto phent = b.read(phentOff, 2); + const auto phnum = b.read(phnumOff, 2); + if (phoff == 0 || phent == 0 || phnum == 0) + return Report{ Outcome::NothingToDo, {}, {} }; // no program headers: no PT_DYNAMIC + + // Program-header field offsets. p_offset/p_vaddr/p_filesz sit in different + // places in the two classes because ELF64 moved p_flags forward. + const std::size_t poOff = elf64 ? 0x08 : 0x04; + const std::size_t pvOff = elf64 ? 0x10 : 0x08; + const std::size_t pfOff = elf64 ? 0x20 : 0x10; + + std::optional dynOff, dynSize; + struct Load { std::uint64_t off, vaddr, filesz; }; + std::vector loads; + for (std::uint64_t i = 0; i < phnum; ++i) { + const auto ph = phoff + i * phent; + if (!b.has(static_cast(ph), static_cast(phent))) + return std::unexpected(std::format( + "'{}' has a program-header table that runs past the file", + image.string())); + const auto type = static_cast(b.read(static_cast(ph), 4)); + if (type == kPtDynamic) { + dynOff = b.read(static_cast(ph + poOff), ptrW); + dynSize = b.read(static_cast(ph + pfOff), ptrW); + } else if (type == kPtLoad) { + loads.push_back({ b.read(static_cast(ph + poOff), ptrW), + b.read(static_cast(ph + pvOff), ptrW), + b.read(static_cast(ph + pfOff), ptrW) }); + } + } + if (!dynOff || !dynSize) return Report{ Outcome::NothingToDo, {}, {} }; + + // Walk the Elf*_Dyn array up to and including its DT_NULL terminator. + const std::size_t slotW = ptrW * 2; + std::vector> slots; // (tag, value) + for (std::uint64_t at = *dynOff; at + slotW <= *dynOff + *dynSize; at += slotW) { + if (!b.has(static_cast(at), slotW)) + return std::unexpected(std::format( + "'{}' has a PT_DYNAMIC that runs past the file", image.string())); + auto tag = b.read(static_cast(at), ptrW); + auto val = b.read(static_cast(at + ptrW), ptrW); + slots.push_back({ tag, val }); + if (tag == kDtNull) break; + } + if (slots.empty()) return Report{ Outcome::NothingToDo, {}, {} }; + + // DT_STRTAB is a VIRTUAL address; map it through PT_LOAD so the removed + // paths can be named in the log. Purely for reporting — a failure to map + // it must not stop the removal. + std::optional strtabFileOff; + for (auto const& [tag, val] : slots) { + if (tag != kDtStrtab) continue; + for (auto const& l : loads) { + if (val >= l.vaddr && val - l.vaddr < l.filesz) { + strtabFileOff = l.off + (val - l.vaddr); + break; + } + } + } + + Report report; + std::vector> kept; + for (auto const& slot : slots) { + if (slot.first == kDtRpath || slot.first == kDtRunpath) { + if (strtabFileOff) + report.paths.push_back(cstr_at(data, static_cast(*strtabFileOff + slot.second))); + continue; + } + kept.push_back(slot); + } + if (kept.size() == slots.size()) return Report{ Outcome::NothingToDo, {}, {} }; + + // Same byte length: the freed slots become DT_NULL padding, which is what + // a linker writes there anyway. Nothing after PT_DYNAMIC moves, so no + // section header, segment offset or relocation needs touching. + kept.resize(slots.size(), { kDtNull, 0 }); + for (std::size_t i = 0; i < kept.size(); ++i) { + const auto at = *dynOff + i * slotW; + b.write(static_cast(at), ptrW, kept[i].first); + b.write(static_cast(at + ptrW), ptrW, kept[i].second); + } + + if (!spit(image, data)) + return std::unexpected(std::format("cannot write '{}'", image.string())); + report.outcome = Outcome::Removed; + return report; +} + +// Mach-O: READ ONLY. +// +// The same defect can exist here as LC_RPATH, and the same edit is possible in +// principle — but mcpp has no macOS artifact to measure it on in the suite +// that gates this code, and shipping an untested byte-editor for a format +// whose load commands carry their own sizes is how a package becomes +// unloadable instead of unportable. So this reads them and hands them back; +// the caller decides what to say. `install_name_tool -delete_rpath` is the +// documented manual step until a macOS-gated test exists. +std::expected report_macho(const std::filesystem::path& image, + const std::string& d) +{ + constexpr std::uint32_t kMagic64 = 0xFEEDFACFu, kMagic32 = 0xFEEDFACEu; + constexpr std::uint32_t kCigam64 = 0xCFFAEDFEu, kCigam32 = 0xCEFAEDFEu; + constexpr std::uint32_t kLcRpath = 0x1Cu; + if (d.size() < 32) return std::unexpected(std::format( + "'{}' is too short to be a Mach-O image", image.string())); + + auto raw32 = [&](std::size_t off, bool little) { + std::uint32_t v = 0; + for (std::size_t i = 0; i < 4; ++i) + v |= static_cast(static_cast(d[off + (little ? i : 3 - i)])) << (8 * i); + return v; + }; + const auto magicLE = raw32(0, true); + bool little = true, wide = true; + if (magicLE == kMagic64) { little = true; wide = true; } + else if (magicLE == kMagic32) { little = true; wide = false; } + else if (magicLE == kCigam64) { little = false; wide = true; } + else if (magicLE == kCigam32) { little = false; wide = false; } + else return Report{ Outcome::Unanalysed, {}, + "a universal (fat) Mach-O; its per-architecture images are not walked" }; + + const auto ncmds = raw32(16, little); + const auto sizeofcmds = raw32(20, little); + std::size_t at = wide ? 32 : 28; + if (at + sizeofcmds > d.size()) return std::unexpected(std::format( + "'{}' has a Mach-O load-command table that runs past the file", image.string())); + + Report report; + for (std::uint32_t i = 0; i < ncmds; ++i) { + if (at + 8 > d.size()) break; + const auto cmd = raw32(at, little); + const auto cmdsize = raw32(at + 4, little); + if (cmdsize < 8 || at + cmdsize > d.size()) break; + if (cmd == kLcRpath) { + const auto strOff = raw32(at + 8, little); + if (strOff < cmdsize) report.paths.push_back(cstr_at(d, at + strOff)); + } + at += cmdsize; + } + report.outcome = Outcome::Reported; + return report; +} + +} // namespace + +std::string_view describe(Outcome o) { + switch (o) { + case Outcome::Removed: return "removed"; + case Outcome::NothingToDo: return "already clean"; + case Outcome::NotApplicable: return "not applicable"; + case Outcome::Reported: return "reported"; + case Outcome::Unanalysed: return "not analysed"; + } + return "not analysed"; +} + +std::expected strip_search_paths(const std::filesystem::path& image) +{ + std::error_code ec; + if (!std::filesystem::is_regular_file(image, ec)) + return std::unexpected(std::format("'{}' is not a file", image.string())); + + auto data = slurp(image); + if (!data) return std::unexpected(std::format("cannot read '{}'", image.string())); + if (data->size() < 8) return Report{ Outcome::NotApplicable, {}, {} }; + + // The format is asked of the BYTES, never of the host or of the file + // extension: a Linux host packing a `--target x86_64-windows-gnu` leg holds + // a PE, and a `.so` built for another architecture is still an ELF. + switch (mcpp::pack::binfmt::identify(image).format) { + case mcpp::pack::binfmt::Format::Elf: + return strip_elf(image, std::move(*data)); + case mcpp::pack::binfmt::Format::MachO: + return report_macho(image, *data); + case mcpp::pack::binfmt::Format::Pe: + // PE has no loader search path baked into the image at all — the + // DLL search order is the process's, not the file's. Nothing to + // remove, and that is an answer rather than a gap. + return Report{ Outcome::NotApplicable, {}, {} }; + case mcpp::pack::binfmt::Format::Unknown: + break; + } + // A static archive lands here, and so does anything else that is not an + // image. Both are `NotApplicable`: an archive's members are relocatable + // objects, which carry no dynamic section to begin with. + return Report{ Outcome::NotApplicable, {}, {} }; +} + +} // namespace mcpp::pack::relocate diff --git a/src/pack/strip.cppm b/src/pack/strip.cppm new file mode 100644 index 00000000..866ff163 --- /dev/null +++ b/src/pack/strip.cppm @@ -0,0 +1,228 @@ +// mcpp.pack.strip — removing debug information from a SHIPPED artifact. +// +// WHY A PACKAGING STEP AND NOT `[profile.].strip` +// +// mcpp already has a `strip` axis: `[profile.dist].strip` appends `-s` to the +// LINK. It cannot do this job, and the reason is not a detail: +// +// * a `-s` link never touches a static archive, and a static archive is what +// most library packages ship; +// * `-s` is strip-ALL, which is the one thing an archive must not have done +// to it (below); +// * a link-time strip discards the symbols instead of separating them, so +// there is no `.debug` file to keep. +// +// So the two are different decisions with different inputs, and they are named +// differently: `[profile].strip` is "how is this built", `[pack].strip` is +// "what travels". Documented in docs/02 and docs/12. +// +// ⚠️ THE DIVISION BY ARTIFACT SHAPE IS LOAD-BEARING, AND IT IS MEASURED +// +// `strip --strip-all` on a `.a` removes the ARCHIVE SYMBOL INDEX, and the +// package then fails at the consumer's link with a message that names neither +// strip nor the publisher: +// +// ld: ./libdep.a: error adding symbols: archive has no index; run ranlib to add one +// +// while `--strip-debug` on the same archive links and runs (measured: 2988 → +// 1244 bytes, `ok=42`). A shared library is the opposite case — it may lose +// its `.symtab` but must keep `.dynsym`, which is what `--strip-unneeded` +// means. So there is a table, and it is dh_strip's: +// +// executable --strip-all (nothing links against it) +// shared library --strip-unneeded (keeps .dynsym = the exports) +// static archive --strip-debug (keeps .symtab = the archive index) +// + --enable-deterministic-archives +// +// `--remove-section=.comment --remove-section=.note` goes on all three, also +// from dh_strip. It does NOT match `.note.gnu.build-id`: section removal is by +// exact name, so the build-id survives and `--add-gnu-debuglink` still has +// something to pair with. +// +// EMPTY TOOL IS NOT ALWAYS AN ERROR +// +// PE/MSVC keeps debug information in a separate `.pdb` by design, so there is +// nothing in-band to remove and no binutils to remove it with. That is +// `NotApplicable`. Every other format carries DWARF inside the image, so a +// missing `strip` there is a REFUSAL — the same stance `run_library_pack` +// already takes for a missing archiver, and for the same reason: shipping the +// artifact anyway is the silent-wrong-answer this feature exists to remove. + +export module mcpp.pack.strip; + +import std; +import mcpp.pack.binfmt; +import mcpp.platform; + +export namespace mcpp::pack { + +// What the artifact IS. Taken from the link unit's kind, never guessed from +// the extension: `libfoo.a` and `foo.lib` are the same shape under different +// names, and a `.so` that is really a linker script is neither. +enum class ArtifactShape { Executable, SharedLibrary, StaticArchive }; + +// The tools for ONE leg, resolved from that leg's toolchain by the caller +// (mcpp.toolchain.binutils_tool). Both may be empty; see the header for when +// that is an answer and when it is a refusal. +struct StripTools { + std::filesystem::path strip; + std::filesystem::path objcopy; + // Does this leg's format carry debug information inside the image? + // False only for PE/MSVC (`.pdb`). Resolved once by the caller so this + // module never has to know what a toolchain is. + bool inBandDebugInfo = true; +}; + +enum class StripOutcome { Stripped, NotApplicable }; + +struct StripResult { + StripOutcome outcome = StripOutcome::NotApplicable; + std::filesystem::path debugFile; // written only when a debug dir was given + std::uintmax_t before = 0; + std::uintmax_t after = 0; +}; + +// The strip arguments for `shape`. Exported for its unit test — the table +// above is the whole feature, and a test that has to spawn a real `strip` to +// read it back would run on one platform out of three. +std::vector strip_args(ArtifactShape shape); + +// Separate the debug information (when `debugDir` is non-empty), then strip. +// +// ORDER IS NOT FREE: `--only-keep-debug` has to read the artifact while it +// still has its debug sections, and `--add-gnu-debuglink` has to write into +// the artifact after they are gone. Doing it the other way round produces a +// `.debug` file with no debug information in it and no diagnostic. +std::expected +strip_artifact(const std::filesystem::path& artifact, + ArtifactShape shape, + const StripTools& tools, + const std::filesystem::path& debugDir); + +} // namespace mcpp::pack + +namespace mcpp::pack { + +namespace { + +std::expected run_tool(const std::string& cmd) { + auto r = mcpp::platform::process::capture(cmd + " 2>&1"); + if (r.exit_code != 0) + return std::unexpected(std::format( + " command: {}\n output : {}", cmd, r.output)); + return {}; +} + +std::string join_quoted(const std::filesystem::path& tool, + const std::vector& args, + const std::filesystem::path& file) +{ + std::string cmd = mcpp::platform::shell::quote(tool.string()); + for (auto const& a : args) cmd += " " + mcpp::platform::shell::quote(a); + cmd += " " + mcpp::platform::shell::quote(file.string()); + return cmd; +} + +std::uintmax_t size_of(const std::filesystem::path& p) { + std::error_code ec; + auto n = std::filesystem::file_size(p, ec); + return ec ? 0 : n; +} + +} // namespace + +std::vector strip_args(ArtifactShape shape) { + // dh_strip's own division. See the header for the measurement behind the + // archive row — it is the one that turns a package into an unlinkable one. + std::vector args{ "--remove-section=.comment", + "--remove-section=.note" }; + switch (shape) { + case ArtifactShape::Executable: + args.push_back("--strip-all"); + break; + case ArtifactShape::SharedLibrary: + args.push_back("--strip-unneeded"); + break; + case ArtifactShape::StaticArchive: + args.push_back("--strip-debug"); + // Zero the member uid/gid/timestamps while we are rewriting the + // archive anyway: two identical packs should produce identical + // bytes, and this is the one place the packer touches them. + args.push_back("--enable-deterministic-archives"); + break; + } + return args; +} + +std::expected +strip_artifact(const std::filesystem::path& artifact, + ArtifactShape shape, + const StripTools& tools, + const std::filesystem::path& debugDir) +{ + StripResult out; + std::error_code ec; + if (!std::filesystem::is_regular_file(artifact, ec)) + return std::unexpected(std::format("'{}' is not a file", artifact.string())); + + if (!tools.inBandDebugInfo) { + // PE/MSVC: the `.pdb` is a separate file and was never inside this one. + out.outcome = StripOutcome::NotApplicable; + return out; + } + if (tools.strip.empty()) + return std::unexpected(std::format( + "no `strip` was resolved for this target, so '{}' would ship with its " + "debug information and the publisher's absolute source paths.\n" + " Pass `--no-strip` to ship it as built, or install the binutils that " + "match this toolchain.", + artifact.filename().string())); + + out.before = size_of(artifact); + + // ── separate, when asked ────────────────────────────────────────── + // + // Not for archives: a `.a` is a container of relocatable objects and + // `--only-keep-debug` on one produces something no debugger consumes. + // dh_strip makes the same exclusion. + const bool separate = !debugDir.empty() && shape != ArtifactShape::StaticArchive; + if (separate) { + if (tools.objcopy.empty()) + return std::unexpected(std::format( + "`--debug-symbols` was given but no `objcopy` was resolved for this " + "target, so the debug information for '{}' cannot be separated.", + artifact.filename().string())); + std::filesystem::create_directories(debugDir, ec); + if (ec) return std::unexpected(std::format( + "cannot create '{}': {}", debugDir.string(), ec.message())); + out.debugFile = debugDir / (artifact.filename().string() + ".debug"); + auto cmd = std::format("{} --only-keep-debug {} {}", + mcpp::platform::shell::quote(tools.objcopy.string()), + mcpp::platform::shell::quote(artifact.string()), + mcpp::platform::shell::quote(out.debugFile.string())); + if (auto r = run_tool(cmd); !r) return std::unexpected(std::format( + "cannot separate debug information from '{}'.\n{}", + artifact.string(), r.error())); + } + + // ── strip ───────────────────────────────────────────────────────── + if (auto r = run_tool(join_quoted(tools.strip, strip_args(shape), artifact)); !r) + return std::unexpected(std::format( + "cannot strip '{}'.\n{}", artifact.string(), r.error())); + + // ── and point the stripped artifact back at its symbols ─────────── + if (separate) { + auto cmd = std::format("{} --add-gnu-debuglink={} {}", + mcpp::platform::shell::quote(tools.objcopy.string()), + mcpp::platform::shell::quote(out.debugFile.string()), + mcpp::platform::shell::quote(artifact.string())); + if (auto r = run_tool(cmd); !r) return std::unexpected(std::format( + "cannot add the debug link to '{}'.\n{}", artifact.string(), r.error())); + } + + out.after = size_of(artifact); + out.outcome = StripOutcome::Stripped; + return out; +} + +} // namespace mcpp::pack diff --git a/src/toolchain/clang.cppm b/src/toolchain/clang.cppm index c7438745..3c7f6034 100644 --- a/src/toolchain/clang.cppm +++ b/src/toolchain/clang.cppm @@ -43,7 +43,6 @@ std::vector std_compat_build_commands(const Toolchain& tc, std::string_view sysrootFlag, std::string_view cppStandardFlag); -std::filesystem::path archive_tool(const Toolchain& tc); // Locate clang-scan-deps in the same bin/ directory as clang++. std::optional find_scan_deps(const Toolchain& tc); @@ -244,13 +243,6 @@ std::vector std_module_build_commands(const Toolchain& tc, #endif } -std::filesystem::path archive_tool(const Toolchain& tc) { - auto llvmAr = tc.binaryPath.parent_path() / - (std::string("llvm-ar") + std::string(mcpp::platform::exe_suffix)); - if (std::filesystem::exists(llvmAr)) return llvmAr; - return {}; -} - std::optional find_scan_deps(const Toolchain& tc) { auto p = tc.binaryPath.parent_path() / (std::string("clang-scan-deps") + std::string(mcpp::platform::exe_suffix)); diff --git a/src/toolchain/registry.cppm b/src/toolchain/registry.cppm index f88a88dc..23e2b67b 100644 --- a/src/toolchain/registry.cppm +++ b/src/toolchain/registry.cppm @@ -215,6 +215,23 @@ struct AvailableIndex { std::vector available_toolchain_indexes(); std::filesystem::path derive_c_compiler(const Toolchain& tc); + +// A binutils-family tool for THIS toolchain's TARGET, named in the GNU +// spelling ("ar", "strip", "objcopy"). +// +// WHY ONE FUNCTION. Four families spell the same tool four ways — llvm- +// beside clang, `-` for a cross, a separate binutils payload for a +// glibc gcc — and until this existed only `ar` knew that. A second tool added +// by copying `archive_tool` would be the same decision derived twice, and the +// copy is the one that silently stops agreeing. +// +// EMPTY IS AN ANSWER, NOT ALWAYS A FAILURE. On MSVC there is no binutils and +// no need for one: PE/MSVC keeps debug information in a separate `.pdb`, so +// there is nothing in-band for `strip` to remove. Callers must distinguish +// "this format has nothing to strip" from "the tool this format needs is +// missing" — see mcpp.pack.strip, which refuses only the second. +std::filesystem::path binutils_tool(const Toolchain& tc, std::string_view name); + std::filesystem::path archive_tool(const Toolchain& tc); std::filesystem::path link_tool(const Toolchain& tc); std::filesystem::path staged_std_bmi_path(const Toolchain& tc, @@ -609,55 +626,69 @@ std::filesystem::path derive_c_compiler(const Toolchain& tc) { return derive_c_compiler_path(tc.binaryPath); } -std::filesystem::path archive_tool(const Toolchain& tc) { - if (tc.compiler == CompilerId::MSVC) { - auto lib = tc.binaryPath.parent_path() / "lib.exe"; - std::error_code ec; - if (std::filesystem::exists(lib, ec)) return lib; +std::filesystem::path binutils_tool(const Toolchain& tc, std::string_view name) { + // MSVC: no binutils, and none wanted — see the declaration. + if (tc.compiler == CompilerId::MSVC) return {}; + + std::error_code ec; + auto dir = tc.binaryPath.parent_path(); + + // Clang ships the whole family as `llvm-` beside the frontend. + if (is_clang(tc)) { + auto llvmTool = dir / (std::string("llvm-") + std::string(name) + + std::string(mcpp::platform::exe_suffix)); + if (std::filesystem::exists(llvmTool, ec)) return llvmTool; return {}; } - if (is_clang(tc)) return mcpp::toolchain::clang::archive_tool(tc); // MinGW bundles its own binutils next to the frontend (self-contained, // like musl) — never an external binutils xpkg. Native (Windows-host) ships // `ar.exe`; the Linux-hosted cross ships the triple-prefixed ELF tool // `x86_64-w64-mingw32-ar`. Try the cross form first, then native. if (is_mingw_target(tc)) { - std::error_code ec; - auto dir = tc.binaryPath.parent_path(); if (!tc.targetTriple.empty()) { - auto crossAr = dir / (tc.targetTriple + "-ar"); - if (std::filesystem::exists(crossAr, ec)) return crossAr; + auto cross = dir / (tc.targetTriple + "-" + std::string(name)); + if (std::filesystem::exists(cross, ec)) return cross; } - auto ar = dir / "ar.exe"; - if (std::filesystem::exists(ar, ec)) return ar; + auto native = dir / (std::string(name) + ".exe"); + if (std::filesystem::exists(native, ec)) return native; return {}; } if (!is_musl_target(tc)) { if (auto binutilsBin = mcpp::toolchain::gcc::find_binutils_bin(tc.binaryPath)) - return *binutilsBin / "ar"; + return *binutilsBin / std::string(name); } - // musl `ar` is the triple-prefixed cross tool (e.g. aarch64-linux-musl-ar), + // A musl tool is the triple-prefixed cross form (e.g. aarch64-linux-musl-ar), // sitting next to the frontend. Derive from the resolved target triple so - // cross targets pick the matching archiver instead of the x86_64 one. - std::string arName = !tc.targetTriple.empty() - ? tc.targetTriple + "-ar" - : "x86_64-linux-musl-ar"; - auto dir = tc.binaryPath.parent_path(); + // cross targets pick the matching tool instead of the x86_64 one. + std::string crossName = (!tc.targetTriple.empty() + ? tc.targetTriple : std::string("x86_64-linux-musl")) + "-" + std::string(name); // Same `.exe` reasoning as the frontend candidates above: a windows-hosted // musl cross payload ships `-ar.exe`. Try it first, then the bare // name (which is what every ELF host has). if constexpr (mcpp::platform::is_windows) { - auto muslArExe = dir / (arName + ".exe"); - if (std::filesystem::exists(muslArExe)) return muslArExe; + auto muslExe = dir / (crossName + ".exe"); + if (std::filesystem::exists(muslExe, ec)) return muslExe; } - auto muslAr = dir / arName; - if (std::filesystem::exists(muslAr)) return muslAr; + auto musl = dir / crossName; + if (std::filesystem::exists(musl, ec)) return musl; return {}; } +std::filesystem::path archive_tool(const Toolchain& tc) { + // The one spelling that is NOT a binutils name: MSVC archives with + // LIB.EXE, which takes a different verb for every operation. + if (tc.compiler == CompilerId::MSVC) { + auto lib = tc.binaryPath.parent_path() / "lib.exe"; + std::error_code ec; + if (std::filesystem::exists(lib, ec)) return lib; + return {}; + } + return binutils_tool(tc, "ar"); +} + std::filesystem::path staged_std_bmi_path(const Toolchain& tc, const std::filesystem::path& outputDir) { if (tc.compiler == CompilerId::MSVC) diff --git a/src/version.cppm b/src/version.cppm index f83be425..c4f41023 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.19.4"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.20.1"; } // namespace mcpp diff --git a/tests/e2e/215_pack_has_no_build_machine_paths.sh b/tests/e2e/215_pack_has_no_build_machine_paths.sh index 24d59d51..1483fd2e 100755 --- a/tests/e2e/215_pack_has_no_build_machine_paths.sh +++ b/tests/e2e/215_pack_has_no_build_machine_paths.sh @@ -29,60 +29,10 @@ TMP=$(mktemp -d) trap "rm -rf $TMP" EXIT export MCPP_HOME=$HOME/.mcpp -read_tag() { -python3 - "$1" <<'PY' -import struct, sys -d = open(sys.argv[1], 'rb').read() -if d[:4] != b'\x7fELF' or d[4] != 2: - print("NOT-ELF64"); raise SystemExit -phoff, = struct.unpack_from('/registry/data/xpkgs/xim-x-glibc/2.44/lib64 +# : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 +# : /registry/subos/default/lib +# +# On another machine those directories do not exist, and — this is the part +# that makes it fatal rather than untidy — a shared object that carries ANY +# DT_RUNPATH makes the loader skip the whole inherited DT_RPATH chain when +# resolving that object's own dependencies. So the consumer's carefully +# computed DT_RPATH (payload + package dir + SubOS farm) is not consulted, and +# the program dies with +# +# error while loading shared libraries: libstdc++.so.6: cannot open shared object file +# +# ⚠️ AND WHY `$ORIGIN` IS NOT THE FIX. Measured: a DT_RUNPATH of `$ORIGIN`, and +# a DT_RUNPATH of the empty string, both fail exactly the same way. It is the +# TAG's presence that disables inheritance, not its contents. The criterion is +# therefore "there is no tag", never "the tag is relative". +# +# ⚠️ AND WHY THIS TEST DELIBERATELY BREAKS THE PACKAGE HALF-WAY THROUGH. +# Asserting only that the fixed package runs cannot tell "the defect is fixed" +# from "this machine happens to satisfy the stale path" — which is precisely +# what made the pre-existing e2e (251) green throughout the bug's life: it +# consumes the package on the machine that built it, where those directories +# are right there. So the defect is put BACK with patchelf, the consumer is +# required to FAIL, and only then is it restored. A guard that cannot observe +# the defect is not a guard. +set -e +source "$(dirname "$0")/_host_path.sh" +source "$(dirname "$0")/_elf_tag.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit-shared] +kind = "shared" +soname = "libmathkit.so.1" +EOF + +cd mathkit +"$MCPP" pack mathkit-shared > pack.log 2>&1 || { cat pack.log; echo "FAIL: pack"; exit 1; } +pkg="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +PKG_HOST="$(host_path "$pkg")" +libdir="$pkg/lib/x86_64-linux-gnu" +[[ -d "$libdir" ]] || libdir="$(dirname "$(find "$pkg/lib" -name 'libmathkit-shared.so' | head -1)")" +so="$libdir/libmathkit-shared.so" +[[ -f "$so" ]] || { find "$pkg" -type f; echo "FAIL: no .so in the package"; exit 1; } + +# ── 1. static: EVERY ELF in the package, and both of the .so's names ──────── +# +# The SONAME alias is checked too, and that is not belt-and-braces: it is a +# symlink here, but on a filesystem where `create_symlink` fails the packer +# falls back to a COPY — and that copy used to be made from the BUILD TREE's +# file rather than from the staged one, i.e. from the unrelocated original, +# under the exact name the loader asks for. +fail=0 +swept=0 +while IFS= read -r obj; do + head -c4 "$obj" 2>/dev/null | grep -q $'\x7fELF' || continue + read -r form tag paths <<<"$(read_tag "$obj")" + [[ "$form" == "NOT-ELF64" ]] && continue + swept=$((swept + 1)) + printf ' %-34s %-14s %s %s\n' "$(basename "$obj")" "$form" "$tag" "$paths" + if [[ "$tag" != "NONE" ]]; then + echo "FAIL: a packaged library carries $tag ($paths)" + echo " A shipped .so must carry NO loader search path: any DT_RUNPATH —" + echo " including \$ORIGIN and the empty string — stops the consumer's" + echo " DT_RPATH from being inherited for this object's dependencies." + fail=1 + fi +done < <(find "$libdir" \( -type f -o -type l \)) +[[ "$swept" -ge 1 ]] || { echo "FAIL: swept no ELF — the test proved nothing"; exit 1; } +[[ "$fail" == "0" ]] || exit 1 + +# ── 2. a consumer of the package starts ──────────────────────────────────── +cd "$TMP" +mkdir -p app/src +cat > app/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("ok=%d\n", mk::answer()); return 0; } +EOF +cat > app/mcpp.toml < build.log 2>&1 ) || { cat app/build.log; echo "FAIL: consumer build"; exit 1; } +exe="$(find app/target -path '*/bin/app' -type f | head -1)" +[[ -x "$exe" ]] || { echo "FAIL: no consumer binary"; exit 1; } + +run_bare() { "$exe" 2>&1; } # NOT `mcpp run`: no injected environment + +out="$(run_bare)" && rc=0 || rc=$? +[[ "$rc" == "0" && "$out" == *"ok=42"* ]] || { + echo "FAIL: the consumer does not start against the packed library" + echo " rc=$rc out=$out"; exit 1; } + +# ── 3. put the defect BACK, and require the consumer to fail ─────────────── +# +# `/nonexistent-machine/...` stands in for "the publisher's store, on somebody +# else's computer". Nothing is rebuilt: only the shipped library's dynamic +# section changes, which is exactly the difference between the two machines. +cp "$so" "$so.packed" +patchelf --set-rpath '/nonexistent-machine/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/lib64' "$so" +out="$(run_bare)" && rc=0 || rc=$? +if [[ "$rc" == "0" ]]; then + echo "FAIL: the consumer still ran with a stale RUNPATH restored on the library." + echo " That means this test cannot observe the defect it exists for —" + echo " the consumer is finding libstdc++ some other way, so a regression" + echo " in the packer would pass unnoticed." + exit 1 +fi +[[ "$out" == *"cannot open shared object file"* ]] || { + echo "FAIL: the restored defect produced an unexpected failure: $out"; exit 1; } +echo " defect restored → consumer fails as expected (rc=$rc)" + +# ── 4. …and restore, so the pass is not an artifact of step 3 ────────────── +cp "$so.packed" "$so" +out="$(run_bare)" && rc=0 || rc=$? +[[ "$rc" == "0" && "$out" == *"ok=42"* ]] || { + echo "FAIL: restoring the packed library did not restore the behaviour" + echo " rc=$rc out=$out"; exit 1; } + +echo "PASS: a packed shared library carries no build-machine search path, and the guard can see the defect" diff --git a/tests/e2e/265_pack_strips_but_stays_usable.sh b/tests/e2e/265_pack_strips_but_stays_usable.sh new file mode 100755 index 00000000..61cf5db7 --- /dev/null +++ b/tests/e2e/265_pack_strips_but_stays_usable.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +# requires: pack python3 +# 265_pack_strips_but_stays_usable.sh — `mcpp pack` ships stripped artifacts, +# and stripping must not make them unusable. +# +# ⚠️ THE ONE THAT BITES: A STATIC ARCHIVE CANNOT BE `--strip-all`ed. +# +# `strip` with no shape argument defaults to strip-all, and on a `.a` that +# removes the ARCHIVE SYMBOL INDEX. The package then fails at the CONSUMER's +# link with +# +# ld: libmathkit.a: error adding symbols: archive has no index; run ranlib to add one +# +# — a message that names neither `strip` nor the publisher, arriving on a +# machine the publisher does not have. Measured: `--strip-debug` keeps the +# index (2988 → 1244 bytes) and the consumer links and runs. So the packer +# uses dh_strip's division, and this test consumes BOTH shapes rather than +# inspecting them: "the archive still has an index" is a proxy, "a program +# linked against it prints the right number" is the thing. +# +# The three configuration channels are pinned from BOTH sides — a test that +# only checks the default cannot tell "the switch works" from "there is no +# switch". +set -e +source "$(dirname "$0")/_host_path.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit] +kind = "lib" +[targets.mathkit-shared] +kind = "shared" +soname = "libmathkit.so.1" +EOF + +# `file` is not required by the harness; ask the ELF itself instead. +has_symtab() { +python3 - "$1" <<'PY' +import struct, sys +d = open(sys.argv[1], 'rb').read() +if d[:4] != b'\x7fELF' or d[4] != 2: + print("no"); raise SystemExit +shoff, = struct.unpack_from(' — build a program against it and run + local pkg="$1" want="$2" tag="$3" + rm -rf "$TMP/consumer" + mkdir -p "$TMP/consumer/src" + cat > "$TMP/consumer/src/main.cpp" <<'EOF' +#include +import mathkit; +int main(){ std::printf("ok=%d\n", mk::answer()); return 0; } +EOF + cat > "$TMP/consumer/mcpp.toml" < run.log 2>&1 ) || { + cat "$TMP/consumer/run.log" + echo "FAIL[$tag]: a consumer cannot use the stripped package"; exit 1; } + grep -q "$want" "$TMP/consumer/run.log" || { + cat "$TMP/consumer/run.log"; echo "FAIL[$tag]: wrong answer"; exit 1; } +} + +cd mathkit + +# ── 1. static: stripped, and STILL LINKABLE ──────────────────────────────── +rm -rf target/dist +"$MCPP" pack mathkit > pack-a.log 2>&1 || { cat pack-a.log; echo "FAIL: static pack"; exit 1; } +grep -q "Stripped" pack-a.log || { cat pack-a.log; echo "FAIL: the static leg was not stripped"; exit 1; } +pkg_a="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +consume "$pkg_a" "ok=42" "static" +echo " static archive: stripped and still linkable" + +# ── 2. shared: stripped, and STILL LOADABLE ──────────────────────────────── +rm -rf target/dist +"$MCPP" pack mathkit-shared > pack-b.log 2>&1 || { cat pack-b.log; echo "FAIL: shared pack"; exit 1; } +grep -q "Stripped" pack-b.log || { cat pack-b.log; echo "FAIL: the shared leg was not stripped"; exit 1; } +pkg_b="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +so="$(find "$pkg_b/lib" -name 'libmathkit-shared.so' -type f | head -1)" +[[ -n "$so" ]] || { echo "FAIL: no .so"; exit 1; } +# `--strip-unneeded` must leave `.dynsym` — it IS the export list. If this +# regressed to `--strip-all` the link below would still work (ld reads +# `.dynsym`), so the run is what proves it. +[[ "$(has_symtab "$so")" == "no" ]] || { + echo "FAIL: the shared library still carries a symbol/debug table after stripping"; exit 1; } +consume "$pkg_b" "ok=42" "shared" +echo " shared library: stripped and still loadable" + +# ── 3. the OTHER side of every switch ────────────────────────────────────── +rm -rf target/dist +"$MCPP" pack mathkit-shared --no-strip > pack-c.log 2>&1 || { cat pack-c.log; echo "FAIL: --no-strip pack"; exit 1; } +grep -q "Stripped" pack-c.log && { cat pack-c.log; echo "FAIL: --no-strip stripped anyway"; exit 1; } +pkg_c="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +so_c="$(find "$pkg_c/lib" -name 'libmathkit-shared.so' -type f | head -1)" +[[ "$(has_symtab "$so_c")" == "yes" ]] || { + echo "FAIL: --no-strip produced an artifact with no symbols — the flag did nothing" + exit 1; } +echo " --no-strip: symbols kept" + +rm -rf target/dist +cp mcpp.toml mcpp.toml.bak +printf '\n[pack]\nstrip = false\n' >> mcpp.toml +"$MCPP" pack mathkit-shared > pack-d.log 2>&1 || { cat pack-d.log; echo "FAIL: [pack] strip pack"; exit 1; } +grep -q "Stripped" pack-d.log && { cat pack-d.log; echo "FAIL: [pack] strip = false was ignored"; exit 1; } +mv mcpp.toml.bak mcpp.toml +echo " [pack] strip = false: honoured" + +# ── 4. --debug-symbols keeps the symbols, beside the artifact ────────────── +rm -rf target/dist +"$MCPP" pack mathkit-shared --debug-symbols dbg > pack-e.log 2>&1 || { + cat pack-e.log; echo "FAIL: --debug-symbols pack"; exit 1; } +[[ -f dbg/libmathkit-shared.so.debug ]] || { + ls -R dbg 2>&1; echo "FAIL: no separated debug file"; exit 1; } +[[ "$(has_symtab dbg/libmathkit-shared.so.debug)" == "yes" ]] || { + echo "FAIL: the separated debug file carries no debug information"; exit 1; } +pkg_e="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +so_e="$(find "$pkg_e/lib" -name 'libmathkit-shared.so' -type f | head -1)" +[[ "$(has_symtab "$so_e")" == "no" ]] || { + echo "FAIL: --debug-symbols left the debug information in the shipped artifact too" + exit 1; } +# The link back. Without it a debugger has the file and no way to find it. +python3 - "$so_e" <<'PY' || { echo "FAIL: no .gnu_debuglink in the stripped artifact"; exit 1; } +import struct, sys +d = open(sys.argv[1], 'rb').read() +shoff, = struct.unpack_from('' +# +# That variable is glibc's. dyld has never heard of it (its counterpart is +# `DYLD_PRINT_LIBRARIES`), so on macOS this does not trace anything — IT RUNS +# THE USER'S PROGRAM. Whatever the program prints is then parsed as a +# dependency table, yields nothing, and a bundle containing just the binary is +# written and reported as `Packed`. A program with side effects performs them. +# An interactive one hangs the packer. +# +# The `_WIN32` branch beside it has refused the same class since it was +# written; macOS was simply never checked, because the e2e harness only grants +# the `pack` capability where `elf` + `patchelf` are both present — i.e. Linux. +# So no job in this suite has ever run `mcpp pack` on a Mac. +# +# ⚠️ BOTH SIDES, ON THE SAME HOST. Asserting only the refusal cannot tell "the +# gate works" from "pack is broken on this machine". So the same run also packs +# a LIBRARY target, which takes a different pipeline and must still succeed. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p proj/src +cat > proj/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > proj/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > proj/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("ok=%d\n", mk::answer()); return 0; } +EOF +cat > proj/mcpp.toml <<'EOF' +[package] +name = "proj" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.proj] +kind = "bin" +main = "src/main.cpp" +[targets.mathkit] +kind = "lib" +EOF + +cd proj + +# ── 1. the program is refused, by name ───────────────────────────────────── +if "$MCPP" pack proj > pack.log 2>&1; then + cat pack.log + echo "FAIL: mcpp pack produced a bundle for a Mach-O program." + echo " Its dependency closure cannot be resolved on this platform, so" + echo " whatever it produced is not one — and producing it RAN the program." + exit 1 +fi +grep -qi "Mach-O" pack.log || { + cat pack.log + echo "FAIL: pack failed, but not with the Mach-O refusal — so this test is" + echo " observing some other failure and proves nothing about the gate." + exit 1; } +grep -q "LD_TRACE_LOADED_OBJECTS" pack.log || { + cat pack.log + echo "FAIL: the refusal does not say WHY. A reader has to be able to tell" + echo " this from 'macOS is unsupported in general'." + exit 1; } +echo " a Mach-O program is refused, and the message names the mechanism" + +# ── 2. …and a library target on the same host still packs ────────────────── +"$MCPP" pack mathkit > packlib.log 2>&1 || { + cat packlib.log + echo "FAIL: a library target does not pack on macOS either — the refusal above" + echo " is therefore not evidence of a working gate." + exit 1; } +pkg="$(find target/dist -maxdepth 1 -type d -name 'proj-0.1.0-*' | head -1)" +[[ -n "$(find "$pkg" -name '*.a' | head -1)" ]] || { + find "$pkg" -type f; echo "FAIL: no archive in the library package"; exit 1; } +echo " a library target still packs" + +echo "PASS: mcpp pack refuses a Mach-O program with the reason, and library packaging is unaffected" diff --git a/tests/e2e/_elf_tag.sh b/tests/e2e/_elf_tag.sh new file mode 100644 index 00000000..325cd9f9 --- /dev/null +++ b/tests/e2e/_elf_tag.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# _elf_tag.sh — read an ELF's FORM and its loader search-path TAG. +# +# `read_tag ` prints three fields: +# +#
+# +# form executable | shared_library | NOT-ELF64 +# tag RPATH | RUNPATH | BOTH | NONE +# paths ':'-joined contents of whichever tag is present +# +# ⚠️ WHY A PARSER AND NOT `strings` +# +# Removing DT_RPATH/DT_RUNPATH removes the ENTRY, not the string it pointed at: +# `.dynstr` is tail-merged by the linker, so a shorter live string can begin +# inside the dead one and deleting those bytes cannot be shown safe. Measured: +# `patchelf --remove-rpath` leaves the identical residue at the identical file +# size, so this is the reference behaviour, not mcpp's shortcut. +# +# Consequence: "this artifact carries no build-machine path" is a question about +# the DYNAMIC ENTRIES. A `grep` over the file's bytes answers a different +# question and reports a correctly relocated artifact as dirty — and it would +# also flip to "clean" for an unrelated reason the day `mcpp pack` started +# stripping DWARF. One check, one thing. +# +# Shared by 215 (application bundles) and 264 (library packages) so the two +# cannot drift into disagreeing about what the criterion is. + +read_tag() { +python3 - "$1" <<'PY' +import struct, sys +d = open(sys.argv[1], 'rb').read() +if d[:4] != b'\x7fELF' or d[4] != 2: + print("NOT-ELF64"); raise SystemExit +phoff, = struct.unpack_from(' + +import std; +import mcpp.pack.relocate; +import mcpp.pack.strip; + +// mcpp.pack.relocate / mcpp.pack.strip — the two halves of "the build machine +// does not travel", tested where CI cannot otherwise reach them. +// +// WHY SYNTHETIC ELF BYTES RATHER THAN A COMPILED .so +// +// The e2e suite already packs a real library and reads the result back, so the +// x86_64/little-endian path has an end-to-end judge. What it CANNOT reach is +// ELF32 and big-endian: no CI job produces one, and `mcpp pack --target` will +// hand exactly those to this code the first time somebody packages for a +// 32-bit or MIPS/PowerPC target. A wrong width or a wrong byte order there +// does not fail — it writes a corrupt dynamic section, which is the failure +// class this whole area exists to remove. +// +// The images below are the smallest thing that is still an ELF for the purpose +// of this question: a header, a PT_LOAD so DT_STRTAB can be mapped, a +// PT_DYNAMIC, and a string table. They are read back by a parser written here, +// deliberately NOT by the module under test — checking an editor with its own +// reader proves only that it is self-consistent. + +namespace { + +struct Dyn { std::uint64_t tag, val; }; + +// A minimal but well-formed ELF image carrying `entries`. +struct SyntheticElf { + bool wide; // true = ELF64 + bool little; + std::string bytes; + + std::size_t ptrW() const { return wide ? 8 : 4; } + std::size_t slotW() const { return ptrW() * 2; } + std::size_t dynOff = 0; + std::size_t strOff = 0; +}; + +void put(std::string& b, std::size_t off, std::size_t width, + std::uint64_t v, bool little) +{ + for (std::size_t i = 0; i < width; ++i) + b[off + (little ? i : width - 1 - i)] = + static_cast((v >> (8 * i)) & 0xFF); +} + +std::uint64_t get(const std::string& b, std::size_t off, std::size_t width, + bool little) +{ + std::uint64_t v = 0; + for (std::size_t i = 0; i < width; ++i) + v |= static_cast( + static_cast(b[off + (little ? i : width - 1 - i)])) + << (8 * i); + return v; +} + +// `strings` are laid out in order; a Dyn whose tag is in `stringTags` has its +// value replaced by that string's offset. +SyntheticElf make_elf(bool wide, bool little, + std::vector entries, + const std::vector& strings, + const std::vector& stringSlots) +{ + SyntheticElf e{ wide, little, {}, }; + const std::size_t ehSize = wide ? 64 : 52; + const std::size_t phEnt = wide ? 56 : 32; + const std::size_t phNum = 2; + const std::size_t phOff = ehSize; + const std::size_t dynOff = phOff + phEnt * phNum; + const std::size_t slotW = (wide ? 8 : 4) * 2; + const std::size_t dynSize = slotW * entries.size(); + const std::size_t strOff = dynOff + dynSize; + + std::string strtab; + std::vector strAt; + strtab.push_back('\0'); // index 0 is the empty string + for (auto const& s : strings) { strAt.push_back(strtab.size()); strtab += s; strtab.push_back('\0'); } + + e.bytes.assign(strOff + strtab.size(), '\0'); + e.dynOff = dynOff; + e.strOff = strOff; + + // ── ELF header ──────────────────────────────────────────────────── + e.bytes[0] = 0x7F; e.bytes[1] = 'E'; e.bytes[2] = 'L'; e.bytes[3] = 'F'; + e.bytes[4] = static_cast(wide ? 2 : 1); + e.bytes[5] = static_cast(little ? 1 : 2); + e.bytes[6] = 1; + put(e.bytes, 16, 2, 3, little); // e_type = ET_DYN + put(e.bytes, 18, 2, wide ? 62 : 3, little); // e_machine + put(e.bytes, wide ? 0x20 : 0x1C, wide ? 8 : 4, phOff, little); + put(e.bytes, wide ? 0x36 : 0x2A, 2, phEnt, little); + put(e.bytes, wide ? 0x38 : 0x2C, 2, phNum, little); + + // ── PT_LOAD covering the whole image, vaddr == file offset ──────── + const std::size_t poOff = wide ? 0x08 : 0x04; + const std::size_t pvOff = wide ? 0x10 : 0x08; + const std::size_t pfOff = wide ? 0x20 : 0x10; + put(e.bytes, phOff, 4, 1, little); // PT_LOAD + put(e.bytes, phOff + poOff, wide ? 8 : 4, 0, little); + put(e.bytes, phOff + pvOff, wide ? 8 : 4, 0, little); + put(e.bytes, phOff + pfOff, wide ? 8 : 4, e.bytes.size(), little); + + // ── PT_DYNAMIC ──────────────────────────────────────────────────── + put(e.bytes, phOff + phEnt, 4, 2, little); // PT_DYNAMIC + put(e.bytes, phOff + phEnt + poOff, wide ? 8 : 4, dynOff, little); + put(e.bytes, phOff + phEnt + pfOff, wide ? 8 : 4, dynSize, little); + + // ── the dynamic array ───────────────────────────────────────────── + for (std::size_t i = 0; i < entries.size(); ++i) { + auto value = entries[i].val; + for (std::size_t k = 0; k < stringSlots.size(); ++k) + if (stringSlots[k] == i) value = strAt[k]; + put(e.bytes, dynOff + i * slotW, wide ? 8 : 4, entries[i].tag, little); + put(e.bytes, dynOff + i * slotW + (wide ? 8 : 4), wide ? 8 : 4, value, little); + } + // DT_STRTAB's value is a VADDR, and vaddr == file offset here by construction. + for (std::size_t i = 0; i < entries.size(); ++i) + if (entries[i].tag == 5) + put(e.bytes, dynOff + i * slotW + (wide ? 8 : 4), wide ? 8 : 4, strOff, little); + + std::memcpy(e.bytes.data() + strOff, strtab.data(), strtab.size()); + return e; +} + +// An independent reader: what tags does this image's dynamic array carry? +std::vector read_tags(const std::string& bytes, std::size_t dynOff, + bool wide, bool little) +{ + const std::size_t ptrW = wide ? 8 : 4; + std::vector tags; + for (std::size_t at = dynOff; at + ptrW * 2 <= bytes.size(); at += ptrW * 2) { + auto tag = get(bytes, at, ptrW, little); + tags.push_back(tag); + if (tag == 0) break; + } + return tags; +} + +std::filesystem::path write_temp(const std::string& bytes, std::string_view stem) { + auto dir = std::filesystem::temp_directory_path() / "mcpp-relocate-test"; + std::filesystem::create_directories(dir); + auto p = dir / std::string(stem); + std::ofstream out(p, std::ios::binary | std::ios::trunc); + out.write(bytes.data(), static_cast(bytes.size())); + out.close(); + return p; +} + +constexpr std::uint64_t kNull = 0, kNeeded = 1, kStrtab = 5, kSoname = 14, + kRpath = 15, kRunpath = 29; + +} // namespace + +// ── the ELF editor ────────────────────────────────────────────────────── + +TEST(PackRelocate, RemovesRunpathFromElf64LittleEndian) { + auto e = make_elf(/*wide=*/true, /*little=*/true, + { {kNeeded, 0}, {kStrtab, 0}, {kRunpath, 0}, {kSoname, 0}, {kNull, 0} }, + { "libc.so.6", "/home/builder/.mcpp/store/lib64", "libfoo.so.1" }, + { 0, 2, 3 }); + const auto before = e.bytes.size(); + auto path = write_temp(e.bytes, "elf64le.so"); + + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()) << (r ? "" : r.error()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::Removed); + ASSERT_EQ(r->paths.size(), 1u); + // The report names what was dropped — a log line that says only "removed" + // cannot tell a stale store path from `$ORIGIN`. + EXPECT_EQ(r->paths[0], "/home/builder/.mcpp/store/lib64"); + + std::ifstream in(path, std::ios::binary); + std::string after{ std::istreambuf_iterator(in), {} }; + // SAME LENGTH: the freed slot becomes DT_NULL padding, so nothing after + // PT_DYNAMIC moves and no offset in the file needs fixing up. + EXPECT_EQ(after.size(), before); + + auto tags = read_tags(after, e.dynOff, true, true); + EXPECT_EQ(std::ranges::count(tags, kRunpath), 0); + // The other entries survive, in order. Losing DT_SONAME here would make the + // library unfindable by the name its consumers link against. + EXPECT_EQ(std::ranges::count(tags, kNeeded), 1); + EXPECT_EQ(std::ranges::count(tags, kSoname), 1); + EXPECT_EQ(std::ranges::count(tags, kStrtab), 1); +} + +TEST(PackRelocate, RemovesRpathFromElf32BigEndian) { + // No CI job produces this shape, and `--target` can. A width or byte-order + // mistake would corrupt the image rather than fail. + auto e = make_elf(/*wide=*/false, /*little=*/false, + { {kNeeded, 0}, {kStrtab, 0}, {kRpath, 0}, {kNull, 0} }, + { "libc.so.6", "/home/builder/store/lib" }, + { 0, 2 }); + const auto before = e.bytes.size(); + auto path = write_temp(e.bytes, "elf32be.so"); + + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()) << (r ? "" : r.error()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::Removed); + ASSERT_EQ(r->paths.size(), 1u); + EXPECT_EQ(r->paths[0], "/home/builder/store/lib"); + + std::ifstream in(path, std::ios::binary); + std::string after{ std::istreambuf_iterator(in), {} }; + EXPECT_EQ(after.size(), before); + auto tags = read_tags(after, e.dynOff, false, false); + EXPECT_EQ(std::ranges::count(tags, kRpath), 0); + EXPECT_EQ(std::ranges::count(tags, kNeeded), 1); +} + +TEST(PackRelocate, RemovesBothTagsWhenAnImageCarriesBoth) { + // `Both` is what a linker driven with `--disable-new-dtags` after an + // explicit -rpath can produce, and the loader honours DT_RUNPATH there. + // Leaving either one behind leaves the defect behind. + auto e = make_elf(true, true, + { {kStrtab, 0}, {kRpath, 0}, {kRunpath, 0}, {kNeeded, 0}, {kNull, 0} }, + { "/a/store/lib", "/b/farm/lib", "libc.so.6" }, + { 1, 2, 3 }); + auto path = write_temp(e.bytes, "elfboth.so"); + + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::Removed); + EXPECT_EQ(r->paths.size(), 2u); + + std::ifstream in(path, std::ios::binary); + std::string after{ std::istreambuf_iterator(in), {} }; + auto tags = read_tags(after, e.dynOff, true, true); + EXPECT_EQ(std::ranges::count(tags, kRpath), 0); + EXPECT_EQ(std::ranges::count(tags, kRunpath), 0); +} + +TEST(PackRelocate, AlreadyCleanIsNotTheSameAsNotChecked) { + auto e = make_elf(true, true, + { {kNeeded, 0}, {kStrtab, 0}, {kNull, 0} }, { "libc.so.6" }, { 0 }); + auto path = write_temp(e.bytes, "elfclean.so"); + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::NothingToDo); + EXPECT_TRUE(r->paths.empty()); +} + +TEST(PackRelocate, AStaticArchiveIsNotApplicableRatherThanAnError) { + // Every leg goes through this step, and a `.a` has no dynamic section to + // begin with. Reporting that as a failure would make the caller branch on + // the format before calling — which is the coupling this module removes. + auto path = write_temp("!\n/ 0 0 0 0 4 `\n", + "libfoo.a"); + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::NotApplicable); +} + +TEST(PackRelocate, AMissingFileIsAnError) { + auto r = mcpp::pack::relocate::strip_search_paths( + std::filesystem::temp_directory_path() / "mcpp-relocate-test" / "nope.so"); + EXPECT_FALSE(r.has_value()); +} + +// ── the strip table ───────────────────────────────────────────────────── + +TEST(PackStrip, StaticArchiveKeepsItsSymbolIndex) { + // ⚠️ THE MEASUREMENT BEHIND THIS TEST. `strip --strip-all` on a `.a` + // removes the archive symbol index, and the consumer's link then fails + // with `archive has no index; run ranlib to add one` — a message that + // names neither strip nor the publisher. `--strip-debug` keeps it + // (measured: 2988 -> 1244 bytes, and the consumer links and runs). + auto args = mcpp::pack::strip_args(mcpp::pack::ArtifactShape::StaticArchive); + EXPECT_NE(std::ranges::find(args, "--strip-debug"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-all"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-unneeded"), args.end()); + // Two identical packs should produce identical bytes; this is the one + // place the packer rewrites an archive. + EXPECT_NE(std::ranges::find(args, "--enable-deterministic-archives"), args.end()); +} + +TEST(PackStrip, SharedLibraryKeepsItsDynamicSymbols) { + // `--strip-unneeded` removes `.symtab` and keeps `.dynsym`, which IS the + // export list. `--strip-all` would also keep `.dynsym`, but dh_strip uses + // the narrower flag and there is no reason to be broader than the + // reference implementation of this exact decision. + auto args = mcpp::pack::strip_args(mcpp::pack::ArtifactShape::SharedLibrary); + EXPECT_NE(std::ranges::find(args, "--strip-unneeded"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-all"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-debug"), args.end()); +} + +TEST(PackStrip, ExecutableIsStrippedWhole) { + auto args = mcpp::pack::strip_args(mcpp::pack::ArtifactShape::Executable); + EXPECT_NE(std::ranges::find(args, "--strip-all"), args.end()); +} + +TEST(PackStrip, EveryShapeDropsCommentAndNote) { + // dh_strip drops both from all three. `.note` is matched by exact name, so + // `.note.gnu.build-id` survives — which is what `--add-gnu-debuglink` and + // debuginfod pair with. + for (auto shape : { mcpp::pack::ArtifactShape::Executable, + mcpp::pack::ArtifactShape::SharedLibrary, + mcpp::pack::ArtifactShape::StaticArchive }) { + auto args = mcpp::pack::strip_args(shape); + EXPECT_NE(std::ranges::find(args, "--remove-section=.comment"), args.end()); + EXPECT_NE(std::ranges::find(args, "--remove-section=.note"), args.end()); + } +} + +TEST(PackStrip, MsvcHasNothingInBandToRemove) { + // PE/MSVC keeps debug information in a separate `.pdb`. An empty `strip` + // tool there is the right answer, not a missing dependency — and the two + // must not be conflated, or every other platform gets a silent no-op. + auto path = write_temp("not an image", "notanimage.bin"); + mcpp::pack::StripTools tools{ {}, {}, /*inBandDebugInfo=*/false }; + auto r = mcpp::pack::strip_artifact(path, mcpp::pack::ArtifactShape::SharedLibrary, + tools, {}); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::StripOutcome::NotApplicable); +} + +TEST(PackStrip, AMissingStripToolIsRefusedWhereDebugInfoIsInBand) { + auto path = write_temp("not an image", "notanimage2.bin"); + mcpp::pack::StripTools tools{ {}, {}, /*inBandDebugInfo=*/true }; + auto r = mcpp::pack::strip_artifact(path, mcpp::pack::ArtifactShape::SharedLibrary, + tools, {}); + ASSERT_FALSE(r.has_value()); + EXPECT_NE(r.error().find("no `strip`"), std::string::npos); +} From dac3902f730eddb7c15ed1b8e1f8cd7d7c77a9cb Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 07:58:02 +0800 Subject: [PATCH 2/8] fix(pack): whether stripping applies is a property of the TARGET, not the compiler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Keying `inBandDebugInfo` on `tc.compiler != MSVC` is wrong in both directions, and each direction is a configuration mcpp ships: clang -> x86_64-windows-msvc produces .pdb debug info, and would have been asked to strip in-band DWARF that is not there. Apple clang on macOS ships no llvm-strip, so the rule would REFUSE every `mcpp pack` on a Mac — for a format whose linked image carries a debug MAP (N_OSO stanzas naming the .o files) and leaves the DWARF outside it. `objcopy --only-keep-debug` there has nothing to copy, and .dSYM is dsymutil's job. `debug_info_is_in_band(canonicalTriple)` answers it segment-wise from the canonical triple, which both packers have already resolved. ELF and PE/MinGW are in-band; Mach-O and the MSVC ABI are not. --- src/pack/library_pipeline.cppm | 7 ++-- src/pack/pipeline.cppm | 12 ++++++- src/pack/strip.cppm | 58 ++++++++++++++++++++++++++----- tests/unit/test_pack_relocate.cpp | 23 ++++++++++++ 4 files changed, 86 insertions(+), 14 deletions(-) diff --git a/src/pack/library_pipeline.cppm b/src/pack/library_pipeline.cppm index 1521af0b..a7167e59 100644 --- a/src/pack/library_pipeline.cppm +++ b/src/pack/library_pipeline.cppm @@ -354,13 +354,12 @@ export int build_and_pack_library(const std::string& targetName, .importLibrary = importLib, // From THIS leg's toolchain, for the same reason `archiveTool` is: // a fat package's foreign leg must not be stripped by the host's - // tool. `inBandDebugInfo` is the one bit of "does this even apply" - // — PE/MSVC keeps debug information in a separate `.pdb`. + // tool. Whether stripping APPLIES is a question about the leg's + // TARGET, not about its compiler — see mcpp.pack.strip. .stripTools = mcpp::pack::StripTools{ .strip = mcpp::toolchain::binutils_tool(ctx->tc, "strip"), .objcopy = mcpp::toolchain::binutils_tool(ctx->tc, "objcopy"), - .inBandDebugInfo = - ctx->tc.compiler != mcpp::toolchain::CompilerId::MSVC, + .inBandDebugInfo = mcpp::pack::debug_info_is_in_band(triple), }, }); mcpp::ui::status("Packed leg", std::format("{} [{}]", triple, tag.str())); diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index 770143fd..bebbe762 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -21,6 +21,7 @@ import mcpp.pack; import mcpp.pack.strip; import mcpp.toolchain.model; import mcpp.toolchain.registry; +import mcpp.toolchain.triple; import mcpp.ui; namespace mcpp::pack { @@ -195,7 +196,16 @@ export int build_and_pack(Options opts, bool modeFromUser, plan->stripTools = mcpp::pack::StripTools{ .strip = mcpp::toolchain::binutils_tool(ctx->tc, "strip"), .objcopy = mcpp::toolchain::binutils_tool(ctx->tc, "objcopy"), - .inBandDebugInfo = ctx->tc.compiler != mcpp::toolchain::CompilerId::MSVC, + // The CANONICAL triple, resolved the same way the library packer + // resolves it: an empty `targetTriple` means "this host", and asking + // the empty string would answer "in-band" for macOS and MSVC alike. + .inBandDebugInfo = mcpp::pack::debug_info_is_in_band( + ctx->tc.targetTriple.empty() + ? mcpp::toolchain::triple::host_triple().str() + : [&] { + auto t = mcpp::toolchain::triple::parse(ctx->tc.targetTriple); + return t ? t->str() : ctx->tc.targetTriple; + }()), }; mcpp::ui::info("Packing", std::format("{} v{} ({}{})", diff --git a/src/pack/strip.cppm b/src/pack/strip.cppm index 866ff163..637ec2e7 100644 --- a/src/pack/strip.cppm +++ b/src/pack/strip.cppm @@ -39,14 +39,30 @@ // exact name, so the build-id survives and `--add-gnu-debuglink` still has // something to pair with. // -// EMPTY TOOL IS NOT ALWAYS AN ERROR +// EMPTY TOOL IS NOT ALWAYS AN ERROR, AND WHICH CASE IT IS BELONGS TO THE +// TARGET FORMAT — NOT TO THE COMPILER // -// PE/MSVC keeps debug information in a separate `.pdb` by design, so there is -// nothing in-band to remove and no binutils to remove it with. That is -// `NotApplicable`. Every other format carries DWARF inside the image, so a -// missing `strip` there is a REFUSAL — the same stance `run_library_pack` -// already takes for a missing archiver, and for the same reason: shipping the -// artifact anyway is the silent-wrong-answer this feature exists to remove. +// Two of the three formats keep debug information OUTSIDE the image, and for +// them an absent `strip` is the right answer rather than a missing dependency: +// +// PE/MSVC a separate `.pdb`, by design. +// Mach-O the linked image carries a DEBUG MAP (N_OSO stanzas naming the +// `.o` files); the DWARF itself never enters the `.dylib` unless +// `dsymutil` is run, and what it then produces is a `.dSYM` +// bundle beside the image. `strip` there would remove the symbol +// table — a different thing — and `objcopy --only-keep-debug` has +// nothing to copy. +// +// ELF and PE/MinGW carry DWARF inside the image, so a missing `strip` there is +// a REFUSAL — the same stance `run_library_pack` already takes for a missing +// archiver, and for the same reason: shipping the artifact anyway is the +// silent-wrong-answer this feature exists to remove. +// +// ⚠️ KEYING THIS ON THE COMPILER WOULD BE WRONG IN BOTH DIRECTIONS. clang +// targeting `x86_64-windows-msvc` produces `.pdb` debug info and would be +// asked to strip in-band; Apple's clang ships no `llvm-strip`, so a +// compiler-keyed rule would REFUSE every `mcpp pack` on macOS for a format that +// has nothing to strip in the first place. export module mcpp.pack.strip; @@ -68,11 +84,19 @@ struct StripTools { std::filesystem::path strip; std::filesystem::path objcopy; // Does this leg's format carry debug information inside the image? - // False only for PE/MSVC (`.pdb`). Resolved once by the caller so this - // module never has to know what a toolchain is. + // Ask `debug_info_is_in_band` rather than filling this by hand — see the + // header for the two formats where the answer is no, and for why the + // question is about the TARGET and not about the compiler. bool inBandDebugInfo = true; }; +// Is debug information carried INSIDE an image built for `canonicalTriple`? +// +// A string question about the canonical triple (`arch-os[-env]`), deliberately: +// this module knows nothing about toolchains, and both packers have already +// resolved that triple by the time they ask. +bool debug_info_is_in_band(std::string_view canonicalTriple); + enum class StripOutcome { Stripped, NotApplicable }; struct StripResult { @@ -131,6 +155,22 @@ std::uintmax_t size_of(const std::filesystem::path& p) { } // namespace +bool debug_info_is_in_band(std::string_view canonicalTriple) { + // `arch-os[-env]`. Splitting rather than substring-matching: an arch or a + // vendor segment could contain either of these words, and mcpp has been + // bitten before by a triple predicate that answered on a substring. + std::vector seg; + for (std::size_t i = 0; i <= canonicalTriple.size(); ) { + auto j = canonicalTriple.find('-', i); + if (j == std::string_view::npos) { seg.push_back(canonicalTriple.substr(i)); break; } + seg.push_back(canonicalTriple.substr(i, j - i)); + i = j + 1; + } + if (seg.size() >= 2 && seg[1] == "macos") return false; // debug map + .dSYM + if (seg.size() >= 3 && seg[2] == "msvc") return false; // separate .pdb + return true; +} + std::vector strip_args(ArtifactShape shape) { // dh_strip's own division. See the header for the measurement behind the // archive row — it is the one that turns a package into an unlinkable one. diff --git a/tests/unit/test_pack_relocate.cpp b/tests/unit/test_pack_relocate.cpp index c9b055b0..ddf76bc0 100644 --- a/tests/unit/test_pack_relocate.cpp +++ b/tests/unit/test_pack_relocate.cpp @@ -307,6 +307,29 @@ TEST(PackStrip, EveryShapeDropsCommentAndNote) { } } +TEST(PackStrip, WhetherStrippingAppliesIsAskedOfTheTargetNotTheCompiler) { + // Keying this on the compiler is wrong in BOTH directions, and each + // direction is a real configuration mcpp ships: + // + // clang -> x86_64-windows-msvc produces .pdb debug info; a + // compiler-keyed rule would try to strip + // in-band DWARF that is not there. + // Apple clang on macOS ships no `llvm-strip`, so a + // compiler-keyed rule would REFUSE every + // `mcpp pack` on a Mac — for a format + // whose image carries a debug MAP and + // leaves the DWARF in the .o files. + EXPECT_TRUE (mcpp::pack::debug_info_is_in_band("x86_64-linux-gnu")); + EXPECT_TRUE (mcpp::pack::debug_info_is_in_band("aarch64-linux-musl")); + EXPECT_TRUE (mcpp::pack::debug_info_is_in_band("x86_64-windows-gnu")); + EXPECT_FALSE(mcpp::pack::debug_info_is_in_band("x86_64-windows-msvc")); + EXPECT_FALSE(mcpp::pack::debug_info_is_in_band("aarch64-macos")); + EXPECT_FALSE(mcpp::pack::debug_info_is_in_band("x86_64-macos")); + // Segment-wise, not substring: mcpp has been bitten by a triple predicate + // that answered on a substring before. + EXPECT_TRUE(mcpp::pack::debug_info_is_in_band("macos64-linux-gnu")); +} + TEST(PackStrip, MsvcHasNothingInBandToRemove) { // PE/MSVC keeps debug information in a separate `.pdb`. An empty `strip` // tool there is the right answer, not a missing dependency — and the two From 54c2a520f59479b6521542cbad0465411537448d Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:00:26 +0800 Subject: [PATCH 3/8] docs(zh): the Mach-O refusal sits where its English counterpart does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit check_docs_style.sh enforces bilingual heading parity, and the zh section had been anchored on the wrong neighbour: it landed after 配置项 as an h3 where the English one is an h4 immediately following the Windows cross-packing note. Same place, same level — the two files are read side by side. --- docs/zh/02-pack-and-release.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/zh/02-pack-and-release.md b/docs/zh/02-pack-and-release.md index e2643f68..1de807c4 100644 --- a/docs/zh/02-pack-and-release.md +++ b/docs/zh/02-pack-and-release.md @@ -295,6 +295,20 @@ mcpp pack --target x86_64-windows-gnu # 在 Linux 宿主上 反方向 —— 在 Windows 上给 Linux / macOS 产物打包 —— 仍然不支持,原因还是最初 那个:那条闭包要由目标自己的动态链接器解析,而 Windows 宿主没有办法运行它。 +#### Mach-O 程序会被拒绝 —— 在所有宿主上,包括 macOS + +同一步闭包解析是靠 `LD_TRACE_LOADED_OBJECTS=1` **运行产物**来问动态链接器要 +依赖表的。这个变量属于 glibc 的 ld.so,dyld 从来不认(它的对应物是 +`DYLD_PRINT_LIBRARIES`)。所以在 Mac 上这条命令不会 trace 任何东西 —— +**它会把用户的程序跑起来**,然后把程序的输出当成依赖表解析。mcpp 现在直接拒绝, +并在信息里点名缺的是哪个机制。 + +判定按产物的**格式**而不是宿主,理由与 Windows 那条完全相同: +`LD_TRACE_LOADED_OBJECTS` 在 Linux 上也 trace 不了一个 Mach-O。 + +`kind = "lib"` / `"shared"` 目标在 macOS 上照常打包 —— 库打包从不运行产物。 +这条限制只针对程序。 + ## 配置项 打包行为通过 `mcpp.toml` 中的 `[pack]` 节配置,常用字段如下: @@ -320,20 +334,6 @@ force_bundle = ["libfoo.so"] # 即使命中 PEP 600 名单也强制打包 `static` 模式还需在 `[target.]` 中配置 musl 工具链,完整写法 参见 [`examples/03-pack-static`](../../examples/03-pack-static/) 的 `mcpp.toml`。 -### Mach-O 程序会被拒绝 —— 在所有宿主上,包括 macOS - -同一步闭包解析是靠 `LD_TRACE_LOADED_OBJECTS=1` **运行产物**来问动态链接器要 -依赖表的。这个变量属于 glibc 的 ld.so,dyld 从来不认(它的对应物是 -`DYLD_PRINT_LIBRARIES`)。所以在 Mac 上这条命令不会 trace 任何东西 —— -**它会把用户的程序跑起来**,然后把程序的输出当成依赖表解析。mcpp 现在直接拒绝, -并在信息里点名缺的是哪个机制。 - -判定按产物的**格式**而不是宿主,理由与 Windows 那条完全相同: -`LD_TRACE_LOADED_OBJECTS` 在 Linux 上也 trace 不了一个 Mach-O。 - -`kind = "lib"` / `"shared"` 目标在 macOS 上照常打包 —— 库打包从不运行产物。 -这条限制只针对程序。 - ## 待支持 macOS **程序** bundling(Mach-O 依赖闭包,走 `otool -L` / `LC_LOAD_DYLIB`, From f4de836c66e851f142c4d6ee11c58bfc60f29770 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:05:11 +0800 Subject: [PATCH 4/8] fix(pack): a fat package's debug files land per triple, and the empty-bundle relocate needs no patchelf MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects found reviewing my own diff. A fat package's legs SHARE an artifact name — `libmathkit-shared.so` for both the gnu and the musl leg is the documented normal case, not a corner one — so a flat `--debug-symbols` directory had the second leg overwrite the first, and the first artifact's .gnu_debuglink then resolved to the other target's symbols. Silently. The debug files now mirror `lib//`, and 265 asserts the layout rather than just the file's existence. And the empty-bundle case sat inside `if (!patchelf.empty())`, which is the exact shape the in-process editor exists to remove: `mcpp pack --mode system` on a host whose sandbox has no patchelf left the build machine's store in the artifact and said nothing at all. --- src/pack/library.cppm | 10 ++++- src/pack/pack.cppm | 41 ++++++++++--------- tests/e2e/265_pack_strips_but_stays_usable.sh | 15 +++++-- 3 files changed, 43 insertions(+), 23 deletions(-) diff --git a/src/pack/library.cppm b/src/pack/library.cppm index fdf13710..0acbef14 100644 --- a/src/pack/library.cppm +++ b/src/pack/library.cppm @@ -374,7 +374,15 @@ run_library_pack(const LibraryPackPlan& plan) if (plan.strip) { const auto shape = leg.shared ? mcpp::pack::ArtifactShape::SharedLibrary : mcpp::pack::ArtifactShape::StaticArchive; - auto r = mcpp::pack::strip_artifact(dst, shape, leg.stripTools, plan.debugDir); + // PER LEG, mirroring `lib//`. A fat package's legs share an + // artifact NAME — `libmathkit-shared.so` for both the gnu and the + // musl leg is the normal case, not a corner one — so a flat debug + // directory would have the second leg overwrite the first, and the + // first artifact's `.gnu_debuglink` would then resolve to the other + // target's symbols. Silently. + const auto legDebugDir = plan.debugDir.empty() + ? std::filesystem::path{} : plan.debugDir / leg.triple; + auto r = mcpp::pack::strip_artifact(dst, shape, leg.stripTools, legDebugDir); if (!r) return std::unexpected(LibraryPackError{ r.error() }); if (r->outcome == mcpp::pack::StripOutcome::Stripped) { mcpp::ui::status("Stripped", std::format("{} {} → {} bytes{}", diff --git a/src/pack/pack.cppm b/src/pack/pack.cppm index d138310b..82c4e864 100644 --- a/src/pack/pack.cppm +++ b/src/pack/pack.cppm @@ -1140,29 +1140,32 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) if (auto r = bundle_libs(toBundle, plan.stagingRoot); !r) return std::unexpected(Error{r.error()}); + // Search path: point at bundled libs, or REMOVE THE TAG if there are none. + // + // The empty case used to be `patchelf --set-rpath ''`, which leaves the + // tag present with an empty string — and a present-but-empty DT_RUNPATH + // is not inert: it suppresses the inherited DT_RPATH chain exactly like + // a stale one does (measured — see mcpp.pack.relocate). Harmless on an + // executable, which is the top of that chain, but there is no reason to + // write a tag that says nothing. + // + // OUTSIDE the patchelf guard, because it no longer needs patchelf. That + // is the whole point of the in-process editor: `--mode system` on a host + // where the sandbox has no patchelf used to leave the build machine's + // store in the artifact and say nothing at all. + if (toBundle.empty()) { + if (auto r = mcpp::pack::relocate::strip_search_paths(bundledBinary); !r) + return std::unexpected(Error{r.error()}); + } + auto patchelf = sandbox_patchelf(cfg); if (!patchelf.empty()) { - // Search path: point at bundled libs (or clear if none). // non-empty bundle → "$ORIGIN/../lib" so the binary finds them - // empty bundle → clear the original dev-sandbox RUNPATH - // (~/.mcpp/registry/... doesn't exist on - // a user's target machine) - // An EMPTY bundle gets the tag REMOVED, not set to "". - // - // `patchelf --set-rpath ''` leaves the tag present with an empty - // string, and a present-but-empty DT_RUNPATH is not inert: it - // suppresses the inherited DT_RPATH chain exactly like a stale one - // does (measured — see mcpp.pack.relocate). Harmless on an - // executable, which is the top of that chain, but there is no - // reason to write a tag that says nothing, and the library packer - // needs the removal path anyway. - if (toBundle.empty()) { - if (auto r = mcpp::pack::relocate::strip_search_paths(bundledBinary); !r) + if (!toBundle.empty()) { + if (auto r = set_search_path(bundledBinary, "$ORIGIN/../lib", + mcpp::build::loader::Form::Executable, + patchelf); !r) return std::unexpected(Error{r.error()}); - } else if (auto r = set_search_path(bundledBinary, "$ORIGIN/../lib", - mcpp::build::loader::Form::Executable, - patchelf); !r) { - return std::unexpected(Error{r.error()}); } // EVERY BUNDLED LIBRARY, not just the executable. diff --git a/tests/e2e/265_pack_strips_but_stays_usable.sh b/tests/e2e/265_pack_strips_but_stays_usable.sh index 61cf5db7..926e13c9 100755 --- a/tests/e2e/265_pack_strips_but_stays_usable.sh +++ b/tests/e2e/265_pack_strips_but_stays_usable.sh @@ -146,9 +146,18 @@ echo " [pack] strip = false: honoured" rm -rf target/dist "$MCPP" pack mathkit-shared --debug-symbols dbg > pack-e.log 2>&1 || { cat pack-e.log; echo "FAIL: --debug-symbols pack"; exit 1; } -[[ -f dbg/libmathkit-shared.so.debug ]] || { - ls -R dbg 2>&1; echo "FAIL: no separated debug file"; exit 1; } -[[ "$(has_symtab dbg/libmathkit-shared.so.debug)" == "yes" ]] || { +# PER TRIPLE, mirroring `lib//`. A fat package's legs share an artifact +# NAME (`libmathkit-shared.so` for both a gnu and a musl leg is the normal case), +# so a flat debug directory would have the second leg overwrite the first — and +# the first artifact's `.gnu_debuglink` would then resolve to the other target's +# symbols, silently. Asserting the layout is what keeps that from regressing. +dbgfile="$(find dbg -name 'libmathkit-shared.so.debug' | head -1)" +[[ -n "$dbgfile" ]] || { ls -R dbg 2>&1; echo "FAIL: no separated debug file"; exit 1; } +[[ "$dbgfile" == dbg/*/libmathkit-shared.so.debug ]] || { + echo "FAIL: the debug file is not under a per-triple directory: $dbgfile" + echo " A fat package's legs share an artifact name; a flat layout loses one." + exit 1; } +[[ "$(has_symtab "$dbgfile")" == "yes" ]] || { echo "FAIL: the separated debug file carries no debug information"; exit 1; } pkg_e="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" so_e="$(find "$pkg_e/lib" -name 'libmathkit-shared.so' -type f | head -1)" From dee080e0d659ba4a752d78bf837a8cbb655e4607 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:13:33 +0800 Subject: [PATCH 5/8] test(e2e): 240 states its profile, because pack no longer builds where a bare build does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The PE cross-pack test drops a stand-in msvcrt.dll into the build tree and then expects `mcpp pack` to find it in the closure. With packaging defaulting to release, the two commands resolve to different profiles and therefore different target/// directories — the file lands in the one pack does not use, and the assertion reads as 'the closure reader failed' when nothing about the closure is wrong. The fixture now states `[build] default-profile`, which settles it for both and doubles as a check that the manifest still outranks pack's fallback. The user-visible half of the same fact is now in docs/02 and the changelog: a file placed beside a built artifact by hand is only visible to pack when both commands resolve to the same profile. The declarative channels are unaffected. --- CHANGELOG.md | 11 ++++++++++- docs/02-pack-and-release.md | 8 ++++++++ docs/zh/02-pack-and-release.md | 7 +++++++ tests/e2e/240_pack_pe_zip_cross.sh | 15 +++++++++++++++ 4 files changed, 40 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c68d34b..7c0c1b7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -74,7 +74,16 @@ 新增 `--profile` / `--no-strip` / `--debug-symbols ` 与 `[pack] strip`、 `[pack] debug_symbols`。`--debug-symbols` 是分离而不是丢弃:写出 - `/<产物>.debug` 并给发货产物加 `.gnu_debuglink`。 + `//<产物>.debug` 并给发货产物加 `.gnu_debuglink`——**按 triple 分目录**, + 因为 fat 包的各条 leg 产物同名(`gnu` 与 `musl` 两条腿都叫 + `libmathkit-shared.so` 是常态),扁平布局会让后一条覆盖前一条,而前一个产物的 + `.gnu_debuglink` 会静默指向另一个目标的符号。 + + ⚠️ **一条要知道的后果**:裸 `mcpp build` 与裸 `mcpp pack` 现在写进**不同的** + `target///` 目录(指纹把 profile 算进去了)。手工放到构建 + 产物旁边的文件只在两条命令解析到同一个 profile 时才被 `pack` 看见;声明式通道 + (`[runtime] deploy_files`、`runtime_search_dirs`)不受影响。e2e 240 因此在 + fixture 里显式写了 `[build] default-profile`。 > `[pack] strip` 与 `[profile.].strip` 是两个决定:后者给**链接**加 `-s` > (碰不到静态归档,也分离不出任何东西),前者管**包里带什么**。 diff --git a/docs/02-pack-and-release.md b/docs/02-pack-and-release.md index caf8166b..23a1f37b 100644 --- a/docs/02-pack-and-release.md +++ b/docs/02-pack-and-release.md @@ -154,6 +154,14 @@ fallback. Only the last step differs, so a project that states a profile still gets the one it stated, and `mcpp pack` never produces an artifact built with flags `mcpp build` would not. +> One consequence to know: a bare `mcpp build` and a bare `mcpp pack` now write +> into **different** `target///` directories, because the +> fingerprint covers the profile. A file placed beside a built artifact by hand +> — a DLL, a data blob — is therefore only visible to `pack` when both commands +> resolve to the same profile: state it in `[build] default-profile`, or pass +> `--profile` to both. The declarative channels (`[runtime] deploy_files`, +> `runtime_search_dirs`) are unaffected. + **Debug information is stripped, and the publisher's paths go with it.** An unstripped artifact carries DWARF, and DWARF carries the absolute paths of the producer's source tree and build directory. What is removed depends on what the diff --git a/docs/zh/02-pack-and-release.md b/docs/zh/02-pack-and-release.md index 1de807c4..d3f982f6 100644 --- a/docs/zh/02-pack-and-release.md +++ b/docs/zh/02-pack-and-release.md @@ -118,6 +118,13 @@ mcpp pack --debug-symbols dbg/ # 把分离出的 *.debug 写到 dbg/ profile 的工程仍然拿到它声明的那个,`mcpp pack` 也不会产出一个 `mcpp build` 产不出来的 flag 组合。 +> 有一条后果要知道:裸 `mcpp build` 与裸 `mcpp pack` 现在会写进**不同的** +> `target///` 目录 —— 指纹把 profile 算进去了。手工放到 +> 构建产物旁边的文件(一个 DLL、一份数据)因此只在两条命令解析到同一个 +> profile 时才被 `pack` 看见:把它写进 `[build] default-profile`,或者两条命令 +> 都带 `--profile`。声明式通道(`[runtime] deploy_files`、 +> `runtime_search_dirs`)不受影响。 + **调试信息会被剥掉,发布者的路径随之消失。** 未 strip 的产物带着 DWARF,而 DWARF 带着发布者源码树与构建目录的绝对路径。剥什么取决于产物**是什么** —— 这是 dh_strip 的分档,而其中归档那一行是要命的: diff --git a/tests/e2e/240_pack_pe_zip_cross.sh b/tests/e2e/240_pack_pe_zip_cross.sh index 17684c96..911be28f 100755 --- a/tests/e2e/240_pack_pe_zip_cross.sh +++ b/tests/e2e/240_pack_pe_zip_cross.sh @@ -39,6 +39,21 @@ cat > mcpp.toml <<'EOF' name = "winpack" version = "0.1.0" +# ⚠️ THIS TEST DROPS A FILE INTO THE BUILD TREE AND EXPECTS `pack` TO SEE IT, +# so the two commands have to agree on WHICH build tree that is. +# +# `mcpp pack` builds with the `release` fallback (a packaged artifact leaves +# this machine), while a bare `mcpp build` uses `dev` — two profiles, two +# fingerprint directories, and the stand-in DLL below would land in the one +# `pack` does not use. Nothing about the closure would be wrong; the file would +# simply not be there, and the assertion would read as "the closure reader +# failed". +# +# Stating the profile in the manifest settles it for both, and doubles as a +# check that `[build] default-profile` still outranks pack's fallback. +[build] +default-profile = "dev" + # `force_bundle` reaching a SYSTEM name is what makes the next assertion # positive rather than vacuous — see the comment at the msvcrt.dll check. [pack.bundle-project] From e7082c46a1ab6e862f869584ab9c4833fac99775 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:19:55 +0800 Subject: [PATCH 6/8] ci: a runner's DNS hiccup is not a red build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured on a macOS ARM64 runner, in a step that had not yet executed a line of mcpp: fatal: unable to access 'https://github.com/openxlings/xlings/': Could not resolve host: github.com Thirty seconds of resolver timeout, then a failed PR check that says nothing about the change under test. Every workflow that reaches an external repository did it with a bare `git clone` — six call sites, one failure mode — while this repository had already learned the same lesson one protocol over: fetch_release.sh carries a long note on `curl --retry` not covering transport-layer errors and `--retry-all-errors` being the flag that does. git has no such flag. .github/tools/git_clone_retry.sh retries EVERY failure, bounded. It does not try to tell a DNS blip from a missing repository: git reports both as exit 128 with only the message to distinguish them, and parsing that message would be a weaker copy of git's own taxonomy that breaks the first time git rephrases one. A genuinely missing repo costs the attempts and then fails with git's own last message intact (measured: 4s, exit 128); a resolver blip costs one backoff. ⚠️ The helper's first draft exited 0 on a clone that never succeeded — `$?` after an `if` is the status of the IF STATEMENT, which is 0 when the body did not run. That turns a hard failure into a green step with a missing checkout. Caught by asserting the exit code in its own test rather than by reading the output, which looked correct. ci-aarch64-fresh-install spells the retry inline instead: it checks the repository out LAST ON PURPOSE (a .xlings.json in the workspace re-points where `xlings install` writes, so an early checkout silently changes what the fresh-install steps are testing), so the shared helper does not exist on disk at those two clones. Same policy, and the comment says why it is not the same file. Verified: real clone exits 0 with content; bad repo exits 128 bounded; unreachable host retries then fails with git's message; a partial destination is cleared before the retry; the inline form survives `set -e`. --- .github/tools/git_clone_retry.sh | 77 +++++++++++++++++++ .../workflows/ci-aarch64-fresh-install.yml | 37 ++++++++- .github/workflows/ci-linux.yml | 3 +- .github/workflows/ci-macos.yml | 7 +- .github/workflows/ci-windows.yml | 3 +- .github/workflows/cross-build-test.yml | 5 +- 6 files changed, 126 insertions(+), 6 deletions(-) create mode 100755 .github/tools/git_clone_retry.sh diff --git a/.github/tools/git_clone_retry.sh b/.github/tools/git_clone_retry.sh new file mode 100755 index 00000000..a5c19060 --- /dev/null +++ b/.github/tools/git_clone_retry.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# git_clone_retry.sh — `git clone`, but a hosted runner's DNS hiccup is not a +# red build. +# +# WHY THIS EXISTS +# +# Measured on a macOS ARM64 runner, in a step that had not yet executed a single +# line of mcpp: +# +# Cloning into '/tmp/xlings-src'... +# fatal: unable to access 'https://github.com/openxlings/xlings/': +# Could not resolve host: github.com +# ##[error]Process completed with exit code 128 +# +# Thirty seconds of resolver timeout, then a failed PR check that says nothing +# about the change under test. Every workflow in this repository that reaches an +# external repository did it with a bare `git clone`, so every one of them could +# fail this way — six call sites, one failure mode. +# +# THIS REPOSITORY ALREADY LEARNED THIS FOR `curl` +# +# `fetch_release.sh` carries the same lesson one protocol over: `curl --retry` +# alone does not cover a transport-layer error, and `--retry-all-errors` is what +# does. `git` has no such flag at all — the retry has to be here. +# +# WHY BOUNDED, AND WHY IT DOES NOT TRY TO BE CLEVER +# +# git reports "the host does not resolve", "the repository does not exist" and +# "the credentials are wrong" as the same exit 128, with only the message to +# tell them apart. Parsing that message would be a second, weaker copy of git's +# own error taxonomy — and it would be wrong the first time git rephrased one. +# +# So this retries EVERY failure, a bounded number of times. A genuinely missing +# repository costs the attempts and then fails with git's own last message +# intact, which is ~50s and a diagnostic that still names the real cause. A +# resolver blip costs one backoff. That trade is the same one `--retry-all-errors` +# makes, and it is the right way round: a false red costs a maintainer a rerun +# and their attention, a slow true red costs 50 seconds. +# +# Usage: git_clone_retry.sh [git clone args...] +# Env: GIT_CLONE_ATTEMPTS (default 4), GIT_CLONE_BACKOFF (default "5 15 30") +set -euo pipefail + +ATTEMPTS="${GIT_CLONE_ATTEMPTS:-4}" +read -r -a BACKOFF <<< "${GIT_CLONE_BACKOFF:-5 15 30}" + +# The destination is the last argument when it is not an option. Removing a +# partial clone before retrying matters: git refuses to clone into a directory +# that exists and is not empty, so without this the second attempt fails for a +# different reason than the first and the log stops making sense. +dest="${*: -1}" + +for (( i = 1; i <= ATTEMPTS; i++ )); do + # `cmd || rc=$?`, NOT `if cmd; then …; fi; rc=$?`. + # + # After an `if`, `$?` is the status of the IF STATEMENT — which is 0 when + # the body did not run. Written that way this script exits 0 on a clone that + # never succeeded, i.e. it turns a hard failure into a green step with a + # missing checkout. Caught by asserting the exit code in the helper's own + # test rather than by reading the output, which looked correct. + rc=0 + git clone "$@" || rc=$? + if (( rc == 0 )); then + exit 0 + fi + if (( i == ATTEMPTS )); then + echo "git_clone_retry: giving up after $ATTEMPTS attempts (last exit $rc)" >&2 + exit "$rc" + fi + if [[ -n "$dest" && "$dest" != -* && -e "$dest" ]]; then + echo "git_clone_retry: removing partial '$dest' before retrying" >&2 + rm -rf -- "$dest" + fi + delay="${BACKOFF[$(( i - 1 < ${#BACKOFF[@]} ? i - 1 : ${#BACKOFF[@]} - 1 ))]}" + echo "git_clone_retry: attempt $i/$ATTEMPTS failed (exit $rc); retrying in ${delay}s" >&2 + sleep "$delay" +done diff --git a/.github/workflows/ci-aarch64-fresh-install.yml b/.github/workflows/ci-aarch64-fresh-install.yml index 12ccf073..b451203e 100644 --- a/.github/workflows/ci-aarch64-fresh-install.yml +++ b/.github/workflows/ci-aarch64-fresh-install.yml @@ -79,7 +79,26 @@ jobs: mcpp index update || true idx="$HOME/.mcpp/registry/data/xim-pkgindex" rm -rf "$idx" - git clone --depth 1 https://github.com/openxlings/xim-pkgindex "$idx" + # A bare `git clone` here fails the whole job on a runner DNS + # hiccup. The policy lives in .github/tools/git_clone_retry.sh — + # but this job checks the repository out LAST ON PURPOSE (a + # `.xlings.json` in the workspace re-points where `xlings install` + # writes, so an early checkout silently changes what the + # fresh-install steps above are testing). The helper therefore does + # not exist on disk yet, and the retry is spelled inline. Same + # policy: retry every failure, bounded, git's own message survives. + clone_retry() { + local rc dest="${@: -1}" + for i in 1 2 3 4; do + rc=0; git clone "$@" || rc=$? + [ "$rc" = 0 ] && return 0 + [ "$i" = 4 ] && return "$rc" + [ -e "$dest" ] && rm -rf -- "$dest" + echo "clone_retry: attempt $i failed (exit $rc); retrying" >&2 + sleep $((i * 5)) + done + } + clone_retry --depth 1 https://github.com/openxlings/xim-pkgindex "$idx" grep -n "skipping relocation\|os.isfile(path.join(bindir" "$idx/pkgs/m/musl-gcc.lua" | head -2 || true - name: Native build + run an `import std` program @@ -156,7 +175,21 @@ jobs: export MCPP_HOME=$(mcpp self env | awk -F'= *' '/^MCPP_HOME/{print $2; exit}') echo "reusing MCPP_HOME=$MCPP_HOME" test -d "$MCPP_HOME" || { echo "could not determine MCPP_HOME"; exit 1; } - git clone --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src + # Same inline retry as the index clone above, and for the same + # reason — the checkout that would provide the shared helper is + # deliberately the last step in this job. + clone_retry() { + local rc dest="${@: -1}" + for i in 1 2 3 4; do + rc=0; git clone "$@" || rc=$? + [ "$rc" = 0 ] && return 0 + [ "$i" = 4 ] && return "$rc" + [ -e "$dest" ] && rm -rf -- "$dest" + echo "clone_retry: attempt $i failed (exit $rc); retrying" >&2 + sleep $((i * 5)) + done + } + clone_retry --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src # "$m", not `mcpp`: the just-built binary is the code under review, # and building xlings with the INSTALLED one meant this half of the diff --git a/.github/workflows/ci-linux.yml b/.github/workflows/ci-linux.yml index 8decb8ef..6a853c4c 100644 --- a/.github/workflows/ci-linux.yml +++ b/.github/workflows/ci-linux.yml @@ -169,7 +169,8 @@ jobs: - name: "Integration: mcpp builds & runs xlings (openxlings/xlings)" run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - git clone --depth 1 --recurse-submodules \ + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP" self config --mirror GLOBAL diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index 4290bbb7..a61ff9a4 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -310,7 +310,12 @@ jobs: run: | MCPP=/tmp/mcpp-fresh # the freshly self-hosted binary built from this PR export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - git clone --depth 1 --recurse-submodules \ + # ⚠️ THIS IS THE STEP THE RETRY WAS MEASURED ON: a macOS runner + # answered `Could not resolve host: github.com` after 30s of + # resolver timeout, failing a PR check before a single line of + # mcpp had run. See .github/tools/git_clone_retry.sh. + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP" self config --mirror GLOBAL diff --git a/.github/workflows/ci-windows.yml b/.github/workflows/ci-windows.yml index a1baeb52..a96089fa 100644 --- a/.github/workflows/ci-windows.yml +++ b/.github/workflows/ci-windows.yml @@ -254,7 +254,8 @@ jobs: XLINGS_NON_INTERACTIVE: '1' run: | export MCPP_VENDORED_XLINGS=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe") - git clone --depth 1 --recurse-submodules \ + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP_SELF" self config --mirror GLOBAL diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 1be786e9..8c2e1517 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -173,7 +173,10 @@ jobs: - name: "Cross-build xlings -> ${{ matrix.target }}" run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - git clone --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src + # A hosted runner's DNS hiccup is not a red build — see + # .github/tools/git_clone_retry.sh for the measurement. + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP" self config --mirror GLOBAL 2>/dev/null || true "$MCPP" build --target ${{ matrix.target }} From 3cf975561a6566b8733b99206a7639aa90ce1d08 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:21:22 +0800 Subject: [PATCH 7/8] test(e2e): 265's consumer manifest goes through a named *_HOST variable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 00_fixture_path_hygiene refuses an inline `$(host_path …)` in a manifest heredoc, and it is right to: the manifest is FILE CONTENT, and on Git Bash a shell-spelled /tmp/... path is read by a native mcpp.exe as 'root of the current drive'. Naming the converted value is what makes the conversion visible where it is used rather than buried in an interpolation. --- tests/e2e/265_pack_strips_but_stays_usable.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/e2e/265_pack_strips_but_stays_usable.sh b/tests/e2e/265_pack_strips_but_stays_usable.sh index 926e13c9..be5351ad 100755 --- a/tests/e2e/265_pack_strips_but_stays_usable.sh +++ b/tests/e2e/265_pack_strips_but_stays_usable.sh @@ -74,6 +74,12 @@ PY consume() { # — build a program against it and run local pkg="$1" want="$2" tag="$3" + # Through a named *_HOST variable, not interpolated inline: the manifest + # below is FILE CONTENT, and on Git Bash a shell-spelled /tmp/... path is + # read by a native mcpp.exe as "root of the current drive". 00_fixture_path + # _hygiene enforces the naming so the conversion is visible at the use site. + local PKG_HOST + PKG_HOST="$(host_path "$pkg")" rm -rf "$TMP/consumer" mkdir -p "$TMP/consumer/src" cat > "$TMP/consumer/src/main.cpp" <<'EOF' @@ -86,7 +92,7 @@ EOF name = "consumer" version = "0.1.0" [dependencies] -mathkit = { path = "$(host_path "$pkg")" } +mathkit = { path = "$PKG_HOST" } [targets.consumer] kind = "bin" main = "src/main.cpp" From bf9e88c357118bfe3dbcecbf70bd127116c53b77 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:35:25 +0800 Subject: [PATCH 8/8] test(e2e): 249/250 check their routing on Mach-O too, and the refusal names the artifact MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both files are about ROUTING — a workspace root hands through to its member's program; `mcpp pack ` reaches the target it was given — and both proved it by looking inside a produced bundle. On macOS that bundle can no longer be produced, so the macOS e2e job went red on two tests that are not about bundling at all. Worth noting what their previous green meant: they passed on macOS while `mcpp pack` was resolving the dependency closure by RUNNING the user's program. The bundle they inspected was the one that produced. The Mach-O refusal now names the artifact it reached. That is not decoration: a refusal that does not say WHICH program it got to is indistinguishable from one that resolved the wrong target — the exact defect route_pack_target exists to prevent — and it is the only evidence available on a platform where no bundle can be inspected. 250 additionally keeps the unknown-name case, which must still fail EARLIER and differently, or 'refuses everything' would pass. Linux paths unchanged and re-verified; the new string is present in the rebuilt binary. --- src/pack/pack.cppm | 11 +++++-- .../e2e/249_pack_workspace_root_unchanged.sh | 26 +++++++++++++++ tests/e2e/250_pack_names_the_target.sh | 32 +++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/src/pack/pack.cppm b/src/pack/pack.cppm index 82c4e864..1ce351a7 100644 --- a/src/pack/pack.cppm +++ b/src/pack/pack.cppm @@ -1031,10 +1031,17 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) // docs/02 lists macOS bundling under "Planned Support"; until it lands, // saying so is strictly better than producing an empty bundle that claims // to be one. + // + // NAMES THE ARTIFACT. Not decoration: `mcpp pack ` routes on the + // target's kind, and a refusal that does not say WHICH program it got to is + // indistinguishable from one that resolved the wrong target — which is the + // exact defect `route_pack_target` exists to prevent. It is also the only + // way an e2e can check that routing on macOS, where no program bundle can + // be produced to inspect. if (mcpp::pack::binfmt::identify(plan.builtBinary).format == mcpp::pack::binfmt::Format::MachO) { - return std::unexpected(Error{ - "cannot package a Mach-O program yet.\n" + return std::unexpected(Error{std::format( + "cannot package the Mach-O program '{}' yet.\n", plan.binaryName) + " The dependency closure for that format is resolved by running the " "artifact under\n" " the target's own dynamic linker, and the mechanism mcpp uses " diff --git a/tests/e2e/249_pack_workspace_root_unchanged.sh b/tests/e2e/249_pack_workspace_root_unchanged.sh index 6eb9447f..bc8f3cd1 100755 --- a/tests/e2e/249_pack_workspace_root_unchanged.sh +++ b/tests/e2e/249_pack_workspace_root_unchanged.sh @@ -52,6 +52,32 @@ int main() { std::printf("ok=%d\n", core_answer()); return 0; } EOF cd ws + +# ── Mach-O: the routing is the claim, and a bundle cannot be produced ── +# +# `mcpp pack` of a PROGRAM is refused on Mach-O (its dependency closure would +# be resolved by RUNNING the artifact under a linker that ignores the tracing +# variable — see 266). So on macOS the question this file asks becomes: did the +# workspace root still hand through to the MEMBER'S PROGRAM? The refusal names +# the artifact, which is exactly the evidence for that — and it is a different +# message from "this package declares no program and no library to pack", which +# is what a routing regression would produce. +if [[ "$(uname -s)" == "Darwin" ]]; then + if "$MCPP" pack --mode system > pack.log 2>&1; then + cat pack.log + echo "FAIL: a Mach-O program bundle was produced; 266 says it must be refused" + exit 1 + fi + grep -q "Mach-O program 'hello'" pack.log || { + cat pack.log + echo "FAIL: the workspace root did not route to the member's program 'hello'." + echo " (A refusal naming some other artifact, or a 'declares no program'" + echo " error, is the routing regression this file exists to catch.)" + exit 1; } + echo "PASS: a workspace root still routes to its member's program" + exit 0 +fi + "$MCPP" pack --mode system > pack.log 2>&1 || { cat pack.log echo "FAIL: packing from a virtual workspace root stopped working" diff --git a/tests/e2e/250_pack_names_the_target.sh b/tests/e2e/250_pack_names_the_target.sh index 9fa079db..a35b7236 100755 --- a/tests/e2e/250_pack_names_the_target.sh +++ b/tests/e2e/250_pack_names_the_target.sh @@ -42,6 +42,38 @@ EOF cd two +# ── Mach-O: which target was CHOSEN is still checkable ───────────────── +# +# A program bundle cannot be produced on Mach-O (266), so "look inside the +# staging dir" is unavailable here. The refusal names the artifact it got to, +# which answers the same question: `pack beta` must reach beta and `pack alpha` +# must reach alpha. Without this branch the file would simply not run on macOS, +# and the defect it exists for — `mcpp pack app2` bundling app1 — is not a +# platform-specific one. +if [[ "$(uname -s)" == "Darwin" ]]; then + for want in beta alpha; do + other=$([[ "$want" == beta ]] && echo alpha || echo beta) + if "$MCPP" pack "$want" --mode system > "$want.log" 2>&1; then + cat "$want.log"; echo "FAIL: a Mach-O program bundle was produced"; exit 1 + fi + grep -q "Mach-O program '$want'" "$want.log" || { + cat "$want.log" + echo "FAIL: 'mcpp pack $want' did not reach $want"; exit 1; } + grep -q "Mach-O program '$other'" "$want.log" && { + cat "$want.log" + echo "FAIL: 'mcpp pack $want' reached $other instead"; exit 1; } + done + # An unknown name must still fail EARLIER and differently — otherwise the + # two assertions above would pass for a build that refuses everything. + if "$MCPP" pack nosuch --mode system > bad.log 2>&1; then + cat bad.log; echo "FAIL: an unknown target name was accepted"; exit 1 + fi + grep -q "no target named 'nosuch'" bad.log || { + cat bad.log; echo "wrong refusal for an unknown name"; exit 1; } + echo "PASS: mcpp pack reaches the target it is given" + exit 0 +fi + # ── the named program is the one that gets bundled ───────────────────── "$MCPP" pack beta --mode system > beta.log 2>&1 || { cat beta.log; echo "pack beta failed"; exit 1; } staged="$(find target/dist -maxdepth 1 -type d -name 'two-0.1.0*' | head -1)"