diff --git a/.agents/docs/2026-08-20-issue460-shared-library-runpath.md b/.agents/docs/2026-08-20-issue460-shared-library-runpath.md new file mode 100644 index 00000000..2e7d1881 --- /dev/null +++ b/.agents/docs/2026-08-20-issue460-shared-library-runpath.md @@ -0,0 +1,605 @@ +# `mcpp pack` 的 `kind = "shared"` 产物带走了构建机:#460 的实测、根因与优化方案 + +> 2026-08-20 · issue #460 · 复现于本仓库 HEAD 构建出的 `mcpp 2026.8.19.4`(与报告的 2026.8.18.3 同形) +> 状态:**P0-1 / P0-2 / P0-3 + P2-1 已实施,发布于 2026.8.20.1**;P1-1 / P1-3 已批准未实施 + +--- + +## 0. 一句话结论 + +报告说的现象是对的,但**它给出的期望行为有一半是错的**。 + +> 期望:打包出的 `.so` 应该是可重定位的——没有 `RUNPATH`(**或为 `$ORIGIN`**) + +实测:`$ORIGIN` 修不好这个 bug,空字符串也修不好。因为在 ELF 装载器里, +**DT_RUNPATH 只要*存在*,消费方可执行文件的 DT_RPATH 就不再被继承**——不管 +DT_RUNPATH 里写的是什么。库自己的 `libstdc++.so.6` 从此无解。 + +所以判据不是「路径是可重定位的」,而是「**这条 tag 不存在**」。 + +| 打包出的 `.so` 上的状态 | 消费方 exe 的 DT_RPATH 是否被继承 | 另一台机器上的结果 | +|---|---|---| +| `DT_RUNPATH = <构建机 store 路径>`(**今天的行为**) | 否 | `libstdc++.so.6: cannot open shared object file`,rc=127 | +| **无 tag**(`patchelf --remove-rpath`) | **是** | **`ok=42`** | +| `DT_RUNPATH = $ORIGIN`(报告建议的另一半) | 否 | 同样 rc=127 | +| `DT_RUNPATH = ""`(`--set-rpath ''`,最容易写出来的「修复」) | 否 | 同样 rc=127 | +| `DT_RPATH = <构建机 store 路径>`(`--force-rpath`) | 是 | `ok=42`(但违反 loader 契约,见 §5.1) | + +四行都是在**真实的 `mcpp pack` 产物 + 真实的 mcpp 消费方**上跑出来的,复现命令见附录 A。 + +--- + +## 1. 现状:两个 packer,一条契约,只有一个实现了它 + +`mcpp pack` 有两条互不相干的实现路径,由 `[targets.].kind` 分流 +(`src/cli/cmd_publish.cppm::cmd_pack` → `mcpp::pack::route_pack_target`): + +| | 应用打包 | **库打包** | +|---|---|---| +| 入口 | `mcpp::pack::build_and_pack` (`src/pack/pipeline.cppm`) | `mcpp::pack::build_and_pack_library` (`src/pack/library_pipeline.cppm:111`) | +| 落地 | `run_pack` (`src/pack/pack.cppm:~954`) | `run_library_pack` (`src/pack/library.cppm:197`) | +| 问「运行时需要什么」 | 是(`ldd` 闭包 + 捆绑) | 否(问的是「消费方要编什么、能链什么」) | +| **ELF 重定位** | **有**(§1.1) | **无**(§1.2) | +| 宿主/格式拒绝 | 有(`_WIN32` 上产 ELF 直接拒绝并给理由) | 无 | + +两个文件都写了很长的注释解释自己在小心什么,但**「产物不能带走构建机」这一条只写在 +应用侧**。 + +### 1.1 应用侧已经做对了,而且写了理由 + +`src/pack/pack.cppm:1028-1096`: + +```cpp +auto patchelf = sandbox_patchelf(cfg); +if (!patchelf.empty()) { + const char* rpath = toBundle.empty() ? "" : "$ORIGIN/../lib"; + set_search_path(bundledBinary, rpath, loader::Form::Executable, patchelf); + + // EVERY BUNDLED LIBRARY, not just the executable. + // ... + // /xim-x-glibc/2.44/lib64 : /xim-x-gcc/16.1.0/lib64 + // : /compat-x-glx-runtime/…/lib : $ORIGIN + // Those directories do not exist on the target ... + for (auto const& dep : toBundle) { ... set_search_path(staged, "$ORIGIN", ...); } +``` + +注释里那段 store 路径,和 #460 贴出来的 `readelf -d` 输出**是同一段**。也就是说: +这个缺陷的形状在 2026-08-11 就已经被完整地描述过一次,并且在应用侧修好了。 + +### 1.2 库侧一行 ELF 处理都没有 + +`src/pack/library.cppm:247-337`,每条 leg 做的事: + +```cpp +auto dst = plan.stagingRoot / "lib" / leg.triple / name; +copy_into(leg.artifact, dst); // 255 —— 只是 copy_file +... +if (leg.shared && !leg.soname.empty()) { create_symlink(name, alias); } // 287-294 +... +.digest = file_digest(dst), // 333 +``` + +全文 grep 佐证: + +``` +$ grep -n "patchelf\|set_search_path\|set_rpath" src/pack/library.cppm src/pack/library_pipeline.cppm +(无输出) +``` + +`.so` 就这样带着链接期的 RUNPATH 原封不动进了 tarball。 + +### 1.3 那条 RUNPATH 是谁写进去的 + +不是 bug,是设计——**dev 构建需要它**: + +- `src/toolchain/linkmodel.cppm:123-134` —— payload/sysroot 模式下,每个 `libDirs` + 条目发一对 `-L -Wl,-rpath,`(glibc payload、gcc payload)。 +- `src/build/flags.cppm:624-637` —— SubOS farm 作为 `ldRuntimeFallback` 挂在最后。 + +于是本机构建出的 `.so` 拿到: + +``` +DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 + : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 + : /registry/subos/default/lib +``` + +`SharedLibrary` 形态拿 **DT_RUNPATH** 而不是 DT_RPATH,是 `src/build/loader_contract.cppm:49-56` +明确规定的,而且是实测出来的(给库强上 DT_RPATH 会让 `eglInitialize` 挂掉, +openxlings/xlings#593)。**这条契约本身没有问题**,问题在于打包时没人把它取下来。 + +### 1.4 消费方本来是能自己解决的 + +`src/pack/manifest_emit.cppm:273-280` 为 shared 包发: + +```toml +[runtime] +runtime_search_dirs = ["lib/x86_64-linux-gnu"] +``` + +这条经 `linkIntent.runtimeSearchDirs` → `src/build/flags.cppm:362` 变成消费方的 +`-Wl,-rpath`。实测消费方 exe 拿到的是: + +``` +0x0f (RPATH) [/…/xim-x-glibc/2.44/lib64 ← Payload + : /…/xim-x-gcc/16.1.0/lib64 ← Payload + : /lib/x86_64-linux-gnu ← Package(来自 runtime_search_dirs) + : /registry/subos/default/lib] ← SubosFarm +``` + +**这正是打包 `.so` 需要的那个闭包,只不过是在「将要运行它的那台机器上」解析的。** +所以正确答案不是给 `.so` 换一个可重定位的路径,而是**把这条 tag 删掉,让它落到 +消费方的 DT_RPATH 上**——那份地址是对的、是本机的、而且是 mcpp 自己算的。 + +> 注意实测里的一个细节:消费方 exe 的 `DT_NEEDED` 里**没有 `libstdc++.so.6`** +> (它自己的代码没有拉到 libstdc++ 符号)。所以进程里通往 libstdc++ 的**唯一**路径 +> 就是 `libmathkit.so.1` 自己的解析——而那条路被它自己的 DT_RUNPATH 堵死了。 + +--- + +## 2. 实测 + +全部在本机跑完,mcpp 用 `target/x86_64-linux-gnu/5adc2f74a17360b2/bin/mcpp`(2026.8.19.4)。 + +### 2.1 复现(与 issue 逐字一致) + +fixture 就是 `tests/e2e/251_pack_library_shared.sh` 里的 `mathkit`: + +``` +$ mcpp pack mathkit-shared + Packed …/target/dist/mathkit-0.1.0-x86_64-linux-gnu-gcc16-libstdcxx16-c++23.tar.gz + +$ readelf -d /lib/x86_64-linux-gnu/libmathkit-shared.so + (NEEDED) [libstdc++.so.6] + (NEEDED) [libm.so.6] + (NEEDED) [libgcc_s.so.1] + (NEEDED) [libc.so.6] + (SONAME) [libmathkit.so.1] + (RUNPATH) [/home/speak/.mcpp/registry/data/xpkgs/xim-x-glibc/2.44/lib64: + /home/speak/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/lib64: + /home/speak/.mcpp/registry/subos/default/lib] +``` + +### 2.2 模拟「另一台机器」的四态实验 + +把包里 `.so` 的 RUNPATH 改写成**同形但不存在**的路径 +(`/other-machine/.mcpp/registry/…`),消费方 exe 不重新链接,直接跑: + +| 状态 | `.so` 的 tag | rc | 输出 | +|---|---|---|---| +| S1 今天的行为 | `RUNPATH=[/other-machine/…]` | **127** | `error while loading shared libraries: libstdc++.so.6: cannot open shared object file` | +| S2 `--remove-rpath` | 无 | **0** | `ok=42` | +| S3 `--set-rpath '$ORIGIN'` | `RUNPATH=[$ORIGIN]` | **127** | 同 S1 | +| S4 `--set-rpath ''` | `RUNPATH=[]` | **127** | 同 S1 | + +S1 的输出**与 #460 报告的错误信息一字不差**。 + +### 2.3 隔离出机制(合成最小例,不依赖 mcpp) + +三个文件:`libdep.so` 在 `deps/`;`libfoo.so` NEEDED `libdep.so`;`app` 的 +`DT_RPATH` 含 `deps/`(`--disable-new-dtags`)。 + +| `libfoo.so` 上的状态 | 结果 | +|---|---| +| `DT_RUNPATH=/nonexistent/…` | `libdep.so: cannot open shared object file`,rc=127 | +| 无 tag | `ok=42` | +| `DT_RUNPATH=$ORIGIN` | rc=127 | +| **`DT_RPATH=/nonexistent/…`**(`--force-rpath`) | **`ok=42`** | + +最后一行是关键对照:**同样是失效路径,写成 DT_RPATH 就不阻断继承,写成 DT_RUNPATH 就阻断**。 +所以这不是「路径找不到」,是「tag 的种类关掉了整条继承链」。 + +### 2.4 `LD_LIBRARY_PATH` 能盖住它,但 `mcpp run` 这次没盖住 + +| 启动方式 | 结果 | +|---|---| +| 裸执行 | rc=127 | +| `LD_LIBRARY_PATH=` 显式 | `ok=42` | +| `mcpp run` | **rc=127** | + +装载器的顺序是 DT_RPATH → `LD_LIBRARY_PATH` → DT_RUNPATH,所以 `LD_LIBRARY_PATH` +**不受** DT_RUNPATH 抑制。但 `compute_run_env` 只把 `plan.runtimeLibraryDirs` +(依赖产物目录)放进去,不含工具链 payload,所以 `mcpp run` 忠实地复现了失败。 + +⇒ **守卫可以用 `mcpp run`,只要让构建机路径不可达。**(见 §5.3) + +--- + +## 3. 根因 + +### 3.1 装载器物理 + +glibc 解析 `libfoo.so` 的 DT_NEEDED 时: + +1. 沿 loader 链走 DT_RPATH(`libfoo.so` → 加载它的 exe → …) +2. `LD_LIBRARY_PATH` +3. **`libfoo.so` 自己的** DT_RUNPATH(只有自己的,不继承) +4. `ld.so.cache` +5. 默认目录 + +而 `elf_get_dynamic_info` 在 **每个对象** 上执行「若有 DT_RUNPATH 则丢弃 DT_RPATH」。 +实测(§2.3 最后一行)说明:对携带 DT_RUNPATH 的对象,**第 1 步整条链都不再生效**。 + +于是:一个 tag 的存在,把消费方精心算好的四段 DT_RPATH 全部作废,只留下构建机上 +那三个不存在的目录 + ld.so.cache。ld.so.cache 里没有 mcpp payload 的 libstdc++ +(它就不该在),于是 127。 + +### 3.2 三层遮蔽,所以这个缺陷是外部用户报上来的 + +**遮蔽一:e2e 251 全程在同一台机器上。** +`tests/e2e/251_pack_library_shared.sh:97-118` 用 `path = ""` 依赖 + `mcpp run`, +包和构建机 store 都在原地。RUNPATH 指向的目录**真的存在**,于是永远绿。 +文件开头自己写的 "the process starts and the loader resolves the SONAME" 是真的—— +只是它解析的是构建机那份。 + +**遮蔽二:`215_pack_has_no_build_machine_paths.sh` 的清扫到不了库包。** +这个测试的标题就是「打包产物不得含构建机路径」,清扫逻辑(`$MCPP_HOME` 出现在任何 +ELF 的 RPATH/RUNPATH 里就失败)**正好能抓住 #460**。但它只跑 `mcpp pack`(应用), +从没跑过 `mcpp pack `。一个覆盖面窄于自身声明的检查,对它看不见的东西 +报「clean」。 + +**遮蔽三:mcpp 自己的运行期闭包模型表达不了这条规则。** +`src/platform/elf_runtime.cppm:325-330`: + +```cpp +std::vector dirs; +for (auto const& raw : requester.runpaths) + append_unique_path(dirs, expand_origin(raw, requester.artifact)); +for (auto const& dir : additionalSearchDirs) append_unique_path(dirs, dir); +for (auto const& dir : binding.libraryDirs) append_unique_path(dirs, dir); +``` + +它把「requester 自己的搜索路径」和「上层传下来的搜索路径」做**并集**,即把继承链 +建模成恒为开、且可叠加。真实规则是:requester 带 DT_RUNPATH ⇒ 后两组**全部不适用**。 + +`ElfRuntimeFacts` 里**已经有** `searchPathTag`(`elf_runtime.cppm:52`,`loader_contract` +就靠它判违规),`resolve_needed` 只是没读它。所以这不是「信息不够」,是「同一份信息 +两处解释,其中一处没跟上」。 + +这个模块自己的注释写着: + +> When the model and the artifact disagree about which loader runs, the model +> wins the report and the artifact wins reality. + +——正是这一条,只不过这次分歧不在 loader 上,在 tag 上。 + +### 3.3 政策模块有一个不存在的调用方 + +`src/platform/runtime_search.cppm:107-116`: + +```cpp +// Is this directory part of THIS machine's private state? +// +// `mcpp pack` asks this to decide what may not be baked into a distributable +// artifact. ... +bool is_machine_local(Origin origin); +``` + +实际调用方: + +``` +src/build/prepare.cppm:6696 写 resolution.json 的 machine_local 字段 +src/doctor.cppm:742 打印 [machine-local] 标记 +tests/unit/test_runtime_search.cpp +``` + +**没有任何 packer 调用它。** 两个 packer 各自硬编码了自己的答案:应用侧无条件改写成 +`$ORIGIN`,库侧什么都不做。注释描述的那个架构(策略在一处、打包器去问)从未接线。 + +### 3.4 同一类缺陷在 Mach-O 上解决过一次,ELF 侧漏了 + +`tests/e2e/259_shared_library_macho.sh` 的开头: + +> a library built in `/private/var/folders/…/target/…/bin` records that path, and +> the moment it is packed and extracted somewhere else, every consumer of it fails +> at load time — **on the publisher's machine it works perfectly**. +> +> ⚠️ AND WHY IT ASSERTS THE PATH IS *GONE*. Checking that the consumer runs in +> place proves nothing … so the library is packed, **the producer's whole build +> tree is DELETED**, and only then is the consumer built and run. + +Mach-O 侧的修法是**链接期**的(`ninja_backend.cppm:247` 无条件发 +`-Wl,-install_name,@rpath/`),守卫是**删掉构建树**。 + +ELF 侧:同样的缺陷类,既没有对应的修法,守卫(251)也恰恰是 259 明确点名「证明不了 +任何事」的那种——原地跑。 + +--- + +## 4. 影响范围 + +| 轴 | 结论 | 证据 | +|---|---|---| +| `kind = "shared"` + ELF | **受影响**,产物在另一台机器上不可用 | 实测 §2 | +| `kind = "lib"`(静态) | 不受影响 —— `.a` 是归档,没有动态段 | 格式事实 | +| PE(`.dll`) | 不受影响 —— PE 没有 rpath;DLL 靠 exe 同目录 / PATH 解析 | 读码 | +| Mach-O(`.dylib`) | **需核验**。`install_name` 已是 `@rpath/`(259 已钉),但 `LC_RPATH` 是否会带上 payload 目录取决于该机是否用 payload clang(`linkmodel.cppm:206-209` 会发 `-Wl,-rpath,`)。判据:`otool -l \| grep -A2 LC_RPATH` | 读码 | +| 已发布的二进制库包 | **同样受影响,且不会因为 mcpp 升级而自愈**。注意机制:`mcpp publish` 走的是**源码**发布(`publish/pipeline.cppm:109` 用 `git archive` 打源码 tarball),**不**调 `build_and_pack_library`;二进制库包的发布是作者手动上传 `mcpp pack` 的产物。所以受影响的是**已经躺在 release / 索引里的那些 tarball**,里面就是本节讨论的 `.so` | 读码 | +| 交叉打包 | `run_library_pack` **没有** `pack.cppm` 那样的宿主/格式拒绝,所以 ELF leg 可以从 Windows / macOS 宿主产出(mcpp 支持 Windows 宿主产 Linux ELF,PR#339);而 `sandbox_patchelf` 在那些宿主上根本不存在 | 读码 | +| 应用打包 | **不受影响**,已修(§1.1)。被捆绑的 `.so` 拿 `$ORIGIN`,其未捆绑的系统依赖走 ld.so.cache;`--mode self-contained` 走 wrapper 的 `--library-path`(等价 `LD_LIBRARY_PATH`,不受 DT_RUNPATH 抑制) | 读码 + §2.4 | + +--- + +## 5. 优化方案 + +按优先级排。P0 三条是一个整体:**不带守卫的修复会以同样的方式再次隐身**。 + +### P0-1 · 库包必须做 ELF 重定位:**删掉 tag**,而不是改写它 + +在 `run_library_pack` 的 leg 循环里,`copy_into(leg.artifact, dst)`(`library.cppm:255`) +之后、`file_digest(dst)`(`:333`)之前,插入一步 relocate: + +``` +对 staging 里的每个 ELF 产物: + 读 PT_DYNAMIC + 若无 DT_RPATH / DT_RUNPATH → 什么都不做(不是错误) + 否则 → 删除这两个 tag 的全部条目 +``` + +**为什么是「删除」而不是「改写成 `$ORIGIN`」**:§2.2 的 S3/S4 已经实测否掉了改写。 +更根本的理由在 §1.4:消费方的 DT_RPATH 是同一个闭包在正确的机器上解析的结果, +包括 payload、包目录、SubOS farm 三段。留下任何非空 DT_RUNPATH 都会把它们全部作废。 + +**顺序上的两个坑(都必须一起改)**: + +1. **digest 必须覆盖重定位后的字节。** `file_digest(dst)` 在 `:333`,relocate 必须 + 排在它前面,否则 manifest 记录的 digest 与包里的文件不一致。 +2. **soname 别名的 copy 回退拷的是原始产物。** `library.cppm:293` 是 + `copy_into(leg.artifact, alias)` —— 拷的是 `leg.artifact` 不是 `dst`。今天两者 + 逐字节相同所以看不出来;relocate 落地后,**符号链接创建失败的机器(Windows、 + 部分网络文件系统)会拿到一份未重定位的别名**,而 SONAME 别名恰恰是装载器真正 + 打开的那个名字。改成从 `dst` 拷。 + +**判据(四条都要,缺一条这个修复就会假绿)**: + +- (a) `readelf -d <包里的 .so>` 里**没有** `RPATH` 也没有 `RUNPATH` 行 —— 判据是 tag + 缺席,不是路径为空(S4); +- (b) SONAME 别名(符号链接或副本)与主文件的 (a) 结论相同; +- (c) 包里任何 ELF 的任何字符串都不含 `$MCPP_HOME`(复用 215 的清扫谓词); +- (d) **消费方在构建机状态不可达时仍能启动**(§5.3 的双向探针)。 + +### P0-2 · 机制:进程内改写 PT_DYNAMIC,不要调 patchelf + +**已写原型并实测通过**(附录 B)。做法:`Elf64_Dyn` 数组按 16 字节槽压缩——删掉目标 +槽、后面的整体前移一槽、末尾补 `DT_NULL`。**文件尺寸不变,没有任何偏移需要修**。 + +``` +removed 1 slot(s), 27 slots rewritten in place +size before/after: 17976 / 17976 +readelf -h: ok; readelf -d: RUNPATH 行消失; 消费方: ok=42 +``` + +为什么不用 `sandbox_patchelf`: + +- **交叉打包会静默跳过。** 库打包支持 `--target` 列表(fat 包),且没有宿主拒绝。 + 从 macOS/Windows 宿主产 ELF leg 时 `sandbox_patchelf()` 返回空——今天应用侧对此 + 的处理是 `if (!patchelf.empty())`,即**安静地不做**。在库侧照抄这个形状,等于把 + #460 保留给一半的宿主。 +- **多一个外部工具就多一个「装了没装」的轴**,而这一步的正确性是包能不能用的前提。 +- mcpp 已经有完整的 ELF 读取器(`mcpp.platform.elf_runtime`),缺的只是一个写侧, + 而这个写侧要做的事**只有删槽**——不需要 patchelf 那些搬 segment 的能力。 + +**边界(必须在实现里显式处理,不能默默跳过)**:ELF32 / 大端(RISC-V/ARM 32 位交叉 +leg 会遇到)、`Both`(同时有 DT_RPATH 和 DT_RUNPATH)、非 ELF 输入(`.a`/`.dll`/`.dylib` +直接返回「不适用」)、只读文件权限。 + +**patchelf 缺席时怎么办**:不适用了——进程内实现没有缺席这一说。这也顺带删掉了 +「库打包依赖 sandbox 里装了 patchelf」这条不成文前提。 + +> 建议把这个能力放在新模块 `mcpp.pack.relocate`(或 `mcpp.platform.elf_write`), +> 由**两个 packer 共用**,并让它按 `mcpp::platform::search::is_machine_local` 做决策—— +> 这样 §3.3 里那句注释就第一次变成真的。应用侧现有的 `set_search_path` 保持不变 +> (它要**写入** `$ORIGIN`,是另一个动作),但「哪些条目不许带走」应该由同一个谓词回答。 + +### P0-3 · 守卫:两侧钉,且必须让构建机状态不可达 + +新增 e2e(建议 `264_pack_library_is_relocatable.sh`),`# requires: elf python3`: + +1. `mcpp pack mathkit-shared`; +2. **静态判据**:包里每个 ELF 都断言 §P0-1 的 (a)(b)(c)——复用 215 的 `read_tag` + python 片段(建议抽到 `tests/e2e/_read_elf_tag.sh`,两个测试共用一份实现); +3. **动态判据,双向**: + - 构建消费方 → `mcpp run` → 断言 `ok=42`; + - **故意打回缺陷**:把包里 `.so` 的 RUNPATH 设成 `/nonexistent/…`,再跑 → 断言 + **失败**且信息里有 `cannot open shared object file`; + - 还原 → 再跑 → 断言恢复 `ok=42`。 + +第三步的中间那半是**这个测试存在的理由**:只钉「修好之后能跑」区分不了「守卫生效」 +和「根本没有这道门」——修复前后它都会绿(§3.2 遮蔽一)。反向那半只在缺陷真的能被 +观测到时才会红。 + +> 打回缺陷这一步需要一个能写 RUNPATH 的工具。若不想让测试依赖 patchelf,可以让 +> `mcpp` 暴露一个只在测试里用的内部子命令,或者直接用 python 就地把 `DT_NULL` 槽 +> 改回 `DT_RUNPATH`——反向操作和 P0-2 的正向操作是同一段代码。 + +另外两处: + +- **把 215 的清扫扩到库包**(或在新测试里复用它的谓词)。它的标题声称的范围本来就 + 包含库包。 +- **251 保留原样**,它钉的是「两个名字都在 + 消费方能起来」,不该被改成移植性测试。 + +### P1-1 · 让运行期闭包模型能表达这条规则 —— **已批准(2026-08-20)** + +`resolve_needed`(`elf_runtime.cppm:310`)按 `requester.searchPathTag` 分支: + +``` +requester 带 DT_RUNPATH(含 Both,因为装载器按 RUNPATH 处理) + → dirs = requester.runpaths (+ 非 hermetic 时的 host 默认目录) +否则 + → dirs = requester.runpaths + additionalSearchDirs + binding.libraryDirs (+ host 默认) +``` + +**收益**:mcpp 能在消费方 `build` 阶段就报出「这个预编译 `.so` 的 DT_RUNPATH 会挡住 +你的 RPATH」,而不是等到用户在另一台机器上拿到 127。对**第三方/手写的** +`[[runtime.artifacts]]` 预编译 `.so`(#433 那条线),这是唯一的防线——那些 `.so` +不是 mcpp 产的,P0-1 管不到。 + +**风险(必须先量再定)**:模型收紧会把今天判 `Pass` 的一些产物变成 `Unresolvable`, +而 `Unresolvable` 是 blocking 的。farm 里那些自带 RUNPATH 的第三方库,其依赖在模型里 +将只能从它们自己的 RUNPATH + host 默认目录找——hermetic binding 下没有 ld.so.cache +这一层,可能出现「真实能跑但模型说找不到」。 + +**建议的落法**:先把新规则作为**诊断**(`Inconclusive` + 明确文案)接进去,跑全量 +e2e 与几个真实工程做对照,确认没有新的红,再决定是否升级为 blocking。不要一步到位。 + +### P1-2 · `is_machine_local` 收敛为唯一策略点 + +让 P0-2 的 relocate 走 `is_machine_local`,并让应用侧的 `set_search_path` 决策也从 +它派生。这样「什么不许带走」只有一处定义;`resolution.json` / `doctor` 报的 +`[machine-local]` 与打包器实际剥掉的东西,第一次成为同一个答案。 + +### P1-3 · 打包 shared 目标时,校验它自己的 DT_NEEDED 闭包(读码,未实测) + +`run_library_pack` 只发 `targetName` 这一个产物。如果这个 `shared` 目标链接了同工程的 +另一个 `shared` 目标,那个 `.so` 不在包里,`check_prebuilt` 也只校验**声明过的** +artifact 存在(`prebuilt.cppm:82-91`),不看 `.so` 自己的 DT_NEEDED。 + +删掉 RUNPATH 之后,packer 手里正好有了做这件事所需的一切:staging 里的 `.so` 用 +`inspect_elf_runtime` 读一遍,每个 NEEDED 若既不是系统库、又不在包里、又不在 +`[dependencies]` 里 → 拒绝或警告(带文件名)。这与 `library.cppm:303` 对「没有 +archiver 就不许打包」的既有立场一致:**这类缺陷正是这个 feature 存在的理由**。 + +### P2-1 · 发布出去的是 debug、未 strip、且含发布者绝对源码路径的产物(实测)—— **已批准:默认 release + strip,并给可配置开关(2026-08-20)。设计见 `2026-08-20-pack-and-consumer-model-review.md` §6.1** + +`build_and_pack_library` 里 `BuildOverrides ov` 只设了 `target_triple`,没有 profile。 +实测产物: + +``` +$ file /lib/…/libmathkit-shared.so +ELF 64-bit LSB shared object, …, with debug_info, not stripped + +$ strings -a … | grep +…/b460/mathkit/src/mathkit.cppm +…/b460/mathkit/target/x86_64-linux-gnu/e16a674b43e1ee6f +…/b460/mathkit/src +``` + +这不是运行期缺陷(DWARF 路径找不到只影响调试),但它是: + +- **第二条构建机泄漏**——如果 §P0-1 的判据 (c) 收紧到「全文不含构建机路径」而不是 + 「RPATH/RUNPATH 不含」,这一条会直接把测试打红。**所以 (c) 必须明确写成只查动态 + 段**,否则守卫会因为一个不同的问题而红,读的人会以为 relocate 没生效。 +- 体积与发布质量问题(参考 2026.7.29.1:未 strip 让镜像上传从 34.81MB 降到 4.62MB + 才是真因)。 + +**这是策略决定,不是 bug。已拍板(2026-08-20):默认 release + strip,并提供可选/可配置 +开关。** 具体的键、CLI 形状、与 `dropObjects`/digest 的交互次序,见 +`2026-08-20-pack-and-consumer-model-review.md` §6.1。 + +### P2-2 · `mcpp doctor` 侧的事后检查 + +对已安装的 mcpp-pack 包(`is_distribution_package`)扫一遍其 `[[runtime.artifacts]]` +指向的 ELF,发现带 DT_RPATH/DT_RUNPATH 就报出来。这能覆盖**已经发布出去的**那批 +tarball(§4 最后一行)——它们不会因为 mcpp 升级而自动变好,只能靠重新打包;在那之前 +至少要让使用者能看见原因,而不是拿到一句 `cannot open shared object file`。 + +--- + +## 6. 明确不建议做的 + +| 方案 | 为什么不 | +|---|---| +| 把 `.so` 的 RUNPATH 改写成 `$ORIGIN` | **实测无效**(S3)。任何非空 DT_RUNPATH 都会关掉继承链。 | +| `--set-rpath ''` / 任何「只清空字符串」的工具 | **实测无效**(S4)。tag 还在。判据必须是 tag 缺席。 | +| 给库强制 DT_RPATH(`--force-rpath`) | 实测**能跑**(§2.3),但违反 `loader_contract` 的实测结论:库上的 DT_RPATH 有传递性,会把自己的搜索路径压进其下每一次查找,`eglInitialize` 因此挂过(xlings#593)。用一个已知会炸的机制换另一个。 | +| 链接期就不给 `shared` 目标发 RUNPATH | 会同时打断 dev 流(工程内 `.so` 的自解析、`ldd` 可读性)和被第三方宿主 `dlopen` 的场景。**dev 产物和 dist 产物的要求本就不同**——差异应该发生在打包这一步,这也正是应用侧的做法。 | +| 把 libstdc++ 捆进库包 | 与 `dist::Role::SharedLibrary` 的契约直接冲突:一个自带静态 libstdc++ 的 `.so` 导出过 777 个 GLOBAL std 符号,劫持了 exe 自己的 `-static-libstdc++`(#336 / 2026.8.11.3)。 | +| 只改文档、把 `patchelf --remove-rpath` 写成已知规避 | 报告里给的规避**是对的**,但它要求每个使用者都知道这件事;而失败发生在下游用户的机器上,发布者永远看不到。 | + +--- + +## 7. Review 对照表 + +| # | 主张 | 证据等级 | +|---|---|---| +| 1 | 库打包路径对 ELF 零处理 | **实测 + grep** | +| 2 | 产物 RUNPATH 与 issue 逐字一致 | **实测** | +| 3 | 非空 DT_RUNPATH 关掉消费方 DT_RPATH 的继承 | **实测**(真实产物 + 合成对照,4+4 态) | +| 4 | `$ORIGIN` 与空串都修不好 | **实测**(S3/S4) | +| 5 | 删 tag 后消费方在同一条件下能跑 | **实测**(S2) | +| 6 | 进程内删 `Elf64_Dyn` 槽可行、尺寸不变、产物有效 | **实测**(原型,附录 B) | +| 7 | 251 因为「同机」而永远绿;215 覆盖不到库包 | **读码 + 实测** | +| 8 | `resolve_needed` 把继承链建模成并集 | 读码(`elf_runtime.cppm:325-330`) | +| 9 | `is_machine_local` 没有 packer 调用方 | **grep** | +| 10 | 已发布的二进制库包同样受影响(但 `mcpp publish` 本身发的是源码,不是这条路径) | 读码(`publish/pipeline.cppm:109`) | +| 11 | soname 别名的 copy 回退拷的是未重定位的源 | 读码(`library.cppm:293`) | +| 12 | 库包产物是 debug、未 strip、含发布者源码路径 | **实测** | +| 13 | 打包 shared 目标不校验其 DT_NEEDED 闭包 | 读码,**未实测** | +| 14 | Mach-O 的 `LC_RPATH` 是否泄漏 | **未核验**,需在 macOS 上跑 | + +--- + +## 附录 A · 复现步骤 + +```bash +# 1. 造 fixture(与 tests/e2e/251 相同) +mkdir -p mathkit/src && cd mathkit +cat > src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit-shared] +kind = "shared" +soname = "libmathkit.so.1" +EOF + +# 2. 打包并看 RUNPATH +mcpp pack mathkit-shared +PKG=$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*') +readelf -d "$PKG/lib/x86_64-linux-gnu/libmathkit-shared.so" | grep -E 'RUNPATH|NEEDED' + +# 3. 造消费方(path 依赖),构建 +# app/mcpp.toml 里 [dependencies] mathkit = { path = "" } +# exe 会拿到含 payload + 包目录 + farm 的 DT_RPATH + +# 4. 模拟另一台机器:把包里 .so 的 RUNPATH 指向不存在的同形路径 +SO="$PKG/lib/x86_64-linux-gnu/libmathkit-shared.so"; cp "$SO" "$SO.orig" +patchelf --set-rpath '/other-machine/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/lib64' "$SO" +./app/target/*/*/bin/app # → libstdc++.so.6: cannot open shared object file + +# 5. 三个对照 +patchelf --remove-rpath "$SO"; ./app/target/*/*/bin/app # ok=42 +cp "$SO.orig" "$SO"; patchelf --set-rpath '$ORIGIN' "$SO"; ./app/target/*/*/bin/app # 127 +cp "$SO.orig" "$SO"; patchelf --set-rpath '' "$SO"; ./app/target/*/*/bin/app # 127 +``` + +## 附录 B · 进程内删 `Elf64_Dyn` 槽的原型 + +```python +# 删除 DT_RPATH(15) / DT_RUNPATH(29):按 16 字节槽压缩,末尾补 DT_NULL。 +# 文件尺寸不变,不动任何 segment/section 偏移。 +phoff, = struct.unpack_from(' 2026-08-20 · 承接 `2026-08-20-issue460-shared-library-runpath.md` +> 覆盖:架构完备性 / 跨平台 / 兼容性 / 易用性 +> 状态:**B0 / B1 / B3 已实施,发布于 2026.8.20.1**(§7 的前三批); +> B2 / B4 / B5 / B6 仍是设计。已批准的两项(#460 P1-1 = B4,P2-1 = B3)见 §6.1 与 §7。 + +--- + +## 0. 总评 + +**消费侧是这套设计里完成度最高的一半,生产侧缺的是同一根骨架。** + +- **消费侧**几乎没有结构性缺口。「包就是一个普通 mcpp 包」这个决定(零新 section、零新 key) + 是整个设计里最好的一步:它同时买到了老客户端可用、多种依赖形态(path/git/index)复用、 + 以及「不需要为分发再写一条解析路径」。ABI tag 的 don't-care 规则、interface digest、 + 中立链接通道与 GNU 拼写双写——每一条都有实测背书。真缺口只有两个(§3.2 / §3.3)。 + +- **生产侧**的三个缺口是**同一个形状**:应用侧做了一半、库侧一点没做,而中间**没有共同的 + 骨架**去强制两边一致。#460 只是这个形状最先炸出来的那一个: + + | 缺的东西 | 应用侧 | 库侧 | + |---|---|---| + | 二进制格式抽象(ELF/PE/Mach-O) | 有一半(`binfmt` 只服务 PE 闭包;ELF 靠跑二进制;Mach-O 没有) | 完全没有(格式无关地 `copy_file`) | + | 「什么不许带走」的策略点 | 硬编码成 `$ORIGIN` | 没有(#460) | + | profile / strip 轴 | 没有 | 没有 | + +- **跨平台的缺口不是「少支持一个平台」,而是「在做不到的平台上不拒绝」。** Windows 宿主 + 有一段写得很好的拒绝;macOS 没有,而 macOS 走的那条路会**执行用户的程序**(§4.1)。 + 这是本次 review 里优先级最高的一条,高于 #460。 + +--- + +## 1. 缺口全景(轴 × 两个 packer) + +| 轴 | 应用打包 `build_and_pack` | 库打包 `build_and_pack_library` | 判断 | +|---|---|---|---| +| 路由输入 | `[targets.].kind`,一处解析(`route.cppm`) | 同 | ✅ 好 | +| 格式判定 | `binfmt::identify` + `plan.targetIsPe` | 无 | ⚠️ 不对称 | +| 依赖闭包 | ELF=跑二进制;PE=读导入表;**Mach-O=无** | **无** | ⚠️ §3.4 / §4.1 | +| 可分发性重定位 | ELF 有(patchelf) | **无** | ❌ #460 | +| mode 轴(system/vendored/self-contained/static) | 四档 | 无(`--mode` 被 warning 忽略) | ✅ 合理(§6.2) | +| target 轴 | 恰好一个 | 多 leg(fat) | ✅ 合理,原因写在代码里 | +| ABI 轴 | n/a | **只有 triple**(`cfg_predicate_for` 只发 arch/os/env) | ⚠️ §3.2 | +| profile / strip | 无 | 无 | ❌ P2-1(已批准) | +| 宿主门 | Windows 拒绝 + 说理由;**macOS 不拒绝** | 无(不跑产物,但需要 relocate) | ❌ §4.1 / §4.3 | +| 归档确定性 | zip 确定(排序 + 无时间戳);tar **不确定** | 同 | ⚠️ §5.3 | +| 产物自洽性校验 | `ldd` 闭包顺带校验 | **无** | ⚠️ §3.4 | +| 老客户端兼容 | n/a | 静态 + 真实两半(e2e 252) | ✅ 很好 | + +--- + +## 2. 生产侧:三个缺口是同一个形状 + +### 2.1 缺一个「二进制格式」抽象层 + +今天格式知识散在四处,各自用不同的方式回答同一个问题: + +| 谁 | 怎么知道格式 | 怎么读 | +|---|---|---| +| `pack.cppm::run` | `plan.targetIsPe`(布尔) | PE 读导入表 / 否则跑二进制 | +| `pack.cppm::ldd_parse` | 不判定,假设 glibc | `LD_TRACE_LOADED_OBJECTS=1` | +| `binfmt.cppm` | `identify()` → `{Elf, Pe, MachO, Unknown}` | 有完整三态,**但只被 PE 路径用** | +| `library.cppm` | 不判定 | 不读 | +| `elf_runtime.cppm` | ELF only | 真正的解析器 | + +`binfmt::Format` 已经是三态了,`elf_runtime` 已经是一个像样的 ELF 读取器了——缺的是把 +「一个可分发二进制」抽象出来的那一层: + +``` +DistributableImage + ├ format() Elf | MachO | Pe | NotAnImage + ├ dependencies() DT_NEEDED / LC_LOAD_DYLIB / import table + ├ search_paths() DT_RPATH+RUNPATH / LC_RPATH / (PE: 无) + ├ strip_machine_local_paths() #460 的动作,按格式实现,PE 上是 no-op + └ id_name() SONAME / install_name / (PE: 无) +``` + +有了它,#460 的修复是「库 packer 多调一个方法」,而不是「再写一遍 patchelf 调用」; +Mach-O 的闭包也有了落点(§4.1);`is_machine_local` 有了唯一的调用方(§2.2)。 + +**这不是重构洁癖**:今天这四个答题者已经给出过两次不一致的答案(#460 是第一次: +应用侧剥、库侧不剥;§4.1 是第二次:PE 有宿主门、Mach-O 没有)。 + +### 2.2 缺一个「什么不许带走」的策略点 + +`runtime_search.cppm::is_machine_local` 的注释写着「`mcpp pack` asks this」,而 +**没有任何 packer 调用它**(grep 只有 `prepare.cppm` 写 `resolution.json`、`doctor` 打印、 +单测)。两个 packer 各自硬编码。 + +**修法**:relocate 一步按 `Origin` 逐条判定,而不是「全删」或「全换成 `$ORIGIN`」。 +注意 #460 实测的约束——ELF 上**过滤后若结果非空,仍然会阻断继承链**,所以库包这一档的 +正确答案仍然是删空 tag;但**判定过程**应该走 `is_machine_local`,这样: + +- 「这条被剥掉了,因为它是 machine-local」可以打印/记录,而不是静默; +- 一条**不是** machine-local 却仍被剥掉的条目(例如 `$ORIGIN`)成为一个可以被诊断的事件 + ——它意味着这个 `.so` 依赖包里没有的兄弟库(§3.4); +- `resolution.json` 里报的 `[machine-local]` 与打包器实际剥掉的东西第一次是同一个答案。 + +### 2.3 缺 profile / strip 轴 + +`BuildOverrides` 里没有 profile,`PackConfig` 里只有 +`default_mode / include / exclude / also_skip / force_bundle`。**两个 packer 发出去的都是 +dev 构建**(#460 实测:未 strip、含发布者绝对源码路径)。已批准修复,设计见 §6.1。 + +--- + +## 3. 消费侧:完备的部分与两个真缺口 + +### 3.1 做对了的(不需要动) + +- **包 = 普通 mcpp 包**。`[[runtime.artifacts]].provenance = "mcpp-pack …"` 作为标记, + 没有新 section、没有新 key ⇒ 老 mcpp 能构建、不能校验,而不是**加载失败**。这一条避开了 + 「新键让整份 manifest 加载失败,已发布包永远无法采用」的老坑。 +- **双写链接通道**:GNU 拼写的 `ldflags` + 方言中立的 `[target..runtime]`, + `merge_conditional_config` 里「中立形式**替换**库引用、但保留 `-Wl,-Bdynamic`」—— + 这个「只替换它能表达的部分」的规则是对的,而且是 e2e 257 逼出来的。 +- **ABI tag 的 don't-care 规则**:C surface 发短 tag,`standard` 按下界比较而不是相等。 + 形状即声明,不需要 flag。 +- **根目录拒绝**:`prepare.cppm:757` 拒绝把分发包本身当源码树构建,并给出可照抄的修法。 +- **不重建**:`plan.cppm:1472` 跳过分发包的 shared target,理由写清楚了(否则 relink 会 + 产出一个缺少所有实现单元的库)。 +- **诊断质量**:`check_prebuilt` 的「最接近的拒绝」+ 逐维度 need/got,是能直接照做的。 + +### 3.2 缺口 A:leg 选择轴只有 triple,而 ABI tag 有四维 + +`cfg_predicate_for` 只发 `arch / os / env`。于是: + +- 一个 fat 包**无法**同时携带同一 triple 的 gcc16 与 clang22 两条 leg——两个 + `[target.'cfg(all(arch="x86_64", os="linux", env="gnu"))'.build]` 会同时匹配, + `-L`/`-l` 各来一份。 +- 实际形态因此是「一个 ABI 一个包」:`dirName` 在单 leg 时带 abiTag,所以两次 pack 产出 + 两个不同的目录/tarball。但**包名与版本相同** ⇒ 索引里只能有一个 `latest`。 +- 结果:`check_prebuilt` 会给出一句很好的拒绝(「pin `[toolchain]` 到发布者用的那个」), + 但**生态层面没有出路**——gcc 用户和 clang 用户不能共用一个包名+版本。 + +**这是设计上的已知边界还是缺陷,需要你定调。** 两条可能的方向: + +- **A1(小)**:承认边界,把它写进 docs/12 的 limitations 表(现在没有这一行), + 并让 `check_prebuilt` 的拒绝里提示「同一版本可能存在其他 ABI 的包」。 +- **A2(大)**:给索引/解析加一条 ABI 轴——包身份从 `(ns, name, version)` 变成 + `(ns, name, version, abi)`,或在一个包里允许 `[target.'abi(...)']` 谓词。 + 代价很大(触及 `SPEC-001` 包身份与索引 schema),**不建议现在做**,但值得记下来: + 这是「二进制生态」真正要长大时必然撞上的墙。 + +### 3.3 缺口 B:库产物的 digest 记录了,但从来没人校验 + +`manifest_emit` 为每条 leg 写 `digest = "fnv1a:…"`,`check_prebuilt` 只校验 +`role == "interface"` 的那一条。于是: + +- 「interface 与二进制是成对产生的、不可分别替换」这个论点**只钉了一半**; +- 一个被替换/截断的 `.so` 只会在链接或运行时报错,而不是在门口被指出来; +- 而记录一个从不被检查的字段,读代码的人会以为它被检查了(这正是 §2.2 的同一种病)。 + +**修法很小**:`check_prebuilt` 的第 1 步(「artifact 在不在」)顺手把 digest 也比了。 +成本是每次 prepare 多读几个 `.so`/`.a`——可以只在 artifact 的 mtime/size 变化时算, +或者干脆接受(fnv1a 很快)。 + +### 3.4 半个缺口:`.so` 自己的依赖闭包没人看(#460 P1-3) + +`run_library_pack` 只发 `targetName` 一个产物;`check_prebuilt` 只校验**声明过的** +artifact 存在。所以一个 `shared` 目标若链接了同工程的另一个 `shared` 目标,包里缺的那个 +`.so` 不会被任何一侧发现。 + +`#460` 的修复正好把所需的一切放到 packer 手里(它已经要读 ELF 了),所以这两件事应该 +**一起做**:剥路径时顺便把 `DT_NEEDED` 读出来,凡是既不是系统库、又不在包里、又不在 +`[dependencies]` 里的,拒绝或警告并点名文件。 + +--- + +## 4. 跨平台 + +### 4.1 ⚠️ 最高优先级:macOS 宿主上 `mcpp pack`(应用)会**执行用户的程序** + +**读码结论,需在 macOS 上核验。** 路径: + +``` +run(plan, cfg) + → plan.targetIsPe ? run_pe // 否 + → #if defined(_WIN32) 拒绝 // macOS 上不成立 + → #else … ldd_parse(bundledBinary) // ← 走到这里 + cmd = "LD_TRACE_LOADED_OBJECTS=1 '' 2>&1" +``` + +`LD_TRACE_LOADED_OBJECTS` 是 **glibc ld.so 的**变量。dyld 不认它(它的对应物是 +`DYLD_PRINT_LIBRARIES`),所以这条命令在 macOS 上就是**把用户的程序跑起来**。之后: + +- 程序退出码为 0 ⇒ 它的 stdout 被当作 ldd 输出解析 ⇒ 一条依赖都解析不出来 ⇒ + `toBundle` 为空 ⇒ `sandbox_patchelf` 在 macOS 上也不存在 ⇒ `if (!patchelf.empty())` + 整段跳过 ⇒ 产出一个只含二进制和 wrapper 的 tarball,并打印 **`Packed`**; +- 程序退出码非 0 ⇒ 报 `ldd failed on : command exited with N`, + 一个既不提 macOS 也不提 dyld 的错误; +- 程序是交互式/长驻的 ⇒ `mcpp pack` 挂住; +- 程序有副作用(写文件、发网络请求)⇒ 打包这个动作把它做了一遍。 + +`docs/02` 把 macOS dylib 列在「Planned Support」,所以**不支持是已知的**; +问题是**代码不拒绝**。而 Windows 那条完全同源的路径写了一段很好的拒绝: + +> The dependency closure for that format is resolved by running the artifact under +> the target's own dynamic linker, which this machine has no way to do. + +macOS 的情况**更糟**而不是更轻:Windows 宿主根本跑不动那个 ELF,macOS 宿主**跑得动** +那个 Mach-O,只是不会产生 trace。 + +**建议(P0,独立于 #460)**: + +1. 立刻加一道拒绝——按 `binfmt::identify(builtBinary).format == MachO` 判定(而不是按宿主 + `__APPLE__`,理由与 `run()` 现有注释一致:这从来不是宿主的问题,是格式的问题), + 文案照 Windows 那段的形状写清楚为什么; +2. 判据必须**双向**:macOS 上跑一次断言「拒绝且信息提到 Mach-O」;同时保留一条断言 + 「Linux 上同一命令仍然成功」——只钉拒绝分不清「门生效」和「pack 整个坏了」; +3. 真正的支持另开:Mach-O 的闭包是 `otool -L` / 解析 `LC_LOAD_DYLIB`,重定位是 + `install_name_tool -change` + `LC_RPATH`。这是 §2.1 那个抽象层的第一个真实客户。 + +> 同一段代码还有一个较小的隐患:即使在 Linux 上,`ldd_parse` 也是**执行产物**。 +> 对交叉产物(比如 `--target aarch64-linux-gnu`)这同样跑不起来。今天靠什么挡住? +> 值得核验一遍——`binfmt::Ident` 里已经有「这台机器能不能跑这个文件」的语义, +> 说明这个问题被想过,但 `run()` 的分支只用了 `targetIsPe`。 + +### 4.2 三种格式的重定位机制不在同一层 + +| 格式 | 「不带走构建机」靠什么 | 在哪一层 | 守卫 | +|---|---|---|---| +| Mach-O | `-Wl,-install_name,@rpath/` | **链接期**,无条件 | e2e 259,**删掉生产者构建树**后才断言 | +| ELF | 什么都没有(#460) | —— | 无(251 原地跑,永远绿) | +| PE | 不需要(无 rpath);DLL 由消费方 deploy 到 `bin/` | 消费期 | e2e 257 + wine | + +三条各自都合理,但**没有一处写下「这三条是同一个问题的三种答案」**。§2.1 的抽象层就是那个 +写下来的地方;在它落地之前,至少应该在 `loader_contract.cppm` 或新的 relocate 模块的头部 +把这张表写进去——那个文件已经证明了「把规则写一次」在这个仓库里是有效的做法。 + +### 4.3 交叉打包:库侧没有宿主门,而修复所需的工具是 Linux-only + +`run_library_pack` 里没有任何 `_WIN32` / 格式判定,这在今天是**对的**(它不跑产物), +而且是有用的:mcpp 支持 Windows 宿主产 Linux ELF(PR#339),所以一个 Windows CI 可以产 +Linux 的库包。 + +但 #460 的修复会给这条路径引入一个 Linux-only 的依赖(`sandbox_patchelf` 只在 +`/patchelf/*/bin/patchelf` 找)。若照抄应用侧的 `if (!patchelf.empty())`, +**非 Linux 宿主会安静地不剥**——把 #460 原样留给一半的宿主。 + +⇒ 这就是 #460 P0-2 选择「进程内改写 PT_DYNAMIC」的架构理由,不只是省一个依赖。 + +### 4.4 macOS 只能服务一个目标 ⇒ 不能产 fat 包 + +已在 docs/12 的 verification scope 里写明并标注 *impossible*(不是 gap)。✅ 无需处理。 + +--- + +## 5. 兼容性 + +### 5.1 做得好的 + +- **零新 section / 零新 key**,并且 e2e 252 用**两半**钉:静态(生成的 manifest 的 section + 集合是既有词汇的子集,字面列出而不是推导)+ 真实(用 `$MCPP_BOOT` 消费)。 + 第二半在 CI 里因 xvm shim 而跳过——**这一点被诚实地写进了 docs/12**,值得保持。 +- **`cfg()` 而不是裸 triple**:裸 triple 只在 `--target` 时匹配,生成 `cfg()` 才是 + 「在每个 mcpp 上都是同一句话」。这一条是踩过坑的。 + +### 5.2 已发布的包无法追溯修复 + +#460 修好之后,**已经躺在 release / 索引里的 shared 包仍然是坏的**,而且坏在下游用户的 +机器上、发布者永远看不到。所以消费侧的检测(#460 P2-2)不是锦上添花,它是这批包唯一的 +出路:让使用者拿到一句能行动的话,而不是 `cannot open shared object file`。 + +**建议把它提到与 P0 同批**:检测的实现成本极低(读 `[[runtime.artifacts]]` 指向的 ELF, +看有没有 RPATH/RUNPATH),收益是把一个「无解的历史包袱」变成「一句可行动的诊断」。 + +### 5.3 确定性:zip 确定,tar 不确定,而文档只说了前者 + +`docs/02` 写着: + +> The archive is **deterministic**: no timestamps are read, so two packs of the same +> tree are byte-identical and a published checksum means something. + +这句话对 **zip 路径**(PE)成立——`run_pe` 里显式排序、`zip::write` 不读时间戳。 +但 tar 路径是 `tar -czf -C `: + +- 目录遍历顺序来自 readdir,不排序; +- tar 记录每个成员的 mtime / uid / gid / mode; +- gzip 头部默认写入时间戳。 + +⇒ **同一棵树两次打包,Linux/macOS 上得到两个不同的 sha256。** 这与「published checksum +means something」直接冲突,而索引条目正是靠 sha256 认包的。 + +**修法(低成本)**:`tar --sort=name --mtime=@0 --owner=0 --group=0 --numeric-owner` ++ `gzip -n`(或 `--format=ustar`)。注意 BSD tar(macOS 自带)不认 `--sort`, +需要按平台分支或改为自己产 tar——**这正好是「先量再改」的地方:先加一条 e2e 断言 +「两次 pack 的 sha256 相同」,它今天应该是红的。** + +### 5.4 未来加键的规则应该写下来 + +现有设计规避了「新键让老 mcpp 整份 manifest 加载失败」的坑,但那是**这次**的选择, +不是一条被写下来的规则。建议在 `manifest_emit.cppm` 头部把它升格为约束: + +> 这个文件只允许输出在 `<某个版本>` 之前就已被解析的键。要表达新东西,先让它成为 +> 一个**已被解析且被忽略**的键在生态里流通一个发布周期,再开始输出。 + +--- + +## 6. 易用性,以及已批准项的落地形状 + +### 6.1 P2-1 落地:默认 release + strip,可配置 —— **已批准** + +**建议的形状**(两个 packer 共用,不是只给库包): + +```toml +[pack] +profile = "release" # 默认。可写 "dev";未来若有更多 profile 直接沿用同一词汇 +strip = true # 默认。false = 保留符号与 debug 段 +``` + +``` +mcpp pack [--profile dev|release] [--no-strip] [--debug-symbols ] +``` + +**六个必须一起定的点**: + +1. **默认值改变是一次行为变更。** 今天所有人拿到的是 dev 产物;改默认之后同一条命令产出 + 不同的二进制。⇒ 输出里必须说一句(`Packing … (release, stripped)`), + 并写进 CHANGELOG 的 breaking 段。 +2. **次序**:`strip` 必须排在 **relocate 之后、`file_digest(dst)` 之前**。三个动作都改 + 字节,而 digest 是包的凭证。次序写死在一个地方,不要让两个 packer 各排一次。 +3. **strip 用什么**:`llvm-strip`/`strip` 来自**当前 leg 的工具链**(和 `archive_tool` + 同一个来源,`dialect_for(ctx->tc)` 已经有这个抽象),**不能用宿主的 strip** —— + 交叉 leg 上宿主 strip 可能不认目标格式。若解析不到,应该像 `archiver` 那条一样**拒绝** + 而不是静默跳过(`library.cppm:303` 已经确立了这个立场)。 +4. **静态库的 strip 要小心**:`strip` 一个 `.a` 会删掉符号表,库就不可链接了。 + 静态归档只能 `--strip-debug`,不能 `--strip-all`。**这一条必须有单独的 e2e**, + 否则「打包成功、消费方 undefined reference」。 +5. **与 `dropObjects` 的关系**:先删成员再 strip,还是反过来?建议先 `ar d` 后 strip, + 这样 strip 处理的是最终归档。两者都改归档 ⇒ 同样在 digest 之前。 +6. **debug 符号不要直接丢掉**:`--debug-symbols ` 或默认在 `target/dist/` 旁边留一份 + `*.debug`(`objcopy --only-keep-debug` + `--add-gnu-debuglink`),否则用户拿到崩溃栈时 + 没有任何东西可用。这一条可以排在后面做,但**接口要现在留出来**,不然默认 strip 之后 + 再补就是第二次行为变更。 + +**判据**:packed `.so`/`.a` 的 `file` 输出为 `stripped`;`strings` 里不含发布者的绝对 +源码路径;**静态库 strip 后消费方仍能链接并跑出正确结果**(第 4 点); +`--no-strip` 与 `[pack] strip = false` 两条通道各钉一次。 + +> 与 #460 守卫的交互(重要):#460 的「无构建机路径」判据**必须写明只查动态段**。 +> 如果写成「全文不含 `$MCPP_HOME`」,它在 P2-1 落地前会因 DWARF 而红,落地后又会因为 +> strip 掉了而变绿——两次都不是因为 relocate。**一个测试只量一件事。** + +### 6.2 `--mode` 对库目标只是 warning + +``` +--mode is an application-bundle depth and does not apply to the library target 'x' yet; ignoring it +``` + +这个处理是对的(拒绝会让 `mcpp pack --mode static` 在混合工程里变得难用),但那个 +**「yet」**暗示以后会有。值得现在就想清楚:**库包需要 mode 轴吗?** + +我的判断:**不需要 mode,但需要一条「第三方 `.so` 怎么办」的答案。** 今天 docs/12 写着 +「bundling dependencies into the package ❌ declare them instead」,这是一个清楚的立场, +而且和 `[dependencies]` 的传递是自洽的。⇒ 建议把 warning 里的「yet」去掉,改成指向那条 +立场的一句话。措辞暗示的路线图,读的人会当承诺。 + +### 6.3 消费者拿到 127 时的可操作性 + +这是当前整条链路里**唯一一处诊断质量明显低于本仓库水准**的地方,而且正好是终端用户所在的 +位置:`mcpp` 在生产侧和链接侧的每一条错误都能照做,而运行期失败落到的是 ld.so 的 +`cannot open shared object file`——它不提包名、不提 mcpp、不提该做什么。 + +**建议**:`mcpp run` / `mcpp test` 在子进程以 127 退出且 stderr 匹配 +`cannot open shared object file` 时,接管这条信息:用已有的 `resolve_runtime_closure` +跑一遍,把「哪个对象需要它 / 它的搜索路径是什么 / 为什么继承没生效」打出来。 +P1-1(闭包模型学会 RUNPATH 抑制规则)落地后,这段诊断才**说得对**——所以它是 P1-1 的 +自然下游,建议排在同一批。 + +### 6.4 缺一个「验证这个包」的入口 + +今天要回答「我发出去的包能用吗」,只有「换一台机器试试」。建议加 +`mcpp pack --verify `(或 `mcpp doctor --package `): + +- 每个 ELF 的 loader tag 契约 + 有无 machine-local 路径; +- `[[runtime.artifacts]]` 指向的文件都在、digest 都对(§3.3); +- `.so` 的 DT_NEEDED 闭包被包 + `[dependencies]` 覆盖(§3.4); +- interface digest 与 `interface/` 一致。 + +它同时是 §5.2 那批**存量坏包**的检测入口,也是 e2e 守卫可以直接调用的东西—— +守卫用产品自己的检查,比守卫各写一份 python 解析器更不容易腐坏。 + +--- + +## 7. 建议的落地顺序 + +| 批次 | 内容 | 理由 | +|---|---|---| +| **B0** | §4.1 macOS 宿主拒绝(格式判定,不是宿主判定)+ 双向 e2e | 唯一一条「会执行用户程序」的缺陷,且修复极小 | +| **B1** | #460 P0-1/P0-2/P0-3:relocate(进程内 ELF 写)+ digest 次序 + 别名从 `dst` 拷 + 双向守卫 | 用户已报;不带守卫的修复会再次隐身 | +| **B2** | §5.2 / #460 P2-2:消费侧与 `doctor` 检测存量坏包 | 已发布的包唯一的出路,成本极低 | +| **B3** | P2-1 release + strip + 可配置(§6.1 六点)| 已批准;必须排在 B1 之后,否则两个改动在同一个判据上互相干扰 | +| **B4** | P1-1 闭包模型学会 RUNPATH 抑制(先诊断、跑全量 e2e 对照、再考虑升级为 blocking)+ §6.3 的 127 诊断 | 已批准;下游有真实收益,但有把绿判红的风险 | +| **B5** | §2.1 `DistributableImage` 抽象 + §2.2 `is_machine_local` 收敛 + §3.3 digest 校验 + §3.4 闭包校验 | 结构性;做完之后 Mach-O 的真实支持才有落点 | +| **B6** | §5.3 归档确定性(先加红测试)、§6.4 `--verify`、§6.2 措辞 | 收尾 | +| **暂不做** | §3.2 A2(包身份加 ABI 轴) | 触及 SPEC-001 与索引 schema;先按 A1 把边界写进文档 | + +--- + +## 8. 判据清单(review 时对照) + +| # | 主张 | 证据等级 | +|---|---|---| +| 1 | macOS 宿主上应用打包走 glibc 路径并执行产物 | **读码**(`pack.cppm:934` 分支 + `ldd_parse:442`),**需 macOS 核验** | +| 2 | e2e 的 `pack` 能力 = ELF + patchelf ⇒ 应用打包在 macOS 上一条 e2e 都不跑 | **读码**(`tests/e2e/run_all.sh:43-76` 的 `Linux)` / `Darwin)` 分支) | +| 3 | `binfmt::Format` 已是三态但只服务 PE 路径 | 读码 | +| 4 | `is_machine_local` 无 packer 调用方 | **grep** | +| 5 | leg 选择只有 arch/os/env 三维 | 读码(`manifest_emit.cppm:142-161`) | +| 6 | 库 leg 的 digest 记录但从不校验 | 读码(`prebuilt.cppm:137-139` 只匹配 `role == "interface"`) | +| 7 | tar 路径不确定、zip 路径确定,而文档只声明了确定性 | 读码(`pack.cppm:797` vs `:917`)+ docs/02:306 | +| 8 | `PackConfig` 无 profile / strip 键 | 读码(`types.cppm:791-800`) | +| 9 | 静态库不能 `--strip-all`(会删符号表) | 通用知识,**落地前必须实测** | +| 10 | 老客户端兼容用静态+真实两半钉,且 CI 上只跑了静态那半 | 读码 + docs/12 自述 | +| 11 | `--mode` 对库目标 warning 且措辞含 "yet" | 读码(`cmd_publish.cppm:78-82`) | diff --git a/.github/tools/git_clone_retry.sh b/.github/tools/git_clone_retry.sh new file mode 100755 index 00000000..a5c19060 --- /dev/null +++ b/.github/tools/git_clone_retry.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# git_clone_retry.sh — `git clone`, but a hosted runner's DNS hiccup is not a +# red build. +# +# WHY THIS EXISTS +# +# Measured on a macOS ARM64 runner, in a step that had not yet executed a single +# line of mcpp: +# +# Cloning into '/tmp/xlings-src'... +# fatal: unable to access 'https://github.com/openxlings/xlings/': +# Could not resolve host: github.com +# ##[error]Process completed with exit code 128 +# +# Thirty seconds of resolver timeout, then a failed PR check that says nothing +# about the change under test. Every workflow in this repository that reaches an +# external repository did it with a bare `git clone`, so every one of them could +# fail this way — six call sites, one failure mode. +# +# THIS REPOSITORY ALREADY LEARNED THIS FOR `curl` +# +# `fetch_release.sh` carries the same lesson one protocol over: `curl --retry` +# alone does not cover a transport-layer error, and `--retry-all-errors` is what +# does. `git` has no such flag at all — the retry has to be here. +# +# WHY BOUNDED, AND WHY IT DOES NOT TRY TO BE CLEVER +# +# git reports "the host does not resolve", "the repository does not exist" and +# "the credentials are wrong" as the same exit 128, with only the message to +# tell them apart. Parsing that message would be a second, weaker copy of git's +# own error taxonomy — and it would be wrong the first time git rephrased one. +# +# So this retries EVERY failure, a bounded number of times. A genuinely missing +# repository costs the attempts and then fails with git's own last message +# intact, which is ~50s and a diagnostic that still names the real cause. A +# resolver blip costs one backoff. That trade is the same one `--retry-all-errors` +# makes, and it is the right way round: a false red costs a maintainer a rerun +# and their attention, a slow true red costs 50 seconds. +# +# Usage: git_clone_retry.sh [git clone args...] +# Env: GIT_CLONE_ATTEMPTS (default 4), GIT_CLONE_BACKOFF (default "5 15 30") +set -euo pipefail + +ATTEMPTS="${GIT_CLONE_ATTEMPTS:-4}" +read -r -a BACKOFF <<< "${GIT_CLONE_BACKOFF:-5 15 30}" + +# The destination is the last argument when it is not an option. Removing a +# partial clone before retrying matters: git refuses to clone into a directory +# that exists and is not empty, so without this the second attempt fails for a +# different reason than the first and the log stops making sense. +dest="${*: -1}" + +for (( i = 1; i <= ATTEMPTS; i++ )); do + # `cmd || rc=$?`, NOT `if cmd; then …; fi; rc=$?`. + # + # After an `if`, `$?` is the status of the IF STATEMENT — which is 0 when + # the body did not run. Written that way this script exits 0 on a clone that + # never succeeded, i.e. it turns a hard failure into a green step with a + # missing checkout. Caught by asserting the exit code in the helper's own + # test rather than by reading the output, which looked correct. + rc=0 + git clone "$@" || rc=$? + if (( rc == 0 )); then + exit 0 + fi + if (( i == ATTEMPTS )); then + echo "git_clone_retry: giving up after $ATTEMPTS attempts (last exit $rc)" >&2 + exit "$rc" + fi + if [[ -n "$dest" && "$dest" != -* && -e "$dest" ]]; then + echo "git_clone_retry: removing partial '$dest' before retrying" >&2 + rm -rf -- "$dest" + fi + delay="${BACKOFF[$(( i - 1 < ${#BACKOFF[@]} ? i - 1 : ${#BACKOFF[@]} - 1 ))]}" + echo "git_clone_retry: attempt $i/$ATTEMPTS failed (exit $rc); retrying in ${delay}s" >&2 + sleep "$delay" +done diff --git a/.github/workflows/ci-aarch64-fresh-install.yml b/.github/workflows/ci-aarch64-fresh-install.yml index 12ccf073..b451203e 100644 --- a/.github/workflows/ci-aarch64-fresh-install.yml +++ b/.github/workflows/ci-aarch64-fresh-install.yml @@ -79,7 +79,26 @@ jobs: mcpp index update || true idx="$HOME/.mcpp/registry/data/xim-pkgindex" rm -rf "$idx" - git clone --depth 1 https://github.com/openxlings/xim-pkgindex "$idx" + # A bare `git clone` here fails the whole job on a runner DNS + # hiccup. The policy lives in .github/tools/git_clone_retry.sh — + # but this job checks the repository out LAST ON PURPOSE (a + # `.xlings.json` in the workspace re-points where `xlings install` + # writes, so an early checkout silently changes what the + # fresh-install steps above are testing). The helper therefore does + # not exist on disk yet, and the retry is spelled inline. Same + # policy: retry every failure, bounded, git's own message survives. + clone_retry() { + local rc dest="${@: -1}" + for i in 1 2 3 4; do + rc=0; git clone "$@" || rc=$? + [ "$rc" = 0 ] && return 0 + [ "$i" = 4 ] && return "$rc" + [ -e "$dest" ] && rm -rf -- "$dest" + echo "clone_retry: attempt $i failed (exit $rc); retrying" >&2 + sleep $((i * 5)) + done + } + clone_retry --depth 1 https://github.com/openxlings/xim-pkgindex "$idx" grep -n "skipping relocation\|os.isfile(path.join(bindir" "$idx/pkgs/m/musl-gcc.lua" | head -2 || true - name: Native build + run an `import std` program @@ -156,7 +175,21 @@ jobs: export MCPP_HOME=$(mcpp self env | awk -F'= *' '/^MCPP_HOME/{print $2; exit}') echo "reusing MCPP_HOME=$MCPP_HOME" test -d "$MCPP_HOME" || { echo "could not determine MCPP_HOME"; exit 1; } - git clone --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src + # Same inline retry as the index clone above, and for the same + # reason — the checkout that would provide the shared helper is + # deliberately the last step in this job. + clone_retry() { + local rc dest="${@: -1}" + for i in 1 2 3 4; do + rc=0; git clone "$@" || rc=$? + [ "$rc" = 0 ] && return 0 + [ "$i" = 4 ] && return "$rc" + [ -e "$dest" ] && rm -rf -- "$dest" + echo "clone_retry: attempt $i failed (exit $rc); retrying" >&2 + sleep $((i * 5)) + done + } + clone_retry --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src # "$m", not `mcpp`: the just-built binary is the code under review, # and building xlings with the INSTALLED one meant this half of the diff --git a/.github/workflows/ci-linux.yml b/.github/workflows/ci-linux.yml index 8decb8ef..6a853c4c 100644 --- a/.github/workflows/ci-linux.yml +++ b/.github/workflows/ci-linux.yml @@ -169,7 +169,8 @@ jobs: - name: "Integration: mcpp builds & runs xlings (openxlings/xlings)" run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - git clone --depth 1 --recurse-submodules \ + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP" self config --mirror GLOBAL diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index 4290bbb7..a61ff9a4 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -310,7 +310,12 @@ jobs: run: | MCPP=/tmp/mcpp-fresh # the freshly self-hosted binary built from this PR export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - git clone --depth 1 --recurse-submodules \ + # ⚠️ THIS IS THE STEP THE RETRY WAS MEASURED ON: a macOS runner + # answered `Could not resolve host: github.com` after 30s of + # resolver timeout, failing a PR check before a single line of + # mcpp had run. See .github/tools/git_clone_retry.sh. + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP" self config --mirror GLOBAL diff --git a/.github/workflows/ci-windows.yml b/.github/workflows/ci-windows.yml index a1baeb52..a96089fa 100644 --- a/.github/workflows/ci-windows.yml +++ b/.github/workflows/ci-windows.yml @@ -254,7 +254,8 @@ jobs: XLINGS_NON_INTERACTIVE: '1' run: | export MCPP_VENDORED_XLINGS=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe") - git clone --depth 1 --recurse-submodules \ + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP_SELF" self config --mirror GLOBAL diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 1be786e9..8c2e1517 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -173,7 +173,10 @@ jobs: - name: "Cross-build xlings -> ${{ matrix.target }}" run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - git clone --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src + # A hosted runner's DNS hiccup is not a red build — see + # .github/tools/git_clone_retry.sh for the measurement. + "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ + --depth 1 https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src "$MCPP" self config --mirror GLOBAL 2>/dev/null || true "$MCPP" build --target ${{ matrix.target }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 5345c966..7c0c1b7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,103 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [2026.8.20.1] — 2026-08-20 + +### 修复 + +- **`mcpp pack` 打出的 `kind = "shared"` 库带走了构建机,产物在别人机器上起不来 + (#460)。** + + 打包只是把链接产物 `copy_file` 进包里,于是 `.so` 保留了链接期的 + `DT_RUNPATH`——一串指向**构建机** `~/.mcpp/` 的绝对路径。消费者在另一台机器上 + 拿到的是 `libstdc++.so.6: cannot open shared object file`。 + + ⚠️ **issue 里建议的 `$ORIGIN` 修不好它,空串也修不好。** 在真实包 + 真实消费者上 + 实测(把构建机 store 变成不可达): + + | 发货 `.so` 上的状态 | 消费方 `DT_RPATH` 被继承 | 结果 | + |---|---|---| + | 失效绝对路径的 `DT_RUNPATH`(此前的行为) | 否 | rc=127 | + | **没有这条 tag** | **是** | ok | + | `DT_RUNPATH = $ORIGIN` | 否 | rc=127 | + | `DT_RUNPATH = ""` | 否 | rc=127 | + + 关掉继承的是这条 tag 的**存在**而不是内容,所以判据是「tag 不存在」。删掉它也 + 不是妥协:消费方自己的 `DT_RPATH` 是同一个闭包,只不过是在真正要运行它的机器上 + 解析的。 + + 实现是**进程内改写 `PT_DYNAMIC`**(删槽、后移、补 `DT_NULL`,文件尺寸不变), + 不是调 patchelf——库打包支持 `--target` 且没有宿主门,非 Linux 宿主上 + `sandbox_patchelf` 会解析为空,照抄应用侧的 `if (!patchelf.empty())` 等于把这个 + 缺陷留给一半的宿主。新增 `mcpp.pack.relocate`,覆盖 ELF32/64 × 大小端(单测), + Mach-O 只读报告 `LC_RPATH`。 + +- **`mcpp pack` 一个 Mach-O 程序会**执行用户的程序**,然后报告 `Packed`。** + + 非 PE 路径靠 `LD_TRACE_LOADED_OBJECTS=1 ''` 向动态链接器要依赖表,而这个 + 变量是 glibc 的;dyld 不认它,于是那条命令在 macOS 上就是把程序跑起来,程序的 + 输出被当成依赖表解析(解析出零条),然后写出一个只含二进制的包。有副作用的程序 + 会把副作用做一遍,交互式的会把打包器挂住。现在按**产物格式**(而非宿主)拒绝, + 理由与旁边那条 `_WIN32` 拒绝完全同源。macOS 上 `kind = "lib"` / `"shared"` 照常 + 打包——库打包从不运行产物。 + + 为什么此前没人发现:e2e 的 `pack` 能力 = `elf` + `patchelf`,只有 `Linux)` 分支 + 给,所以应用打包在 macOS 上**一条 e2e 都没跑过**。 + +- **共享库包的 SONAME 别名在符号链接失败时会拷到未处理的原始产物。** + + 别名的 copy 回退读的是 `leg.artifact`(构建树里的文件)而不是暂存后的 `dst`。 + 在没有重定位/strip 之前两者逐字节相同,所以看不出来;之后它会在 + `create_symlink` 失败的机器上,用装载器真正要打开的那个名字,发出一份未重定位、 + 未 strip 的库。 + +### 变更 + +- **`mcpp pack` 默认走 release 并 strip 发货产物。** + + 此前两个打包器发的都是 dev 构建:未 strip、带着发布者的绝对源码路径。现在 + profile 的**兜底**从 `dev` 改为 `release`——其余优先级不变(`--profile` > + `[build] default-profile` > 兜底),所以声明过 profile 的工程仍然拿到它声明的 + 那个。 + + 剥什么取决于产物**是什么**,用的是 dh_strip 的分档: + + | 产物 | 参数 | 为什么不能更狠 | + |---|---|---| + | 可执行文件 | `--strip-all` | 没有人链接它 | + | 共享库 | `--strip-unneeded` | 保留 `.dynsym`——那**就是**导出表 | + | 静态归档 | `--strip-debug --enable-deterministic-archives` | ⚠️ `--strip-all` 会删掉归档的**符号索引**,消费方链接时报 `archive has no index; run ranlib to add one`(实测) | + + 被捆绑进 bundle 的第三方 `.so` **不**剥——它们不是 mcpp 构建的。 + + 新增 `--profile` / `--no-strip` / `--debug-symbols ` 与 `[pack] strip`、 + `[pack] debug_symbols`。`--debug-symbols` 是分离而不是丢弃:写出 + `//<产物>.debug` 并给发货产物加 `.gnu_debuglink`——**按 triple 分目录**, + 因为 fat 包的各条 leg 产物同名(`gnu` 与 `musl` 两条腿都叫 + `libmathkit-shared.so` 是常态),扁平布局会让后一条覆盖前一条,而前一个产物的 + `.gnu_debuglink` 会静默指向另一个目标的符号。 + + ⚠️ **一条要知道的后果**:裸 `mcpp build` 与裸 `mcpp pack` 现在写进**不同的** + `target///` 目录(指纹把 profile 算进去了)。手工放到构建 + 产物旁边的文件只在两条命令解析到同一个 profile 时才被 `pack` 看见;声明式通道 + (`[runtime] deploy_files`、`runtime_search_dirs`)不受影响。e2e 240 因此在 + fixture 里显式写了 `[build] default-profile`。 + + > `[pack] strip` 与 `[profile.].strip` 是两个决定:后者给**链接**加 `-s` + > (碰不到静态归档,也分离不出任何东西),前者管**包里带什么**。 + +### 内部 + +- `mcpp::toolchain::binutils_tool(tc, name)`:四个工具链家族对同一个 binutils 工具 + 的四种拼法,此前只有 `ar` 知道。`archive_tool` 现在由它表达(MSVC 的 `lib.exe` + 仍是特例,因为它不是 binutils 的名字)。 +- `tests/e2e/_elf_tag.sh`:215 与新增的 264 共用同一个 ELF 读取器,两份拷贝会变成 + 「构建机路径」的两个定义。 +- **判据只查动态段,不查文件字节**:重定位删的是条目,字符串留在 `.dynstr` + (`patchelf --remove-rpath` 实测残留完全相同,`.dynstr` 有尾部合并,删不安全)。 + 一个 `grep` 式的判据会把正确重定位的产物报成脏的,而且会在 strip 落地那天因为 + 另一个原因变绿——两次都不是因为重定位。 + ## [2026.8.18.3] — 2026-08-18 ### 新增 diff --git a/docs/02-pack-and-release.md b/docs/02-pack-and-release.md index ff44a63f..23a1f37b 100644 --- a/docs/02-pack-and-release.md +++ b/docs/02-pack-and-release.md @@ -133,6 +133,9 @@ mcpp pack --target aarch64-linux-musl # ARM64 equivalent mcpp pack --format dir # output as a directory, no tarball mcpp pack -o myapp.tar.gz # filename only: lands at target/dist/myapp.tar.gz mcpp pack -o /abs/path/myapp.tar.gz # includes a directory: output to the literal path +mcpp pack --profile dev # build with a different profile (default: release) +mcpp pack --no-strip # ship the artifacts as built +mcpp pack --debug-symbols dbg/ # write the separated *.debug files under dbg/ ``` When `-o` is given a bare filename, the output is placed under `target/dist/`; @@ -140,6 +143,55 @@ when it includes a directory (relative or absolute), the literal path is used. For the full set of options, see `mcpp pack --help`. +### What a packed artifact is built with, and what travels inside it + +Two things differ from `mcpp build`, and both exist because a package leaves +this machine: + +**The profile falls back to `release`, not `dev`.** Precedence is unchanged +otherwise — `--profile` beats `[build] default-profile`, which beats the +fallback. Only the last step differs, so a project that states a profile still +gets the one it stated, and `mcpp pack` never produces an artifact built with +flags `mcpp build` would not. + +> One consequence to know: a bare `mcpp build` and a bare `mcpp pack` now write +> into **different** `target///` directories, because the +> fingerprint covers the profile. A file placed beside a built artifact by hand +> — a DLL, a data blob — is therefore only visible to `pack` when both commands +> resolve to the same profile: state it in `[build] default-profile`, or pass +> `--profile` to both. The declarative channels (`[runtime] deploy_files`, +> `runtime_search_dirs`) are unaffected. + +**Debug information is stripped, and the publisher's paths go with it.** An +unstripped artifact carries DWARF, and DWARF carries the absolute paths of the +producer's source tree and build directory. What is removed depends on what the +artifact *is* — this is dh_strip's division, and the archive row is the one that +matters: + +| artifact | strip flags | why not more | +|---|---|---| +| executable | `--strip-all` | nothing links against it | +| shared library | `--strip-unneeded` | keeps `.dynsym` — that IS the export list | +| static archive | `--strip-debug --enable-deterministic-archives` | `--strip-all` removes the archive **symbol index**, and the consumer's link then fails with `archive has no index; run ranlib to add one` | + +All three also drop `.comment` and `.note`. Section removal is by exact name, so +`.note.gnu.build-id` survives and still pairs with `--add-gnu-debuglink`. + +`--no-strip` (or `[pack] strip = false`) ships the artifacts exactly as built. +`--debug-symbols ` separates the information instead of discarding it: +`/.debug` is written and the shipped artifact gets a +`.gnu_debuglink` pointing at it, which is what a debugger and `debuginfod` +follow. + +> `[pack] strip` is not `[profile.].strip`. The profile key appends `-s` +> to the **link**, which never touches a static archive and cannot separate +> anything; this one governs what the **package** carries. Two different +> decisions, two different names. + +**Bundled libraries are never stripped.** They came out of the store or off the +host, mcpp did not build them, and rewriting somebody else's shared payload for +this bundle's benefit is not the packer's business. + ## Output Layout The tarball contents are wrapped in a single top-level directory whose name @@ -310,6 +362,21 @@ The reverse direction — packing a Linux or macOS artifact *from* Windows — still does not work, and for the original reason: that closure is resolved by the target's own dynamic linker, which a Windows host has no way to run. +#### Packing a Mach-O program is refused — on every host, including macOS + +The same closure step asks the dynamic linker for the dependency list by running +the artifact with `LD_TRACE_LOADED_OBJECTS=1`. That variable is glibc's; dyld +has never heard of it. So on a Mac the command does not trace anything — **it +runs the program**, and whatever the program prints is then parsed as a +dependency table. mcpp refuses instead, and says which mechanism is missing. + +The refusal is keyed on the artifact's **format**, not on the host, for the same +reason the Windows one is: `LD_TRACE_LOADED_OBJECTS` cannot trace a Mach-O from +Linux either. + +A `kind = "lib"` / `"shared"` target packs normally on macOS — a library package +never runs the artifact. This restriction is only for programs. + ## Configuration Packaging behavior is configured via the `[pack]` section in `mcpp.toml`. The @@ -317,9 +384,11 @@ common fields are: ```toml [pack] -default_mode = "static" # override the normal vendored default for bare `mcpp pack` -include = ["share/**", "config/*.toml"] # extra files to bundle -exclude = ["debug/**"] +default_mode = "static" # override the normal vendored default for bare `mcpp pack` +strip = true # default. false ships the artifacts as built +debug_symbols = "dist/debug" # separate the debug info here instead of discarding it +include = ["share/**", "config/*.toml"] # extra files to bundle +exclude = ["debug/**"] # Fine-tune the vendored filtering policy. The configuration key keeps its # established `bundle-project` spelling. @@ -339,7 +408,11 @@ The `static` mode additionally requires a musl toolchain configured under ## Planned Support -macOS dylib, Windows DLL, and distribution formats such as `.deb` / `.rpm` / -AppImage are still on the roadmap. This document evolves alongside the +macOS **program** bundling (the Mach-O dependency closure, via `otool -L` / +`LC_LOAD_DYLIB`, and `install_name_tool` for relocation) is still on the +roadmap; until it lands `mcpp pack ` refuses on that format rather than +producing something that only looks like a bundle. Windows DLL bundling beyond +the current `.zip`, and distribution formats such as `.deb` / `.rpm` / AppImage, +are also on the roadmap. This document evolves alongside the `mcpp pack` implementation; for the latest options, refer to `mcpp pack --help`. diff --git a/docs/12-binary-distribution.md b/docs/12-binary-distribution.md index 6373a778..c54f51da 100644 --- a/docs/12-binary-distribution.md +++ b/docs/12-binary-distribution.md @@ -256,16 +256,77 @@ That is a degradation, not a break, and it is the right direction. But it means **the gate protects new clients only**, which belongs in the release notes of any package published to a mixed-version audience. +## What travels inside a package, and what deliberately does not + +A published package must work on a machine that is not the publisher's. Two +steps enforce that, and both run on every artifact the packer stages. + +### The build machine's loader paths are removed + +A dev build bakes the toolchain's own directories into every shared object: + +```text +DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 + : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 + : /registry/subos/default/lib +``` + +That is correct for a dev build and fatal for a package (issue #460), because +of one rule in the ELF loader: **an object that carries any `DT_RUNPATH` makes +the loader skip the entire inherited `DT_RPATH` chain when resolving that +object's own dependencies.** The consumer's `DT_RPATH` — payload, package +directory, SubOS farm, all computed on the machine that will actually run it — +is therefore not consulted, and the program dies with + +```text +error while loading shared libraries: libstdc++.so.6: cannot open shared object file +``` + +⚠️ **`$ORIGIN` is not the fix.** Measured on a real package with the build +machine's store made unreachable: + +| state on the shipped `.so` | consumer's `DT_RPATH` inherited? | result | +|---|---|---| +| stale absolute `DT_RUNPATH` | no | fails | +| **no tag at all** | **yes** | **runs** | +| `DT_RUNPATH = $ORIGIN` | no | fails | +| `DT_RUNPATH = ""` | no | fails | + +It is the tag's *presence* that disables inheritance, not its contents. So +`mcpp pack` removes the entry rather than rewriting it — and removing it is not +a compromise, it is the right answer: the consumer's own `DT_RPATH` is the same +closure, resolved where it means something. + +The path *string* stays in `.dynstr`, unreferenced. `.dynstr` is tail-merged by +the linker, so a shorter live string can begin inside the dead one and deleting +those bytes cannot be shown safe; `patchelf --remove-rpath` leaves the identical +residue at the identical file size. **A guard for this must therefore read the +dynamic entries, never `grep` the file's bytes** — see +`tests/e2e/_elf_tag.sh`. + +On Mach-O the packer reads `LC_RPATH` and warns when a package would carry one; +rewriting it (`install_name_tool -delete_rpath`) is not automated yet, because +no test in this suite produces a `.dylib` to measure the edit on. + +### Debug information is removed + +See [docs/02](02-pack-and-release.md) for the flags, the per-shape table, and +`--debug-symbols`. The rule that matters for a *library* package: a static +archive is only ever `--strip-debug`ed, because `--strip-all` removes the +archive symbol index and the consumer's link then fails with `archive has no +index; run ranlib to add one`. + ## Current limitations | | status | |---|---| | `kind = "lib"` (static) | ✅ every target, tested on all three | -| `kind = "shared"` on Linux/ELF | ✅ — the package carries both the link name and the SONAME | +| `kind = "shared"` on Linux/ELF | ✅ — the package carries both the link name and the SONAME, and no build-machine loader path | | `kind = "shared"` on PE / MinGW (`*-windows-gnu`) | ✅ — the package carries the `.dll` **and** its import library | -| `kind = "shared"` on Mach-O (`*-macos`) | ✅ — install name is `@rpath/`, so the `.dylib` relocates | +| `kind = "shared"` on Mach-O (`*-macos`) | ✅ — install name is `@rpath/`, so the `.dylib` relocates. `LC_RPATH` is reported, not yet rewritten | | `kind = "shared"` on PE / MSVC (`*-windows-msvc`) | ✅ — mcpp generates the `.def`; see below | | `kind = "shared"` on `*-musl` | ❌ a musl target links statically | +| one package carrying two ABIs for the same triple (gcc **and** clang) | ❌ leg selection is `cfg(arch/os/env)`; publish one package per ABI | | shipping prebuilt BMIs | ❌ not attempted; BMIs are compiler-build-exact | | bundling dependencies into the package | ❌ declare them instead (above) | | consuming a package with **native `cl.exe`** | ✅ — via the neutral link intent; see below | @@ -415,6 +476,9 @@ The e2e suite gates each test on host capabilities, so "the suite is green" and | Mach-O shared library relocating out of its build tree | — | ✅ | — | | MSVC refusing `kind = "shared"` for the export reason | — | — | ✅ | | a released mcpp consuming a package this one produced | local only | local only | local only | +| a packed `.so` carries no build-machine loader path, **and the guard can see the defect when it is put back** | ✅ | — | — | +| a stripped static archive still links; a stripped shared library still loads; `--no-strip` / `[pack] strip` / `--debug-symbols` from both sides | ✅ | — | — | +| the ELF editor on ELF32 and big-endian | unit test | unit test | unit test | *impossible* is not a gap: a macOS host can serve exactly one target (`host_can_serve`, `registry.cppm`), so a package with two legs cannot be produced diff --git a/docs/zh/02-pack-and-release.md b/docs/zh/02-pack-and-release.md index 08700fb0..d3f982f6 100644 --- a/docs/zh/02-pack-and-release.md +++ b/docs/zh/02-pack-and-release.md @@ -99,6 +99,9 @@ mcpp pack --target aarch64-linux-musl # ARM64 等价写法 mcpp pack --format dir # 输出为目录,不打包 tarball mcpp pack -o myapp.tar.gz # 仅文件名:落到 target/dist/myapp.tar.gz mcpp pack -o /abs/path/myapp.tar.gz # 含目录:按字面路径输出 +mcpp pack --profile dev # 换一个 profile 构建(默认 release) +mcpp pack --no-strip # 按构建原样发货,不剥符号 +mcpp pack --debug-symbols dbg/ # 把分离出的 *.debug 写到 dbg/ ``` `-o` 接受裸文件名时自动归到 `target/dist/`;含目录(相对或绝对) @@ -106,6 +109,46 @@ mcpp pack -o /abs/path/myapp.tar.gz # 含目录:按字面路径输出 完整选项参见 `mcpp pack --help`。 +### 打包产物用什么构建,里面带什么走 + +与 `mcpp build` 有两点不同,都因为「这个产物要离开本机」: + +**profile 的兜底是 `release` 而不是 `dev`。** 其余优先级不变 —— +`--profile` > `[build] default-profile` > 兜底。只有最后一步不同,所以声明过 +profile 的工程仍然拿到它声明的那个,`mcpp pack` 也不会产出一个 `mcpp build` +产不出来的 flag 组合。 + +> 有一条后果要知道:裸 `mcpp build` 与裸 `mcpp pack` 现在会写进**不同的** +> `target///` 目录 —— 指纹把 profile 算进去了。手工放到 +> 构建产物旁边的文件(一个 DLL、一份数据)因此只在两条命令解析到同一个 +> profile 时才被 `pack` 看见:把它写进 `[build] default-profile`,或者两条命令 +> 都带 `--profile`。声明式通道(`[runtime] deploy_files`、 +> `runtime_search_dirs`)不受影响。 + +**调试信息会被剥掉,发布者的路径随之消失。** 未 strip 的产物带着 DWARF,而 +DWARF 带着发布者源码树与构建目录的绝对路径。剥什么取决于产物**是什么** —— +这是 dh_strip 的分档,而其中归档那一行是要命的: + +| 产物 | strip 参数 | 为什么不能更狠 | +|---|---|---| +| 可执行文件 | `--strip-all` | 没有人链接它 | +| 共享库 | `--strip-unneeded` | 保留 `.dynsym` —— 那**就是**导出表 | +| 静态归档 | `--strip-debug --enable-deterministic-archives` | `--strip-all` 会删掉归档的**符号索引**,消费方链接时报 `archive has no index; run ranlib to add one` | + +三档都会去掉 `.comment` 与 `.note`。段删除按**精确名字**匹配,所以 +`.note.gnu.build-id` 会保留,`--add-gnu-debuglink` 仍然有东西可配对。 + +`--no-strip`(或 `[pack] strip = false`)按构建原样发货。 +`--debug-symbols <目录>` 则是分离而不是丢弃:写出 `<目录>/<产物>.debug`, +并给发货的产物加上指向它的 `.gnu_debuglink` —— 调试器与 `debuginfod` 认这个。 + +> `[pack] strip` 不是 `[profile.].strip`。后者是给**链接**加 `-s`, +> 既碰不到静态归档、也无法分离出任何东西;前者管的是**包里带什么**。 +> 两个不同的决定,两个不同的名字。 + +**被捆绑进来的库永远不 strip。** 它们来自 store 或宿主,不是 mcpp 构建的, +为了这一个 bundle 去改写别人的共享载荷不是打包器该做的事。 + ## 产物布局 tarball 内容包在一个顶层目录里,该目录的名字与 tarball 文件名(去掉 @@ -259,15 +302,31 @@ mcpp pack --target x86_64-windows-gnu # 在 Linux 宿主上 反方向 —— 在 Windows 上给 Linux / macOS 产物打包 —— 仍然不支持,原因还是最初 那个:那条闭包要由目标自己的动态链接器解析,而 Windows 宿主没有办法运行它。 +#### Mach-O 程序会被拒绝 —— 在所有宿主上,包括 macOS + +同一步闭包解析是靠 `LD_TRACE_LOADED_OBJECTS=1` **运行产物**来问动态链接器要 +依赖表的。这个变量属于 glibc 的 ld.so,dyld 从来不认(它的对应物是 +`DYLD_PRINT_LIBRARIES`)。所以在 Mac 上这条命令不会 trace 任何东西 —— +**它会把用户的程序跑起来**,然后把程序的输出当成依赖表解析。mcpp 现在直接拒绝, +并在信息里点名缺的是哪个机制。 + +判定按产物的**格式**而不是宿主,理由与 Windows 那条完全相同: +`LD_TRACE_LOADED_OBJECTS` 在 Linux 上也 trace 不了一个 Mach-O。 + +`kind = "lib"` / `"shared"` 目标在 macOS 上照常打包 —— 库打包从不运行产物。 +这条限制只针对程序。 + ## 配置项 打包行为通过 `mcpp.toml` 中的 `[pack]` 节配置,常用字段如下: ```toml [pack] -default_mode = "static" # 覆盖裸 `mcpp pack` 的正常 vendored 默认值 -include = ["share/**", "config/*.toml"] # 额外打包的文件 -exclude = ["debug/**"] +default_mode = "static" # 覆盖裸 `mcpp pack` 的正常 vendored 默认值 +strip = true # 默认值。false = 按构建原样发货 +debug_symbols = "dist/debug" # 把调试信息分离到这里,而不是丢弃 +include = ["share/**", "config/*.toml"] # 额外打包的文件 +exclude = ["debug/**"] # 微调 vendored 的过滤策略。配置键保留既有的 `bundle-project` 拼写。 [pack.bundle-project] @@ -284,6 +343,8 @@ force_bundle = ["libfoo.so"] # 即使命中 PEP 600 名单也强制打包 ## 待支持 -macOS dylib、Windows DLL,以及 `.deb` / `.rpm` / AppImage 等分发格式 -尚在规划中。本文档随 `mcpp pack` 实现演进,最新选项以 +macOS **程序** bundling(Mach-O 依赖闭包,走 `otool -L` / `LC_LOAD_DYLIB`, +重定位走 `install_name_tool`)仍在规划中;在它落地之前,`mcpp pack <程序>` +会在该格式上拒绝,而不是产出一个只是看起来像 bundle 的东西。当前 `.zip` +之外的 Windows DLL 分发,以及 `.deb` / `.rpm` / AppImage 等格式,同样在规划中。本文档随 `mcpp pack` 实现演进,最新选项以 `mcpp pack --help` 为准。 diff --git a/docs/zh/12-binary-distribution.md b/docs/zh/12-binary-distribution.md index 4a636872..2b5cb5a4 100644 --- a/docs/zh/12-binary-distribution.md +++ b/docs/zh/12-binary-distribution.md @@ -231,16 +231,69 @@ ldflags = ["-Llib/x86_64-linux-musl", "-lmathkit"] 这是**降级**而不是变砖,方向是对的。但它意味着**闸门只保护新客户端**, 面向混合版本用户群发布时,这一条应写进发布说明。 +## 包里带什么走,以及刻意不带什么 + +发布出去的包必须能在**不是发布者的**机器上工作。两个步骤保证这件事, +它们作用在打包器暂存的每一个产物上。 + +### 构建机的 loader 搜索路径会被删掉 + +dev 构建会把工具链自己的目录烙进每一个共享对象: + +```text +DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 + : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 + : /registry/subos/default/lib +``` + +这对 dev 构建是对的,对一个包则是致命的(issue #460),原因是 ELF 装载器的 +一条规则:**一个携带任何 `DT_RUNPATH` 的对象,会让装载器在解析它自己的依赖时 +跳过整条继承来的 `DT_RPATH` 链。** 于是消费方那条 `DT_RPATH` —— 载荷、包目录、 +SubOS farm,全都是在真正要运行它的那台机器上算出来的 —— 根本不被查,程序死在 + +```text +error while loading shared libraries: libstdc++.so.6: cannot open shared object file +``` + +⚠️ **`$ORIGIN` 不是解药。** 在真实的包上、把构建机的 store 变成不可达之后实测: + +| 发货 `.so` 上的状态 | 消费方 `DT_RPATH` 被继承? | 结果 | +|---|---|---| +| 失效的绝对路径 `DT_RUNPATH` | 否 | 失败 | +| **完全没有这条 tag** | **是** | **能跑** | +| `DT_RUNPATH = $ORIGIN` | 否 | 失败 | +| `DT_RUNPATH = ""` | 否 | 失败 | + +关掉继承的是这条 tag 的**存在**,不是它的内容。所以 `mcpp pack` 删掉这个条目而 +不是改写它 —— 而且删掉不是妥协,它就是正确答案:消费方自己的 `DT_RPATH` 是同一个 +闭包,只不过是在它有意义的那台机器上解析的。 + +路径**字符串**会留在 `.dynstr` 里,没有人再指向它。`.dynstr` 被链接器做了尾部合并, +一个更短的活字符串可能从这条死字符串的中间开始,删掉这些字节无法被证明是安全的; +`patchelf --remove-rpath` 留下的残留在尺寸上逐字节相同。**所以这件事的守卫必须读 +动态段的条目,绝不能 `grep` 文件字节** —— 见 `tests/e2e/_elf_tag.sh`。 + +Mach-O 上打包器会读出 `LC_RPATH` 并在包会携带它时告警;自动改写 +(`install_name_tool -delete_rpath`)尚未做,因为这个套件里还没有任何测试能产出 +一个 `.dylib` 来给这次字节编辑做判据。 + +### 调试信息会被剥掉 + +参数、分档表与 `--debug-symbols` 见 [docs/02](02-pack-and-release.md)。 +对**库**包最要紧的一条:静态归档只做 `--strip-debug`,因为 `--strip-all` 会删掉 +归档的符号索引,消费方链接时会报 `archive has no index; run ranlib to add one`。 + ## 当前边界 | | 状态 | |---|---| | `kind = "lib"`(静态) | ✅ 所有 target,三平台都测了 | -| `kind = "shared"` on Linux/ELF | ✅ —— 包里同时带链接名与 SONAME | +| `kind = "shared"` on Linux/ELF | ✅ —— 包里同时带链接名与 SONAME,且不含构建机的 loader 路径 | | `kind = "shared"` on PE / MinGW(`*-windows-gnu`) | ✅ —— 包里同时带 `.dll` **和它的导入库** | -| `kind = "shared"` on Mach-O(`*-macos`) | ✅ —— install name 是 `@rpath/`,`.dylib` 可重定位 | +| `kind = "shared"` on Mach-O(`*-macos`) | ✅ —— install name 是 `@rpath/`,`.dylib` 可重定位。`LC_RPATH` 只报告,尚未改写 | | `kind = "shared"` on PE / MSVC(`*-windows-msvc`) | ✅ —— mcpp 生成 `.def`;见下 | | `kind = "shared"` on `*-musl` | ❌ musl target 是静态链接的 | +| 一个包同时携带同一 triple 的两套 ABI(gcc **与** clang) | ❌ leg 选择是 `cfg(arch/os/env)`;一个 ABI 发一个包 | | 发布预编译 BMI | ❌ 未尝试;BMI 与编译器构建逐位绑定 | | 把依赖打包进去 | ❌ 改为声明依赖(见上) | | 用**原生 `cl.exe`** 消费这种包 | ✅ —— 经方言中立的链接意图;见下 | @@ -364,6 +417,9 @@ e2e 套件按宿主能力给每条测试开门,所以「套件是绿的」和「 | Mach-O 共享库离开构建树仍可加载 | — | ✅ | — | | MSVC 以「导出」为理由拒绝 `kind = "shared"` | — | — | ✅ | | 已发布的 mcpp 消费本版产出的包 | 仅本机 | 仅本机 | 仅本机 | +| 打包出的 `.so` 不含构建机 loader 路径,**且把缺陷放回去时守卫看得见** | ✅ | — | — | +| strip 过的静态归档仍可链接、strip 过的共享库仍可加载,`--no-strip` / `[pack] strip` / `--debug-symbols` 两侧都钉 | ✅ | — | — | +| ELF 编辑器在 ELF32 与大端上的行为 | 单测 | 单测 | 单测 | *不可能* 不是缺口:macOS 宿主只能服务一个 target(`host_can_serve`, `registry.cppm`),那里根本产不出两个 target 的产物的包。 diff --git a/mcpp.toml b/mcpp.toml index 8f51eeeb..dd921dcd 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.19.4" +version = "2026.8.20.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 95c552ef..781e9bfa 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -575,14 +575,23 @@ export CacheMode resolve_cache_mode(const mcpp::manifest::Manifest& m, // place. The rule is pure (manifest + one override string), so both sides can // evaluate it without resolving a toolchain or scanning the module graph. // -// Precedence: --profile/--release/--dev > [build].default-profile > "dev". +// Precedence: --profile/--release/--dev > [build].default-profile > `fallback`. // The global default is "dev" (-O0 -g) per the dominant convention // (Cargo/Meson/CMake/Zig/Bazel/MSBuild all default to debug). +// +// `fallback` exists for ONE caller: `mcpp pack`, where the artifact leaves this +// machine and an unoptimized build with the publisher's absolute source paths +// in it is never what was meant. It changes the LAST step only, so a manifest +// that states `[build] default-profile` still decides — packaging an artifact +// with different flags than `mcpp build` produces would be its own surprise. +// Adding a parameter here rather than a second resolver keeps the precedence +// rule in one function, which is why this function exists at all. export std::string resolve_profile_name(const mcpp::manifest::Manifest& m, - std::string_view override_name) { + std::string_view override_name, + std::string_view fallback = "dev") { if (!override_name.empty()) return std::string(override_name); if (!m.buildConfig.defaultProfile.empty()) return m.buildConfig.defaultProfile; - return "dev"; + return fallback.empty() ? std::string("dev") : std::string(fallback); } // Command-level overrides (--target / --static). @@ -642,6 +651,10 @@ export struct BuildOverrides { bool force_static = false; // --static (or implied by musl target) std::string package_filter; // -p : only build this workspace member std::string profile; // --profile (default "release") + // What `resolve_profile_name` falls back to when neither the command line + // nor `[build] default-profile` says. Empty = "dev", which is every + // interactive command. `mcpp pack` sets "release": see resolve_profile_name. + std::string profile_fallback; std::string features; // --features a,b,c (root package activation) bool strict = false; // --strict: schema warnings become errors std::string capabilities; // --cap blas=openblas,lapack=mkl (provider pins) @@ -1121,7 +1134,7 @@ prepare_build(bool print_fingerprint, // wants its plain `mcpp build` optimized sets // [build].default-profile = "release" (mcpp's own mcpp.toml does this, // so the released binary stays -O2). - pname = resolve_profile_name(*m, overrides.profile); + pname = resolve_profile_name(*m, overrides.profile, overrides.profile_fallback); mcpp::manifest::Profile pr; if (pname == "dev" || pname == "debug") { pr.optLevel = "0"; pr.debug = true; } else if (pname == "dist") { pr.optLevel = "3"; pr.strip = true; } diff --git a/src/cli.cppm b/src/cli.cppm index bd4431c6..5e3818b7 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -447,6 +447,15 @@ int run(int argc, char** argv) { .help("tar (default; .zip for a Windows target) | dir")) .option(cl::Option("output").short_name('o').takes_value() .help("Override output path")) + // Packaging builds RELEASE by default — the artifact leaves this + // machine. `[build] default-profile` still wins when it is set; + // this only replaces the "dev" fallback every other command uses. + .option(cl::Option("profile").takes_value() + .help("Build profile (default: [build] default-profile, else release)")) + .option(cl::Option("no-strip") + .help("Ship the artifacts as built (default: strip debug info)")) + .option(cl::Option("debug-symbols").takes_value().value_name("DIR") + .help("Write the separated *.debug files here (default: discard)")) .action(wrap_rc(cmd_pack))) // ─── emit (one nested subcommand: xpkg) ──────────────────────── diff --git a/src/cli/cmd_publish.cppm b/src/cli/cmd_publish.cppm index d9d6d93a..dab84430 100644 --- a/src/cli/cmd_publish.cppm +++ b/src/cli/cmd_publish.cppm @@ -58,6 +58,16 @@ export int cmd_pack(const mcpplibs::cmdline::ParsedArgs& parsed) { } if (auto v = parsed.value("output")) opts.output = *v; + // ⚠️ `value()`, not `option_or_empty()`, for `profile` — and NOT for + // anything whose name a positional shares. `ParsedArgs::value()` falls back + // to a same-named positional when the option is unset, which is how + // `mcpp run q` once became `--target=q`. `pack`'s positional is `target`, + // so `--target` is read through `option()` above and stays unaffected; + // `profile` and `debug-symbols` have no positional twin. + if (auto v = parsed.value("profile")) opts.profile = *v; + if (parsed.is_flag_set("no-strip")) opts.strip = false; + if (auto v = parsed.value("debug-symbols")) opts.debugSymbols = *v; + // `--target` is repeatable: one leg per triple, which is how a library // package ships for several targets at once. The application path has // always taken exactly one, and still does — packing one executable for diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index cd80af7b..26131662 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -1359,6 +1359,10 @@ std::expected parse_string(std::string_view content, } m.packConfig.defaultMode = s; } + if (auto v = doc->get_bool("pack.strip")) + m.packConfig.strip = *v; + if (auto v = doc->get_string("pack.debug_symbols")) + m.packConfig.debugSymbols = *v; if (auto v = doc->get_string_array("pack.include")) m.packConfig.include = *v; if (auto v = doc->get_string_array("pack.exclude")) diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 5c83888c..ececfd44 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -790,6 +790,20 @@ struct LibConfig { // "bundle-all" — bundle every dynamic dep including libc / libstdc++ struct PackConfig { std::string defaultMode; // empty → "bundle-project" + // ⚠️ THERE IS DELIBERATELY NO `[pack] profile`. Which profile `mcpp pack` + // builds with is `--profile` > `[build] default-profile` > "release" — + // packaging only changes the LAST step (from "dev"), because a fourth + // precedence level would have to be resolved before `prepare_build` runs + // and this manifest is what `prepare_build` produces. + // Strip the SHIPPED artifacts (not a link-time `-s`; see mcpp.pack.strip). + // Tri-state: unset = the default (strip), which is what a published binary + // wants. `false` ships the artifact exactly as built. + std::optional strip; + // Where the separated `*.debug` files go, package-root-relative or + // absolute. Empty = do not separate, which is the default: most publishers + // do not ship a debug package, and writing one by default would double the + // output of every `mcpp pack`. + std::string debugSymbols; std::vector include; // extra files/globs to ship std::vector exclude; // patterns to drop from include // Mode C overrides — let the user expand or contract the PEP 600 diff --git a/src/pack/library.cppm b/src/pack/library.cppm index 4260b3bd..0acbef14 100644 --- a/src/pack/library.cppm +++ b/src/pack/library.cppm @@ -38,9 +38,12 @@ export module mcpp.pack.library; import std; import mcpp.pack.digest; import mcpp.pack.manifest_emit; +import mcpp.pack.relocate; +import mcpp.pack.strip; import mcpp.source_kind; // builtin_extension_table — what needs no declaring import mcpp.pack.zip; import mcpp.platform; +import mcpp.ui; export namespace mcpp::pack { @@ -71,6 +74,11 @@ struct LibraryLeg { // package that no linker can use, so the package carries both and the // emitted manifest points consumers at this one. std::filesystem::path importLibrary; + // Debug-information removal for THIS leg's toolchain, resolved by the + // caller from `mcpp::toolchain::binutils_tool`. Per leg for the same + // reason `archiveTool` is: a fat package's aarch64 leg must not be + // stripped by the x86_64 host's tool. + mcpp::pack::StripTools stripTools; }; struct LibraryPackPlan { @@ -95,6 +103,11 @@ struct LibraryPackPlan { std::vector> dependencies; std::vector extras; // README / LICENSE / [pack].include hits + // Remove debug information from the shipped artifacts (mcpp.pack.resolve_strip). + bool strip = true; + // Where the separated `*.debug` files go, absolute. Empty = do not separate. + std::filesystem::path debugDir; + std::vector legs; }; @@ -273,26 +286,6 @@ run_library_pack(const LibraryPackPlan& plan) return std::unexpected(r.error()); } - // A shared library needs BOTH of its names present. - // - // `-lmathkit-shared` resolves `libmathkit-shared.so` at link time, but - // the object records `SONAME libmathkit.so.1`, and that is the name the - // loader asks for. Ship only the built file and the consumer links, - // then fails to start — mcpp's own runtime-closure check reports - // "libmathkit.so.1 not found on the search path this artifact will - // actually use", which is how this was caught. - // - // A symlink is what a distribution ships; a copy is the fallback for - // filesystems (and archives) that cannot carry one. - if (leg.shared && !leg.soname.empty() && leg.soname != name) { - auto alias = dst.parent_path() / leg.soname; - std::error_code linkEc; - std::filesystem::remove(alias, linkEc); - std::filesystem::create_symlink(name, alias, linkEc); - if (linkEc) - if (auto r = copy_into(leg.artifact, alias); !r) return std::unexpected(r.error()); - } - // Delete the objects of the units published as source. The consumer // compiles those itself; leaving them in the archive means two // definitions of the module initialiser, resolved by link order. @@ -324,6 +317,107 @@ run_library_pack(const LibraryPackPlan& plan) } } + // ── THE ORDER, AND IT IS NOT FREE ───────────────────────────── + // + // Four steps change the artifact's bytes and one records them. They + // are written here, once, because every one of them is a way to ship + // a package whose manifest describes a file that is not the one in the + // archive: + // + // 1. copy (above) + // 2. drop objects (above) — changes the archive's members + // 3. relocate — remove the build machine's loader paths + // 4. strip — remove debug info (and separate it) + // 5. soname alias — a symlink to, or a COPY OF, the FINAL file + // 6. digest — the package's evidence, over the final bytes + // + // 5 after 3–4 is the one that used to be wrong in a way nothing could + // see: the alias' copy fallback read `leg.artifact` (the BUILD tree's + // file), which was byte-identical only because nothing here modified + // anything. With 3 and 4 in place it would ship an unrelocated, + // unstripped library under the exact name the loader asks for — and + // only on the machines where `create_symlink` fails, which is where + // nobody looks. + + // 3. The build machine does not travel. See mcpp.pack.relocate for why + // this removes the tag rather than rewriting it to `$ORIGIN`. + { + auto r = mcpp::pack::relocate::strip_search_paths(dst); + if (!r) return std::unexpected(LibraryPackError{ std::format( + "cannot make '{}' relocatable: {}", dst.string(), r.error()) }); + using O = mcpp::pack::relocate::Outcome; + if (r->outcome == O::Removed) { + std::string what; + for (auto const& p : r->paths) { if (!what.empty()) what += " "; what += p; } + mcpp::ui::status("Relocated", + std::format("{} ({} dropped from the loader search path)", + name, what.empty() ? std::string("build-machine paths") : what)); + } else if (r->outcome == O::Reported && !r->paths.empty()) { + // Mach-O: read, not rewritten. Saying nothing here would let a + // `.dylib` carry the publisher's LC_RPATH into a package while + // the ELF leg beside it is clean. + std::string what; + for (auto const& p : r->paths) { if (!what.empty()) what += ", "; what += p; } + mcpp::ui::warning(std::format( + "{} carries LC_RPATH entries that mcpp does not yet rewrite: {}\n" + " If any of them names a directory on THIS machine, the package is " + "not relocatable. Remove it with `install_name_tool -delete_rpath " + " ` before publishing.", name, what)); + } else if (r->outcome == O::Unanalysed) { + mcpp::ui::warning(std::format( + "{} could not be checked for build-machine loader paths: {}", + name, r->note)); + } + } + + // 4. Debug information does not travel either — unless asked. + if (plan.strip) { + const auto shape = leg.shared ? mcpp::pack::ArtifactShape::SharedLibrary + : mcpp::pack::ArtifactShape::StaticArchive; + // PER LEG, mirroring `lib//`. A fat package's legs share an + // artifact NAME — `libmathkit-shared.so` for both the gnu and the + // musl leg is the normal case, not a corner one — so a flat debug + // directory would have the second leg overwrite the first, and the + // first artifact's `.gnu_debuglink` would then resolve to the other + // target's symbols. Silently. + const auto legDebugDir = plan.debugDir.empty() + ? std::filesystem::path{} : plan.debugDir / leg.triple; + auto r = mcpp::pack::strip_artifact(dst, shape, leg.stripTools, legDebugDir); + if (!r) return std::unexpected(LibraryPackError{ r.error() }); + if (r->outcome == mcpp::pack::StripOutcome::Stripped) { + mcpp::ui::status("Stripped", std::format("{} {} → {} bytes{}", + name, r->before, r->after, + r->debugFile.empty() ? std::string{} + : std::format(" (debug: {})", + r->debugFile.filename().string()))); + } + // The IMPORT LIBRARY is deliberately not stripped: it is an archive + // of linker stubs with no debug information to remove, and dh_strip + // makes the same exclusion. + } + + // 5. A shared library needs BOTH of its names present. + // + // `-lmathkit-shared` resolves `libmathkit-shared.so` at link time, but + // the object records `SONAME libmathkit.so.1`, and that is the name the + // loader asks for. Ship only the built file and the consumer links, + // then fails to start — mcpp's own runtime-closure check reports + // "libmathkit.so.1 not found on the search path this artifact will + // actually use", which is how this was caught. + // + // A symlink is what a distribution ships; a copy is the fallback for + // filesystems (and archives) that cannot carry one — and it copies + // `dst`, never `leg.artifact`. See the order note above. + if (leg.shared && !leg.soname.empty() && leg.soname != name) { + auto alias = dst.parent_path() / leg.soname; + std::error_code linkEc; + std::filesystem::remove(alias, linkEc); + std::filesystem::create_symlink(name, alias, linkEc); + if (linkEc) + if (auto r = copy_into(dst, alias); !r) return std::unexpected(r.error()); + } + + // 6. Evidence, over the bytes that actually ship. docLegs.push_back(PackageLeg{ .triple = leg.triple, .libFile = name, diff --git a/src/pack/library_pipeline.cppm b/src/pack/library_pipeline.cppm index 844adfef..a7167e59 100644 --- a/src/pack/library_pipeline.cppm +++ b/src/pack/library_pipeline.cppm @@ -35,6 +35,8 @@ import mcpp.pack; import mcpp.pack.abi_tag; import mcpp.pack.interface; import mcpp.pack.library; +import mcpp.pack.strip; +import mcpp.toolchain.model; import mcpp.platform; import mcpp.toolchain.dialect; import mcpp.toolchain.registry; @@ -130,6 +132,11 @@ export int build_and_pack_library(const std::string& targetName, for (auto const& want : legs) { mcpp::build::BuildOverrides ov; ov.target_triple = want; + // A packaged artifact leaves this machine, so the fallback is release + // rather than the interactive "dev". `[build] default-profile` still + // decides when the project states one — see resolve_profile_name. + ov.profile = opts.profile; + ov.profile_fallback = "release"; auto ctx = mcpp::build::prepare_build(false, /*includeDevDeps=*/false, {}, ov); if (!ctx) { mcpp::ui::error(ctx.error()); return 2; } @@ -287,6 +294,12 @@ export int build_and_pack_library(const std::string& targetName, declaredPlatforms = ctx->manifest.package.platforms; plan.dependencies = publishable_dependencies(ctx->manifest); plan.extras = extras_of(ctx->manifest, ctx->projectRoot); + // Read from the FIRST leg's manifest, like every other package-wide + // fact here: one package, one answer. A `[pack] strip` that differed + // per target would describe two packages. + plan.strip = mcpp::pack::resolve_strip(opts, ctx->manifest.packConfig); + plan.debugDir = mcpp::pack::resolve_debug_dir( + opts, ctx->manifest.packConfig, ctx->projectRoot); plan.interfaceSources = closure.published; plan.dropObjects = published_object_names(closure); for (auto const& d : ctx->manifest.buildConfig.includeDirs) { @@ -339,6 +352,15 @@ export int build_and_pack_library(const std::string& targetName, .soname = target->soname, .shared = shared, .importLibrary = importLib, + // From THIS leg's toolchain, for the same reason `archiveTool` is: + // a fat package's foreign leg must not be stripped by the host's + // tool. Whether stripping APPLIES is a question about the leg's + // TARGET, not about its compiler — see mcpp.pack.strip. + .stripTools = mcpp::pack::StripTools{ + .strip = mcpp::toolchain::binutils_tool(ctx->tc, "strip"), + .objcopy = mcpp::toolchain::binutils_tool(ctx->tc, "objcopy"), + .inBandDebugInfo = mcpp::pack::debug_info_is_in_band(triple), + }, }); mcpp::ui::status("Packed leg", std::format("{} [{}]", triple, tag.str())); } diff --git a/src/pack/pack.cppm b/src/pack/pack.cppm index bd20e0a5..1ce351a7 100644 --- a/src/pack/pack.cppm +++ b/src/pack/pack.cppm @@ -39,6 +39,8 @@ import mcpp.build.loader_contract; import mcpp.config; import mcpp.pack.binfmt; import mcpp.pack.host_requirements; +import mcpp.pack.relocate; +import mcpp.pack.strip; import mcpp.pack.zip; import mcpp.platform; import mcpp.platform.xlings; @@ -83,8 +85,38 @@ struct Options { // caller resolves the contract; this is the one bit of it that packaging // acts on. bool carryToolchainRuntime = false; + + // ── how the shipped artifact is BUILT and what travels inside it ── + // + // `profile` is the `--profile` override only. The default is not spelled + // here: `mcpp pack` passes `BuildOverrides::profile_fallback = "release"`, + // so `[build] default-profile` still decides when it is set, and the + // precedence rule stays in `resolve_profile_name` where the rest of mcpp + // reads it. + std::string profile; + // Tri-state on purpose. `nullopt` = "nobody said", which is what lets + // `[pack] strip` be consulted at all — a plain `bool` defaulted to true + // would make the manifest key unreachable from the CLI's point of view. + std::optional strip; + // `--debug-symbols `: where the separated `*.debug` files go. Empty = + // do not separate. + std::filesystem::path debugSymbols; }; +// The strip decision for this run: `--strip`/`--no-strip` > `[pack] strip` > +// stripped. +// +// Spelled once because both packers ask it and a distribution that strips its +// libraries but not its programs is a distribution whose rule nobody can state. +bool resolve_strip(const Options& opts, const mcpp::manifest::PackConfig& cfg); + +// Where the separated debug files go, absolute. Empty = do not separate. +// A manifest-relative path is resolved against the project root, like every +// other path a manifest names. +std::filesystem::path resolve_debug_dir(const Options& opts, + const mcpp::manifest::PackConfig& cfg, + const std::filesystem::path& projectRoot); + // Resolved plan — all paths absolute, all decisions baked in. struct Plan { Options opts; @@ -110,6 +142,14 @@ struct Plan { // The search set the PE closure resolves names against, after the // contract has had its say (see make_plan). std::vector searchDirs; + // ── debug information: the RESOLVED decision, not the request ───── + // + // On the Plan rather than in Options because `Options` is what the user + // asked for and this is what that came out as once the manifest and the + // toolchain had their say. The library packer keeps the same split. + bool strip = true; + std::filesystem::path debugDir; // absolute; empty = discard + mcpp::pack::StripTools stripTools; }; struct Error { std::string message; }; @@ -222,6 +262,23 @@ std::string wrapper_dirname_from_archive(const std::filesystem::path& archive) { } // namespace detail +bool resolve_strip(const Options& opts, const mcpp::manifest::PackConfig& cfg) { + if (opts.strip) return *opts.strip; + if (cfg.strip) return *cfg.strip; + return true; +} + +std::filesystem::path resolve_debug_dir(const Options& opts, + const mcpp::manifest::PackConfig& cfg, + const std::filesystem::path& projectRoot) +{ + auto raw = !opts.debugSymbols.empty() + ? opts.debugSymbols + : std::filesystem::path(cfg.debugSymbols); + if (raw.empty()) return {}; + return raw.is_absolute() ? raw : projectRoot / raw; +} + std::expected make_plan(const mcpp::manifest::Manifest& manifest, const mcpp::config::GlobalConfig& /*cfg*/, @@ -551,6 +608,26 @@ set_interpreter(const std::filesystem::path& binary, return {}; } +// Remove the program's debug information — and ONLY the program's. +// +// A bundled `.so` is somebody else's file: it came out of the store or off the +// host, mcpp did not build it, and stripping it would change a shared payload's +// bytes for no gain to this bundle. dh_strip draws the same line (a package +// strips what it built). +// +// Shared with `run_pe` deliberately: a MinGW `.exe` carries DWARF in-band just +// like an ELF one, so "does the bundle ship debug info" must not depend on +// which output family it lands in. +std::expected +strip_program(const Plan& plan, const std::filesystem::path& staged) +{ + if (!plan.strip) return {}; + auto r = mcpp::pack::strip_artifact(staged, mcpp::pack::ArtifactShape::Executable, + plan.stripTools, plan.debugDir); + if (!r) return std::unexpected(Error{r.error()}); + return {}; +} + // Bundle all `deps` into /lib/. We dereference any // symlinks so the bundle is self-contained even if /usr/lib/foo.so → /usr/lib/foo.so.1. std::expected @@ -890,6 +967,8 @@ run_pe(const Plan& plan) } } + if (auto r = strip_program(plan, stagedExe); !r) return r; + if (plan.opts.format != Format::Tar) return {}; std::vector entries; @@ -933,6 +1012,49 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) // the host: `LD_TRACE_LOADED_OBJECTS` cannot trace a PE from Linux either. if (plan.targetIsPe) return detail::run_pe(plan); + // A Mach-O artifact is REFUSED, on every host including macOS. + // + // The closure below asks the dynamic linker for the dependency list by + // running the artifact with `LD_TRACE_LOADED_OBJECTS=1`. That variable + // belongs to glibc's ld.so; dyld has never heard of it (its counterpart is + // `DYLD_PRINT_LIBRARIES`), so on macOS the command does not trace anything + // — IT RUNS THE USER'S PROGRAM. Whatever that program prints is then parsed + // as a dependency table, which yields nothing, and the bundle is written + // and reported as `Packed`. A program with side effects performs them; an + // interactive one hangs the packer. + // + // ASKED OF THE FORMAT, NOT OF THE HOST — the same correction the `_WIN32` + // branch below already carries. `LD_TRACE_LOADED_OBJECTS` cannot trace a + // Mach-O from Linux either, and a macOS host is not the thing that makes + // this impossible. + // + // docs/02 lists macOS bundling under "Planned Support"; until it lands, + // saying so is strictly better than producing an empty bundle that claims + // to be one. + // + // NAMES THE ARTIFACT. Not decoration: `mcpp pack ` routes on the + // target's kind, and a refusal that does not say WHICH program it got to is + // indistinguishable from one that resolved the wrong target — which is the + // exact defect `route_pack_target` exists to prevent. It is also the only + // way an e2e can check that routing on macOS, where no program bundle can + // be produced to inspect. + if (mcpp::pack::binfmt::identify(plan.builtBinary).format + == mcpp::pack::binfmt::Format::MachO) { + return std::unexpected(Error{std::format( + "cannot package the Mach-O program '{}' yet.\n", plan.binaryName) + + " The dependency closure for that format is resolved by running the " + "artifact under\n" + " the target's own dynamic linker, and the mechanism mcpp uses " + "(LD_TRACE_LOADED_OBJECTS)\n" + " is glibc's — dyld ignores it and simply RUNS the program, which is " + "why this is\n" + " refused rather than attempted.\n" + " A `kind = \"lib\"` / `\"shared\"` target packs normally on macOS " + "(`mcpp pack `);\n" + " for a program, ship the build tree or use a platform bundler until " + "macOS support lands."}); + } + #if defined(_WIN32) // A NON-PE artifact on a Windows host: a cross build to Linux or macOS. // The closure below asks the dynamic linker by running the binary, which @@ -1025,18 +1147,33 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) if (auto r = bundle_libs(toBundle, plan.stagingRoot); !r) return std::unexpected(Error{r.error()}); + // Search path: point at bundled libs, or REMOVE THE TAG if there are none. + // + // The empty case used to be `patchelf --set-rpath ''`, which leaves the + // tag present with an empty string — and a present-but-empty DT_RUNPATH + // is not inert: it suppresses the inherited DT_RPATH chain exactly like + // a stale one does (measured — see mcpp.pack.relocate). Harmless on an + // executable, which is the top of that chain, but there is no reason to + // write a tag that says nothing. + // + // OUTSIDE the patchelf guard, because it no longer needs patchelf. That + // is the whole point of the in-process editor: `--mode system` on a host + // where the sandbox has no patchelf used to leave the build machine's + // store in the artifact and say nothing at all. + if (toBundle.empty()) { + if (auto r = mcpp::pack::relocate::strip_search_paths(bundledBinary); !r) + return std::unexpected(Error{r.error()}); + } + auto patchelf = sandbox_patchelf(cfg); if (!patchelf.empty()) { - // Search path: point at bundled libs (or clear if none). // non-empty bundle → "$ORIGIN/../lib" so the binary finds them - // empty bundle → clear the original dev-sandbox RUNPATH - // (~/.mcpp/registry/... doesn't exist on - // a user's target machine) - const char* rpath = toBundle.empty() ? "" : "$ORIGIN/../lib"; - if (auto r = set_search_path(bundledBinary, rpath, - mcpp::build::loader::Form::Executable, - patchelf); !r) - return std::unexpected(Error{r.error()}); + if (!toBundle.empty()) { + if (auto r = set_search_path(bundledBinary, "$ORIGIN/../lib", + mcpp::build::loader::Form::Executable, + patchelf); !r) + return std::unexpected(Error{r.error()}); + } // EVERY BUNDLED LIBRARY, not just the executable. // @@ -1123,6 +1260,14 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) return std::unexpected(Error{r.error()}); } + // 4b. Debug information does not travel either. + // + // AFTER every byte-changing step above (patchelf's search path, PT_INTERP) + // and before the archive: strip must see the final image, and the archive + // must see the stripped one. Same ordering rule the library packer states + // at its leg loop. + if (auto r = strip_program(plan, bundledBinary); !r) return r; + // 5. Output. if (plan.opts.format == Format::Tar) { if (auto r = make_tarball(plan.stagingRoot, plan.archivePath); !r) diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index ab21b1da..bebbe762 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -18,6 +18,10 @@ import mcpp.build.plan; import mcpp.config; import mcpp.fetcher.progress; import mcpp.pack; +import mcpp.pack.strip; +import mcpp.toolchain.model; +import mcpp.toolchain.registry; +import mcpp.toolchain.triple; import mcpp.ui; namespace mcpp::pack { @@ -47,6 +51,10 @@ export int build_and_pack(Options opts, bool modeFromUser, ov.target_triple = "x86_64-linux-musl"; else ov.target_triple = opts.targetTriple; + // A bundled program leaves this machine: release is the fallback, not dev. + // `[build] default-profile` still decides when the project states one. + ov.profile = opts.profile; + ov.profile_fallback = "release"; auto ctx = mcpp::build::prepare_build(/*print_fp=*/false, /*includeDevDeps=*/false, /*extraTargets=*/{}, ov); @@ -76,7 +84,9 @@ export int build_and_pack(Options opts, bool modeFromUser, && ctx->tc.targetTriple.find("-musl") == std::string::npos) { // Need to re-prepare the build with the musl target. mcpp::build::BuildOverrides ov2; - ov2.target_triple = "x86_64-linux-musl"; + ov2.target_triple = "x86_64-linux-musl"; + ov2.profile = opts.profile; + ov2.profile_fallback = "release"; auto ctx2 = mcpp::build::prepare_build(false, false, {}, ov2); if (!ctx2) { mcpp::ui::error(ctx2.error()); return 2; } ctx = std::move(ctx2); @@ -176,9 +186,32 @@ export int build_and_pack(Options opts, bool modeFromUser, ctx->plan.runtimeRequirements); if (!plan) { mcpp::ui::error(plan.error().message); return 1; } - mcpp::ui::info("Packing", std::format("{} v{} ({})", + // The RESOLVED debug-information decision. On the plan, not in Options: + // Options is the request, this is what it came out as once the manifest + // and the toolchain had their say. Tools come from the build's own + // toolchain so a cross bundle is stripped by the cross tool. + plan->strip = mcpp::pack::resolve_strip(opts, ctx->manifest.packConfig); + plan->debugDir = mcpp::pack::resolve_debug_dir(opts, ctx->manifest.packConfig, + ctx->projectRoot); + plan->stripTools = mcpp::pack::StripTools{ + .strip = mcpp::toolchain::binutils_tool(ctx->tc, "strip"), + .objcopy = mcpp::toolchain::binutils_tool(ctx->tc, "objcopy"), + // The CANONICAL triple, resolved the same way the library packer + // resolves it: an empty `targetTriple` means "this host", and asking + // the empty string would answer "in-band" for macOS and MSVC alike. + .inBandDebugInfo = mcpp::pack::debug_info_is_in_band( + ctx->tc.targetTriple.empty() + ? mcpp::toolchain::triple::host_triple().str() + : [&] { + auto t = mcpp::toolchain::triple::parse(ctx->tc.targetTriple); + return t ? t->str() : ctx->tc.targetTriple; + }()), + }; + + mcpp::ui::info("Packing", std::format("{} v{} ({}{})", plan->packageName, plan->packageVersion, - mcpp::pack::mode_cli_name(plan->opts.mode))); + mcpp::pack::mode_cli_name(plan->opts.mode), + plan->strip ? ", stripped" : "")); auto r = mcpp::pack::run(*plan, *cfg); if (!r) { diff --git a/src/pack/relocate.cppm b/src/pack/relocate.cppm new file mode 100644 index 00000000..6d21cdab --- /dev/null +++ b/src/pack/relocate.cppm @@ -0,0 +1,377 @@ +// mcpp.pack.relocate — removing the BUILD MACHINE from a distributable image. +// +// THE RULE IS "THE TAG IS GONE", NOT "THE PATH IS RELATIVE" +// +// A dev build bakes the toolchain's own directories into every artifact: +// +// DT_RUNPATH = /registry/data/xpkgs/xim-x-glibc/2.44/lib64 +// : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 +// : /registry/subos/default/lib +// +// That is correct for a dev build and wrong for a package, and the obvious fix +// — rewrite it to `$ORIGIN` — DOES NOT WORK. Measured on a real `mcpp pack` +// product consumed by a real mcpp-built program, with the build machine's +// store made unreachable (issue #460): +// +// state on the shipped .so consumer's DT_RPATH inherited? result +// ------------------------------ ----------------------------- ------ +// DT_RUNPATH = no 127 +// no tag at all YES ok +// DT_RUNPATH = $ORIGIN no 127 +// DT_RUNPATH = "" (empty string) no 127 +// DT_RPATH = yes ok +// +// The ELF loader drops the whole inherited DT_RPATH chain for any object that +// carries a DT_RUNPATH — whatever that RUNPATH says. So a non-empty +// replacement is not "less wrong", it is exactly as broken as the original, +// and an EMPTY one (which is what `patchelf --set-rpath ''` writes, and what +// any "clear the string" tool leaves behind) is broken too. +// +// Removing the tag is not merely the least-bad option, it is the RIGHT one: +// the consumer's own DT_RPATH is the same closure — payload, package dir, +// SubOS farm — resolved on the machine that will actually run it. +// +// WHY THIS EDITS BYTES INSTEAD OF SHELLING OUT TO patchelf +// +// A library package is cross-target by construction (`--target` is repeatable, +// and `run_library_pack` has no host gate because it never runs the artifact). +// `sandbox_patchelf` resolves only under a Linux xlings store, so a macOS or +// Windows host producing an ELF leg would find nothing — and the application +// packer's shape for that case is `if (!patchelf.empty())`, i.e. SILENTLY DO +// NOTHING. Copying that shape here would hand half of the hosts the very +// defect this module exists to remove. +// +// The edit needed is also far smaller than what patchelf is built for: delete +// one entry from the `Elf*_Dyn` array by moving the tail up one slot and +// padding the end with DT_NULL. Same file length, no segment moves, no offset +// fixups. +// +// WHAT THIS DELIBERATELY DOES NOT DO +// +// The path STRING stays in `.dynstr`. Nothing points at it any more, so it is +// not reachable by the loader — but `strings` still finds it. That is not +// sloppiness and it is not a regression against the reference tool: measured, +// `patchelf --remove-rpath` leaves the identical residue at the identical file +// size. `.dynstr` is TAIL-MERGED by the linker, so a shorter live string may +// begin inside the dead one, and removing those bytes cannot be shown safe +// without a full reference analysis of `.dynstr`'s three consumers (DT_* +// strings, `.dynsym` names, `.gnu.version_r`). Silently truncating a symbol +// name is a worse outcome than a dead string. +// +// ⚠️ CONSEQUENCE FOR ANY GUARD: "this artifact carries no build-machine path" +// must be asked of the DYNAMIC ENTRIES, never of the file's bytes. A +// byte-pattern check reports a correctly relocated artifact as dirty. + +export module mcpp.pack.relocate; + +import std; +import mcpp.pack.binfmt; + +export namespace mcpp::pack::relocate { + +// What happened to one image. +// +// FIVE-VALUED, and the last two are the point: an image this module could not +// analyse has NOT been shown to be clean, and reporting that as `NothingToDo` +// would make "nothing was wrong" and "nothing was checked" the same answer. +enum class Outcome { + Removed, // an ELF search-path tag was present and is now gone + NothingToDo, // ELF, analysed, carried no DT_RPATH / DT_RUNPATH + NotApplicable, // PE, an archive, or not an image — no such concept + Reported, // Mach-O: entries were read and are reported, bytes untouched + Unanalysed, // recognised as an image, but this module cannot read it +}; + +struct Report { + Outcome outcome = Outcome::NotApplicable; + // What the removed (ELF) or found (Mach-O) entries said, for the log. + std::vector paths; + // Why, when `outcome == Unanalysed`. Empty otherwise. + std::string note; +}; + +std::string_view describe(Outcome o); + +// Remove every loader search-path entry from `image`, in place. +// +// Errors are reserved for "the file is there and something is wrong with it" +// (unreadable, truncated, not writable). A file that simply has no such +// concept — a `.a`, a `.dll`, a `.lib` — is `NotApplicable`, not an error: +// the caller packs every leg through the same step and must not have to know +// the format first. +std::expected strip_search_paths(const std::filesystem::path& image); + +} // namespace mcpp::pack::relocate + +namespace mcpp::pack::relocate { + +namespace { + +// ── endian- and class-aware byte access ───────────────────────────────── +// +// ELF32 and big-endian are not hypothetical for mcpp: a `--target` list can +// name a 32-bit or big-endian triple, and the packer is the same code for all +// of them. Getting this wrong would corrupt an artifact rather than refuse it, +// so the accessors are parameterised rather than assumed. +struct Bytes { + std::string* d = nullptr; + bool little = true; + + std::uint64_t read(std::size_t off, std::size_t width) const { + std::uint64_t v = 0; + for (std::size_t i = 0; i < width; ++i) { + auto byte = static_cast((*d)[off + (little ? i : width - 1 - i)]); + v |= static_cast(byte) << (8 * i); + } + return v; + } + void write(std::size_t off, std::size_t width, std::uint64_t v) { + for (std::size_t i = 0; i < width; ++i) { + auto byte = static_cast((v >> (8 * i)) & 0xFF); + (*d)[off + (little ? i : width - 1 - i)] = byte; + } + } + bool has(std::size_t off, std::size_t len) const { + return d && off <= d->size() && d->size() - off >= len; + } +}; + +std::optional slurp(const std::filesystem::path& p) { + std::ifstream in(p, std::ios::binary); + if (!in) return std::nullopt; + std::ostringstream ss; + ss << in.rdbuf(); + if (!in && !in.eof()) return std::nullopt; + return ss.str(); +} + +bool spit(const std::filesystem::path& p, const std::string& data) { + std::ofstream out(p, std::ios::binary | std::ios::trunc); + if (!out) return false; + out.write(data.data(), static_cast(data.size())); + return static_cast(out); +} + +// Read a NUL-terminated string at `off`, bounded by the file. +std::string cstr_at(const std::string& d, std::size_t off) { + if (off >= d.size()) return {}; + auto end = d.find('\0', off); + return d.substr(off, end == std::string::npos ? std::string::npos : end - off); +} + +constexpr std::uint64_t kDtNull = 0; +constexpr std::uint64_t kDtStrtab = 5; +constexpr std::uint64_t kDtRpath = 15; +constexpr std::uint64_t kDtRunpath = 29; +constexpr std::uint32_t kPtLoad = 1; +constexpr std::uint32_t kPtDynamic = 2; + +std::expected strip_elf(const std::filesystem::path& image, + std::string data) +{ + const bool elf64 = static_cast(data[4]) == 2; + const std::uint8_t dataEnc = static_cast(data[5]); + if (dataEnc != 1 && dataEnc != 2) + return std::unexpected(std::format( + "'{}' declares an unknown ELF data encoding ({})", image.string(), dataEnc)); + + Bytes b{ &data, dataEnc == 1 }; + const std::size_t ptrW = elf64 ? 8 : 4; + + // e_phoff / e_phentsize / e_phnum — the two layouts differ only in offsets. + const std::size_t phoffOff = elf64 ? 0x20 : 0x1C; + const std::size_t phentOff = elf64 ? 0x36 : 0x2A; + const std::size_t phnumOff = elf64 ? 0x38 : 0x2C; + if (!b.has(phnumOff, 2)) return std::unexpected(std::format( + "'{}' is truncated before its ELF header ends", image.string())); + + const auto phoff = b.read(phoffOff, ptrW); + const auto phent = b.read(phentOff, 2); + const auto phnum = b.read(phnumOff, 2); + if (phoff == 0 || phent == 0 || phnum == 0) + return Report{ Outcome::NothingToDo, {}, {} }; // no program headers: no PT_DYNAMIC + + // Program-header field offsets. p_offset/p_vaddr/p_filesz sit in different + // places in the two classes because ELF64 moved p_flags forward. + const std::size_t poOff = elf64 ? 0x08 : 0x04; + const std::size_t pvOff = elf64 ? 0x10 : 0x08; + const std::size_t pfOff = elf64 ? 0x20 : 0x10; + + std::optional dynOff, dynSize; + struct Load { std::uint64_t off, vaddr, filesz; }; + std::vector loads; + for (std::uint64_t i = 0; i < phnum; ++i) { + const auto ph = phoff + i * phent; + if (!b.has(static_cast(ph), static_cast(phent))) + return std::unexpected(std::format( + "'{}' has a program-header table that runs past the file", + image.string())); + const auto type = static_cast(b.read(static_cast(ph), 4)); + if (type == kPtDynamic) { + dynOff = b.read(static_cast(ph + poOff), ptrW); + dynSize = b.read(static_cast(ph + pfOff), ptrW); + } else if (type == kPtLoad) { + loads.push_back({ b.read(static_cast(ph + poOff), ptrW), + b.read(static_cast(ph + pvOff), ptrW), + b.read(static_cast(ph + pfOff), ptrW) }); + } + } + if (!dynOff || !dynSize) return Report{ Outcome::NothingToDo, {}, {} }; + + // Walk the Elf*_Dyn array up to and including its DT_NULL terminator. + const std::size_t slotW = ptrW * 2; + std::vector> slots; // (tag, value) + for (std::uint64_t at = *dynOff; at + slotW <= *dynOff + *dynSize; at += slotW) { + if (!b.has(static_cast(at), slotW)) + return std::unexpected(std::format( + "'{}' has a PT_DYNAMIC that runs past the file", image.string())); + auto tag = b.read(static_cast(at), ptrW); + auto val = b.read(static_cast(at + ptrW), ptrW); + slots.push_back({ tag, val }); + if (tag == kDtNull) break; + } + if (slots.empty()) return Report{ Outcome::NothingToDo, {}, {} }; + + // DT_STRTAB is a VIRTUAL address; map it through PT_LOAD so the removed + // paths can be named in the log. Purely for reporting — a failure to map + // it must not stop the removal. + std::optional strtabFileOff; + for (auto const& [tag, val] : slots) { + if (tag != kDtStrtab) continue; + for (auto const& l : loads) { + if (val >= l.vaddr && val - l.vaddr < l.filesz) { + strtabFileOff = l.off + (val - l.vaddr); + break; + } + } + } + + Report report; + std::vector> kept; + for (auto const& slot : slots) { + if (slot.first == kDtRpath || slot.first == kDtRunpath) { + if (strtabFileOff) + report.paths.push_back(cstr_at(data, static_cast(*strtabFileOff + slot.second))); + continue; + } + kept.push_back(slot); + } + if (kept.size() == slots.size()) return Report{ Outcome::NothingToDo, {}, {} }; + + // Same byte length: the freed slots become DT_NULL padding, which is what + // a linker writes there anyway. Nothing after PT_DYNAMIC moves, so no + // section header, segment offset or relocation needs touching. + kept.resize(slots.size(), { kDtNull, 0 }); + for (std::size_t i = 0; i < kept.size(); ++i) { + const auto at = *dynOff + i * slotW; + b.write(static_cast(at), ptrW, kept[i].first); + b.write(static_cast(at + ptrW), ptrW, kept[i].second); + } + + if (!spit(image, data)) + return std::unexpected(std::format("cannot write '{}'", image.string())); + report.outcome = Outcome::Removed; + return report; +} + +// Mach-O: READ ONLY. +// +// The same defect can exist here as LC_RPATH, and the same edit is possible in +// principle — but mcpp has no macOS artifact to measure it on in the suite +// that gates this code, and shipping an untested byte-editor for a format +// whose load commands carry their own sizes is how a package becomes +// unloadable instead of unportable. So this reads them and hands them back; +// the caller decides what to say. `install_name_tool -delete_rpath` is the +// documented manual step until a macOS-gated test exists. +std::expected report_macho(const std::filesystem::path& image, + const std::string& d) +{ + constexpr std::uint32_t kMagic64 = 0xFEEDFACFu, kMagic32 = 0xFEEDFACEu; + constexpr std::uint32_t kCigam64 = 0xCFFAEDFEu, kCigam32 = 0xCEFAEDFEu; + constexpr std::uint32_t kLcRpath = 0x1Cu; + if (d.size() < 32) return std::unexpected(std::format( + "'{}' is too short to be a Mach-O image", image.string())); + + auto raw32 = [&](std::size_t off, bool little) { + std::uint32_t v = 0; + for (std::size_t i = 0; i < 4; ++i) + v |= static_cast(static_cast(d[off + (little ? i : 3 - i)])) << (8 * i); + return v; + }; + const auto magicLE = raw32(0, true); + bool little = true, wide = true; + if (magicLE == kMagic64) { little = true; wide = true; } + else if (magicLE == kMagic32) { little = true; wide = false; } + else if (magicLE == kCigam64) { little = false; wide = true; } + else if (magicLE == kCigam32) { little = false; wide = false; } + else return Report{ Outcome::Unanalysed, {}, + "a universal (fat) Mach-O; its per-architecture images are not walked" }; + + const auto ncmds = raw32(16, little); + const auto sizeofcmds = raw32(20, little); + std::size_t at = wide ? 32 : 28; + if (at + sizeofcmds > d.size()) return std::unexpected(std::format( + "'{}' has a Mach-O load-command table that runs past the file", image.string())); + + Report report; + for (std::uint32_t i = 0; i < ncmds; ++i) { + if (at + 8 > d.size()) break; + const auto cmd = raw32(at, little); + const auto cmdsize = raw32(at + 4, little); + if (cmdsize < 8 || at + cmdsize > d.size()) break; + if (cmd == kLcRpath) { + const auto strOff = raw32(at + 8, little); + if (strOff < cmdsize) report.paths.push_back(cstr_at(d, at + strOff)); + } + at += cmdsize; + } + report.outcome = Outcome::Reported; + return report; +} + +} // namespace + +std::string_view describe(Outcome o) { + switch (o) { + case Outcome::Removed: return "removed"; + case Outcome::NothingToDo: return "already clean"; + case Outcome::NotApplicable: return "not applicable"; + case Outcome::Reported: return "reported"; + case Outcome::Unanalysed: return "not analysed"; + } + return "not analysed"; +} + +std::expected strip_search_paths(const std::filesystem::path& image) +{ + std::error_code ec; + if (!std::filesystem::is_regular_file(image, ec)) + return std::unexpected(std::format("'{}' is not a file", image.string())); + + auto data = slurp(image); + if (!data) return std::unexpected(std::format("cannot read '{}'", image.string())); + if (data->size() < 8) return Report{ Outcome::NotApplicable, {}, {} }; + + // The format is asked of the BYTES, never of the host or of the file + // extension: a Linux host packing a `--target x86_64-windows-gnu` leg holds + // a PE, and a `.so` built for another architecture is still an ELF. + switch (mcpp::pack::binfmt::identify(image).format) { + case mcpp::pack::binfmt::Format::Elf: + return strip_elf(image, std::move(*data)); + case mcpp::pack::binfmt::Format::MachO: + return report_macho(image, *data); + case mcpp::pack::binfmt::Format::Pe: + // PE has no loader search path baked into the image at all — the + // DLL search order is the process's, not the file's. Nothing to + // remove, and that is an answer rather than a gap. + return Report{ Outcome::NotApplicable, {}, {} }; + case mcpp::pack::binfmt::Format::Unknown: + break; + } + // A static archive lands here, and so does anything else that is not an + // image. Both are `NotApplicable`: an archive's members are relocatable + // objects, which carry no dynamic section to begin with. + return Report{ Outcome::NotApplicable, {}, {} }; +} + +} // namespace mcpp::pack::relocate diff --git a/src/pack/strip.cppm b/src/pack/strip.cppm new file mode 100644 index 00000000..637ec2e7 --- /dev/null +++ b/src/pack/strip.cppm @@ -0,0 +1,268 @@ +// mcpp.pack.strip — removing debug information from a SHIPPED artifact. +// +// WHY A PACKAGING STEP AND NOT `[profile.].strip` +// +// mcpp already has a `strip` axis: `[profile.dist].strip` appends `-s` to the +// LINK. It cannot do this job, and the reason is not a detail: +// +// * a `-s` link never touches a static archive, and a static archive is what +// most library packages ship; +// * `-s` is strip-ALL, which is the one thing an archive must not have done +// to it (below); +// * a link-time strip discards the symbols instead of separating them, so +// there is no `.debug` file to keep. +// +// So the two are different decisions with different inputs, and they are named +// differently: `[profile].strip` is "how is this built", `[pack].strip` is +// "what travels". Documented in docs/02 and docs/12. +// +// ⚠️ THE DIVISION BY ARTIFACT SHAPE IS LOAD-BEARING, AND IT IS MEASURED +// +// `strip --strip-all` on a `.a` removes the ARCHIVE SYMBOL INDEX, and the +// package then fails at the consumer's link with a message that names neither +// strip nor the publisher: +// +// ld: ./libdep.a: error adding symbols: archive has no index; run ranlib to add one +// +// while `--strip-debug` on the same archive links and runs (measured: 2988 → +// 1244 bytes, `ok=42`). A shared library is the opposite case — it may lose +// its `.symtab` but must keep `.dynsym`, which is what `--strip-unneeded` +// means. So there is a table, and it is dh_strip's: +// +// executable --strip-all (nothing links against it) +// shared library --strip-unneeded (keeps .dynsym = the exports) +// static archive --strip-debug (keeps .symtab = the archive index) +// + --enable-deterministic-archives +// +// `--remove-section=.comment --remove-section=.note` goes on all three, also +// from dh_strip. It does NOT match `.note.gnu.build-id`: section removal is by +// exact name, so the build-id survives and `--add-gnu-debuglink` still has +// something to pair with. +// +// EMPTY TOOL IS NOT ALWAYS AN ERROR, AND WHICH CASE IT IS BELONGS TO THE +// TARGET FORMAT — NOT TO THE COMPILER +// +// Two of the three formats keep debug information OUTSIDE the image, and for +// them an absent `strip` is the right answer rather than a missing dependency: +// +// PE/MSVC a separate `.pdb`, by design. +// Mach-O the linked image carries a DEBUG MAP (N_OSO stanzas naming the +// `.o` files); the DWARF itself never enters the `.dylib` unless +// `dsymutil` is run, and what it then produces is a `.dSYM` +// bundle beside the image. `strip` there would remove the symbol +// table — a different thing — and `objcopy --only-keep-debug` has +// nothing to copy. +// +// ELF and PE/MinGW carry DWARF inside the image, so a missing `strip` there is +// a REFUSAL — the same stance `run_library_pack` already takes for a missing +// archiver, and for the same reason: shipping the artifact anyway is the +// silent-wrong-answer this feature exists to remove. +// +// ⚠️ KEYING THIS ON THE COMPILER WOULD BE WRONG IN BOTH DIRECTIONS. clang +// targeting `x86_64-windows-msvc` produces `.pdb` debug info and would be +// asked to strip in-band; Apple's clang ships no `llvm-strip`, so a +// compiler-keyed rule would REFUSE every `mcpp pack` on macOS for a format that +// has nothing to strip in the first place. + +export module mcpp.pack.strip; + +import std; +import mcpp.pack.binfmt; +import mcpp.platform; + +export namespace mcpp::pack { + +// What the artifact IS. Taken from the link unit's kind, never guessed from +// the extension: `libfoo.a` and `foo.lib` are the same shape under different +// names, and a `.so` that is really a linker script is neither. +enum class ArtifactShape { Executable, SharedLibrary, StaticArchive }; + +// The tools for ONE leg, resolved from that leg's toolchain by the caller +// (mcpp.toolchain.binutils_tool). Both may be empty; see the header for when +// that is an answer and when it is a refusal. +struct StripTools { + std::filesystem::path strip; + std::filesystem::path objcopy; + // Does this leg's format carry debug information inside the image? + // Ask `debug_info_is_in_band` rather than filling this by hand — see the + // header for the two formats where the answer is no, and for why the + // question is about the TARGET and not about the compiler. + bool inBandDebugInfo = true; +}; + +// Is debug information carried INSIDE an image built for `canonicalTriple`? +// +// A string question about the canonical triple (`arch-os[-env]`), deliberately: +// this module knows nothing about toolchains, and both packers have already +// resolved that triple by the time they ask. +bool debug_info_is_in_band(std::string_view canonicalTriple); + +enum class StripOutcome { Stripped, NotApplicable }; + +struct StripResult { + StripOutcome outcome = StripOutcome::NotApplicable; + std::filesystem::path debugFile; // written only when a debug dir was given + std::uintmax_t before = 0; + std::uintmax_t after = 0; +}; + +// The strip arguments for `shape`. Exported for its unit test — the table +// above is the whole feature, and a test that has to spawn a real `strip` to +// read it back would run on one platform out of three. +std::vector strip_args(ArtifactShape shape); + +// Separate the debug information (when `debugDir` is non-empty), then strip. +// +// ORDER IS NOT FREE: `--only-keep-debug` has to read the artifact while it +// still has its debug sections, and `--add-gnu-debuglink` has to write into +// the artifact after they are gone. Doing it the other way round produces a +// `.debug` file with no debug information in it and no diagnostic. +std::expected +strip_artifact(const std::filesystem::path& artifact, + ArtifactShape shape, + const StripTools& tools, + const std::filesystem::path& debugDir); + +} // namespace mcpp::pack + +namespace mcpp::pack { + +namespace { + +std::expected run_tool(const std::string& cmd) { + auto r = mcpp::platform::process::capture(cmd + " 2>&1"); + if (r.exit_code != 0) + return std::unexpected(std::format( + " command: {}\n output : {}", cmd, r.output)); + return {}; +} + +std::string join_quoted(const std::filesystem::path& tool, + const std::vector& args, + const std::filesystem::path& file) +{ + std::string cmd = mcpp::platform::shell::quote(tool.string()); + for (auto const& a : args) cmd += " " + mcpp::platform::shell::quote(a); + cmd += " " + mcpp::platform::shell::quote(file.string()); + return cmd; +} + +std::uintmax_t size_of(const std::filesystem::path& p) { + std::error_code ec; + auto n = std::filesystem::file_size(p, ec); + return ec ? 0 : n; +} + +} // namespace + +bool debug_info_is_in_band(std::string_view canonicalTriple) { + // `arch-os[-env]`. Splitting rather than substring-matching: an arch or a + // vendor segment could contain either of these words, and mcpp has been + // bitten before by a triple predicate that answered on a substring. + std::vector seg; + for (std::size_t i = 0; i <= canonicalTriple.size(); ) { + auto j = canonicalTriple.find('-', i); + if (j == std::string_view::npos) { seg.push_back(canonicalTriple.substr(i)); break; } + seg.push_back(canonicalTriple.substr(i, j - i)); + i = j + 1; + } + if (seg.size() >= 2 && seg[1] == "macos") return false; // debug map + .dSYM + if (seg.size() >= 3 && seg[2] == "msvc") return false; // separate .pdb + return true; +} + +std::vector strip_args(ArtifactShape shape) { + // dh_strip's own division. See the header for the measurement behind the + // archive row — it is the one that turns a package into an unlinkable one. + std::vector args{ "--remove-section=.comment", + "--remove-section=.note" }; + switch (shape) { + case ArtifactShape::Executable: + args.push_back("--strip-all"); + break; + case ArtifactShape::SharedLibrary: + args.push_back("--strip-unneeded"); + break; + case ArtifactShape::StaticArchive: + args.push_back("--strip-debug"); + // Zero the member uid/gid/timestamps while we are rewriting the + // archive anyway: two identical packs should produce identical + // bytes, and this is the one place the packer touches them. + args.push_back("--enable-deterministic-archives"); + break; + } + return args; +} + +std::expected +strip_artifact(const std::filesystem::path& artifact, + ArtifactShape shape, + const StripTools& tools, + const std::filesystem::path& debugDir) +{ + StripResult out; + std::error_code ec; + if (!std::filesystem::is_regular_file(artifact, ec)) + return std::unexpected(std::format("'{}' is not a file", artifact.string())); + + if (!tools.inBandDebugInfo) { + // PE/MSVC: the `.pdb` is a separate file and was never inside this one. + out.outcome = StripOutcome::NotApplicable; + return out; + } + if (tools.strip.empty()) + return std::unexpected(std::format( + "no `strip` was resolved for this target, so '{}' would ship with its " + "debug information and the publisher's absolute source paths.\n" + " Pass `--no-strip` to ship it as built, or install the binutils that " + "match this toolchain.", + artifact.filename().string())); + + out.before = size_of(artifact); + + // ── separate, when asked ────────────────────────────────────────── + // + // Not for archives: a `.a` is a container of relocatable objects and + // `--only-keep-debug` on one produces something no debugger consumes. + // dh_strip makes the same exclusion. + const bool separate = !debugDir.empty() && shape != ArtifactShape::StaticArchive; + if (separate) { + if (tools.objcopy.empty()) + return std::unexpected(std::format( + "`--debug-symbols` was given but no `objcopy` was resolved for this " + "target, so the debug information for '{}' cannot be separated.", + artifact.filename().string())); + std::filesystem::create_directories(debugDir, ec); + if (ec) return std::unexpected(std::format( + "cannot create '{}': {}", debugDir.string(), ec.message())); + out.debugFile = debugDir / (artifact.filename().string() + ".debug"); + auto cmd = std::format("{} --only-keep-debug {} {}", + mcpp::platform::shell::quote(tools.objcopy.string()), + mcpp::platform::shell::quote(artifact.string()), + mcpp::platform::shell::quote(out.debugFile.string())); + if (auto r = run_tool(cmd); !r) return std::unexpected(std::format( + "cannot separate debug information from '{}'.\n{}", + artifact.string(), r.error())); + } + + // ── strip ───────────────────────────────────────────────────────── + if (auto r = run_tool(join_quoted(tools.strip, strip_args(shape), artifact)); !r) + return std::unexpected(std::format( + "cannot strip '{}'.\n{}", artifact.string(), r.error())); + + // ── and point the stripped artifact back at its symbols ─────────── + if (separate) { + auto cmd = std::format("{} --add-gnu-debuglink={} {}", + mcpp::platform::shell::quote(tools.objcopy.string()), + mcpp::platform::shell::quote(out.debugFile.string()), + mcpp::platform::shell::quote(artifact.string())); + if (auto r = run_tool(cmd); !r) return std::unexpected(std::format( + "cannot add the debug link to '{}'.\n{}", artifact.string(), r.error())); + } + + out.after = size_of(artifact); + out.outcome = StripOutcome::Stripped; + return out; +} + +} // namespace mcpp::pack diff --git a/src/toolchain/clang.cppm b/src/toolchain/clang.cppm index c7438745..3c7f6034 100644 --- a/src/toolchain/clang.cppm +++ b/src/toolchain/clang.cppm @@ -43,7 +43,6 @@ std::vector std_compat_build_commands(const Toolchain& tc, std::string_view sysrootFlag, std::string_view cppStandardFlag); -std::filesystem::path archive_tool(const Toolchain& tc); // Locate clang-scan-deps in the same bin/ directory as clang++. std::optional find_scan_deps(const Toolchain& tc); @@ -244,13 +243,6 @@ std::vector std_module_build_commands(const Toolchain& tc, #endif } -std::filesystem::path archive_tool(const Toolchain& tc) { - auto llvmAr = tc.binaryPath.parent_path() / - (std::string("llvm-ar") + std::string(mcpp::platform::exe_suffix)); - if (std::filesystem::exists(llvmAr)) return llvmAr; - return {}; -} - std::optional find_scan_deps(const Toolchain& tc) { auto p = tc.binaryPath.parent_path() / (std::string("clang-scan-deps") + std::string(mcpp::platform::exe_suffix)); diff --git a/src/toolchain/registry.cppm b/src/toolchain/registry.cppm index f88a88dc..23e2b67b 100644 --- a/src/toolchain/registry.cppm +++ b/src/toolchain/registry.cppm @@ -215,6 +215,23 @@ struct AvailableIndex { std::vector available_toolchain_indexes(); std::filesystem::path derive_c_compiler(const Toolchain& tc); + +// A binutils-family tool for THIS toolchain's TARGET, named in the GNU +// spelling ("ar", "strip", "objcopy"). +// +// WHY ONE FUNCTION. Four families spell the same tool four ways — llvm- +// beside clang, `-` for a cross, a separate binutils payload for a +// glibc gcc — and until this existed only `ar` knew that. A second tool added +// by copying `archive_tool` would be the same decision derived twice, and the +// copy is the one that silently stops agreeing. +// +// EMPTY IS AN ANSWER, NOT ALWAYS A FAILURE. On MSVC there is no binutils and +// no need for one: PE/MSVC keeps debug information in a separate `.pdb`, so +// there is nothing in-band for `strip` to remove. Callers must distinguish +// "this format has nothing to strip" from "the tool this format needs is +// missing" — see mcpp.pack.strip, which refuses only the second. +std::filesystem::path binutils_tool(const Toolchain& tc, std::string_view name); + std::filesystem::path archive_tool(const Toolchain& tc); std::filesystem::path link_tool(const Toolchain& tc); std::filesystem::path staged_std_bmi_path(const Toolchain& tc, @@ -609,55 +626,69 @@ std::filesystem::path derive_c_compiler(const Toolchain& tc) { return derive_c_compiler_path(tc.binaryPath); } -std::filesystem::path archive_tool(const Toolchain& tc) { - if (tc.compiler == CompilerId::MSVC) { - auto lib = tc.binaryPath.parent_path() / "lib.exe"; - std::error_code ec; - if (std::filesystem::exists(lib, ec)) return lib; +std::filesystem::path binutils_tool(const Toolchain& tc, std::string_view name) { + // MSVC: no binutils, and none wanted — see the declaration. + if (tc.compiler == CompilerId::MSVC) return {}; + + std::error_code ec; + auto dir = tc.binaryPath.parent_path(); + + // Clang ships the whole family as `llvm-` beside the frontend. + if (is_clang(tc)) { + auto llvmTool = dir / (std::string("llvm-") + std::string(name) + + std::string(mcpp::platform::exe_suffix)); + if (std::filesystem::exists(llvmTool, ec)) return llvmTool; return {}; } - if (is_clang(tc)) return mcpp::toolchain::clang::archive_tool(tc); // MinGW bundles its own binutils next to the frontend (self-contained, // like musl) — never an external binutils xpkg. Native (Windows-host) ships // `ar.exe`; the Linux-hosted cross ships the triple-prefixed ELF tool // `x86_64-w64-mingw32-ar`. Try the cross form first, then native. if (is_mingw_target(tc)) { - std::error_code ec; - auto dir = tc.binaryPath.parent_path(); if (!tc.targetTriple.empty()) { - auto crossAr = dir / (tc.targetTriple + "-ar"); - if (std::filesystem::exists(crossAr, ec)) return crossAr; + auto cross = dir / (tc.targetTriple + "-" + std::string(name)); + if (std::filesystem::exists(cross, ec)) return cross; } - auto ar = dir / "ar.exe"; - if (std::filesystem::exists(ar, ec)) return ar; + auto native = dir / (std::string(name) + ".exe"); + if (std::filesystem::exists(native, ec)) return native; return {}; } if (!is_musl_target(tc)) { if (auto binutilsBin = mcpp::toolchain::gcc::find_binutils_bin(tc.binaryPath)) - return *binutilsBin / "ar"; + return *binutilsBin / std::string(name); } - // musl `ar` is the triple-prefixed cross tool (e.g. aarch64-linux-musl-ar), + // A musl tool is the triple-prefixed cross form (e.g. aarch64-linux-musl-ar), // sitting next to the frontend. Derive from the resolved target triple so - // cross targets pick the matching archiver instead of the x86_64 one. - std::string arName = !tc.targetTriple.empty() - ? tc.targetTriple + "-ar" - : "x86_64-linux-musl-ar"; - auto dir = tc.binaryPath.parent_path(); + // cross targets pick the matching tool instead of the x86_64 one. + std::string crossName = (!tc.targetTriple.empty() + ? tc.targetTriple : std::string("x86_64-linux-musl")) + "-" + std::string(name); // Same `.exe` reasoning as the frontend candidates above: a windows-hosted // musl cross payload ships `-ar.exe`. Try it first, then the bare // name (which is what every ELF host has). if constexpr (mcpp::platform::is_windows) { - auto muslArExe = dir / (arName + ".exe"); - if (std::filesystem::exists(muslArExe)) return muslArExe; + auto muslExe = dir / (crossName + ".exe"); + if (std::filesystem::exists(muslExe, ec)) return muslExe; } - auto muslAr = dir / arName; - if (std::filesystem::exists(muslAr)) return muslAr; + auto musl = dir / crossName; + if (std::filesystem::exists(musl, ec)) return musl; return {}; } +std::filesystem::path archive_tool(const Toolchain& tc) { + // The one spelling that is NOT a binutils name: MSVC archives with + // LIB.EXE, which takes a different verb for every operation. + if (tc.compiler == CompilerId::MSVC) { + auto lib = tc.binaryPath.parent_path() / "lib.exe"; + std::error_code ec; + if (std::filesystem::exists(lib, ec)) return lib; + return {}; + } + return binutils_tool(tc, "ar"); +} + std::filesystem::path staged_std_bmi_path(const Toolchain& tc, const std::filesystem::path& outputDir) { if (tc.compiler == CompilerId::MSVC) diff --git a/src/version.cppm b/src/version.cppm index f83be425..c4f41023 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.19.4"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.20.1"; } // namespace mcpp diff --git a/tests/e2e/215_pack_has_no_build_machine_paths.sh b/tests/e2e/215_pack_has_no_build_machine_paths.sh index 24d59d51..1483fd2e 100755 --- a/tests/e2e/215_pack_has_no_build_machine_paths.sh +++ b/tests/e2e/215_pack_has_no_build_machine_paths.sh @@ -29,60 +29,10 @@ TMP=$(mktemp -d) trap "rm -rf $TMP" EXIT export MCPP_HOME=$HOME/.mcpp -read_tag() { -python3 - "$1" <<'PY' -import struct, sys -d = open(sys.argv[1], 'rb').read() -if d[:4] != b'\x7fELF' or d[4] != 2: - print("NOT-ELF64"); raise SystemExit -phoff, = struct.unpack_from(' mcpp.toml <<'EOF' name = "winpack" version = "0.1.0" +# ⚠️ THIS TEST DROPS A FILE INTO THE BUILD TREE AND EXPECTS `pack` TO SEE IT, +# so the two commands have to agree on WHICH build tree that is. +# +# `mcpp pack` builds with the `release` fallback (a packaged artifact leaves +# this machine), while a bare `mcpp build` uses `dev` — two profiles, two +# fingerprint directories, and the stand-in DLL below would land in the one +# `pack` does not use. Nothing about the closure would be wrong; the file would +# simply not be there, and the assertion would read as "the closure reader +# failed". +# +# Stating the profile in the manifest settles it for both, and doubles as a +# check that `[build] default-profile` still outranks pack's fallback. +[build] +default-profile = "dev" + # `force_bundle` reaching a SYSTEM name is what makes the next assertion # positive rather than vacuous — see the comment at the msvcrt.dll check. [pack.bundle-project] diff --git a/tests/e2e/249_pack_workspace_root_unchanged.sh b/tests/e2e/249_pack_workspace_root_unchanged.sh index 6eb9447f..bc8f3cd1 100755 --- a/tests/e2e/249_pack_workspace_root_unchanged.sh +++ b/tests/e2e/249_pack_workspace_root_unchanged.sh @@ -52,6 +52,32 @@ int main() { std::printf("ok=%d\n", core_answer()); return 0; } EOF cd ws + +# ── Mach-O: the routing is the claim, and a bundle cannot be produced ── +# +# `mcpp pack` of a PROGRAM is refused on Mach-O (its dependency closure would +# be resolved by RUNNING the artifact under a linker that ignores the tracing +# variable — see 266). So on macOS the question this file asks becomes: did the +# workspace root still hand through to the MEMBER'S PROGRAM? The refusal names +# the artifact, which is exactly the evidence for that — and it is a different +# message from "this package declares no program and no library to pack", which +# is what a routing regression would produce. +if [[ "$(uname -s)" == "Darwin" ]]; then + if "$MCPP" pack --mode system > pack.log 2>&1; then + cat pack.log + echo "FAIL: a Mach-O program bundle was produced; 266 says it must be refused" + exit 1 + fi + grep -q "Mach-O program 'hello'" pack.log || { + cat pack.log + echo "FAIL: the workspace root did not route to the member's program 'hello'." + echo " (A refusal naming some other artifact, or a 'declares no program'" + echo " error, is the routing regression this file exists to catch.)" + exit 1; } + echo "PASS: a workspace root still routes to its member's program" + exit 0 +fi + "$MCPP" pack --mode system > pack.log 2>&1 || { cat pack.log echo "FAIL: packing from a virtual workspace root stopped working" diff --git a/tests/e2e/250_pack_names_the_target.sh b/tests/e2e/250_pack_names_the_target.sh index 9fa079db..a35b7236 100755 --- a/tests/e2e/250_pack_names_the_target.sh +++ b/tests/e2e/250_pack_names_the_target.sh @@ -42,6 +42,38 @@ EOF cd two +# ── Mach-O: which target was CHOSEN is still checkable ───────────────── +# +# A program bundle cannot be produced on Mach-O (266), so "look inside the +# staging dir" is unavailable here. The refusal names the artifact it got to, +# which answers the same question: `pack beta` must reach beta and `pack alpha` +# must reach alpha. Without this branch the file would simply not run on macOS, +# and the defect it exists for — `mcpp pack app2` bundling app1 — is not a +# platform-specific one. +if [[ "$(uname -s)" == "Darwin" ]]; then + for want in beta alpha; do + other=$([[ "$want" == beta ]] && echo alpha || echo beta) + if "$MCPP" pack "$want" --mode system > "$want.log" 2>&1; then + cat "$want.log"; echo "FAIL: a Mach-O program bundle was produced"; exit 1 + fi + grep -q "Mach-O program '$want'" "$want.log" || { + cat "$want.log" + echo "FAIL: 'mcpp pack $want' did not reach $want"; exit 1; } + grep -q "Mach-O program '$other'" "$want.log" && { + cat "$want.log" + echo "FAIL: 'mcpp pack $want' reached $other instead"; exit 1; } + done + # An unknown name must still fail EARLIER and differently — otherwise the + # two assertions above would pass for a build that refuses everything. + if "$MCPP" pack nosuch --mode system > bad.log 2>&1; then + cat bad.log; echo "FAIL: an unknown target name was accepted"; exit 1 + fi + grep -q "no target named 'nosuch'" bad.log || { + cat bad.log; echo "wrong refusal for an unknown name"; exit 1; } + echo "PASS: mcpp pack reaches the target it is given" + exit 0 +fi + # ── the named program is the one that gets bundled ───────────────────── "$MCPP" pack beta --mode system > beta.log 2>&1 || { cat beta.log; echo "pack beta failed"; exit 1; } staged="$(find target/dist -maxdepth 1 -type d -name 'two-0.1.0*' | head -1)" diff --git a/tests/e2e/264_pack_library_is_relocatable.sh b/tests/e2e/264_pack_library_is_relocatable.sh new file mode 100755 index 00000000..ab14996a --- /dev/null +++ b/tests/e2e/264_pack_library_is_relocatable.sh @@ -0,0 +1,155 @@ +#!/usr/bin/env bash +# requires: pack python3 +# 264_pack_library_is_relocatable.sh — a packed `kind = "shared"` library must +# not carry the BUILD MACHINE's loader search path (issue #460). +# +# WHAT WAS WRONG, AND WHY NOBODY SAW IT +# +# `run_library_pack` copied the built `.so` into the package and did nothing +# else, so it kept the DT_RUNPATH the link gave it: +# +# /registry/data/xpkgs/xim-x-glibc/2.44/lib64 +# : /registry/data/xpkgs/xim-x-gcc/16.1.0/lib64 +# : /registry/subos/default/lib +# +# On another machine those directories do not exist, and — this is the part +# that makes it fatal rather than untidy — a shared object that carries ANY +# DT_RUNPATH makes the loader skip the whole inherited DT_RPATH chain when +# resolving that object's own dependencies. So the consumer's carefully +# computed DT_RPATH (payload + package dir + SubOS farm) is not consulted, and +# the program dies with +# +# error while loading shared libraries: libstdc++.so.6: cannot open shared object file +# +# ⚠️ AND WHY `$ORIGIN` IS NOT THE FIX. Measured: a DT_RUNPATH of `$ORIGIN`, and +# a DT_RUNPATH of the empty string, both fail exactly the same way. It is the +# TAG's presence that disables inheritance, not its contents. The criterion is +# therefore "there is no tag", never "the tag is relative". +# +# ⚠️ AND WHY THIS TEST DELIBERATELY BREAKS THE PACKAGE HALF-WAY THROUGH. +# Asserting only that the fixed package runs cannot tell "the defect is fixed" +# from "this machine happens to satisfy the stale path" — which is precisely +# what made the pre-existing e2e (251) green throughout the bug's life: it +# consumes the package on the machine that built it, where those directories +# are right there. So the defect is put BACK with patchelf, the consumer is +# required to FAIL, and only then is it restored. A guard that cannot observe +# the defect is not a guard. +set -e +source "$(dirname "$0")/_host_path.sh" +source "$(dirname "$0")/_elf_tag.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit-shared] +kind = "shared" +soname = "libmathkit.so.1" +EOF + +cd mathkit +"$MCPP" pack mathkit-shared > pack.log 2>&1 || { cat pack.log; echo "FAIL: pack"; exit 1; } +pkg="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +PKG_HOST="$(host_path "$pkg")" +libdir="$pkg/lib/x86_64-linux-gnu" +[[ -d "$libdir" ]] || libdir="$(dirname "$(find "$pkg/lib" -name 'libmathkit-shared.so' | head -1)")" +so="$libdir/libmathkit-shared.so" +[[ -f "$so" ]] || { find "$pkg" -type f; echo "FAIL: no .so in the package"; exit 1; } + +# ── 1. static: EVERY ELF in the package, and both of the .so's names ──────── +# +# The SONAME alias is checked too, and that is not belt-and-braces: it is a +# symlink here, but on a filesystem where `create_symlink` fails the packer +# falls back to a COPY — and that copy used to be made from the BUILD TREE's +# file rather than from the staged one, i.e. from the unrelocated original, +# under the exact name the loader asks for. +fail=0 +swept=0 +while IFS= read -r obj; do + head -c4 "$obj" 2>/dev/null | grep -q $'\x7fELF' || continue + read -r form tag paths <<<"$(read_tag "$obj")" + [[ "$form" == "NOT-ELF64" ]] && continue + swept=$((swept + 1)) + printf ' %-34s %-14s %s %s\n' "$(basename "$obj")" "$form" "$tag" "$paths" + if [[ "$tag" != "NONE" ]]; then + echo "FAIL: a packaged library carries $tag ($paths)" + echo " A shipped .so must carry NO loader search path: any DT_RUNPATH —" + echo " including \$ORIGIN and the empty string — stops the consumer's" + echo " DT_RPATH from being inherited for this object's dependencies." + fail=1 + fi +done < <(find "$libdir" \( -type f -o -type l \)) +[[ "$swept" -ge 1 ]] || { echo "FAIL: swept no ELF — the test proved nothing"; exit 1; } +[[ "$fail" == "0" ]] || exit 1 + +# ── 2. a consumer of the package starts ──────────────────────────────────── +cd "$TMP" +mkdir -p app/src +cat > app/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("ok=%d\n", mk::answer()); return 0; } +EOF +cat > app/mcpp.toml < build.log 2>&1 ) || { cat app/build.log; echo "FAIL: consumer build"; exit 1; } +exe="$(find app/target -path '*/bin/app' -type f | head -1)" +[[ -x "$exe" ]] || { echo "FAIL: no consumer binary"; exit 1; } + +run_bare() { "$exe" 2>&1; } # NOT `mcpp run`: no injected environment + +out="$(run_bare)" && rc=0 || rc=$? +[[ "$rc" == "0" && "$out" == *"ok=42"* ]] || { + echo "FAIL: the consumer does not start against the packed library" + echo " rc=$rc out=$out"; exit 1; } + +# ── 3. put the defect BACK, and require the consumer to fail ─────────────── +# +# `/nonexistent-machine/...` stands in for "the publisher's store, on somebody +# else's computer". Nothing is rebuilt: only the shipped library's dynamic +# section changes, which is exactly the difference between the two machines. +cp "$so" "$so.packed" +patchelf --set-rpath '/nonexistent-machine/.mcpp/registry/data/xpkgs/xim-x-gcc/16.1.0/lib64' "$so" +out="$(run_bare)" && rc=0 || rc=$? +if [[ "$rc" == "0" ]]; then + echo "FAIL: the consumer still ran with a stale RUNPATH restored on the library." + echo " That means this test cannot observe the defect it exists for —" + echo " the consumer is finding libstdc++ some other way, so a regression" + echo " in the packer would pass unnoticed." + exit 1 +fi +[[ "$out" == *"cannot open shared object file"* ]] || { + echo "FAIL: the restored defect produced an unexpected failure: $out"; exit 1; } +echo " defect restored → consumer fails as expected (rc=$rc)" + +# ── 4. …and restore, so the pass is not an artifact of step 3 ────────────── +cp "$so.packed" "$so" +out="$(run_bare)" && rc=0 || rc=$? +[[ "$rc" == "0" && "$out" == *"ok=42"* ]] || { + echo "FAIL: restoring the packed library did not restore the behaviour" + echo " rc=$rc out=$out"; exit 1; } + +echo "PASS: a packed shared library carries no build-machine search path, and the guard can see the defect" diff --git a/tests/e2e/265_pack_strips_but_stays_usable.sh b/tests/e2e/265_pack_strips_but_stays_usable.sh new file mode 100755 index 00000000..be5351ad --- /dev/null +++ b/tests/e2e/265_pack_strips_but_stays_usable.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +# requires: pack python3 +# 265_pack_strips_but_stays_usable.sh — `mcpp pack` ships stripped artifacts, +# and stripping must not make them unusable. +# +# ⚠️ THE ONE THAT BITES: A STATIC ARCHIVE CANNOT BE `--strip-all`ed. +# +# `strip` with no shape argument defaults to strip-all, and on a `.a` that +# removes the ARCHIVE SYMBOL INDEX. The package then fails at the CONSUMER's +# link with +# +# ld: libmathkit.a: error adding symbols: archive has no index; run ranlib to add one +# +# — a message that names neither `strip` nor the publisher, arriving on a +# machine the publisher does not have. Measured: `--strip-debug` keeps the +# index (2988 → 1244 bytes) and the consumer links and runs. So the packer +# uses dh_strip's division, and this test consumes BOTH shapes rather than +# inspecting them: "the archive still has an index" is a proxy, "a program +# linked against it prints the right number" is the thing. +# +# The three configuration channels are pinned from BOTH sides — a test that +# only checks the default cannot tell "the switch works" from "there is no +# switch". +set -e +source "$(dirname "$0")/_host_path.sh" + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p mathkit/src +cat > mathkit/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > mathkit/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > mathkit/mcpp.toml <<'EOF' +[package] +name = "mathkit" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.mathkit] +kind = "lib" +[targets.mathkit-shared] +kind = "shared" +soname = "libmathkit.so.1" +EOF + +# `file` is not required by the harness; ask the ELF itself instead. +has_symtab() { +python3 - "$1" <<'PY' +import struct, sys +d = open(sys.argv[1], 'rb').read() +if d[:4] != b'\x7fELF' or d[4] != 2: + print("no"); raise SystemExit +shoff, = struct.unpack_from(' — build a program against it and run + local pkg="$1" want="$2" tag="$3" + # Through a named *_HOST variable, not interpolated inline: the manifest + # below is FILE CONTENT, and on Git Bash a shell-spelled /tmp/... path is + # read by a native mcpp.exe as "root of the current drive". 00_fixture_path + # _hygiene enforces the naming so the conversion is visible at the use site. + local PKG_HOST + PKG_HOST="$(host_path "$pkg")" + rm -rf "$TMP/consumer" + mkdir -p "$TMP/consumer/src" + cat > "$TMP/consumer/src/main.cpp" <<'EOF' +#include +import mathkit; +int main(){ std::printf("ok=%d\n", mk::answer()); return 0; } +EOF + cat > "$TMP/consumer/mcpp.toml" < run.log 2>&1 ) || { + cat "$TMP/consumer/run.log" + echo "FAIL[$tag]: a consumer cannot use the stripped package"; exit 1; } + grep -q "$want" "$TMP/consumer/run.log" || { + cat "$TMP/consumer/run.log"; echo "FAIL[$tag]: wrong answer"; exit 1; } +} + +cd mathkit + +# ── 1. static: stripped, and STILL LINKABLE ──────────────────────────────── +rm -rf target/dist +"$MCPP" pack mathkit > pack-a.log 2>&1 || { cat pack-a.log; echo "FAIL: static pack"; exit 1; } +grep -q "Stripped" pack-a.log || { cat pack-a.log; echo "FAIL: the static leg was not stripped"; exit 1; } +pkg_a="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +consume "$pkg_a" "ok=42" "static" +echo " static archive: stripped and still linkable" + +# ── 2. shared: stripped, and STILL LOADABLE ──────────────────────────────── +rm -rf target/dist +"$MCPP" pack mathkit-shared > pack-b.log 2>&1 || { cat pack-b.log; echo "FAIL: shared pack"; exit 1; } +grep -q "Stripped" pack-b.log || { cat pack-b.log; echo "FAIL: the shared leg was not stripped"; exit 1; } +pkg_b="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +so="$(find "$pkg_b/lib" -name 'libmathkit-shared.so' -type f | head -1)" +[[ -n "$so" ]] || { echo "FAIL: no .so"; exit 1; } +# `--strip-unneeded` must leave `.dynsym` — it IS the export list. If this +# regressed to `--strip-all` the link below would still work (ld reads +# `.dynsym`), so the run is what proves it. +[[ "$(has_symtab "$so")" == "no" ]] || { + echo "FAIL: the shared library still carries a symbol/debug table after stripping"; exit 1; } +consume "$pkg_b" "ok=42" "shared" +echo " shared library: stripped and still loadable" + +# ── 3. the OTHER side of every switch ────────────────────────────────────── +rm -rf target/dist +"$MCPP" pack mathkit-shared --no-strip > pack-c.log 2>&1 || { cat pack-c.log; echo "FAIL: --no-strip pack"; exit 1; } +grep -q "Stripped" pack-c.log && { cat pack-c.log; echo "FAIL: --no-strip stripped anyway"; exit 1; } +pkg_c="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +so_c="$(find "$pkg_c/lib" -name 'libmathkit-shared.so' -type f | head -1)" +[[ "$(has_symtab "$so_c")" == "yes" ]] || { + echo "FAIL: --no-strip produced an artifact with no symbols — the flag did nothing" + exit 1; } +echo " --no-strip: symbols kept" + +rm -rf target/dist +cp mcpp.toml mcpp.toml.bak +printf '\n[pack]\nstrip = false\n' >> mcpp.toml +"$MCPP" pack mathkit-shared > pack-d.log 2>&1 || { cat pack-d.log; echo "FAIL: [pack] strip pack"; exit 1; } +grep -q "Stripped" pack-d.log && { cat pack-d.log; echo "FAIL: [pack] strip = false was ignored"; exit 1; } +mv mcpp.toml.bak mcpp.toml +echo " [pack] strip = false: honoured" + +# ── 4. --debug-symbols keeps the symbols, beside the artifact ────────────── +rm -rf target/dist +"$MCPP" pack mathkit-shared --debug-symbols dbg > pack-e.log 2>&1 || { + cat pack-e.log; echo "FAIL: --debug-symbols pack"; exit 1; } +# PER TRIPLE, mirroring `lib//`. A fat package's legs share an artifact +# NAME (`libmathkit-shared.so` for both a gnu and a musl leg is the normal case), +# so a flat debug directory would have the second leg overwrite the first — and +# the first artifact's `.gnu_debuglink` would then resolve to the other target's +# symbols, silently. Asserting the layout is what keeps that from regressing. +dbgfile="$(find dbg -name 'libmathkit-shared.so.debug' | head -1)" +[[ -n "$dbgfile" ]] || { ls -R dbg 2>&1; echo "FAIL: no separated debug file"; exit 1; } +[[ "$dbgfile" == dbg/*/libmathkit-shared.so.debug ]] || { + echo "FAIL: the debug file is not under a per-triple directory: $dbgfile" + echo " A fat package's legs share an artifact name; a flat layout loses one." + exit 1; } +[[ "$(has_symtab "$dbgfile")" == "yes" ]] || { + echo "FAIL: the separated debug file carries no debug information"; exit 1; } +pkg_e="$TMP/mathkit/$(find target/dist -maxdepth 1 -type d -name 'mathkit-0.1.0-*' | head -1)" +so_e="$(find "$pkg_e/lib" -name 'libmathkit-shared.so' -type f | head -1)" +[[ "$(has_symtab "$so_e")" == "no" ]] || { + echo "FAIL: --debug-symbols left the debug information in the shipped artifact too" + exit 1; } +# The link back. Without it a debugger has the file and no way to find it. +python3 - "$so_e" <<'PY' || { echo "FAIL: no .gnu_debuglink in the stripped artifact"; exit 1; } +import struct, sys +d = open(sys.argv[1], 'rb').read() +shoff, = struct.unpack_from('' +# +# That variable is glibc's. dyld has never heard of it (its counterpart is +# `DYLD_PRINT_LIBRARIES`), so on macOS this does not trace anything — IT RUNS +# THE USER'S PROGRAM. Whatever the program prints is then parsed as a +# dependency table, yields nothing, and a bundle containing just the binary is +# written and reported as `Packed`. A program with side effects performs them. +# An interactive one hangs the packer. +# +# The `_WIN32` branch beside it has refused the same class since it was +# written; macOS was simply never checked, because the e2e harness only grants +# the `pack` capability where `elf` + `patchelf` are both present — i.e. Linux. +# So no job in this suite has ever run `mcpp pack` on a Mac. +# +# ⚠️ BOTH SIDES, ON THE SAME HOST. Asserting only the refusal cannot tell "the +# gate works" from "pack is broken on this machine". So the same run also packs +# a LIBRARY target, which takes a different pipeline and must still succeed. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +mkdir -p proj/src +cat > proj/src/mathkit.cppm <<'EOF' +export module mathkit; +export namespace mk { int answer(); } +EOF +cat > proj/src/impl.cpp <<'EOF' +module mathkit; +namespace mk { int answer() { return 42; } } +EOF +cat > proj/src/main.cpp <<'EOF' +#include +import mathkit; +int main(){ std::printf("ok=%d\n", mk::answer()); return 0; } +EOF +cat > proj/mcpp.toml <<'EOF' +[package] +name = "proj" +version = "0.1.0" +[build] +sources = ["src/*.cppm", "src/*.cpp"] +[targets.proj] +kind = "bin" +main = "src/main.cpp" +[targets.mathkit] +kind = "lib" +EOF + +cd proj + +# ── 1. the program is refused, by name ───────────────────────────────────── +if "$MCPP" pack proj > pack.log 2>&1; then + cat pack.log + echo "FAIL: mcpp pack produced a bundle for a Mach-O program." + echo " Its dependency closure cannot be resolved on this platform, so" + echo " whatever it produced is not one — and producing it RAN the program." + exit 1 +fi +grep -qi "Mach-O" pack.log || { + cat pack.log + echo "FAIL: pack failed, but not with the Mach-O refusal — so this test is" + echo " observing some other failure and proves nothing about the gate." + exit 1; } +grep -q "LD_TRACE_LOADED_OBJECTS" pack.log || { + cat pack.log + echo "FAIL: the refusal does not say WHY. A reader has to be able to tell" + echo " this from 'macOS is unsupported in general'." + exit 1; } +echo " a Mach-O program is refused, and the message names the mechanism" + +# ── 2. …and a library target on the same host still packs ────────────────── +"$MCPP" pack mathkit > packlib.log 2>&1 || { + cat packlib.log + echo "FAIL: a library target does not pack on macOS either — the refusal above" + echo " is therefore not evidence of a working gate." + exit 1; } +pkg="$(find target/dist -maxdepth 1 -type d -name 'proj-0.1.0-*' | head -1)" +[[ -n "$(find "$pkg" -name '*.a' | head -1)" ]] || { + find "$pkg" -type f; echo "FAIL: no archive in the library package"; exit 1; } +echo " a library target still packs" + +echo "PASS: mcpp pack refuses a Mach-O program with the reason, and library packaging is unaffected" diff --git a/tests/e2e/_elf_tag.sh b/tests/e2e/_elf_tag.sh new file mode 100644 index 00000000..325cd9f9 --- /dev/null +++ b/tests/e2e/_elf_tag.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# _elf_tag.sh — read an ELF's FORM and its loader search-path TAG. +# +# `read_tag ` prints three fields: +# +#
+# +# form executable | shared_library | NOT-ELF64 +# tag RPATH | RUNPATH | BOTH | NONE +# paths ':'-joined contents of whichever tag is present +# +# ⚠️ WHY A PARSER AND NOT `strings` +# +# Removing DT_RPATH/DT_RUNPATH removes the ENTRY, not the string it pointed at: +# `.dynstr` is tail-merged by the linker, so a shorter live string can begin +# inside the dead one and deleting those bytes cannot be shown safe. Measured: +# `patchelf --remove-rpath` leaves the identical residue at the identical file +# size, so this is the reference behaviour, not mcpp's shortcut. +# +# Consequence: "this artifact carries no build-machine path" is a question about +# the DYNAMIC ENTRIES. A `grep` over the file's bytes answers a different +# question and reports a correctly relocated artifact as dirty — and it would +# also flip to "clean" for an unrelated reason the day `mcpp pack` started +# stripping DWARF. One check, one thing. +# +# Shared by 215 (application bundles) and 264 (library packages) so the two +# cannot drift into disagreeing about what the criterion is. + +read_tag() { +python3 - "$1" <<'PY' +import struct, sys +d = open(sys.argv[1], 'rb').read() +if d[:4] != b'\x7fELF' or d[4] != 2: + print("NOT-ELF64"); raise SystemExit +phoff, = struct.unpack_from(' + +import std; +import mcpp.pack.relocate; +import mcpp.pack.strip; + +// mcpp.pack.relocate / mcpp.pack.strip — the two halves of "the build machine +// does not travel", tested where CI cannot otherwise reach them. +// +// WHY SYNTHETIC ELF BYTES RATHER THAN A COMPILED .so +// +// The e2e suite already packs a real library and reads the result back, so the +// x86_64/little-endian path has an end-to-end judge. What it CANNOT reach is +// ELF32 and big-endian: no CI job produces one, and `mcpp pack --target` will +// hand exactly those to this code the first time somebody packages for a +// 32-bit or MIPS/PowerPC target. A wrong width or a wrong byte order there +// does not fail — it writes a corrupt dynamic section, which is the failure +// class this whole area exists to remove. +// +// The images below are the smallest thing that is still an ELF for the purpose +// of this question: a header, a PT_LOAD so DT_STRTAB can be mapped, a +// PT_DYNAMIC, and a string table. They are read back by a parser written here, +// deliberately NOT by the module under test — checking an editor with its own +// reader proves only that it is self-consistent. + +namespace { + +struct Dyn { std::uint64_t tag, val; }; + +// A minimal but well-formed ELF image carrying `entries`. +struct SyntheticElf { + bool wide; // true = ELF64 + bool little; + std::string bytes; + + std::size_t ptrW() const { return wide ? 8 : 4; } + std::size_t slotW() const { return ptrW() * 2; } + std::size_t dynOff = 0; + std::size_t strOff = 0; +}; + +void put(std::string& b, std::size_t off, std::size_t width, + std::uint64_t v, bool little) +{ + for (std::size_t i = 0; i < width; ++i) + b[off + (little ? i : width - 1 - i)] = + static_cast((v >> (8 * i)) & 0xFF); +} + +std::uint64_t get(const std::string& b, std::size_t off, std::size_t width, + bool little) +{ + std::uint64_t v = 0; + for (std::size_t i = 0; i < width; ++i) + v |= static_cast( + static_cast(b[off + (little ? i : width - 1 - i)])) + << (8 * i); + return v; +} + +// `strings` are laid out in order; a Dyn whose tag is in `stringTags` has its +// value replaced by that string's offset. +SyntheticElf make_elf(bool wide, bool little, + std::vector entries, + const std::vector& strings, + const std::vector& stringSlots) +{ + SyntheticElf e{ wide, little, {}, }; + const std::size_t ehSize = wide ? 64 : 52; + const std::size_t phEnt = wide ? 56 : 32; + const std::size_t phNum = 2; + const std::size_t phOff = ehSize; + const std::size_t dynOff = phOff + phEnt * phNum; + const std::size_t slotW = (wide ? 8 : 4) * 2; + const std::size_t dynSize = slotW * entries.size(); + const std::size_t strOff = dynOff + dynSize; + + std::string strtab; + std::vector strAt; + strtab.push_back('\0'); // index 0 is the empty string + for (auto const& s : strings) { strAt.push_back(strtab.size()); strtab += s; strtab.push_back('\0'); } + + e.bytes.assign(strOff + strtab.size(), '\0'); + e.dynOff = dynOff; + e.strOff = strOff; + + // ── ELF header ──────────────────────────────────────────────────── + e.bytes[0] = 0x7F; e.bytes[1] = 'E'; e.bytes[2] = 'L'; e.bytes[3] = 'F'; + e.bytes[4] = static_cast(wide ? 2 : 1); + e.bytes[5] = static_cast(little ? 1 : 2); + e.bytes[6] = 1; + put(e.bytes, 16, 2, 3, little); // e_type = ET_DYN + put(e.bytes, 18, 2, wide ? 62 : 3, little); // e_machine + put(e.bytes, wide ? 0x20 : 0x1C, wide ? 8 : 4, phOff, little); + put(e.bytes, wide ? 0x36 : 0x2A, 2, phEnt, little); + put(e.bytes, wide ? 0x38 : 0x2C, 2, phNum, little); + + // ── PT_LOAD covering the whole image, vaddr == file offset ──────── + const std::size_t poOff = wide ? 0x08 : 0x04; + const std::size_t pvOff = wide ? 0x10 : 0x08; + const std::size_t pfOff = wide ? 0x20 : 0x10; + put(e.bytes, phOff, 4, 1, little); // PT_LOAD + put(e.bytes, phOff + poOff, wide ? 8 : 4, 0, little); + put(e.bytes, phOff + pvOff, wide ? 8 : 4, 0, little); + put(e.bytes, phOff + pfOff, wide ? 8 : 4, e.bytes.size(), little); + + // ── PT_DYNAMIC ──────────────────────────────────────────────────── + put(e.bytes, phOff + phEnt, 4, 2, little); // PT_DYNAMIC + put(e.bytes, phOff + phEnt + poOff, wide ? 8 : 4, dynOff, little); + put(e.bytes, phOff + phEnt + pfOff, wide ? 8 : 4, dynSize, little); + + // ── the dynamic array ───────────────────────────────────────────── + for (std::size_t i = 0; i < entries.size(); ++i) { + auto value = entries[i].val; + for (std::size_t k = 0; k < stringSlots.size(); ++k) + if (stringSlots[k] == i) value = strAt[k]; + put(e.bytes, dynOff + i * slotW, wide ? 8 : 4, entries[i].tag, little); + put(e.bytes, dynOff + i * slotW + (wide ? 8 : 4), wide ? 8 : 4, value, little); + } + // DT_STRTAB's value is a VADDR, and vaddr == file offset here by construction. + for (std::size_t i = 0; i < entries.size(); ++i) + if (entries[i].tag == 5) + put(e.bytes, dynOff + i * slotW + (wide ? 8 : 4), wide ? 8 : 4, strOff, little); + + std::memcpy(e.bytes.data() + strOff, strtab.data(), strtab.size()); + return e; +} + +// An independent reader: what tags does this image's dynamic array carry? +std::vector read_tags(const std::string& bytes, std::size_t dynOff, + bool wide, bool little) +{ + const std::size_t ptrW = wide ? 8 : 4; + std::vector tags; + for (std::size_t at = dynOff; at + ptrW * 2 <= bytes.size(); at += ptrW * 2) { + auto tag = get(bytes, at, ptrW, little); + tags.push_back(tag); + if (tag == 0) break; + } + return tags; +} + +std::filesystem::path write_temp(const std::string& bytes, std::string_view stem) { + auto dir = std::filesystem::temp_directory_path() / "mcpp-relocate-test"; + std::filesystem::create_directories(dir); + auto p = dir / std::string(stem); + std::ofstream out(p, std::ios::binary | std::ios::trunc); + out.write(bytes.data(), static_cast(bytes.size())); + out.close(); + return p; +} + +constexpr std::uint64_t kNull = 0, kNeeded = 1, kStrtab = 5, kSoname = 14, + kRpath = 15, kRunpath = 29; + +} // namespace + +// ── the ELF editor ────────────────────────────────────────────────────── + +TEST(PackRelocate, RemovesRunpathFromElf64LittleEndian) { + auto e = make_elf(/*wide=*/true, /*little=*/true, + { {kNeeded, 0}, {kStrtab, 0}, {kRunpath, 0}, {kSoname, 0}, {kNull, 0} }, + { "libc.so.6", "/home/builder/.mcpp/store/lib64", "libfoo.so.1" }, + { 0, 2, 3 }); + const auto before = e.bytes.size(); + auto path = write_temp(e.bytes, "elf64le.so"); + + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()) << (r ? "" : r.error()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::Removed); + ASSERT_EQ(r->paths.size(), 1u); + // The report names what was dropped — a log line that says only "removed" + // cannot tell a stale store path from `$ORIGIN`. + EXPECT_EQ(r->paths[0], "/home/builder/.mcpp/store/lib64"); + + std::ifstream in(path, std::ios::binary); + std::string after{ std::istreambuf_iterator(in), {} }; + // SAME LENGTH: the freed slot becomes DT_NULL padding, so nothing after + // PT_DYNAMIC moves and no offset in the file needs fixing up. + EXPECT_EQ(after.size(), before); + + auto tags = read_tags(after, e.dynOff, true, true); + EXPECT_EQ(std::ranges::count(tags, kRunpath), 0); + // The other entries survive, in order. Losing DT_SONAME here would make the + // library unfindable by the name its consumers link against. + EXPECT_EQ(std::ranges::count(tags, kNeeded), 1); + EXPECT_EQ(std::ranges::count(tags, kSoname), 1); + EXPECT_EQ(std::ranges::count(tags, kStrtab), 1); +} + +TEST(PackRelocate, RemovesRpathFromElf32BigEndian) { + // No CI job produces this shape, and `--target` can. A width or byte-order + // mistake would corrupt the image rather than fail. + auto e = make_elf(/*wide=*/false, /*little=*/false, + { {kNeeded, 0}, {kStrtab, 0}, {kRpath, 0}, {kNull, 0} }, + { "libc.so.6", "/home/builder/store/lib" }, + { 0, 2 }); + const auto before = e.bytes.size(); + auto path = write_temp(e.bytes, "elf32be.so"); + + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()) << (r ? "" : r.error()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::Removed); + ASSERT_EQ(r->paths.size(), 1u); + EXPECT_EQ(r->paths[0], "/home/builder/store/lib"); + + std::ifstream in(path, std::ios::binary); + std::string after{ std::istreambuf_iterator(in), {} }; + EXPECT_EQ(after.size(), before); + auto tags = read_tags(after, e.dynOff, false, false); + EXPECT_EQ(std::ranges::count(tags, kRpath), 0); + EXPECT_EQ(std::ranges::count(tags, kNeeded), 1); +} + +TEST(PackRelocate, RemovesBothTagsWhenAnImageCarriesBoth) { + // `Both` is what a linker driven with `--disable-new-dtags` after an + // explicit -rpath can produce, and the loader honours DT_RUNPATH there. + // Leaving either one behind leaves the defect behind. + auto e = make_elf(true, true, + { {kStrtab, 0}, {kRpath, 0}, {kRunpath, 0}, {kNeeded, 0}, {kNull, 0} }, + { "/a/store/lib", "/b/farm/lib", "libc.so.6" }, + { 1, 2, 3 }); + auto path = write_temp(e.bytes, "elfboth.so"); + + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::Removed); + EXPECT_EQ(r->paths.size(), 2u); + + std::ifstream in(path, std::ios::binary); + std::string after{ std::istreambuf_iterator(in), {} }; + auto tags = read_tags(after, e.dynOff, true, true); + EXPECT_EQ(std::ranges::count(tags, kRpath), 0); + EXPECT_EQ(std::ranges::count(tags, kRunpath), 0); +} + +TEST(PackRelocate, AlreadyCleanIsNotTheSameAsNotChecked) { + auto e = make_elf(true, true, + { {kNeeded, 0}, {kStrtab, 0}, {kNull, 0} }, { "libc.so.6" }, { 0 }); + auto path = write_temp(e.bytes, "elfclean.so"); + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::NothingToDo); + EXPECT_TRUE(r->paths.empty()); +} + +TEST(PackRelocate, AStaticArchiveIsNotApplicableRatherThanAnError) { + // Every leg goes through this step, and a `.a` has no dynamic section to + // begin with. Reporting that as a failure would make the caller branch on + // the format before calling — which is the coupling this module removes. + auto path = write_temp("!\n/ 0 0 0 0 4 `\n", + "libfoo.a"); + auto r = mcpp::pack::relocate::strip_search_paths(path); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::relocate::Outcome::NotApplicable); +} + +TEST(PackRelocate, AMissingFileIsAnError) { + auto r = mcpp::pack::relocate::strip_search_paths( + std::filesystem::temp_directory_path() / "mcpp-relocate-test" / "nope.so"); + EXPECT_FALSE(r.has_value()); +} + +// ── the strip table ───────────────────────────────────────────────────── + +TEST(PackStrip, StaticArchiveKeepsItsSymbolIndex) { + // ⚠️ THE MEASUREMENT BEHIND THIS TEST. `strip --strip-all` on a `.a` + // removes the archive symbol index, and the consumer's link then fails + // with `archive has no index; run ranlib to add one` — a message that + // names neither strip nor the publisher. `--strip-debug` keeps it + // (measured: 2988 -> 1244 bytes, and the consumer links and runs). + auto args = mcpp::pack::strip_args(mcpp::pack::ArtifactShape::StaticArchive); + EXPECT_NE(std::ranges::find(args, "--strip-debug"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-all"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-unneeded"), args.end()); + // Two identical packs should produce identical bytes; this is the one + // place the packer rewrites an archive. + EXPECT_NE(std::ranges::find(args, "--enable-deterministic-archives"), args.end()); +} + +TEST(PackStrip, SharedLibraryKeepsItsDynamicSymbols) { + // `--strip-unneeded` removes `.symtab` and keeps `.dynsym`, which IS the + // export list. `--strip-all` would also keep `.dynsym`, but dh_strip uses + // the narrower flag and there is no reason to be broader than the + // reference implementation of this exact decision. + auto args = mcpp::pack::strip_args(mcpp::pack::ArtifactShape::SharedLibrary); + EXPECT_NE(std::ranges::find(args, "--strip-unneeded"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-all"), args.end()); + EXPECT_EQ(std::ranges::find(args, "--strip-debug"), args.end()); +} + +TEST(PackStrip, ExecutableIsStrippedWhole) { + auto args = mcpp::pack::strip_args(mcpp::pack::ArtifactShape::Executable); + EXPECT_NE(std::ranges::find(args, "--strip-all"), args.end()); +} + +TEST(PackStrip, EveryShapeDropsCommentAndNote) { + // dh_strip drops both from all three. `.note` is matched by exact name, so + // `.note.gnu.build-id` survives — which is what `--add-gnu-debuglink` and + // debuginfod pair with. + for (auto shape : { mcpp::pack::ArtifactShape::Executable, + mcpp::pack::ArtifactShape::SharedLibrary, + mcpp::pack::ArtifactShape::StaticArchive }) { + auto args = mcpp::pack::strip_args(shape); + EXPECT_NE(std::ranges::find(args, "--remove-section=.comment"), args.end()); + EXPECT_NE(std::ranges::find(args, "--remove-section=.note"), args.end()); + } +} + +TEST(PackStrip, WhetherStrippingAppliesIsAskedOfTheTargetNotTheCompiler) { + // Keying this on the compiler is wrong in BOTH directions, and each + // direction is a real configuration mcpp ships: + // + // clang -> x86_64-windows-msvc produces .pdb debug info; a + // compiler-keyed rule would try to strip + // in-band DWARF that is not there. + // Apple clang on macOS ships no `llvm-strip`, so a + // compiler-keyed rule would REFUSE every + // `mcpp pack` on a Mac — for a format + // whose image carries a debug MAP and + // leaves the DWARF in the .o files. + EXPECT_TRUE (mcpp::pack::debug_info_is_in_band("x86_64-linux-gnu")); + EXPECT_TRUE (mcpp::pack::debug_info_is_in_band("aarch64-linux-musl")); + EXPECT_TRUE (mcpp::pack::debug_info_is_in_band("x86_64-windows-gnu")); + EXPECT_FALSE(mcpp::pack::debug_info_is_in_band("x86_64-windows-msvc")); + EXPECT_FALSE(mcpp::pack::debug_info_is_in_band("aarch64-macos")); + EXPECT_FALSE(mcpp::pack::debug_info_is_in_band("x86_64-macos")); + // Segment-wise, not substring: mcpp has been bitten by a triple predicate + // that answered on a substring before. + EXPECT_TRUE(mcpp::pack::debug_info_is_in_band("macos64-linux-gnu")); +} + +TEST(PackStrip, MsvcHasNothingInBandToRemove) { + // PE/MSVC keeps debug information in a separate `.pdb`. An empty `strip` + // tool there is the right answer, not a missing dependency — and the two + // must not be conflated, or every other platform gets a silent no-op. + auto path = write_temp("not an image", "notanimage.bin"); + mcpp::pack::StripTools tools{ {}, {}, /*inBandDebugInfo=*/false }; + auto r = mcpp::pack::strip_artifact(path, mcpp::pack::ArtifactShape::SharedLibrary, + tools, {}); + ASSERT_TRUE(r.has_value()); + EXPECT_EQ(r->outcome, mcpp::pack::StripOutcome::NotApplicable); +} + +TEST(PackStrip, AMissingStripToolIsRefusedWhereDebugInfoIsInBand) { + auto path = write_temp("not an image", "notanimage2.bin"); + mcpp::pack::StripTools tools{ {}, {}, /*inBandDebugInfo=*/true }; + auto r = mcpp::pack::strip_artifact(path, mcpp::pack::ArtifactShape::SharedLibrary, + tools, {}); + ASSERT_FALSE(r.has_value()); + EXPECT_NE(r.error().find("no `strip`"), std::string::npos); +}