Skip to content

Commit ad2e446

Browse files
committed
feat(pack): read the import table instead of running the binary (§4)
`mcpp pack` refused Windows with `#if defined(_WIN32)`, and the reason given was that the tools were POSIX-only. That was the symptom. The cause is one layer down: the dependency closure comes from LD_TRACE_LOADED_OBJECTS=1 '<binary>' which RUNS the artifact — so it can cross neither an OS (a Linux box cannot execute a PE) nor an ARCHITECTURE (an x86_64 box cannot execute an aarch64 ELF, same OS or not). Porting `tar` would not have helped, and every tool the 2026-05-19 design proposed — dumpbin, ImageNtHeader, Compress-Archive — would have reintroduced the obstacle one layer down, because each exists only on the platform where the problem had already gone away. mcpp.pack.binfmt reads it out of the file instead: ELF DT_NEEDED through the segment table, PE imports AND delay-imports (a missing delay-load does not fail at startup — it fails at the first call through it, which is strictly worse to debug). Cross-OS packaging is then not a feature that had to be added; it is what remains once nothing has to be executed. mcpp.pack.zip writes the archive, for the same reason: no zip tool exists on every host (GNU tar cannot write zip, `zip(1)` is often absent, Compress-Archive is Windows-only). Entries are STORED, which is a real size cost and the honest trade — a DEFLATE encoder is the one part that could produce an archive that unpacks WRONG rather than failing loudly, and mcpp has no zlib to borrow one from. Deterministic by construction: no timestamps are read, so a published checksum means something. THE CONTRACT NOW REACHES PACKAGING (§4.3). `cxx_runtime` used to stop at the compile and link flags, so the step that decides which files actually travel could not see what had been promised — on ELF the `ldd` closure agreed with it by luck, on PE nothing did. It is now an input: toolchain-coupled the toolchain's runtime directory joins the search set host-coupled it stays OUT, so a vcruntime140.dll in the toolset is not silently swept into a package that promised the host would provide it --mode system/static + toolchain-coupled → refused, naming the way out PE layout is flat, and that is the relocation mechanism rather than a style: the Win32 loader resolves a DLL from the directory of the executable, and there is no rpath to point elsewhere. Windows' own DLLs are never bundled — two of something that must be unique is a broken program, not a heavier one — but `force_bundle` overrides that, as it always did on ELF. Verified on a Linux host against a real cross-built PE, not only synthesised fixtures: e2e 240 builds a mingw target, drops a stand-in for a DLL the EXE imports, packs, and has PYTHON verify the archive. The msvcrt.dll assertion is the positive half (a parser that read nothing could not have produced it) and the kernel32.dll assertion the negative half; together they are decisive. It runs in the mingw-cross job because running it on Windows would prove nothing. Docs: the Windows layout, the cross-host story and both size/determinism consequences in 02-pack-and-release (en+zh); the MSVC half of `cxx_runtime` in 05-mcpp-toml (en+zh), replacing a claim about /MT that stopped being true; SDK-by-origin and the `@system` rule in 03-toolchains (en; the zh MSVC section was rewritten — it still described msvc as a system-only toolchain and `msvc@19.44` as a pin-verify). 2026-05-19-pack-windows-design.md is marked superseded with what it got wrong and why, since the mistake is instructive.
1 parent a557310 commit ad2e446

14 files changed

Lines changed: 1908 additions & 76 deletions

.agents/docs/2026-05-19-pack-windows-design.md

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,37 @@
11
# Windows Pack Design
22

33
**Date:** 2026-05-19
4-
**Status:** Planned (stub guard in place, implementation not yet started)
4+
**Status:** SUPERSEDED and implemented, 2026-08-17. See
5+
`2026-08-16-windows-toolchain-three-axes-design.md` §4 for the design that
6+
shipped, and `src/pack/binfmt.cppm` / `src/pack/zip.cppm` for the code.
7+
8+
> ## What this document got wrong, and it is worth keeping
9+
>
10+
> Everything below assumes **pack runs on Windows**. That assumption is
11+
> visible in every proposal: `dumpbin /dependents`, `ImageNtHeader` from
12+
> `<windows.h>`, PowerShell's `Compress-Archive`, and an implementation
13+
> "under `#if defined(_WIN32)`".
14+
>
15+
> The assumption came from the guard it was trying to remove. `pack` refused
16+
> Windows, so the problem looked like "pack has no Windows branch". It was
17+
> not: the ELF closure is derived by RUNNING the artifact
18+
> (`LD_TRACE_LOADED_OBJECTS=1 '<binary>'`), so it can cross neither an OS nor
19+
> an ARCHITECTURE — a Linux box cannot trace a PE, and an x86_64 box cannot
20+
> trace an aarch64 ELF either. The `#if defined(_WIN32)` was that limitation
21+
> surfacing at the nearest place a user would hit it.
22+
>
23+
> Reading the import table statically removes both limits at once, and then
24+
> cross-OS packaging is not a feature that had to be added — it is what
25+
> remains when the obstacle is gone. Each Windows-only tool above would have
26+
> reintroduced the obstacle one layer down.
27+
>
28+
> What did survive from here: the `.zip` output, the flat
29+
> DLLs-beside-the-`.exe` layout, no wrapper script, and the skip-list concept
30+
> (with one correction — `vcruntime*.dll` is listed below as a system DLL, and
31+
> it is not: it belongs to the TOOLSET, and whether it travels is
32+
> `cxx_runtime`'s decision).
33+
34+
---
535

636
## Current state
737

.github/workflows/cross-build-test.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,18 @@ jobs:
308308
export MCPP_VENDORED_XLINGS="$XLINGS_BIN"
309309
bash tests/e2e/198_windows_resources_cross.sh
310310
311+
# Packaging a Windows program FROM LINUX — and this job is the only
312+
# place that can happen, for the same reason as the two above.
313+
#
314+
# Running it on a Windows runner would prove nothing: the point of
315+
# reading the import table instead of executing the artifact
316+
# (mcpp.pack.binfmt) is precisely that the packaging host need not be
317+
# the target. A same-OS pack cannot tell the two implementations apart.
318+
- name: "e2e: pack a PE from Linux (zip + DLL closure)"
319+
run: |
320+
export MCPP_VENDORED_XLINGS="$XLINGS_BIN"
321+
bash tests/e2e/240_pack_pe_zip_cross.sh
322+
311323
# ── windows → linux ───────────────────────────────────────────────────────
312324
# The mirror of mingw-cross-wine. Two jobs because a Windows runner cannot
313325
# execute the ELF it produces; the artefact is handed to a Linux job and

docs/02-pack-and-release.md

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -236,6 +236,72 @@ own resolution, say — use `--mode vendored` instead. It repoints `PT_INTERP`
236236
at the host loader, at the cost of requiring the host's glibc to be at least
237237
as new as the one the artifact was built against.
238238

239+
### Windows (PE) — a `.zip`, and the DLLs sit beside the `.exe`
240+
241+
A Windows target produces a **`.zip`**, not a `.tar.gz`, and the layout is
242+
flat:
243+
244+
```
245+
target/dist/myapp-0.1.0-x86_64-pc-windows-msvc.zip
246+
└── myapp-0.1.0-x86_64-pc-windows-msvc/
247+
├── myapp.exe
248+
├── vcruntime140.dll ← only under cxx_runtime = "toolchain-coupled"
249+
├── mydep.dll ← third-party dependencies
250+
├── README.md
251+
└── LICENSE
252+
```
253+
254+
There is no `bin/` + `lib/` split and no entry-point wrapper, and neither is a
255+
style choice. The Win32 loader resolves a DLL from **the directory of the
256+
executable**; PE has no `RUNPATH` to point anywhere else, so "next to the
257+
`.exe`" *is* the mechanism that `$ORIGIN/../lib` provides on ELF.
258+
259+
**Windows' own DLLs are never bundled**`kernel32.dll`, `ntdll.dll`,
260+
`ucrtbase.dll`, the `api-ms-win-*` API sets. Shipping a private copy of an OS
261+
component is a broken program rather than a heavier one (the process ends up
262+
with two of something that must be unique), and Microsoft's redistribution
263+
terms say the same thing from the other side. `[pack.bundle-project]
264+
force_bundle` still overrides this, as it does the ELF skip list.
265+
266+
`vcruntime140.dll` and `msvcp140.dll` are **not** Windows' own: they belong to
267+
the MSVC toolset, exactly as `libstdc++.so` belongs to gcc. Whether they
268+
travel is decided by `cxx_runtime` (see `docs/05-mcpp-toml.md`), not by this
269+
list — and `mcpp pack` refuses a combination that cannot deliver what the
270+
contract promised:
271+
272+
```
273+
$ mcpp pack --mode system # with cxx_runtime = "toolchain-coupled"
274+
error: cxx_runtime = "toolchain-coupled" and --mode system contradict each other.
275+
```
276+
277+
#### Packing a Windows program from Linux or macOS
278+
279+
This works, and it is not a special mode — just build for a Windows target and
280+
pack:
281+
282+
```bash
283+
mcpp pack --target x86_64-windows-gnu # from a Linux host
284+
```
285+
286+
`mcpp pack` used to refuse Windows outright. The reason was not the archiver:
287+
the ELF dependency closure is obtained by **running the artifact** under
288+
`LD_TRACE_LOADED_OBJECTS`, which cannot cross an OS *or* an architecture. A PE
289+
closure is read out of the file's import table instead, so nothing has to be
290+
executed and the packaging host is free. The archive is written by mcpp itself
291+
for the same reason — there is no zip tool present on every host.
292+
293+
Two consequences worth knowing:
294+
295+
- Entries are **stored, not deflated**, so a Windows package is roughly the
296+
size of its contents. Compression is a size optimization, not a correctness
297+
one, and it is not implemented yet.
298+
- The archive is **deterministic**: no timestamps are read, so two packs of
299+
the same tree are byte-identical and a published checksum means something.
300+
301+
The reverse direction — packing a Linux or macOS artifact *from* Windows —
302+
still does not work, and for the original reason: that closure is resolved by
303+
the target's own dynamic linker, which a Windows host has no way to run.
304+
239305
## Configuration
240306

241307
Packaging behavior is configured via the `[pack]` section in `mcpp.toml`. The

docs/03-toolchains.md

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,17 @@ questions. `msvc@system` asks *"use what this developer already has"*;
228228
`msvc@14.44.35207` asks *"build this project with exactly this compiler"*.
229229
Pinned toolsets coexist with each other and with a system Visual Studio.
230230

231+
> **`@system` is an MSVC-only spelling.** There is no `gcc@system` or
232+
> `llvm@system`, and that is deliberate rather than an omission: mcpp is built
233+
> on xlings, a user-space OS, and the design drives host dependencies to a
234+
> minimum — a toolchain comes from a payload the manifest names, so every
235+
> machine builds with the same compiler. Windows is the one place where
236+
> refusing to use what is already installed would cost more than it buys:
237+
> Visual Studio is very often present and cannot always be redistributed.
238+
> `<family>@system` for any other family is an error that names both things
239+
> you might have meant. (The family-less `[toolchain] … = "system"` — the PATH
240+
> compiler — is a separate and deliberate escape hatch, and is unaffected.)
241+
231242
### `msvc@system` — the machine's own Visual Studio
232243

233244
mcpp locates and identifies an installed Visual Studio / Build Tools; it never
@@ -304,18 +315,37 @@ environment from the VC tools + Windows SDK (no `vcvarsall` involved), stages
304315
`/interface /TP /ifcOutput`, scans with `/scanDependencies`, and links with
305316
`link.exe`/`lib.exe` through response files.
306317

307-
The Windows SDK is located in this order: **`WindowsSdkDir`** (+
308-
`WindowsSdkVersion`) if declared, then the `xim:windows-sdk` payload beside a
309-
pinned toolset in mcpp's store, then `C:\Program Files (x86)\Windows Kits\10`.
310-
A missing SDK fails the build with guidance (`mcpp self doctor` reports SDK
311-
status).
318+
**The Windows SDK follows the origin**, because the two origins answer
319+
different questions and so must the SDK:
320+
321+
| origin | how the SDK is chosen |
322+
|---|---|
323+
| `msvc@<toolset>` | the `xim:windows-sdk` payload installed **with that toolset**, in mcpp's own store. `WindowsSdkDir` / `WindowsSdkVersion` in the environment are **ignored**, and mcpp prints a `note:` saying so. |
324+
| `msvc@system` | **`WindowsSdkDir`** (+ `WindowsSdkVersion`) if declared, then `C:\Program Files (x86)\Windows Kits\10`. |
325+
326+
The asymmetry is the point. A pinned toolset is a promise that two machines
327+
compile the same source against the same headers; an environment variable that
328+
can quietly redirect it turns the pin into a preference. A machine's own SDK,
329+
on the other hand, can only be found by looking, and there a declared answer
330+
outranks a scan — the same precedence `VSINSTALLDIR` has over `vswhere`.
331+
332+
If a pinned toolset has no SDK payload beside it (an older install, say), mcpp
333+
falls back to the machine's SDK rather than failing — and says so, because that
334+
build is no longer reproducible and nothing else would record it.
312335

313336
A root only counts as an SDK when it has **both** halves — `Include\<v>\ucrt\
314337
corecrt.h` *and* `Lib\<v>\um\<arch>\kernel32.lib`. A root with headers and no
315338
import libraries is skipped rather than selected, so a partially unpacked
316339
payload cannot outrank the machine's complete SDK and turn into
317340
`LNK1104: cannot open file 'kernel32.lib'` at the very end of a build.
318341

342+
The resolved SDK version is part of the build's **runtime identity**
343+
(`ucrt@10.0.26100.0`) and therefore of the fingerprint that keys the build
344+
cache: changing SDK changes the cache key, exactly as changing compiler does.
345+
It is a **compatibility floor declaration**, not a payload binding like
346+
`glibc@2.39` on Linux — `ucrtbase.dll` is a Windows component and mcpp neither
347+
ships nor substitutes it.
348+
319349
**CRT model.** `/MD` (host-coupled) by default; `/MT` when either
320350

321351
```toml

docs/05-mcpp-toml.md

Lines changed: 33 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -395,10 +395,39 @@ default applies only when nobody said anything.
395395
`self-contained`, `false` means `host-coupled`. An explicit `cxx_runtime` wins.
396396

397397
**A contract that cannot be honored is reported, never silently downgraded.** If a
398-
toolchain ships no `libc++.a`, or a contract has no mechanism on that platform
399-
(`self-contained` under the MSVC runtime would need `/MT`, which mcpp does not emit
400-
yet), the build prints what it fell back to instead of quietly producing a
401-
different artifact than the manifest asked for.
398+
toolchain ships no `libc++.a`, or a contract has no mechanism on that platform,
399+
the build prints what it fell back to instead of quietly producing a different
400+
artifact than the manifest asked for.
401+
402+
#### On the MSVC runtime
403+
404+
The CRT model is the mechanism here, and it is a **whole-project** switch: cl
405+
bakes `_MSVC_MT`/`_MSVC_MD` into the one `std` module a project builds, so a
406+
per-role contract that disagrees with the project's cannot be honoured and is
407+
reported rather than ignored.
408+
409+
| value | what it is on MSVC |
410+
|---|---|
411+
| `self-contained` | `/MT` — the static CRT. `linkage = "static"` selects the same thing from the libc axis. |
412+
| `host-coupled` (default under `/MD`) | the target provides `vcruntime140.dll` / `msvcp140.dll` — i.e. Visual Studio or the redistributable is installed there. |
413+
| `toolchain-coupled` | the toolset's **own** copy of those DLLs travels with the artifact. |
414+
415+
`toolchain-coupled` is worth spelling out, because the obvious reading is
416+
wrong. `ucrtbase.dll` *is* a Windows component (since Windows 10) and mcpp
417+
never ships it. `vcruntime140.dll` and `msvcp140.dll` are **not**: every MSVC
418+
toolset carries them under `VC\Redist\MSVC\<version>\<arch>\`, exactly the
419+
way a gcc payload carries `libstdc++.so`. Under this contract mcpp stages them
420+
beside the artifact — which is what makes a default `/MD` build runnable on a
421+
machine that has only the pinned toolset and no Visual Studio at all.
422+
423+
The debug CRT (`vcruntime140d.dll` and friends, under `debug_nonredist\`) is
424+
never staged: it may not be redistributed.
425+
426+
Combining it with `/MT` is a contradiction rather than a missing feature — a
427+
static CRT leaves no DLL to couple to — so it is reported and resolved to
428+
`self-contained`. `mcpp pack` enforces the other half: a mode that bundles
429+
nothing (`--mode system`, `--mode static`) cannot deliver `toolchain-coupled`
430+
and refuses.
402431

403432
**Scope.** The contract governs the C++ runtime only. Static **libc** is a separate
404433
axis (`linkage = "static"` / `--static`, e.g. a musl target), and the deployment

docs/zh/02-pack-and-release.md

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,65 @@ if (!base) {
172172
它把 `PT_INTERP` 重指到宿主 loader,`/proc/self/exe` 正常,代价是要求宿主
173173
glibc 不低于构建时所用的那份。
174174

175+
### Windows(PE)—— 产物是 `.zip`,DLL 就放在 `.exe` 旁边
176+
177+
Windows 目标产出的是 **`.zip`** 而不是 `.tar.gz`,并且是扁平布局:
178+
179+
```
180+
target/dist/myapp-0.1.0-x86_64-pc-windows-msvc.zip
181+
└── myapp-0.1.0-x86_64-pc-windows-msvc/
182+
├── myapp.exe
183+
├── vcruntime140.dll ← 仅在 cxx_runtime = "toolchain-coupled" 时
184+
├── mydep.dll ← 第三方依赖
185+
├── README.md
186+
└── LICENSE
187+
```
188+
189+
没有 `bin/` + `lib/` 的分层,也没有入口 wrapper —— 这两点都不是风格选择。
190+
Win32 loader 解析 DLL 的第一顺位就是**可执行文件所在目录**,而 PE 没有
191+
`RUNPATH` 可以指向别处,所以"放在 `.exe` 旁边"**就是** ELF 上
192+
`$ORIGIN/../lib` 所提供的那个机制。
193+
194+
**Windows 自己的 DLL 永远不会被打进包里** —— `kernel32.dll``ntdll.dll`
195+
`ucrtbase.dll``api-ms-win-*` API set。带一份系统组件的私有拷贝不是"包变大
196+
了",而是**程序坏了**(进程里出现了两份本应唯一的东西),而 Microsoft 的再分发
197+
条款也从另一侧说了同一件事。`[pack.bundle-project] force_bundle` 仍然可以覆盖
198+
这条,和它覆盖 ELF 跳过表一样。
199+
200+
`vcruntime140.dll` / `msvcp140.dll` **不是** Windows 自己的:它们属于 MSVC
201+
toolset,就像 `libstdc++.so` 属于 gcc。它们要不要跟着产物走,由 `cxx_runtime`
202+
决定(见 `docs/zh/05-mcpp-toml.md`),不由这张表决定 —— 而 `mcpp pack` 会拒绝
203+
那些无法兑现契约的组合:
204+
205+
```
206+
$ mcpp pack --mode system # 且 cxx_runtime = "toolchain-coupled"
207+
error: cxx_runtime = "toolchain-coupled" and --mode system contradict each other.
208+
```
209+
210+
#### 在 Linux / macOS 上给 Windows 打包
211+
212+
这是可以的,而且不是什么特殊模式 —— 指定 Windows 目标构建,然后打包即可:
213+
214+
```bash
215+
mcpp pack --target x86_64-windows-gnu # 在 Linux 宿主上
216+
```
217+
218+
`mcpp pack` 以前直接拒绝 Windows。真正的原因不是打包工具:ELF 的依赖闭包是靠
219+
**把产物跑起来**(`LD_TRACE_LOADED_OBJECTS`)求出来的,所以它既跨不了 OS 也跨
220+
不了架构。PE 的闭包改为从文件的导入表里****出来,于是不需要执行任何东西,
221+
打包宿主也就自由了。压缩包本身也由 mcpp 自己写,理由相同 —— 没有哪个 zip 工具
222+
在每个宿主上都存在。
223+
224+
有两点值得知道:
225+
226+
- 条目是 **stored(不压缩)** 的,所以 Windows 包的体积约等于其内容之和。压缩
227+
是体积优化而不是正确性问题,目前尚未实现。
228+
- 压缩包是**确定性**的:不读取任何时间戳,同一棵树打两次字节一致,公布的校验和
229+
才有意义。
230+
231+
反方向 —— 在 Windows 上给 Linux / macOS 产物打包 —— 仍然不支持,原因还是最初
232+
那个:那条闭包要由目标自己的动态链接器解析,而 Windows 宿主没有办法运行它。
233+
175234
## 配置项
176235

177236
打包行为通过 `mcpp.toml` 中的 `[pack]` 节配置,常用字段如下:

0 commit comments

Comments
 (0)