From 6c8099e91691766024f3debe7a6440d95b23e34c Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:10:38 +0200 Subject: [PATCH 1/7] Native arm64 for dtest: Chrome for Testing 153 Beta, CI on ARM runners Chrome for Testing began publishing linux-arm64 at 153 - chrome AND chromedriver - so the amd64 pin is no longer forced. .dev/cft-version moves to 153.0.8010.5 (the first Beta with it; a fixed version string, so nothing floats), the Dockerfile and bin/setup-test-env pick the CfT platform from the build/host arch instead of hardcoding linux64, and both `platform: linux/amd64` pins come out of compose so the container follows the host. The win is the emulation tax: the container was running amd64 under emulation on an ARM Mac, which is what mem_limit: 4g and the "Chrome 152 needs >2g under amd64 emulation (OOM tab crashed at 2g)" note were paying for. Verified native: uname -m reports aarch64, Chrome for Testing 153.0.8010.5, chromedriver matching, Debian 13. CI moves to ubuntu-24.04-arm in the same commit - free for a public repo - so both sides stay on one architecture. Doing it together matters: bumping 152 -> 153 invalidates the linux/ baselines by itself, so splitting the work would pay for a full re-record twice. Two things this needed that were not in the plan: - Gemfile.lock carried aarch64-linux-musl (Alpine) but not aarch64-linux (glibc), so nokogiri had no native candidate for a Debian ARM container. Added via `bundle lock --add-platform aarch64-linux`. That gap would have broken the ARM runners too. - bin/dc's DOCKER_DEFAULT_PLATFORM=linux/arm64/v8 export is deleted rather than kept-and-now-correct. It never affected the test services (they pinned linux/amd64 and won) and it is what made four separate "must be an arch problem" diagnoses look plausible on 2026-08-22. MEASURED, and the reason this is safe: after changing BOTH the architecture and the Chrome major version, 47 of 55 screenshots still match baselines recorded on amd64 Chrome 152. The 8 that do not are exactly the mobile codeblocks family that already differed before this change, at the same magnitudes - so that divergence is neither arch nor Chrome version. It is the remaining Debian-container vs Ubuntu-runner font stack, and it closes only by running CI inside this image (see .okf/build/rendering-stack.md). Re-record linux/ via an update-baselines dispatch ON MASTER after this merges - recorder and tester must see the same tree. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- .dev/Dockerfile | 21 ++++++++++++++------- .dev/cft-version | 2 +- .dev/compose.yml | 16 ++++++++-------- .github/workflows/test.yml | 4 +++- Gemfile.lock | 4 ++++ bin/dc | 13 +++++-------- bin/setup-test-env | 18 +++++++++++------- 7 files changed, 46 insertions(+), 32 deletions(-) diff --git a/.dev/Dockerfile b/.dev/Dockerfile index 1939b149a..3116565bf 100644 --- a/.dev/Dockerfile +++ b/.dev/Dockerfile @@ -32,8 +32,8 @@ ENV \ BUNDLE_JOBS=4 \ BUNDLE_PATH=/opt/bundle \ BUNDLE_RETRY=3 \ - CHROME_BIN=/opt/cft/chrome-linux64/chrome \ - CHROMEDRIVER_PATH=/opt/cft/chromedriver-linux64/chromedriver \ + CHROME_BIN=/opt/cft/chrome/chrome \ + CHROMEDRIVER_PATH=/opt/cft/chromedriver/chromedriver \ DEBUG_COLORS=true \ DISPLAY=:99 \ DOCKER=true \ @@ -79,11 +79,18 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ # Pinned Chrome for Testing + matching chromedriver (the rendering pin that # makes linux/ baselines reproducible - version in .dev/cft-version). COPY .dev/cft-version /tmp/cft-version -RUN CFT=$(tr -d '[:space:]' < /tmp/cft-version) && mkdir -p /opt/cft && cd /opt/cft && \ - curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/${CFT}/linux64/chrome-linux64.zip" && \ - curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/${CFT}/linux64/chromedriver-linux64.zip" && \ - unzip -q chrome-linux64.zip && unzip -q chromedriver-linux64.zip && \ - rm chrome-linux64.zip chromedriver-linux64.zip && \ +RUN CFT=$(tr -d '[:space:]' < /tmp/cft-version) && \ + case "$(dpkg --print-architecture)" in \ + arm64) CFT_PLAT=linux-arm64 ;; \ + *) CFT_PLAT=linux64 ;; \ + esac && \ + mkdir -p /opt/cft && cd /opt/cft && \ + curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/${CFT}/${CFT_PLAT}/chrome-${CFT_PLAT}.zip" && \ + curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/${CFT}/${CFT_PLAT}/chromedriver-${CFT_PLAT}.zip" && \ + unzip -q "chrome-${CFT_PLAT}.zip" && unzip -q "chromedriver-${CFT_PLAT}.zip" && \ + rm "chrome-${CFT_PLAT}.zip" "chromedriver-${CFT_PLAT}.zip" && \ + ln -sfn "/opt/cft/chrome-${CFT_PLAT}" /opt/cft/chrome && \ + ln -sfn "/opt/cft/chromedriver-${CFT_PLAT}" /opt/cft/chromedriver && \ "$CHROME_BIN" --version && "$CHROMEDRIVER_PATH" --version # Deterministic font rendering (see .dev/fonts.conf header). diff --git a/.dev/cft-version b/.dev/cft-version index 17853bf7e..87b9a379a 100644 --- a/.dev/cft-version +++ b/.dev/cft-version @@ -1 +1 @@ -152.0.7977.54 +153.0.8010.5 diff --git a/.dev/compose.yml b/.dev/compose.yml index 0659a9ac5..e2b668bd9 100644 --- a/.dev/compose.yml +++ b/.dev/compose.yml @@ -25,11 +25,11 @@ services: # Interactive shell (optimized) sh: image: jetthoughts.com-test:1.0.0 - # amd64, not the host arch: this image pins Chrome for Testing linux64 - # (.dev/cft-version) and records the linux/ screenshot baselines that CI - # (ubuntu-latest = amd64) renders. Building/running arm64 downloads an - # amd64 chrome it can only exec under emulation and drifts pixels from CI. - platform: linux/amd64 + # Follows the HOST arch since 2026-08-22. Chrome for Testing began + # publishing linux-arm64 at 153 (.dev/cft-version pins 153.0.8010.5, the + # first Beta with it), and the Dockerfile picks the matching build - so an + # ARM Mac runs native instead of emulating amd64. CI moved to arm64 + # runners in the same change, so both sides still render alike. build: context: ../ dockerfile: ./.dev/Dockerfile @@ -48,8 +48,7 @@ services: # Test runner with optimized dependency management t: image: jetthoughts.com-test:1.0.0 - # amd64 to match CI's rendering (see `sh` service note above). - platform: linux/amd64 + # Host arch, matching CI's arm64 runners (see `sh` service note above). build: context: ../ dockerfile: ./.dev/Dockerfile @@ -64,7 +63,8 @@ services: HUGO_CACHEDIR: "/tmp/hugo_cache_dtest" CAPYBARA_SCREENSHOT_ON_FAILURE: "true" RUBY_THREAD_VM_STACK_SIZE: 1048576 - # Chrome 152 needs >2g under amd64 emulation (OOM "tab crashed" at 2g). + # 4g was sized for amd64 EMULATION ("tab crashed" OOM at 2g). Native arm64 + # should need less; left at 4g until measured rather than guessed down. mem_limit: 4g cpus: '4.0' ulimits: diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ed0302b20..309d7cf77 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -65,7 +65,9 @@ jobs: screenshots: name: Screenshot Tests if: ${{ github.event_name == 'pull_request' || inputs.screenshots || inputs.update-baselines }} - runs-on: ubuntu-latest + # arm64 to match the local container (both run CfT linux-arm64 since + # .dev/cft-version moved to 153). Free for public repos. + runs-on: ubuntu-24.04-arm timeout-minutes: 20 # BLOCKING since 2026-08-22: the report-only soak (2026-07-31, #413) ended — # its motivating defects were fixed (#560 tolerance, #566 baselines, #570 diff --git a/Gemfile.lock b/Gemfile.lock index 0cb5feba0..cee96f223 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -64,6 +64,7 @@ GEM faraday-net_http (3.4.4) net-http (~> 0.5) ffi (1.17.4) + ffi (1.17.4-aarch64-linux-gnu) ffi (1.17.4-aarch64-linux-musl) ffi (1.17.4-arm64-darwin) ffi (1.17.4-x86_64-linux-gnu) @@ -108,6 +109,8 @@ GEM net-http (0.9.1) uri (>= 0.11.1) nio4r (2.7.5) + nokogiri (1.19.4-aarch64-linux-gnu) + racc (~> 1.4) nokogiri (1.19.4-aarch64-linux-musl) racc (~> 1.4) nokogiri (1.19.4-arm64-darwin) @@ -198,6 +201,7 @@ GEM zeitwerk (2.8.3) PLATFORMS + aarch64-linux aarch64-linux-musl arm64-darwin-23 arm64-darwin-25 diff --git a/bin/dc b/bin/dc index a0a4a3405..084d897ba 100755 --- a/bin/dc +++ b/bin/dc @@ -3,14 +3,11 @@ set -euo pipefail -# Set Docker platform for ARM-based Macs (M1/M2 chips) -if [ "$(uname -s)" = "Darwin" ]; then - case "$(uname -m)" in - arm*) - export DOCKER_DEFAULT_PLATFORM=linux/arm64/v8 - ;; - esac -fi +# No DOCKER_DEFAULT_PLATFORM override. Docker already defaults to the host +# arch; forcing linux/arm64/v8 here did nothing for the test services (they +# pinned linux/amd64 and won) while making four separate "it must be an arch +# problem" diagnoses look plausible on 2026-08-22. See +# .okf/build/rendering-stack.md. # One compose project PER WORKTREE. The project name namespaces containers, # networks and volumes, so a fixed "jtcom" meant two worktrees running diff --git a/bin/setup-test-env b/bin/setup-test-env index 2bb9e4100..0fe95b4e9 100755 --- a/bin/setup-test-env +++ b/bin/setup-test-env @@ -16,9 +16,13 @@ set -euo pipefail cd "$(dirname "$0")/.." CFT=$(tr -d '[:space:]' < .dev/cft-version) -CACHE="${XDG_CACHE_HOME:-$HOME/.cache}/jt-cft/$CFT" -CHROME="$CACHE/chrome-linux64/chrome" -DRIVER="$CACHE/chromedriver-linux64/chromedriver" +case "$(uname -m)" in + aarch64|arm64) CFT_PLAT=linux-arm64 ;; + *) CFT_PLAT=linux64 ;; +esac +CACHE="${XDG_CACHE_HOME:-$HOME/.cache}/jt-cft/$CFT-$CFT_PLAT" +CHROME="$CACHE/chrome-$CFT_PLAT/chrome" +DRIVER="$CACHE/chromedriver-$CFT_PLAT/chromedriver" if [ "${1:-}" = "--print-env" ]; then echo "export CHROME_BIN=\"$CHROME\"" @@ -35,10 +39,10 @@ fi if [ ! -x "$CHROME" ] || [ ! -x "$DRIVER" ]; then echo "==> Downloading Chrome for Testing $CFT to $CACHE" mkdir -p "$CACHE" && cd "$CACHE" - curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/$CFT/linux64/chrome-linux64.zip" - curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/$CFT/linux64/chromedriver-linux64.zip" - unzip -oq chrome-linux64.zip && unzip -oq chromedriver-linux64.zip - rm -f chrome-linux64.zip chromedriver-linux64.zip + curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/$CFT/$CFT_PLAT/chrome-$CFT_PLAT.zip" + curl -sSLO "https://storage.googleapis.com/chrome-for-testing-public/$CFT/$CFT_PLAT/chromedriver-$CFT_PLAT.zip" + unzip -oq "chrome-$CFT_PLAT.zip" && unzip -oq "chromedriver-$CFT_PLAT.zip" + rm -f "chrome-$CFT_PLAT.zip" "chromedriver-$CFT_PLAT.zip" cd - >/dev/null fi "$CHROME" --version From 16d88ff03da327874b81320ad54fc7ca74f95a8a Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:19:50 +0200 Subject: [PATCH 2/7] Docker: fix two defects the optimization pass surfaced, then optimize Defects first, because they were silent: - `bundle config set --local` wrote /app/.bundle/config, and the `..:/app` bind mount replaces /app at runtime, so none of those settings reached the running container. BUNDLE_PATH/JOBS/RETRY happened to be ENV as well and covered for it; `without development:assets` did not. Now BUNDLE_WITHOUT alongside the others. Proven: `bundle config get without` in the container reports "Set via BUNDLE_WITHOUT: [:development, :assets]". - compose declared `node_modules:` and `bundle:` named volumes that no service referenced. Removed. Then the optimizations, per current Docker guidance: - `# syntax=docker/dockerfile:1`, so cache-mount syntax does not depend on bin/dc exporting DOCKER_BUILDKIT. - apt cache mounts on /var/cache/apt and /var/lib/apt, with `rm -f /etc/apt/apt.conf.d/docker-clean` FIRST - Debian's hook deletes the downloaded debs, so without removing it the cache mount does nothing. The old `rm -rf /var/lib/apt/lists/*` came out with it: once the lists live in a mount, purging them is wrong rather than redundant. - Both VOLUME lines dropped. `VOLUME /opt/bundle` was the harmful one - a persisted volume there serves stale gems after any image rebuild; `bundle check` now reports satisfied from the image layer. The node_modules shield moved to an explicit ANONYMOUS mount in compose, which keeps the host's macOS binaries out (354 entries present in-container) without persisting stale modules between runs. Not claimed: a rebuild speedup. The apt cache only pays when that layer re-runs and no cold-vs-warm comparison was measured. The compose `hugo` service was checked and KEPT - bin/build drives production builds through it, so it is not vestigial as first assumed. dtest after the change: same 8 codeblocks failures as before, no regression. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- .dev/Dockerfile | 22 +++++++++++----------- .dev/compose.yml | 12 ++++++++---- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/.dev/Dockerfile b/.dev/Dockerfile index 3116565bf..34b0aa903 100644 --- a/.dev/Dockerfile +++ b/.dev/Dockerfile @@ -1,3 +1,4 @@ +# syntax=docker/dockerfile:1 # Multi-stage Dockerfile for JetThoughts test environment. # # Debian (glibc) base, NOT Alpine: this image records the canonical @@ -32,6 +33,7 @@ ENV \ BUNDLE_JOBS=4 \ BUNDLE_PATH=/opt/bundle \ BUNDLE_RETRY=3 \ + BUNDLE_WITHOUT=development:assets \ CHROME_BIN=/opt/cft/chrome/chrome \ CHROMEDRIVER_PATH=/opt/cft/chromedriver/chromedriver \ DEBUG_COLORS=true \ @@ -55,7 +57,10 @@ ENV \ RUBY_YJIT_ENABLE=1 \ TEST_SERVER_PORT=1314 -RUN apt-get update && apt-get install -y --no-install-recommends \ +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt,sharing=locked \ + rm -f /etc/apt/apt.conf.d/docker-clean \ + && apt-get update && apt-get install -y --no-install-recommends \ bash \ build-essential git pkg-config curl ca-certificates unzip \ libjpeg62-turbo libvips42 \ @@ -73,8 +78,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ libxkbcommon0 libxrandr2; do \ apt-cache show "$p" >/dev/null 2>&1 && deps="$deps $p"; \ done \ - && apt-get install -y --no-install-recommends $deps \ - && rm -rf /var/lib/apt/lists/* + && apt-get install -y --no-install-recommends $deps # Pinned Chrome for Testing + matching chromedriver (the rendering pin that # makes linux/ baselines reproducible - version in .dev/cft-version). @@ -104,19 +108,15 @@ WORKDIR /app COPY Gemfile Gemfile.lock ./ +# No `bundle config set --local`: that writes /app/.bundle/config, and the +# `..:/app` bind mount replaces /app at runtime, so those settings never reach +# the running container. Every one of them is an ENV above instead, which does +# survive. RUN --mount=type=cache,target=/usr/local/bundle/cache,sharing=locked \ --mount=type=cache,target=/root/.bundle \ echo "gem: --no-document" > /root/.gemrc && \ - bundle config set --local deployment false && \ - bundle config set --local path /opt/bundle && \ - bundle config set --local jobs 4 && \ - bundle config set --local retry 3 && \ - bundle config set --local without development:assets && \ bundle install -VOLUME /app/node_modules -VOLUME /opt/bundle - COPY --from=node-deps /temp/dev/node_modules ./node_modules COPY .dev/docker-entrypoint.sh /docker-entrypoint.sh diff --git a/.dev/compose.yml b/.dev/compose.yml index e2b668bd9..00a60725e 100644 --- a/.dev/compose.yml +++ b/.dev/compose.yml @@ -39,6 +39,10 @@ services: tty: true volumes: - ..:/app:delegated + # Anonymous mount, not a named volume: it shields the image's + # Linux-built node_modules from the bind mount (the host's are + # macOS binaries) without persisting stale modules between runs. + - /app/node_modules - history:/usr/local/hist environment: HISTFILE: /usr/local/hist/.bash_history @@ -57,6 +61,10 @@ services: command: bin/test volumes: - ..:/app:delegated + # Anonymous mount, not a named volume: it shields the image's + # Linux-built node_modules from the bind mount (the host's are + # macOS binaries) without persisting stale modules between runs. + - /app/node_modules - hugo_cache_dtest:/tmp/hugo_cache_dtest environment: HUGO_DEFAULT_PATH: "_dest/public-dtest" @@ -73,10 +81,6 @@ services: hard: -1 volumes: - node_modules: - driver: local - bundle: - driver: local history: driver: local hugo_cache: From f44ddf064d5d58f419ff81cdd2a556155734539a Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:48:26 +0200 Subject: [PATCH 3/7] Sync STATUS + OKF to the ARM migration, and withdraw a conclusion I asserted The rendering-stack concept and STATUS.md both still described a stack that no longer exists: "wait for Chrome 153 Stable, then go all-ARM", with a two-stack table claiming x86_64 on both sides and Chrome 152. Paul overrode the pin-to-Stable policy on 2026-08-21 and the migration shipped, so the trigger row moves out of "Blocked on Paul" into Now/WIP with the re-record as its next step. The substantive change is a retraction. The previous OKF entry concluded the 8 mobile/blog/special/codeblocks failures were caused by Debian-vs-Ubuntu, and #589's commit message repeated it. That conclusion is not established: local ARM (Debian) and CI ARM (Ubuntu) fail the SAME 8 keys, which reads as confirmation but cannot discriminate, because both are compared against baselines recorded on the old amd64/Chrome-152 stack - every hypothesis predicts red on both. That is the fifth instance of the exact failure shape the concept already catalogues four of: a mechanical-sounding cause asserted without measuring the thing it names. It is now recorded as the fifth, with the discriminating experiment written down - re-record on master, then run local bin/dtest; green means the distro never mattered, same 8 red means it does. Also fixes an indexing gap: rendering-stack.md was never listed in .okf/build/index.md, so the concept was unreachable by progressive disclosure from the section index. Bundle validates with zero ERRORs (--strict still exits 1 on the pre-existing known-red warning set, tracked in the 2608 README). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- .okf/build/index.md | 1 + .okf/build/rendering-stack.md | 76 +++++++++++++++++++++++------------ .okf/log.md | 22 +++++++++- STATUS.md | 2 +- 4 files changed, 74 insertions(+), 27 deletions(-) diff --git a/.okf/build/index.md b/.okf/build/index.md index f053b936e..ce4d5a8c8 100644 --- a/.okf/build/index.md +++ b/.okf/build/index.md @@ -2,6 +2,7 @@ * [Hugo build pipeline](hugo-build.md) - bin/hugo-build with the 8 course validators; also the PurgeCSS cold-start race and the minified-unquoted-attribute audit-tool trap * [Test gates](test-gates.md) - the local suites, when each is a commit blocker, the 0.0001 default tolerance, why the SECTION_CONFIGS shield was deleted and what measuring it cost, why a green run never refreshes a baseline, and why below-fold content is invisible at any tolerance, bin/record-baselines for accepting only the baselines you meant to move, and why a deleted source file still serves from every local _dest/ tree, plus the NULL CHANGE - a diff that passes every gate and alters nothing - and what `okf_validate` actually guards (shape, not truth; error-only conformance) with the two-spec trap, and why computed-style assertions beat more screenshots for below-fold coverage and for contrast (a screenshot returns a pixel delta, never a ratio) +* [A screenshot baseline is a recording of a rendering stack](rendering-stack.md) - which environment records `linux/`, why local dtest and CI can diverge, the all-arm64 migration (CfT 153 Beta, `ubuntu-24.04-arm`), and five wrong explanations for one drift - including the withdrawn "Debian vs Ubuntu" answer and why same-failure-on-both is not confirmation * [CI gates](ci-gates.md) - what GitHub Actions enforces: build, unit, path-scoped link check (visual regression is report-only), and what gates a PR never sees * [Template PDFs](pdf-templates.md) - regenerating the downloadable course PDFs * [Classes only on <th> get purged](hugo-stats-th-classes.md) - Hugo 0.165 writeStats records no class attrs on ``, so PurgeCSS deletes those rules from the production bundle while dev builds look fine and the visual gate stays green - unguardable by a screenshot, guardable by a set-diff unit test (`next_purge_guard_test.rb`); also two sibling false-greens on the /next/ rail (site-scripts in a landing baseof, computed `resources.Get` paths defeating the orphan guard) diff --git a/.okf/build/rendering-stack.md b/.okf/build/rendering-stack.md index 80f42f73b..caccd18fe 100644 --- a/.okf/build/rendering-stack.md +++ b/.okf/build/rendering-stack.md @@ -16,15 +16,16 @@ produced the baseline and which produced the candidate**. | | local `bin/dtest` | CI (records `linux/`) | |---|---|---| -| arch | x86_64 | x86_64 | -| Chrome | 152.0.7977.54, pinned by `.dev/cft-version` | same pin, cached by that file's hash | +| arch | **arm64** | **arm64** (`ubuntu-24.04-arm`) | +| Chrome | 153.0.8010.5, pinned by `.dev/cft-version` | same pin, cached by that file's hash | | `fonts.conf` + font packages | `.dev/fonts.conf`, noto-core / freefont-ttf / dejavu-core | same, via `bin/setup-test-env` | -| **base OS** | **Debian 13 trixie** (freetype 2.13.3) | **ubuntu-latest** | +| **base OS** | **Debian 13 trixie** (freetype 2.13.3) | **Ubuntu 24.04** | -Architecture, browser and fonts were deliberately pinned to match. The base OS -was not, so freetype/harfbuzz differ - and that is enough to move dense -monospace text. Measured 2026-08-22: 8 `mobile/blog/special/codeblocks/*` -screenshots differ ~0.055-0.063 between the two, with no defect present. +Architecture, browser and fonts are deliberately pinned to match. The base OS +is not - so freetype/harfbuzz may differ, which would be enough to move dense +monospace text. 8 `mobile/blog/special/codeblocks/*` screenshots differ +~0.055-0.063 with no defect present, and **the base OS is the last unpinned +variable, not a demonstrated cause** - see the open question below. **macOS is a third stack.** `bin/test` records `macos/`; `bin/dtest` and CI record `linux/`. Neither OS's set is a subset of the other, and a candidate @@ -39,7 +40,8 @@ named. Listed because the *shape* recurs, not the specific causes: pins `platform: linux/amd64` on the `t` service and that DOES override `bin/dc`'s `DOCKER_DEFAULT_PLATFORM=linux/arm64/v8` - which is a real booby-trap in the tooling, since it makes the arm64 story look right. One - `uname -m` in the container ends the debate. + `uname -m` in the container ends the debate. (Both the pin and the export + were deleted in #589 - the trap is gone, the lesson is not.) 2. **"The tolerance change (#560) regressed it."** That test pins its own `tolerance: 0.03` inline, so the default never applied to it. 3. **"The PR merge-commit checkout differs from the dispatch checkout."** True @@ -63,7 +65,7 @@ been first: read the container's OS and library versions. - A green visual run that prints no `[snap_diff] N screenshots compared` line compared nothing - see [test-gates](/build/test-gates.md). -## ARM on Linux: not a dead end, a WAITING one (re-checked 2026-08-22) +## ARM on Linux: SHIPPED 2026-08-22 (was "waiting for Stable") Chrome for Testing **does** publish `linux-arm64` - chrome and chromedriver both - from **153.0.8001.0** onward. An earlier check here concluded "no @@ -76,34 +78,58 @@ is one major release too early. Query the manifest, not the pin. | Beta | 153.0.8010.5 | yes | | Dev / Canary | 154.x | yes | -`.dev/cft-version` pins current Stable, which is the right policy - so ARM is -blocked only until **153 promotes to Stable**. Google shipped official Chrome -for ARM64 Linux on 2026-07-30, so this is a channel-timing question now, not an -availability one. +The policy had been "pin current Stable", which would have blocked ARM until 153 +promoted. **Paul overrode it (2026-08-21): take Beta now.** `.dev/cft-version` +pins `153.0.8010.5` - a fixed version string, so nothing floats, and the pin +returns to Stable when 153 promotes. Migration shipped in +[#589](https://github.com/jetthoughts/jetthoughts.github.io/pull/589). -**TRIGGER: when CfT Stable >= 153, migrate the whole stack to arm64.** It is -strictly better than today on every axis: +The reasoning that made it worth doing, all of which held: -- the container currently runs **amd64 under emulation on an ARM Mac** - that - is why `.dev/compose.yml` carries `mem_limit: 4g` with the note "Chrome 152 - needs >2g under amd64 emulation (OOM 'tab crashed' at 2g)". Native arm64 - removes that tax entirely. +- the container **ran amd64 under emulation on an ARM Mac** - that is why + `.dev/compose.yml` carried `mem_limit: 4g` with the note "Chrome 152 needs + >2g under amd64 emulation (OOM 'tab crashed' at 2g)". Native arm64 removes + that tax entirely. - GitHub's `ubuntu-24.04-arm` runners are free for this public repo, so CI can match. - same arch on both sides, and if CI also runs this image, the same distro - which closes the Debian-vs-Ubuntu gap below at the same time. -Cost: bump `.dev/cft-version`, drop the `platform: linux/amd64` pins, switch -`runs-on`, and re-record every `linux/` baseline once on the new stack. +Two things the plan did not anticipate, both of which would have broken CI too: -**Do not chase the Debian-vs-Ubuntu difference before that trigger** - the ARM -migration forces a full re-record anyway, so paying for parity twice is waste. +- `Gemfile.lock` carried `aarch64-linux-musl` (Alpine) but not `aarch64-linux` + (glibc), so nokogiri had no native candidate on a Debian ARM container. Fix: + `bundle lock --add-platform aarch64-linux`. +- `bin/dc` exported `DOCKER_DEFAULT_PLATFORM=linux/arm64/v8`, which never took + effect because the services pinned `linux/amd64` and won. Deleted rather than + kept-and-now-correct: it is precisely what made wrong explanation #1 below + look plausible. + +### The 8 codeblocks keys: still UNRESOLVED, and beware the false confirmation + +After changing both architecture and Chrome major version, 47 of 55 screenshots +still matched baselines recorded on amd64/Chrome-152. The 8 that did not are the +same `mobile/blog/special/codeblocks/*` family, at the same magnitudes - so the +divergence is neither arch nor Chrome version. + +**It does NOT follow that the base OS is the cause.** Local ARM (Debian) and CI +ARM (Ubuntu) fail the *same 8 keys*, which reads like confirmation and is not: +both are being compared against baselines recorded on the OLD stack, so every +hypothesis predicts red on both. The observation cannot discriminate. An earlier +commit message in this very migration asserted the distro cause anyway - the +fifth instance of the exact shape catalogued above. + +**The discriminating experiment, still to run:** re-record `linux/` via an +`update-baselines` dispatch on master, then run local `bin/dtest`. Green means +the distro never mattered; the same 8 red means it does. Until that runs, the +cause is unknown, not "Debian vs Ubuntu". ## The open decision: one rendering stack, or two? Running CI inside this same container makes local and CI identical by -construction and lets `bin/dtest` be authoritative for visuals; CI is amd64 -native so there is no emulation, and the cost is image build/pull per job - +construction and lets `bin/dtest` be authoritative for visuals; both sides are +now arm64 native so there is no emulation either way, and the cost is image +build/pull per job - **measure it before committing.** Feasibility checked 2026-08-22: the repo is PUBLIC so GHCR is free, the image is 2.54 GB uncompressed, and GitHub Actions can run a whole job inside it via the job-level `container:` key. Publish on diff --git a/.okf/log.md b/.okf/log.md index 226e74f2c..118fac248 100644 --- a/.okf/log.md +++ b/.okf/log.md @@ -51,7 +51,27 @@ make it green: restructure same-day entries under one heading, and add `timestamp` to the 23 concepts missing it (anchored to each file's last commit time, which is verifiable - never invented). -## 2026-08-22 - the codeblocks drift was Debian vs Ubuntu, after four wrong answers +## 2026-08-22 - ARM migration shipped, and the "Debian vs Ubuntu" answer is withdrawn + +**`rendering-stack`**: the whole stack moved to arm64 - CfT pinned to +`153.0.8010.5` (first Beta publishing `linux-arm64`), container native, CI on +`ubuntu-24.04-arm` ([#589](https://github.com/jetthoughts/jetthoughts.github.io/pull/589)). +Paul overrode the pin-to-Stable policy to take Beta now rather than wait for 153 +to promote. Two unplanned blockers, both of which would have hit CI as well: a +`Gemfile.lock` missing the `aarch64-linux` platform (nokogiri had no native +candidate), and `bin/dc`'s dead `DOCKER_DEFAULT_PLATFORM` export. + +**The correction that matters**: the entry below concluded the 8 codeblocks +failures were caused by Debian-vs-Ubuntu. That conclusion is **not established +and has been withdrawn from the concept.** Local ARM (Debian) and CI ARM +(Ubuntu) fail the same 8 keys, which looks like confirmation but cannot +discriminate - both compare against baselines recorded on the OLD stack, so +every hypothesis predicts red on both. Asserting it anyway was the fifth +instance of the same shape the entry below catalogues four of, committed in the +migration's own commit message. Cause is UNKNOWN until the re-record on master +lands and local `bin/dtest` runs against it. + +## 2026-08-22 - the codeblocks drift was blamed on Debian vs Ubuntu (SUPERSEDED - see above) The 8 `mobile/blog/special/codeblocks/*` failures in `bin/dtest` were blamed on arm64-vs-amd64 three separate times, and on the tolerance change, the PR diff --git a/STATUS.md b/STATUS.md index c718ef8d7..4b03d2080 100644 --- a/STATUS.md +++ b/STATUS.md @@ -24,6 +24,7 @@ | 2605 course | v2 live; measuring. Diagnosis: **arrival, not content**; course SEO/AEO **closed** (Paul 2026-08-21) | LinkedIn arrival-test cards **LI-0…LI-D** in [`content-plan`](linkedin-posts/content-plan.md); no new funnel posts on the unproven bridge | [`TASK-TRACKER`](docs/projects/2605-tech-for-non-technical-founders/TASK-TRACKER.md) | | 2607 campaign tasks | Cold-public-sourcing premise tested and **failed** (3 sweeps, 4 venues, 0 verified-fresh rows; Reddit still un-openable) | Sept-restart Paul decision: retire the cold lane or buy Reddit API access (backlog §c) | [`2607 backlog`](docs/projects/2607-vibe-code-rescue/backlog.md) | | Test/CI hygiene | **Gates rebuilt 2026-08-22.** Fault-injecting 8 realistic defects caught **3**; now catches **8** ([#576](https://github.com/jetthoughts/jetthoughts.github.io/pull/576), audit in [`20.11`](docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md)). Link job was excluding 90% of links — 114,050 checked now vs 15,642, and it found 5 real site defects ([#574](https://github.com/jetthoughts/jetthoughts.github.io/pull/574)). `bin/dtest` was comparing **nothing** from a worktree ([#578](https://github.com/jetthoughts/jetthoughts.github.io/pull/578)). CI Linux screenshot job green (run 32565008850) | **Paul decides the dtest arch policy** (below); otherwise nothing queued | [`test-gates`](.okf/build/test-gates.md) | +| Rendering stack → **all-ARM** | **In flight — [#589](https://github.com/jetthoughts/jetthoughts.github.io/pull/589) open.** Paul chose Beta over waiting for Stable (2026-08-21), so CfT pins `153.0.8010.5` (first with `linux-arm64`), container runs native arm64, CI on `ubuntu-24.04-arm`. Non-visual CI green; **Screenshot Tests red is expected** — `linux/` baselines are amd64/Chrome-152 recordings of pre-[#583](https://github.com/jetthoughts/jetthoughts.github.io/pull/583) content | Merge #589 → dispatch `update-baselines` **on master** (recorder and tester must see one tree) → local `bin/dtest` to settle whether the 8 codeblocks keys were ever distro-caused | [`rendering-stack`](.okf/build/rendering-stack.md) | **Not in flight**: 2604 typography — closed 2026-08-08 (P2 leftovers re-homed). 2509 CSS migration — **project complete 2026-07-19** (tracker in maintenance mode; the dormant remainder is the Phase-E groomed backlog + trigger-conditioned items); the 2608 clean-slate rail is the live CSS strategy ([ADR-0006](docs/adr/0006-clean-slate-dual-run.md)). @@ -32,7 +33,6 @@ | What | Where the full ask lives | |---|---| | Five 2608 decisions (register pick, 3 claims, first page, legacy CTO sweep, Sept measurement gate) | [`2608 README`](docs/projects/2608-site-design-system/README.md) | -| **Rendering stack: wait for Chrome 153 Stable, then go all-ARM.** dtest fails 8 codeblocks screenshots (local Debian 13 vs CI ubuntu-latest). Don't chase that parity now — Chrome for Testing ships `linux-arm64` from 153 (Beta today, Stable pins at 152), and migrating to arm64 both sides removes the emulation tax AND forces one re-record anyway. **Trigger: CfT Stable ≥ 153** | [`rendering-stack`](.okf/build/rendering-stack.md) | | **Joy Adamson override** (1 min; Paul's one override candidate from outreach batch 1, still publicly unanswered) | [`20.09 §1`](docs/projects/2510-seo-content-strategy/20-29-strategy/20.09-content-plan-revision-aug-2026.md) desk table + 2607 backlog card #12 | | Three 2605 fabricated-fact findings (five-tech-words client claim, $78K/$400 story, SVG chart stats) | [`2605 TASK-TRACKER`](docs/projects/2605-tech-for-non-technical-founders/TASK-TRACKER.md) §Aug-20 sweep | | 2607 T3 (Gmail warm-source consent) + T10 (split strategy docs, [#449](https://github.com/jetthoughts/jetthoughts.github.io/issues/449)) | [`2607 backlog`](docs/projects/2607-vibe-code-rescue/backlog.md) | From 6c78693914d1e144b15b9e58bc6bacda49282723 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:50:48 +0200 Subject: [PATCH 4/7] Retract two stale rendering-stack explanations the ARM migration falsified Both of these were left behind by #589 and both point a reader the wrong way. bin/dtest justified discarding local screenshot candidates by arch: "bin/dc pins DOCKER_DEFAULT_PLATFORM=linux/arm64/v8 while the committed linux/ baselines are recorded by CI on amd64, so a candidate produced here is never committable". #589 deleted that export and moved CI to arm64, so the stated reason is now false while the conclusion is still right. That combination is the dangerous one: a reader who checks the premise finds the arches now match and concludes local candidates ARE committable, which is one `git add -A` from corrupting the baseline set. Rewritten to name the reason that actually still holds - the base OS is the last unpinned rendering variable. docs 20.10 O3 recorded "the 10 red Linux baselines are NOT stale - they are emulation drift: green on CI-native amd64, red only under local Apple-Silicon emulation." Emulation ended on 2026-08-22 and the reds did not, so emulation was never the cause. Marked withdrawn; the operational rule it carried (never re-record locally, trust CI) is unchanged and still correct. No behaviour change - comments and prose only. bash -n clean, unit 289 runs / 6185 assertions / 0 failures, hugo-build green. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- bin/dtest | 11 ++++++++--- .../20.10-visual-suite-speed-research-reference.md | 10 ++++++---- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/bin/dtest b/bin/dtest index 0eaa7897f..7fd47e9f8 100755 --- a/bin/dtest +++ b/bin/dtest @@ -22,10 +22,15 @@ bin/build-if-stale _dest/public-dtest # AFTER, via trap, so a red or interrupted run cleans up after itself. The # old restore sat on the success path only, below `set -e`. # -# Discarding those candidates costs nothing: bin/dc pins -# DOCKER_DEFAULT_PLATFORM=linux/arm64/v8 on an ARM Mac while the committed -# linux/ baselines are recorded by CI on amd64, so a candidate produced here +# Discarding those candidates costs nothing, because a candidate produced here # is never committable - and it is one `git add -A` from corrupting the set. +# +# Since 2026-08-22 the ARCHITECTURE argument for that no longer applies: both +# this container and CI run arm64 on the same pinned Chrome. Do not read that +# as permission to commit local candidates. The base OS still differs (Debian +# trixie here, Ubuntu 24.04 on the runner) and that is the last unpinned +# rendering variable - see .okf/build/rendering-stack.md, which also records +# why "same failures on both sides" is NOT evidence the two agree. # The only legitimate linux re-record is the CI `update-baselines` dispatch # (.okf/build/test-gates.md). Diff artifacts (*.diff.png, *.heatmap.diff.png) # are untracked and survive the restore, so a red run stays inspectable. diff --git a/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md b/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md index 376bb0955..bfd2cef9c 100644 --- a/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md +++ b/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md @@ -68,10 +68,12 @@ minutes. Puma per process; `JOBS=1` in the dtest container — port pin + ~2 emulated CPUs.) - **O3 — Docker vs host: CONFIRMED** (~18% faster on smoke, 41.6s vs 50.5s). - The 10 red Linux baselines are NOT stale — they are emulation drift: green - on CI-native amd64, red only under local Apple-Silicon emulation. Do NOT - re-record locally (would break green CI); trust CI for those 10. See memory - `project-stale-linux-baselines-pending`. + The 10 red Linux baselines are NOT stale. **The "emulation drift" diagnosis + recorded here (green on CI-native amd64, red only under local Apple-Silicon + emulation) is WITHDRAWN as of 2026-08-22** — emulation ended when the stack + went all-arm64, and the reds did not, so emulation was never the cause. The + operational rule is unchanged and still correct: do NOT re-record locally; + trust CI. The cause is open — see [`rendering-stack`](../../../.okf/build/rendering-stack.md). - **O4 — direct-visit instead of menu-walk.** Several desktop tests reach their subject page via `visit "/"` + hover/click when a direct URL exists (`test_about_us`, `test_contact_us`, `test_free_consultation`, ...). Each pays From d61d74955f52a90c9eb499a127f75829d48d1ea6 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:54:01 +0200 Subject: [PATCH 5/7] Pin the screenshot containers to arm64 instead of following the host Codex review caught this on the branch and it is a real hole. Dropping the `platform: linux/amd64` pins made Compose follow the HOST arch, which is only equivalent to CI's arm64 when the host happens to be ARM. On an Intel Mac or x86_64 Linux box the container would render amd64 Chrome and compare it against test/fixtures/screenshots/linux/ - which CI records on ubuntu-24.04-arm. Local bin/dtest and the required CI check would then disagree by construction, on a gate CLAUDE.md marks BLOCKING. Worse, the comment on the `t` service asserted the guarantee ("Host arch, matching CI's arm64 runners") that the code only delivers on an ARM host, so a reader had no reason to look. Fix is a pin to arm64 - CI's arch - on the two services that build screenshot candidates (`t`, `sh`). This is not the amd64 pin coming back: it pins to the arch that records the baselines rather than to the wrong one. On an ARM Mac the pin IS the host arch, so the emulation win the migration was for is untouched; it costs only an x86 host, which emulates arm64 and is correct-but-slow instead of fast-and-wrong. `hugo` is deliberately left following the host - it serves pages and produces no screenshots. bin/setup-test-env (bare-metal Linux) keeps following the host, because there is nothing to pin to: an x86 CPU cannot execute arm64 Chrome. That host is structurally unable to reproduce the baselines, so it now says so on stderr rather than emitting a suite of false reds that someone could "fix" by re-recording. VERIFIED, not assumed: `docker compose config` shows platform linux/arm64 on both services and none on hugo; the container reports `aarch64` and Chrome for Testing 153.0.8010.5 under the pin; the x86 branch of setup-test-env was exercised directly and fires the warning; bash -n clean on the script; unit 289 runs / 6185 assertions / 0 failures; hugo-build green. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- .dev/compose.yml | 22 ++++++++++++++++------ bin/setup-test-env | 15 ++++++++++++++- 2 files changed, 30 insertions(+), 7 deletions(-) diff --git a/.dev/compose.yml b/.dev/compose.yml index 00a60725e..c34862456 100644 --- a/.dev/compose.yml +++ b/.dev/compose.yml @@ -25,11 +25,19 @@ services: # Interactive shell (optimized) sh: image: jetthoughts.com-test:1.0.0 - # Follows the HOST arch since 2026-08-22. Chrome for Testing began - # publishing linux-arm64 at 153 (.dev/cft-version pins 153.0.8010.5, the - # first Beta with it), and the Dockerfile picks the matching build - so an - # ARM Mac runs native instead of emulating amd64. CI moved to arm64 - # runners in the same change, so both sides still render alike. + # Pinned to arm64 since 2026-08-22, because that is what records the + # committed linux/ baselines: CI runs ubuntu-24.04-arm. Chrome for Testing + # began publishing linux-arm64 at 153 (.dev/cft-version pins 153.0.8010.5, + # the first Beta with it) and the Dockerfile picks the matching build. + # + # PIN, not "follow the host". Following the host is what an earlier draft + # did, and it silently breaks the visual gate on an x86 developer machine: + # Chrome would render amd64 while every linux/ baseline is an arm64 + # recording, so bin/dtest and required CI disagree by construction. On an + # ARM Mac this pin IS the host arch, so it is free - it costs only an x86 + # host, which then emulates arm64 and is correct-but-slow rather than + # fast-and-wrong. The gate is BLOCKING; correctness wins. + platform: linux/arm64 build: context: ../ dockerfile: ./.dev/Dockerfile @@ -52,7 +60,9 @@ services: # Test runner with optimized dependency management t: image: jetthoughts.com-test:1.0.0 - # Host arch, matching CI's arm64 runners (see `sh` service note above). + # Pinned to CI's arch, not the host's - this is the service that produces + # screenshot candidates. See the `sh` service note above for why. + platform: linux/arm64 build: context: ../ dockerfile: ./.dev/Dockerfile diff --git a/bin/setup-test-env b/bin/setup-test-env index 0fe95b4e9..3f65f2894 100755 --- a/bin/setup-test-env +++ b/bin/setup-test-env @@ -16,9 +16,22 @@ set -euo pipefail cd "$(dirname "$0")/.." CFT=$(tr -d '[:space:]' < .dev/cft-version) +# Follows the host here, unlike the container (.dev/compose.yml pins arm64): +# bare metal cannot run an arm64 Chrome on an x86 CPU, so there is nothing to +# pin TO. That makes an x86 host structurally unable to reproduce the linux/ +# baselines, which CI records on arm64 - so say so instead of rendering a +# whole suite of false reds and letting someone re-record from it. case "$(uname -m)" in aarch64|arm64) CFT_PLAT=linux-arm64 ;; - *) CFT_PLAT=linux64 ;; + *) + CFT_PLAT=linux64 + echo "WARNING: host is $(uname -m), but test/fixtures/screenshots/linux/" >&2 + echo " is recorded on arm64 (CI: ubuntu-24.04-arm). Chrome here renders" >&2 + echo " amd64, so visual comparisons will differ from CI for reasons that" >&2 + echo " are NOT defects. Non-visual suites are unaffected." >&2 + echo " Do NOT re-record linux/ from this host. Use .dev/compose.yml (it" >&2 + echo " pins arm64) or the CI update-baselines dispatch." >&2 + ;; esac CACHE="${XDG_CACHE_HOME:-$HOME/.cache}/jt-cft/$CFT-$CFT_PLAT" CHROME="$CACHE/chrome-$CFT_PLAT/chrome" From 1579f3641cb4f0d4311f95f43eb34fd896205edb Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:54:32 +0200 Subject: [PATCH 6/7] OKF: record the arch-pin rule, and make the concept internally consistent Adds the rule the Codex finding produced: pin the test container to CI's arch, never to "the host's". The two coincide only while every developer is on the same architecture, so "follow the host" is a guarantee that expires silently on the first machine that differs - and no suite fails to announce it, because the divergent host is one nobody has yet. That is why review caught it and gates did not, which is the part worth remembering. Also repairs inconsistencies my own earlier edit introduced in the same file: the section still said "Four wrong explanations" after a fifth was documented below it, explanation #1 described the container in the present tense as x86_64 when it is now arm64, and a cross-reference pointed "below" at something that had moved above it. The frontmatter description still advertised four. Bundle validates: 0 ERRORs (--strict exits 1 on the pre-existing known-red warning set only, tracked in the 2608 README). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- .okf/build/rendering-stack.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/.okf/build/rendering-stack.md b/.okf/build/rendering-stack.md index caccd18fe..a4c3821fb 100644 --- a/.okf/build/rendering-stack.md +++ b/.okf/build/rendering-stack.md @@ -1,7 +1,7 @@ --- type: build-concept title: A screenshot baseline is a recording of a rendering stack -description: Which environment records the linux/ baselines, why local dtest differs from CI, and the four wrong explanations that cost a session +description: Which environment records the linux/ baselines, why local dtest can differ from CI, the all-arm64 migration, and the five wrong explanations for one drift - two of them formally withdrawn tags: [testing, screenshots, docker, ci, rendering] timestamp: 2026-08-22T00:00:00Z --- @@ -31,12 +31,12 @@ variable, not a demonstrated cause** - see the open question below. record `linux/`. Neither OS's set is a subset of the other, and a candidate rendered on one must never be committed as a baseline for another. -## Four wrong explanations, in the order they were believed +## Five wrong explanations, in the order they were believed Each sounded mechanical and each was asserted without measuring the thing it named. Listed because the *shape* recurs, not the specific causes: -1. **"arm64 vs amd64 drift."** The container is x86_64. `.dev/compose.yml` +1. **"arm64 vs amd64 drift."** The container was x86_64 at the time. `.dev/compose.yml` pins `platform: linux/amd64` on the `t` service and that DOES override `bin/dc`'s `DOCKER_DEFAULT_PLATFORM=linux/arm64/v8` - which is a real booby-trap in the tooling, since it makes the arm64 story look right. One @@ -49,9 +49,14 @@ named. Listed because the *shape* recurs, not the specific causes: both see the same tree. 4. **"A date-gated post appeared."** Both candidate posts were dated before the baseline was recorded. +5. **"Debian container vs Ubuntu runner."** Believed 2026-08-22 and asserted in + #589's commit message. The base OS is the last unpinned variable, which makes + it the standing candidate - but it has never been measured either. Withdrawn; + see the codeblocks section below for the observation that fooled it. -The measurement that actually resolved it took one command and should have -been first: read the container's OS and library versions. +The measurement that resolved #1-#4 took one command and should have been +first: read the container's OS and library versions. #5 has no such measurement +yet, which is exactly why it is listed here rather than in Rules. ## Rules @@ -62,6 +67,14 @@ been first: read the container's OS and library versions. dispatch on **master** so recorder and tester agree. That is what fixed `mobile/blog/index/_pagination` (a real 0.0425 content difference). - **Never re-record `linux/` from a Mac**, whatever the container reports. +- **Pin the test container to CI's arch, never to "the host's".** They coincide + only while every developer is on the same architecture, so "follow the host" + is a guarantee that silently expires on the first machine that differs - + amd64 Chrome compared against arm64 baselines, on a BLOCKING gate. `.dev/ + compose.yml` pins `linux/arm64` on `t` and `sh` for this reason. Bare metal + cannot pin (an x86 CPU cannot run arm64 Chrome), so `bin/setup-test-env` + warns instead. Caught by review on #589, not by any gate - no suite fails + when the only x86 host is one nobody has yet. - A green visual run that prints no `[snap_diff] N screenshots compared` line compared nothing - see [test-gates](/build/test-gates.md). @@ -102,7 +115,7 @@ Two things the plan did not anticipate, both of which would have broken CI too: `bundle lock --add-platform aarch64-linux`. - `bin/dc` exported `DOCKER_DEFAULT_PLATFORM=linux/arm64/v8`, which never took effect because the services pinned `linux/amd64` and won. Deleted rather than - kept-and-now-correct: it is precisely what made wrong explanation #1 below + kept-and-now-correct: it is precisely what made wrong explanation #1 above look plausible. ### The 8 codeblocks keys: still UNRESOLVED, and beware the false confirmation From b2b1e014a4a50f58542103d8d1f6583e678f99c4 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:06:03 +0200 Subject: [PATCH 7/7] Record two Paul decisions as settled: arm64 is the stack, and 4g stays Both close questions my own comments had left open, which is the defect worth fixing here - a comment that reads "left until measured" is an invitation to reopen, and the next session would have spent time re-deciding. "Our last stack is arm until we will find that cannot support it" (Paul, 2026-08-22). arm64 is THE stack - container, CI, baselines - not a preference and not a trial. Reversed only on evidence something genuinely CANNOT support it. Two things that do not qualify, named explicitly because both are close to hand: "an x86 box would be slower" is a cost, not an inability; and the 8 unexplained codeblocks reds are a measurement job, not grounds to retreat - they predate the migration and survived it unchanged, so they are not evidence against ARM at all. That loose end is exactly what would otherwise invite a retreat to the old stack, so it is called out where the decision is recorded. This also sharpens the compose comment from the previous commit, which framed x86 as a supported-but-slower path. It is not a supported rendering host; the pin is the policy. `mem_limit: 4g` stays, settled rather than pending. The prior comment said "left at 4g until measured rather than guessed down", which invited exactly the optimization Paul has now ruled out. Headroom on a test container is free, and an OOM-killed Chrome is expensive twice over because it presents as a rendering difference rather than as an OOM. Recorded in .okf/build/rendering-stack.md, STATUS.md and ruflo memory (decision-arm64-is-the-stack) so a cold session finds it before relitigating. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- .dev/compose.yml | 22 ++++++++++++++-------- .okf/build/rendering-stack.md | 14 +++++++++++++- STATUS.md | 2 +- 3 files changed, 28 insertions(+), 10 deletions(-) diff --git a/.dev/compose.yml b/.dev/compose.yml index c34862456..230eae71c 100644 --- a/.dev/compose.yml +++ b/.dev/compose.yml @@ -30,13 +30,16 @@ services: # began publishing linux-arm64 at 153 (.dev/cft-version pins 153.0.8010.5, # the first Beta with it) and the Dockerfile picks the matching build. # - # PIN, not "follow the host". Following the host is what an earlier draft - # did, and it silently breaks the visual gate on an x86 developer machine: - # Chrome would render amd64 while every linux/ baseline is an arm64 + # arm64 IS THE STACK (Paul, 2026-08-22) - not a preference, not a trial. + # It is reversed only if something is found that genuinely cannot support + # it, and "an x86 machine would be slower" is not that. + # + # So this is a PIN, not "follow the host". Following the host is what an + # earlier draft did, and it silently breaks the visual gate on an x86 + # machine: Chrome renders amd64 while every linux/ baseline is an arm64 # recording, so bin/dtest and required CI disagree by construction. On an - # ARM Mac this pin IS the host arch, so it is free - it costs only an x86 - # host, which then emulates arm64 and is correct-but-slow rather than - # fast-and-wrong. The gate is BLOCKING; correctness wins. + # ARM Mac the pin IS the host arch, so it costs nothing here; on x86 it + # emulates, which is the correct trade because the gate is BLOCKING. platform: linux/arm64 build: context: ../ @@ -81,8 +84,11 @@ services: HUGO_CACHEDIR: "/tmp/hugo_cache_dtest" CAPYBARA_SCREENSHOT_ON_FAILURE: "true" RUBY_THREAD_VM_STACK_SIZE: 1048576 - # 4g was sized for amd64 EMULATION ("tab crashed" OOM at 2g). Native arm64 - # should need less; left at 4g until measured rather than guessed down. + # 4g was originally sized for amd64 EMULATION ("tab crashed" OOM at 2g). + # Native arm64 probably needs less - and 4g STAYS anyway (Paul, 2026-08-22). + # Settled, not pending: do not "optimize" this down. Headroom on a test + # container is free; an OOM-killed Chrome costs a red suite and an hour of + # diagnosis, and it fails as a rendering difference rather than as an OOM. mem_limit: 4g cpus: '4.0' ulimits: diff --git a/.okf/build/rendering-stack.md b/.okf/build/rendering-stack.md index a4c3821fb..198615b5a 100644 --- a/.okf/build/rendering-stack.md +++ b/.okf/build/rendering-stack.md @@ -78,7 +78,19 @@ yet, which is exactly why it is listed here rather than in Rules. - A green visual run that prints no `[snap_diff] N screenshots compared` line compared nothing - see [test-gates](/build/test-gates.md). -## ARM on Linux: SHIPPED 2026-08-22 (was "waiting for Stable") +## arm64 is THE stack (Paul, 2026-08-22) - shipped, not on trial + +**Standing decision: arm64 everywhere - container, CI, baselines. It is +reversed only on evidence that something genuinely cannot support it.** Slower +on some hypothetical x86 box does not qualify; neither does an unexplained +pixel diff, which is a measurement job, not grounds to go back. Treat "should +we return to amd64?" as answered unless you can name the thing ARM cannot do. + +This matters because the 8 unexplained codeblocks keys below are exactly the +kind of loose end that invites a retreat to the old stack. They predate the +migration and survived it unchanged - so they are not evidence against ARM. + +### How it became available (was "waiting for Stable") Chrome for Testing **does** publish `linux-arm64` - chrome and chromedriver both - from **153.0.8001.0** onward. An earlier check here concluded "no diff --git a/STATUS.md b/STATUS.md index 4b03d2080..e66bf279f 100644 --- a/STATUS.md +++ b/STATUS.md @@ -24,7 +24,7 @@ | 2605 course | v2 live; measuring. Diagnosis: **arrival, not content**; course SEO/AEO **closed** (Paul 2026-08-21) | LinkedIn arrival-test cards **LI-0…LI-D** in [`content-plan`](linkedin-posts/content-plan.md); no new funnel posts on the unproven bridge | [`TASK-TRACKER`](docs/projects/2605-tech-for-non-technical-founders/TASK-TRACKER.md) | | 2607 campaign tasks | Cold-public-sourcing premise tested and **failed** (3 sweeps, 4 venues, 0 verified-fresh rows; Reddit still un-openable) | Sept-restart Paul decision: retire the cold lane or buy Reddit API access (backlog §c) | [`2607 backlog`](docs/projects/2607-vibe-code-rescue/backlog.md) | | Test/CI hygiene | **Gates rebuilt 2026-08-22.** Fault-injecting 8 realistic defects caught **3**; now catches **8** ([#576](https://github.com/jetthoughts/jetthoughts.github.io/pull/576), audit in [`20.11`](docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md)). Link job was excluding 90% of links — 114,050 checked now vs 15,642, and it found 5 real site defects ([#574](https://github.com/jetthoughts/jetthoughts.github.io/pull/574)). `bin/dtest` was comparing **nothing** from a worktree ([#578](https://github.com/jetthoughts/jetthoughts.github.io/pull/578)). CI Linux screenshot job green (run 32565008850) | **Paul decides the dtest arch policy** (below); otherwise nothing queued | [`test-gates`](.okf/build/test-gates.md) | -| Rendering stack → **all-ARM** | **In flight — [#589](https://github.com/jetthoughts/jetthoughts.github.io/pull/589) open.** Paul chose Beta over waiting for Stable (2026-08-21), so CfT pins `153.0.8010.5` (first with `linux-arm64`), container runs native arm64, CI on `ubuntu-24.04-arm`. Non-visual CI green; **Screenshot Tests red is expected** — `linux/` baselines are amd64/Chrome-152 recordings of pre-[#583](https://github.com/jetthoughts/jetthoughts.github.io/pull/583) content | Merge #589 → dispatch `update-baselines` **on master** (recorder and tester must see one tree) → local `bin/dtest` to settle whether the 8 codeblocks keys were ever distro-caused | [`rendering-stack`](.okf/build/rendering-stack.md) | +| Rendering stack → **arm64 is THE stack** | **Settled (Paul 2026-08-22): arm64 everywhere, reversed only if something is found that genuinely cannot support it.** Shipping in [#589](https://github.com/jetthoughts/jetthoughts.github.io/pull/589): CfT pins `153.0.8010.5` (first with `linux-arm64`), container + CI both `arm64` (`ubuntu-24.04-arm`), compose PINS arm64 rather than following the host. Non-visual CI green; **Screenshot Tests red is expected** — `linux/` baselines are amd64/Chrome-152 recordings of pre-[#583](https://github.com/jetthoughts/jetthoughts.github.io/pull/583) content. `mem_limit: 4g` stays (Paul) — settled, not pending | Merge #589 → dispatch `update-baselines` **on master** (recorder and tester must see one tree) → local `bin/dtest`. The 8 codeblocks keys are an open *measurement*, NOT a reason to revisit arm64 — they predate the migration and survived it unchanged | [`rendering-stack`](.okf/build/rendering-stack.md) | **Not in flight**: 2604 typography — closed 2026-08-08 (P2 leftovers re-homed). 2509 CSS migration — **project complete 2026-07-19** (tracker in maintenance mode; the dormant remainder is the Phase-E groomed backlog + trigger-conditioned items); the 2608 clean-slate rail is the live CSS strategy ([ADR-0006](docs/adr/0006-clean-slate-dual-run.md)).