ci: harden the .deb job's apt bootstrap against bullseye mirror-skew 404s

[MichaelTaylor3d]

DO NOT MERGE -- gate round in progress

Version: 0.254.81 (main is at 0.254.79 after #561; #562 holds 0.254.80 -- 0.254.81 assigned by
the coordinator, since only they can see every sibling's claim). Rebased onto origin/main cleanly --
no dropping in the rebase output, no conflicts (this branch never touched Cargo.toml before the
bump, so there was nothing to drop). Cargo.lock's dig-node-service entry refreshed via
cargo update -p dig-node-service --offline -- exactly one line changed, no other dependency moved.

Fixing the deb (linux-amd64)/deb (linux-arm64) apt-bootstrap 404s (Native install packages
workflow). Draft until the two deb legs have run green for real on this branch.

Root cause -- FINAL (superseded two earlier hypotheses; keep this so nobody re-derives them)

deb.debian.org's debian-security POOL is missing objects for this suite that the canonical
security.debian.org serves. Same path, same filename, different host, checked directly:

404  http://deb.debian.org/debian-security/pool/updates/main/p/perl/libperl5.32_5.32.1-4+deb11u5_amd64.deb
200  http://security.debian.org/debian-security/pool/updates/main/p/perl/libperl5.32_5.32.1-4+deb11u5_amd64.deb

The index was never stale -- checked directly, which is what rules out re-fetching (retries,
rm -rf /var/lib/apt/lists/*, a fresh apt-get update) as a fix:

Last-Modified: Mon, 31 Aug 2026 19:41:45 GMT     Age: 67     X-Cache: HIT
Package: libperl5.32   Version: 5.32.1-4+deb11u5

A correctly-current index named this exact file, on a host whose pool does not have it. u5, u6
and u7 of libperl5.32 all 404 the same way on deb.debian.org -- a persistent gap, not a
point-in-time rotation. Reproduces specifically on amd64 because GitHub's ubuntu-latest pool is
routed (by deb.debian.org's anycast/CDN) to an edge (146.75.30.132) with this gap; other network
paths (this session's own sandbox, and evidently every path to security.debian.org) reach a
working edge -- which is exactly what made this look like transient mirror/CDN skew across two
earlier rounds of hypothesis-and-fix in this PR.

Fix, verified locally end-to-end (docker run debian:11, not blind): sed the
debian-security line in /etc/apt/sources.list to security.debian.org before any apt-get
call. Without it, apt-get install ca-certificates curl git 404s on libperl5.32 every time; with
it, every package installs and configures cleanly.

Second finding -- a real deadline, reported as its own decision, NOT resolved here.
bullseye-security's Release file states Valid-Until: Mon, 07 Sep 2026 21:13:04 UTC -- two
days out at the time of this PR. Bullseye is oldoldstable; once that date passes, apt rejects
the index outright ("Release file expired") and every leg fails again, for a different reason,
regardless of host. Do not silence this with Acquire::Check-Valid-Until "false" -- that
suppresses a real signal that the base image is past end of life. This is exactly the glibc-floor
base-image question from the family's original brief (debian:11 = glibc 2.31, deliberately pinned
per dig_ecosystem #1736/#1741), now with a concrete deadline instead of being theoretical. Raising
the base without a deliberate floor decision would silently regress the floor the pin exists to
protect -- so this is flagged for the coordinator to decide, not changed in this PR. Tracked as
dig-node#565 (filed by the coordinator) -- not this PR's to fix.

A SECOND, different cause -- found after the host fix landed and was confirmed working

The security.debian.org fix is correct and confirmed: on commit 8720049b, libperl5.32 fetched
successfully (Get:12 http://security.debian.org/... libperl5.32 ... [4102 kB]) where it 404'd on
deb.debian.org every time before. That was Cause 1.

But that same run still failed -- on libexpat1 (u7, current and present; u8 does not exist), from
THREE different Fastly POP IPs across three attempts (151.101.194.132, 151.101.130.132,
151.101.2.132), and on a different package each time (libexpat1, then curl, then
libexpat1 again). Measured independently: libexpat1 u7 from security.debian.org is 20/20 ->
200 from one vantage. The object exists; a specific edge is serving a stale or negative cache
entry for it at the moment CI asks.

This is Cause 2, and it needed a different remedy than Cause 1: -o Acquire::http::No-Cache=true
added to the install (forces revalidation so a retry cannot be re-served the same cached response --
rm -rf /var/lib/apt/lists/* already forces a fresh INDEX; this forces fresh PACKAGE fetches, the
layer actually failing), plus a longer backoff (attempt*10 instead of attempt*5 -- the previous
5-attempt run completed in ~25s, not long enough for an edge to matter). Comments in all three files
rewritten to describe BOTH causes with their measurements, replacing the earlier "pool prunes
superseded files as point releases land" framing, which matched neither.

Earlier (superseded) root-cause hypotheses -- kept only so these dead ends are not re-walked

.github/workflows/package.yml:88 runs the deb job in container: debian:11. Its bootstrap step
(package.yml:101-105) already runs apt-get update -qq immediately before
apt-get install ... ca-certificates curl git, in the same run: block — confirmed from the
failing job log (update at 05:52:31.269Z, install at 05:52:32.494Z, ~1.2s apart). So this is
not a Docker-layer-caching issue.

It is a genuine, intermittent inconsistency between the Debian bullseye (bullseye-security) index
and the pool state served by whichever mirror/edge a given request lands on: a Packages index
fetched from one edge can briefly list a .deb that a different edge/mirror has already pruned
after a point release. Evidence: a rerun 404s on a different package each time (libexpat1
arm64 in run 33948322114, libperl5.32 amd64 in the same run) -- exactly the rotating-pool
signature, not a fixed broken URL. A same-request retry of the identical URL does not reliably help
(it can hit the same edge); a live probe just now got 200 on both of those exact files from
deb.debian.org, confirming the origin/mirror network is not actually missing them -- it is a
point-in-time skew, and it self-heals.

Bullseye is not archived/EOL: archive.debian.org does not yet carry bullseye-security
(404 on its Release file), while deb.debian.org serves it fine. So this is ordinary
oldstable-security churn, not the start of an archival cutover.

Retry loop + state reset -- KEPT, but demoted: this is NOT what fixes the failure above

These were built across two earlier (superseded) hypotheses and are kept as genuine
defense-in-depth for real transients -- they are cheap and correct -- but five independent,
verified-fresh re-fetches all returned the identical 404 in the run that tested them alone
(33949346891's successor runs), which is exactly the tell that nothing about retrying or
refreshing state was ever going to fix a file that is persistently absent from one host's pool.

Wrap the whole apt-get update && apt-get install sequence in a bounded retry loop (5 attempts,
linear backoff) in all three places this exact failure mode exists:

1. package.yml:101-105 -- the deb job's own bootstrap (the reported failure).
2. build-binaries.yml:184-188 -- the build-linux job's identical bootstrap. This is the job
   that produces the actual shipped release binaries, in the same container: debian:11, with the
   literal same two-line apt-get update -qq / apt-get install ... ca-certificates curl git. A
   local composite action cannot run before actions/checkout
   (setup-linux-build/action.yml:14-16: "each calling job bootstraps exactly those three packages
   first"), so this duplication is structural and cannot be hoisted into one shared step -- both
   copies need the same fix independently.
3. setup-linux-build/action.yml:38-48 -- the shared composite action's OWN toolchain install
   (build-essential pkg-config perl make cmake clang binutils, same container, same failure
   class). This one genuinely is centralized already (both callers invoke it), so fixing it here
   covers the toolchain-install half for both deb and build-linux in one place.

Each retry re-runs apt-get update AND apt-get install together, so every attempt gets a genuinely
fresh index -- this is NOT "retries alone" against a cached/stale index. -o Acquire::Retries=3
(transport-level HTTP retry) stays as a harmless belt-and-brace.

--fix-missing was in the first pushed version of this fix and has been REMOVED after a real
run exposed a worse failure mode than the one it was meant to guard: it let apt-get install
proceed past the 404 and unpack/attempt-configure the packages it COULD fetch, leaving
git/perl/liberror-perl half-configured (dpkg: "git depends on perl; however: Package perl is
not configured yet"). Every subsequent retry attempt then saw git/curl/ca-certificates
"already the newest version" and never touched the network for the missing file again -- so the
retry loop degraded into 5 identical dependency-resolution failures instead of 5 genuinely fresh
attempts. Without --fix-missing, a failed fetch aborts the whole transaction before unpacking
anything (confirmed against the ORIGINAL failing log, which shows zero "Setting up" lines), so every
retry now starts from a clean container state, which is what the loop needs to work at all.

Additional state reset added (dpkg --configure -a || true, apt-get -f install -y || true,
apt-get clean, rm -rf /var/lib/apt/lists/* before every retry): this DID work as designed --
confirmed on the run that tested it, the Unmet dependencies cascade above was gone and all 5
attempts became genuine, independent re-fetches. It just turned out there was nothing wrong to
refetch (see Root Cause above) -- it's still correct to keep for a genuine future transient.

Verification -- pinned to head SHA, per the correction that earlier reports weren't

build-binaries.yml is workflow_call-only and never fires standalone, so a normal PR cannot
exercise fix #2. Piggybacking the sanctioned no-op path: release.yml has a bare
workflow_dispatch (no inputs) that runs build (→ build-binaries.yml) but skips
publish/verify/promote (all gated if: github.ref_type == 'tag'), so dispatching it against
THIS branch exercises the fixed build-linux job with zero risk of publishing anything.

• Run 1, commit 2e756793 (--fix-missing present) -- Native install packages #33949346891:
  deb (linux-amd64) FAILED after 5 attempts, all on the same libperl5.32 404 from the same
  IP (146.75.30.132) -- the --fix-missing poisoning. deb (linux-arm64) SUCCEEDED.
• Run 2, commit 3cd37ea7 (--fix-missing removed only): amd64's bootstrap step passed --
  removing --fix-missing alone was enough for THAT attempt, but did not address the actual
  100%-reproducible pool gap (see below).
• Run 3, commit f944ce5a (+ state reset -- dpkg --configure -a/apt-get clean/rm -rf /var/lib/apt/lists/*): confirmed the Unmet dependencies cascade is gone and all 5 attempts
  become genuine re-fetches -- and confirmed they all still 404 identically, which is what proved
  the pool gap rather than staleness.
• Run 4, commit 8720049b (the security.debian.org sed): confirmed libperl5.32 fetched
  successfully -- Cause 1 fixed. deb (linux-amd64) still FAILED, this time on libexpat1
  from three different Fastly POPs across three attempts -- Cause 2, described above.
• Run 5, commit 579a6c2d (current HEAD -- + No-Cache + longer backoff):
  Native install packages run 33951066050, Release (stable) canary run 33951102579, both
  in progress at time of writing. Will report their deb/build-linux leg conclusions from THIS
  head SHA specifically once terminal.

Base image -- kept debian:11 in THIS PR; the decision is flagged, not made here

debian:11 (bullseye, glibc 2.31) is a deliberate pin (setup-linux-build/action.yml:1-36,
dig_ecosystem #1736/#1741): the container's own glibc IS the shipped floor (asserted by
ldd --version in that action), chosen to cover Ubuntu 20.04/22.04, Debian 11/12, RHEL 9 and
Amazon Linux 2023. Raising the base image would silently raise the floor unless accompanied by a
deliberate floor bump + SPEC.md §11.3 + docs update.

This is no longer purely theoretical. bullseye-security's Release file carries
Valid-Until: Mon, 07 Sep 2026 21:13:04 UTC -- two days out. Once it lapses, apt rejects the index
outright regardless of host, and every leg fails again for a different reason. This PR does NOT
paper over that with Acquire::Check-Valid-Until "false" and does NOT change the base image --
that call belongs to the coordinator, who can weigh the floor guarantee against the imminent
expiry, and it's reported as its own decision above.

Scope

CI-infra only. No Rust source touched, no version bump (workflow-only change; ask before taking a
version number if the gate demands one).

Release-path trace (requested by the coordinator, not fixed here)

Both the stable and nightly release pipelines already fail closed against a failing deb leg, via
mechanisms that predate this PR:

• Stable (release.yml + verify-release-assets.yml): package.yml's own publish job
  (needs: [deb, pkg, msi], the whole matrix must be green) never attaches any .deb if either arch
  fails. verify-release-assets.yml (check-release-assets/action.yml:90-106) requires
  dig-node_<ver>_amd64.deb and _arm64.deb explicitly -- "stricter than feedsign's own failure
  condition" by the action's own comment -- and polls for up to 75 minutes before failing. promote
  (the job that flips releases/latest) needs verify, so an incomplete release never becomes
  latest; the previous good release keeps serving installs.
• Nightly (nightly-release.yml): nightly-publish needs [nightly-meta, nightly-build, nightly-packages] with no failure override, so a failed deb leg inside nightly-packages skips
  nightly-publish entirely -- that night's rolling nightly tag simply stays on the previous
  night's build (stale, not broken/silent-partial). nightly-publish also has its own explicit
  fail-closed assertion on the three native-package names as belt-and-braces
  (nightly-release.yml ~420-445).

So the worst-case impact of this bug, as it stands, is a delayed/stalled stable or nightly
release (visible as a real red on verify/nightly-packages), never a silently incomplete or stale
artifact promoted to users. No release-gating change is proposed or needed for that reason.