Linux build base (bullseye) is disintegrating NOW — index advertises packages its pool lacks, per-arch; every .deb build is red

[MichaelTaylor3d]

The Linux build base is past end-of-life and its apt index EXPIRES 2026-09-07

Measured directly from deb.debian.org on 2026-09-05:

Suite:        oldoldstable-security
Codename:     bullseye-security
Date:         Mon, 31 Aug 2026 21:13:04 UTC
Valid-Until:  Mon, 07 Sep 2026 21:13:04 UTC

Debian 11 "bullseye" is oldoldstable, and the Release file it is built against expires in two days. Once Valid-Until lapses, apt rejects the index outright — Release file ... is not valid yet / has expired — and every .deb build fails, on every architecture, for a reason unrelated to whatever else is fixed by then.

This is NOT the same defect as the current red

The deb legs are red right now for a different cause, already root-caused: deb.debian.org's security pool is missing objects that canonical security.debian.org serves.

404  http://deb.debian.org/debian-security/pool/updates/main/p/perl/libperl5.32_5.32.1-4%2bdeb11u5_amd64.deb
200  http://security.debian.org/debian-security/pool/updates/main/p/perl/libperl5.32_5.32.1-4%2bdeb11u5_amd64.deb

Same path, same filename, different host. The index was verified fresh and correct (Last-Modified recent, Age: 67, naming a real current version). That fix is in flight in dig-node PR #564 and will make CI green. It buys days, not a solution — this ticket is the solution.

Why the obvious workaround is the wrong answer

Acquire::Check-Valid-Until "false" will silence the expiry and keep builds green. Do not reach for it as the fix. It disables the freshness signal on a security suite that has stopped being maintained — so the build would silently keep installing packages from an archive nobody is patching, and the next real problem arrives with no warning at all.

It is acceptable only as an explicitly-labelled, dated stopgap while the real decision below is made.

The real decision, which needs a person to make it

The base is bullseye on purpose — it sets the glibc floor for the shipped binaries, and dig-node has a live, tracked platform-floor concern (#1741 / #1736, plus dig-app#238's missing glibc guard). Raising the base raises the floor and drops support for older Linux hosts, which is precisely the audience the old base exists to serve.

So this is not a mechanical bump. State, explicitly, in whatever change lands:

1. What glibc floor the new base implies, and which distributions that excludes.
2. Whether the floor is still asserted after the change — the composite action has a Verify the container glibc IS the declared floor step, and that assertion must be updated deliberately rather than allowed to pass vacuously against a new base.
3. Whether bookworm (current oldstable, glibc 2.36) is acceptable, or whether the floor genuinely requires staying on an EOL base with a pinned snapshot mirror.

A pinned snapshot.debian.org source is the option that keeps the floor AND stops the rot — it makes the build reproducible and immune to both pool pruning and index expiry, at the cost of never receiving security updates for the build container. That trade may well be right for a build container; it should be chosen, not defaulted into.

Done when

• The deb builds are green after 2026-09-07 without Check-Valid-Until disabled, or with it disabled behind an explicit dated comment naming this ticket and the chosen long-term path.
• The declared glibc floor and its verification step agree with whatever base is in use.

Related

• dig-node PR ci: harden the .deb job's apt bootstrap against bullseye mirror-skew 404s #564 — the mirror-host fix that makes CI green now (does not address expiry)
• #1741 / #1736 — the arm64/glibc platform floor

[MichaelTaylor3d]

Escalation: this is no longer two days away. The suite is already coming apart, and it is breaking every .deb build right now.

The failure mode is not the Valid-Until expiry this ticket was filed for. It is worse and it arrived first: bullseye-security's index advertises versions its own pool does not carry, and the disagreement differs per architecture.

Measured directly against security.debian.org, 2026-09-05:

libglib2.0-0  2.66.8-1+deb11u8    amd64 -> 404      (index names u8)
openssl       1.1.1w-0+deb11u8    arm64 -> 404      (index names u8)
openssl       1.1.1w-0+deb11u8    amd64 -> 200
libcurl4      7.74.0-1.3+deb11u16 arm64 -> 200      (yet 404'd in CI)

No mirror choice and no retry strategy can fix an index that disagrees with its own pool. Four hypotheses were tested and discarded getting here — stale index (refuted, index verified fresh), poisoned container (real, fixed, not the cause), wrong mirror host (real, fixed — libperl5.32 u5 is 6/6 from security.debian.org and 0/6 from deb.debian.org), and transient per-edge 404s (partly real, retries do not clear them).

The mitigation landing now, and why it needs no decision

dig-node PR #564 pins both archives to snapshot.debian.org at a fixed timestamp. Verified internally consistent — it carries every package that 404s live, and its index names the same versions:

snapshot 20260901T000000Z
200  libglib2.0-0_2.66.8-1+deb11u8_amd64.deb
200  openssl_1.1.1w-0+deb11u8_arm64.deb
200  libperl5.32_5.32.1-4+deb11u5_amd64.deb
200  libexpat1_2.2.10-2+deb11u7_amd64.deb

Critically, it does not change the base image, so the glibc floor is untouched. That is what makes it safe to land without answering this ticket's question.

One correction to this ticket, so it is not misread

This ticket says "do not reach for Acquire::Check-Valid-Until false as the fix." That warning stands for a live suite, where it silences a real signal about an unmaintained archive.

It does not apply to a deliberately-pinned snapshot. A snapshot's Release file is frozen and will inevitably age past Valid-Until; disabling the check is the documented way to consume snapshot.debian.org, and there is no signal to hide because the content cannot change. PR #564 adds it for that reason and says so in a comment.

What still needs deciding — unchanged, and now on a clock

The snapshot pin buys time; it does not answer the question. A pinned build container never receives security updates, which is a defensible trade for a build container and an indefensible one to drift into unnoticed.

Still required, per the original ticket:

1. What glibc floor a newer base implies, and which distributions it excludes.
2. Whether the Verify the container glibc IS the declared floor step still asserts something real afterwards, rather than passing vacuously.
3. Whether bookworm (glibc 2.36) is acceptable, or the floor genuinely requires staying on an EOL base with a pinned snapshot permanently.

If the answer is "stay pinned permanently", that is a legitimate outcome — but it should be chosen and written down, with the no-security-updates consequence stated, rather than becoming the default because the snapshot pin made the pain stop.

[MichaelTaylor3d]

Mitigation MERGED — dig-node PR #564, 0.254.81. Both .deb legs are green again. This ticket stays open; the pin buys time, it does not answer the question.

Verified green for the right reason, not merely green. The risk with this class of fix is a build that passes by silently skipping a dependency, so I read the log rather than the check mark:

deb http://snapshot.debian.org/archive/debian/20260901T000000Z bullseye main
deb http://snapshot.debian.org/archive/debian/20260901T000000Z bullseye-updates main
deb http://snapshot.debian.org/archive/debian-security/20260901T000000Z bullseye-security main
echo 'Acquire::Check-Valid-Until "false";' > /etc/apt/apt.conf.d/99snapshot-no-valid-until

91 packages actually fetched. Zero 404s. Zero retry attempts. Among them openssl 1.1.1w-0+deb11u8 — the exact package that 404s on the live mirror for amd64's sibling architecture. Both archives are pinned, not just security; pinning only security would have left the main archive exposed to the same rot.

--fix-missing remains deliberately absent, so a fetch failure still aborts rather than proceeding with a partial install.

What is now true

• The bleeding has stopped, and it stopped without touching the base image — so the glibc floor is unchanged and no decision was pre-empted.
• The build container no longer receives security updates. That is the cost of a pinned snapshot, it is real, and it is invisible in CI. Nothing will go red when a vulnerability lands in a pinned package.

What still needs deciding

Unchanged from the original ticket, and the deadline pressure is off — but the question is now easier to forget, which is the reason for writing this down:

1. What glibc floor a newer base implies, and which distributions it excludes.
2. Whether the Verify the container glibc IS the declared floor step still asserts something real afterwards.
3. Whether bookworm (glibc 2.36) is acceptable, or the floor genuinely requires staying pinned.

"Stay pinned permanently" is a legitimate answer for a build container — but it must be chosen, with the no-security-updates consequence stated, rather than inherited by default because the pin made the symptom disappear.

One maintenance obligation the pin creates: the snapshot timestamp 20260901T000000Z is now a fixed input to every Linux build. Moving it is a deliberate act; leaving it is also a deliberate act. Whoever answers the questions above should say which.