THRIFT-6234: Configure apt retries and timeouts in GitHub Actions workflows

[slachiewicz]

Every Linux job installs packages with apt-get at apt's defaults. This adds one apt configuration fragment, .github/apt/99-ci-network, and copies it into /etc/apt/apt.conf.d/ before each of the 22 apt-get update steps in build.yml, cmake.yml, sca.yml and make-dist.yml.

The runner images resolve packages through a mirror list (mirror+file:/etc/apt/apt-mirrors.txt: azure.archive.ubuntu.com, then archive.ubuntu.com, then security.ubuntu.com). apt only hands an item to the next mirror once it has failed on the current one, and on a transient failure it first retries the same mirror Acquire::Retries times, which already defaults to 3 in the apt 2.4 and 2.8 the runners ship. The http method also reconnects and resends once on its own before it reports a failure. A stalled mirror therefore costs 4 attempts x 2 x 30 s plus back-off, about four minutes per item, before apt falls over. That is how the compiler job of run 34574616061 fetched 8 of 132 packages in its 10 minute step budget.

The fragment sets Acquire::Retries to 1 and the http and https inactivity timeouts to 20 s, so a stalled item falls over to the next mirror after about 80 s instead of four minutes. One retry is kept because the first CI run of this PR showed a package stalling on all three mirror names within 80 s (job cross-test (rb.thin, java,kotlin) of run 34595468854); with no retries that failed the job outright, while the other stalls in the same run fell over to the next mirror after 40 s and their jobs passed. It also sets DPkg::Lock::Timeout to 120 s, which is an addition beyond the retry and timeout change: apt-get fails immediately when another apt process holds the lock, and this makes it wait instead.

This bounds intermittent stalls, which is what the failed run showed. A primary mirror that stalls on every connection still costs 80 s per package and would exceed a 10 minute step; only dropping the mirror from the list would fix that. The https timeout path is not exercised by the measurement below, and the loopback stall is a stand-in for the Azure network.

No open PR touches apt configuration. #3816 adds job-level timeouts to build.yml on different lines; whichever lands second rebases. Ticket: THRIFT-6234.

Verified: apt-config dump with the fragment mounted on ubuntu:22.04 (apt 2.4.14) and ubuntu:24.04 (apt 2.8.3) → all four settings present. Against a local mirror that accepts connections and never answers, apt-get install sl on ubuntu:22.04 with the fragment → .deb ignored on the stalled mirror at 41 s and 82 s, fetched from the fallback mirror, install complete at 82 s (40 s with Acquire::Retries 0); the apt-get update phase with defaults had not reached the fallback mirror after 500 s. zizmor 1.30.1 on the four workflows → no findings.

This change was created with AI assistance.

[slachiewicz]

Force-pushed: Acquire::Retries changed from 0 to 1 after the first CI run. Job cross-test (rb.thin, java,kotlin) of run 34595468854 had libboost-atomic-dev stall on all three mirror names within 80 s and failed with no retries left, while the other stalls in that run fell over to the next mirror and passed. One retry keeps the failover at about 80 s per item and gives each mirror a second attempt. PR body updated with the measurements.

This comment was created with AI assistance.

[slachiewicz]

The Azure apt mirror has been stalling intermittently all morning, so the Build workflow runs since 10:30 UTC give a same-conditions comparison of the apt install steps (45 Linux jobs per run) with and without this change.

Build run	median	p90	worst step	steps over 60 s	failed
master 10:35, without	31 s	52 s	622 s (lib-netstd)	3	0
master 10:46, without	33 s	52 s	203 s (lib-dart 3.2.0)	3	0
this PR run 1, retries 0	32 s	74 s	134 s (cross-test cpp)	5	1
this PR run 2, retries 1	30 s	46 s	179 s (cross-test nodejs)	3	0
THRIFT-4244-doc 12:12, without	33 s	53 s	501 s (cross-test cpp)	2	0
THRIFT-6235 12:22, without	31 s	55 s	323 s (lib-python 3.12)	2	0

Medians are the same everywhere, so the normal path is unchanged. The difference is the tail. Without the change a stalled package is retried on the same mirror four times and nothing is logged: the cross-test cpp step in the THRIFT-4244-doc run waited 311 s and then 155 s on two packages from azure.archive.ubuntu.com, 501 s in total, and the lib-netstd step on master took 622 s. With the change the worst step shows both stalled packages abandoned after the one retry and fetched from the next mirror, 179 s in total.

The one failure was run 1 with retries at 0: one package stalled on all three mirror names within 80 s and apt had nothing left to try. Retries 1 covers that at the cost of a slower failover per package (about 80 s instead of 40 s).

@Jens-G @fishy could you take a look and decide? Open points from my side are the retry count (1 as pushed, or something else), whether the DPkg::Lock::Timeout addition is welcome, and whether a 20 s inactivity timeout is acceptable for the runners.

This comment was created with AI assistance.