Skip to content

Publish the full DocumentDB package set and prove the repository installs #149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guanzhousongmicrosoft merged 2 commits into from
Aug 24, 2026
Merged

Publish the full DocumentDB package set and prove the repository installs #149

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
227f428
Publish the full DocumentDB package set and prove the repo installs
GuanzhouSong Aug 24, 2026
871e34c
Never fill gaps from releases newer than a pinned version
GuanzhouSong Aug 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
316 changes: 235 additions & 81 deletions .github/scripts/download_packages.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,26 +73,165 @@ generate_hashes() {

echo "Downloading packages from $REPO releases"

if [ "$DOCUMENTDB_VERSION" = "latest" ]; then
if release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases" | python3 -c "import sys, json; releases = json.load(sys.stdin); print(json.dumps(releases[0])) if releases else sys.exit(1)")
then
tag="$(echo "$release" | python3 -c "import sys, json; print(json.load(sys.stdin)['tag_name'])")"
echo "Using latest release: $tag"
else
echo "Error: Could not fetch latest release information"
exit 1
fi
else
tag="$DOCUMENTDB_VERSION"
if ! release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases/tags/$tag")
then
echo "Error: Version $tag not found in releases"
exit 1
fi
echo "Using specified release: $tag"
# ---------------------------------------------------------------------------
# Release selection
#
# The primary release supplies the site's release-info.json and is the version
# users are told about. But a release only ships the distributions that were
# in its own build matrix: v0.116-0, for example, ships Tier-1 (ubuntu24 +
# rhel9) only, while v0.114-0 shipped seven distributions. Rebuilding the
# repository from the primary release alone would therefore DELETE the
# deb11/deb12/deb13/ubuntu22 components and the whole rhel8 repository from
# documentdb.io, and every host already pointed at one of them would start
# failing `apt update` with "Component 'ubuntu22' is not defined". That is a
# client-visible outage, not a cosmetic regression.
#
# So the repository is built additively: the primary release fills every
# distribution it ships, then progressively older releases are consulted ONLY
# to fill distributions still missing. A distribution is claimed by the newest
# release that ships it and is never overwritten by an older one. Once a
# release ships every distribution again, the older ones stop contributing
# by themselves - no cleanup required.
# ---------------------------------------------------------------------------
MAX_RELEASES="${MAX_RELEASES:-8}"

RELEASES_JSON=$(mktemp)
if ! curl -fqs "https://api.github.com/repos/${REPO}/releases?per_page=100" > "$RELEASES_JSON"; then
echo "Error: Could not fetch release list"
exit 1
fi

# Ordered list of tags to consider, newest first. Drafts and prereleases are
# skipped: they are not what a repository-backed `apt install` should serve.
TAG_LIST=$(DOCUMENTDB_VERSION="$DOCUMENTDB_VERSION" python3 - "$RELEASES_JSON" <<'PY'
import json, os, sys

releases = json.load(open(sys.argv[1]))
published = [r for r in releases if not r.get("draft") and not r.get("prerelease")]
if not published:
sys.exit("Error: no published releases found")

requested = os.environ.get("DOCUMENTDB_VERSION", "latest")
if requested != "latest":
# The API returns releases newest-first.
index = next((i for i, r in enumerate(published)
if r["tag_name"] == requested), None)
if index is None:
sys.exit(f"Error: Version {requested} not found in releases")
# A pin means "serve this version". Only the pinned release and OLDER ones
# may contribute: pulling gap-fillers from NEWER releases would defeat the
# pin, and worse, it would mix releases that depend on each other. Pinning
# to a release that predates the multi-package layout would otherwise add a
# newer `documentdb` meta package whose `documentdb-N (>= X)` dependency the
# pinned extension cannot satisfy - an unsatisfiable repository.
ordered = published[index:]
else:
ordered = published

print("\n".join(r["tag_name"] for r in ordered))
PY
)

PRIMARY_TAG=$(printf '%s\n' "$TAG_LIST" | head -n 1)
echo "Primary release: $PRIMARY_TAG"

# Packages already placed, keyed by "pool|name|arch". A newer release always
# wins; an older release contributes only packages the newer ones do not
# provide at all. That is what keeps `postgresql-16-documentdb` alive on
# ubuntu24/rhel9 after v0.116-0 narrowed Tier 1 to PostgreSQL 17/18, instead of
# silently deleting a package the install docs still tell people to use.
SEEN_FILE=$(mktemp)

is_claimed() { case " $1 " in *" $2 "*) return 0 ;; *) return 1 ;; esac; }

# "name|arch" identity of a package file, independent of its version.
# postgresql-16-documentdb_0.114-0_amd64.deb -> postgresql-16-documentdb|amd64
# documentdb_0.116.0_all.deb -> documentdb|all
# postgresql16-documentdb-0.114.0-1.el8.x86_64.rpm -> postgresql16-documentdb|x86_64
# documentdb-17-0.116.0-1.noarch.rpm -> documentdb-17|noarch
package_identity() {
local f="$1"
case "$f" in
*.deb)
printf '%s|%s' "${f%%_*}" "$(printf '%s' "$f" | sed -E 's/.*_([^_]+)\.deb$/\1/')"
;;
*.rpm)
local base="${f%.rpm}"
local arch="${base##*.}"
local nvr="${base%.*}"
# Drop the trailing VERSION and RELEASE fields to leave the package name.
printf '%s|%s' "$(printf '%s' "$nvr" | sed -E 's/-[^-]+-[^-]+$//')" "$arch"
;;
esac
}

claim_package() {
# claim_package <pool> <filename> -> 0 when this is a new package for the pool
local key="$1|$(package_identity "$2")"
if grep -qxF "$key" "$SEEN_FILE" 2>/dev/null; then
return 1
fi
printf '%s\n' "$key" >> "$SEEN_FILE"
return 0
}

# Map an asset filename to its APT component, or empty when it is not a
# distribution-prefixed .deb. Any DocumentDB package for that distribution
# matches - the extension, the gateway, the tools, documentdb-common, the
# per-major stand-alone and the meta package - because the v0.116-0 packaging
# redesign ships all of them and an extension-only filter would silently drop
# every package that makes `apt install documentdb` resolve.
deb_component_for() {
case "$1" in
*dbgsym*) echo "" ;;
deb11-*.deb) echo "deb11" ;;
deb12-*.deb) echo "deb12" ;;
deb13-*.deb) echo "deb13" ;;
ubuntu22.04-*.deb) echo "ubuntu22" ;;
ubuntu24.04-*.deb) echo "ubuntu24" ;;
*) echo "" ;;
esac
}

deb_pool_for() {
case "$1" in
deb11) echo "$DEB_POOL_DEB11" ;;
deb12) echo "$DEB_POOL_DEB12" ;;
deb13) echo "$DEB_POOL_DEB13" ;;
ubuntu22) echo "$DEB_POOL_UBUNTU22" ;;
ubuntu24) echo "$DEB_POOL_UBUNTU24" ;;
esac
}

# RPM naming is less uniform than DEB. The extension RPMs carry a distro
# prefix (rhel9-...), the gateway carries a dist tag (....el9.x86_64.rpm), and
# the meta / per-major / common / tools RPMs are noarch with NO dist tag at
# all, so they cannot be routed by name. Those EL-agnostic packages are placed
# in exactly the pools this release populated via its prefixed assets - never
# into a pool served by a different release, where their >= dependencies on a
# same-version documentdb-N would be unsatisfiable.
rpm_pool_for() {
case "$1" in
*debuginfo*|*debugsource*) echo "" ;;
rhel8-*.rpm) echo "rhel8" ;;
rhel9-*.rpm) echo "rhel9" ;;
*.el8.*.rpm) echo "rhel8" ;;
*.el9.*.rpm) echo "rhel9" ;;
*) echo "" ;;
esac
}

mkdir -p out/packages

for tag in $TAG_LIST; do
MAX_RELEASES=$((MAX_RELEASES - 1))
[ "$MAX_RELEASES" -lt 0 ] && break

if ! release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases/tags/$tag"); then
echo "::warning::Could not fetch release $tag, skipping"
continue
fi

ASSETS_FILE=$(mktemp)
echo "$release" | python3 -c "
import sys, json
Expand All @@ -101,73 +240,88 @@ for asset in data.get('assets', []):
print(f\"{asset['name']}|{asset['browser_download_url']}\")
" > "$ASSETS_FILE"

# Process each asset
while IFS='|' read -r filename download_url
do
if [ -z "$filename" ]; then
continue
fi

if [[ "$filename" == *.deb ]]; then
wget -q -P out/packages "$download_url"

if [[ "$filename" =~ ^deb11-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
GOT_DEB=1
mkdir -p "$DEB_POOL_DEB11"
clean_name=$(echo "$filename" | sed 's/^deb11-//')
cp "out/packages/$filename" "$DEB_POOL_DEB11/$clean_name"
sign_deb_package "$DEB_POOL_DEB11/$clean_name"
elif [[ "$filename" =~ ^deb12-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
GOT_DEB=1
mkdir -p "$DEB_POOL_DEB12"
clean_name=$(echo "$filename" | sed 's/^deb12-//')
cp "out/packages/$filename" "$DEB_POOL_DEB12/$clean_name"
sign_deb_package "$DEB_POOL_DEB12/$clean_name"
elif [[ "$filename" =~ ^deb13-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
GOT_DEB=1
mkdir -p "$DEB_POOL_DEB13"
clean_name=$(echo "$filename" | sed 's/^deb13-//')
cp "out/packages/$filename" "$DEB_POOL_DEB13/$clean_name"
sign_deb_package "$DEB_POOL_DEB13/$clean_name"
elif [[ "$filename" =~ ^ubuntu22\.04-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
GOT_DEB=1
mkdir -p "$DEB_POOL_UBUNTU22"
clean_name=$(echo "$filename" | sed 's/^ubuntu22\.04-//')
cp "out/packages/$filename" "$DEB_POOL_UBUNTU22/$clean_name"
sign_deb_package "$DEB_POOL_UBUNTU22/$clean_name"
elif [[ "$filename" =~ ^ubuntu24\.04-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
# First pass: which RPM pools does this release populate? Needed to route the
# EL-agnostic noarch packages, which carry no distro hint in their names.
serve_rpm=""
while IFS='|' read -r filename _; do
[ -z "$filename" ] && continue
case "$filename" in
*.rpm)
pool=$(rpm_pool_for "$filename")
if [ -n "$pool" ] && ! is_claimed "$serve_rpm" "$pool"; then
serve_rpm="$serve_rpm $pool"
fi ;;
esac
done < "$ASSETS_FILE"

added=0
while IFS='|' read -r filename download_url; do
[ -z "$filename" ] && continue

case "$filename" in
*.deb)
comp=$(deb_component_for "$filename")
[ -z "$comp" ] && continue
claim_package "$comp" "$filename" || continue
wget -q -P out/packages "$download_url" || { echo "::warning::download failed: $filename"; continue; }
GOT_DEB=1
mkdir -p "$DEB_POOL_UBUNTU24"
clean_name=$(echo "$filename" | sed 's/^ubuntu24\.04-//')
cp "out/packages/$filename" "$DEB_POOL_UBUNTU24/$clean_name"
sign_deb_package "$DEB_POOL_UBUNTU24/$clean_name"
fi
elif [[ "$filename" == *.rpm ]]; then
echo "Processing RPM: $filename"
wget -q -P out/packages "$download_url"

if [[ "$filename" =~ ^rhel8-postgresql[0-9]+-documentdb.*\.rpm$ ]]; then
GOT_RPM=1
mkdir -p "$RPM_POOL_RHEL8"
clean_name=$(echo "$filename" | sed 's/^rhel8-//')
echo " Adding to RHEL 8: $filename -> $clean_name"
cp "out/packages/$filename" "$RPM_POOL_RHEL8/$clean_name"
elif [[ "$filename" =~ ^rhel9-postgresql[0-9]+-documentdb.*\.rpm$ ]]; then
GOT_RPM=1
mkdir -p "$RPM_POOL_RHEL9"
clean_name=$(echo "$filename" | sed 's/^rhel9-//')
echo " Adding to RHEL 9: $filename -> $clean_name"
cp "out/packages/$filename" "$RPM_POOL_RHEL9/$clean_name"
else
echo " Skipping RPM (does not match patterns): $filename"
fi
else
wget -q -P out/packages "$download_url"
fi
pool=$(deb_pool_for "$comp")
mkdir -p "$pool"
# The distro prefix disambiguates release assets; it is not part of
# the package name and must not survive into the pool.
clean_name=$(echo "$filename" | sed -E 's/^(deb1[123]|ubuntu2[24]\.04)-//')
cp "out/packages/$filename" "$pool/$clean_name"
sign_deb_package "$pool/$clean_name"
added=$((added + 1)) ;;

*.rpm)
pool_name=$(rpm_pool_for "$filename")
if [ -n "$pool_name" ]; then
targets="$pool_name"
else
case "$filename" in
*.noarch.rpm) targets="$serve_rpm" ;;
*) targets="" ;;
esac
fi
[ -z "$targets" ] && continue

downloaded=0
clean_name=$(echo "$filename" | sed -E 's/^rhel[89]-//')
for t in $targets; do
claim_package "$t" "$filename" || continue
case "$t" in
rhel8) dest="$RPM_POOL_RHEL8" ;;
rhel9) dest="$RPM_POOL_RHEL9" ;;
*) continue ;;
esac
if [ "$downloaded" -eq 0 ]; then
wget -q -P out/packages "$download_url" || { echo "::warning::download failed: $filename"; break; }
downloaded=1
GOT_RPM=1
fi
mkdir -p "$dest"
echo " Adding to ${t}: $filename -> $clean_name"
cp "out/packages/$filename" "$dest/$clean_name"
added=$((added + 1))
done ;;

*)
# Non-package assets (SHA256SUMS, manifest.txt, ...) are mirrored only
# for the primary release, which is what the site links to.
[ "$tag" = "$PRIMARY_TAG" ] && wget -q -P out/packages "$download_url" ;;
esac
done < "$ASSETS_FILE"

rm -f "$ASSETS_FILE"

echo "Release $tag contributed $added package file(s)"
rm -f "$ASSETS_FILE"
done

rm -f "$RELEASES_JSON" "$SEEN_FILE"

# release-info.json describes the primary release only: it is the "what is the
# current version" feed for the site, not an inventory of the pool.
release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases/tags/${PRIMARY_TAG}")
echo "$release" | python3 -c "
import sys, json
data = json.load(sys.stdin)
Expand Down
Loading