Skip to content

Fix air-gapped install blockers found by a cold-start test of the published page - #160

Merged
guanzhousongmicrosoft merged 1 commit into
documentdb:mainfrom
GuanzhouSong:users/guanzhousong/offline-install-fixes
Aug 25, 2026
Merged

Fix air-gapped install blockers found by a cold-start test of the published page#160
guanzhousongmicrosoft merged 1 commit into
documentdb:mainfrom
GuanzhouSong:users/guanzhousong/offline-install-fixes

Conversation

@GuanzhouSong

Copy link
Copy Markdown
Contributor

Two testers followed https://documentdb.io/docs/getting-started/packages/ verbatim in containers with no prior knowledge of DocumentDB. Results split sharply:

Path Verdict
Online (APT + RPM → wizard → mongosh) ✅ worked first try on both distros, zero errors
Air-gapped could not be completed on either path

Every fix below is from an observed failure, not review.


1. DEB staging silently produces a broken bundle

dpkg-scanpackages lives in dpkg-dev, which is not in ubuntu:24.04. The string dpkg-dev appeared zero times on the page.

dpkg-scanpackages . /dev/null > Packages && gzip -k Packages
/w/stage_deb.sh: line 50: dpkg-scanpackages: command not found      # exit 127

This is worse than a normal failure because it fails silently. The > redirection still creates the file and && short-circuits before gzip, so the bundle ends up with a 0-byte Packages and no Packages.gz — right after 205 successful downloads printed:

DEB_COUNT=205   BUNDLE_SIZE=203M
-rw-r--r-- 1 root root  0  Packages
ls: cannot access 'Packages.gz': No such file or directory

On the target, apt-get update then exits 0 (the only clue is Err:7 ... Method gave a blank filename, buried among expected offline warnings), and the install dies an hour and 200 MB later:

E: Unable to locate package documentdb-18        # exit 100

Fix: install dpkg-dev first — mirroring the RPM block, which already installs createrepo_c.

2. RPM air-gapped install aborts on the target's stock repos

DNF aborts the whole transaction if any enabled repository is unreachable, and every RHEL-compatible image ships baseos, appstream, extras enabled. The bundle was read fine (341 kB of metadata) and the install still failed:

Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist  # exit 1

--disablerepo / --enablerepo appeared zero times on the page. With them added, the bundle proved complete: Install 164 Packages / Upgrade 2 / 118 M / Complete!

Fix: add the flags and explain the asymmetry — APT only warns about unreachable sources and continues; DNF aborts.

3. Offline installs lacked -y

The online blocks have it, the offline ones didn't, so both aborted at the prompt in a container — on a page that explicitly caters to containers. Note DEBIAN_FRONTEND=noninteractive does not answer apt's continue prompt.

4. jq missing from the "no bundle needed" preconditions

Built exactly the documented precondition, offline, then installed the files:

documentdb-common : Depends: jq but it is not installable
documentdb-postgresql-tools : Depends: jq but it is not installable    # exit 100

jq appeared zero times on the page. Adding it to the precondition → exit 0. On an air-gapped host jq cannot be resolved from anywhere, so this case did not work as written.

5. Two traps outside the offline section

  • sudo -u documentdb-local psql collides with the blanket "in a container, drop sudo". Applied literally that gives -u documentdb-local psqlbash: - : invalid option. sudo -u switches user rather than elevates; now called out.
  • mongosh ... -u admin -p — a bare -p prompts. Non-interactive, that sends an empty password and fails with MongoServerError: Invalid key, which names neither auth nor the empty credential. This form was introduced by my own earlier change and is now inline.

6. --restore claim is distro-dependent

The page said --restore stops the gateway on non-systemd hosts. I tested both:

Result
ubuntu:24.04 Stopping orphan (non-systemd) gateway daemon (pid 7770) → port DOWN
rockylinux:9 ❌ reports success, gateway keeps serving authenticated traffic

Root cause is in documentdb-setup:

for _pid in $(pgrep -f /usr/lib/documentdb-gateway/documentdb-gateway-daemon 2>/dev/null); do

Without pgrep the loop runs zero times and the function silently succeeds; 2>/dev/null swallows the command not found. Ubuntu ships procps; minimal RHEL images do not. The downstream symptom is the next failure too — a later re-run hits Port 10260 is already in use.

This page now documents the prerequisite, but the real fix is packaging: documentdb-common declares neither procps-ng (RPM) nor procps (DEB). Reported separately against documentdb/documentdb; this PR only stops the page from asserting something that is false on RHEL.


Verification

Every changed command re-run verbatim. Staging on a connected container, installing into one started with --network none (air-gap proven by DNS failure before each install):

Path Observed
DEB bundle 205 pkgs / 203 MB → Need to get 0 B/95.2 MBexit 0
RPM bundle 271 pkgs / 173 MB → Install 164 / Upgrade 2exit 0
Extension-only, one file exit 0
--restore on ubuntu:24.04 exit 0, port down

The page's stated sizes were accurate — "~200 packages / 200 MB (DEB) or ~270 / 170 MB (RPM)" vs observed 205/203 MB and 271/173 MB.

npm test 130 passed · eslint clean · tsc --noEmit clean.

Related

documentdb/docs#70 — same mongosh -p fix on the docs-repo page.

Two testers followed https://documentdb.io/docs/getting-started/packages/
verbatim in containers, with no prior knowledge of DocumentDB. The online
path worked first try on both distributions. The air-gapped path could not
be completed on either, and two smaller traps were found elsewhere.

Air-gapped, DEB: the staging block calls dpkg-scanpackages, which lives in
dpkg-dev and is not installed in ubuntu:24.04. The word dpkg-dev did not
appear on the page. This failed silently in the worst way: the shell
redirection still created a 0-byte Packages file, the && short-circuited
before gzip, apt-get update on the target then exited 0, and the install
failed an hour and 200 MB later with "E: Unable to locate package
documentdb-18". Install dpkg-dev first, mirroring the RPM block, which
already installs createrepo_c.

Air-gapped, RPM: dnf aborts the entire transaction when any enabled
repository is unreachable, and every RHEL-compatible image ships baseos,
appstream and extras enabled. The bundle was read successfully and the
install still failed with "Failed to download metadata for repo 'baseos'".
Add --disablerepo='*' --enablerepo=documentdb-offline and explain the
asymmetry with APT, which only warns and continues.

Both offline install commands lacked -y while the online ones have it, so
both aborted at the confirmation prompt in a container. Add it.

"Smaller offline cases" listed the preconditions for skipping the bundle
but omitted jq, which documentdb-common and documentdb-postgresql-tools
both depend on. On an air-gapped host jq cannot be resolved from anywhere,
so the case did not work as written.

Also state which part of the install block to run when staging (up to
apt update, not the final install), and note that what makes the bundle
work is the index inside it, not the package files.

Elsewhere: the blanket "in a container, drop sudo" advice breaks
"sudo -u documentdb-local psql", which switches user rather than elevates
-- dropping sudo there yields "bash: - : invalid option". And a bare -p
makes mongosh prompt, so a non-interactive shell sends an empty password
and fails with "MongoServerError: Invalid key"; that form was introduced
by my own earlier change and is now inline.

Finally, the non-systemd note promised that --restore stops the gateway.
That is true on Debian/Ubuntu and false on a minimal RHEL image, because
documentdb-setup finds the directly-started gateway with pgrep and
procps-ng is not installed there. Verified both ways: on ubuntu:24.04
--restore logged "Stopping orphan (non-systemd) gateway daemon (pid 7770)"
and the port went down; on rockylinux:9 it reported success while the
gateway kept serving authenticated traffic. Note the prerequisite.

That last one is a packaging bug rather than a documentation one --
documentdb-common declares neither procps-ng (RPM) nor procps (DEB) --
and is reported separately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f18515db-c52f-4197-aa50-d81359c7c763
Signed-off-by: Guanzhou Song <guanzhou.song@gmail.com>
@guanzhousongmicrosoft
guanzhousongmicrosoft merged commit c136455 into documentdb:main Aug 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants