diff --git a/.changeset/deploy-runbook.md b/.changeset/deploy-runbook.md new file mode 100644 index 00000000..2ec0fc89 --- /dev/null +++ b/.changeset/deploy-runbook.md @@ -0,0 +1,4 @@ +--- +--- + +docs(deploy): add sandbox deploy runbook and recreate-relay script diff --git a/deploy/README.md b/deploy/README.md index 937878db..159da259 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -6,7 +6,9 @@ that image. This guide assumes a Linux host with Docker Engine and the Compose plugin installed. Container images are published automatically after CI succeeds on pushes to -`main`. See [`docs/DEPLOYMENT.md`](../docs/DEPLOYMENT.md) for the CI/CD flow. +`main`. See [`docs/DEPLOYMENT.md`](../docs/DEPLOYMENT.md) for the CI/CD flow and +[`docs/DEPLOY-RUNBOOK.md`](../docs/DEPLOY-RUNBOOK.md) for the step-by-step update +procedure after a new image lands on the host. ## What the server keeps locally @@ -14,7 +16,7 @@ installed. Container images are published automatically after CI succeeds on pus |------|----------|------------------------| | `.env` | **Yes** | No — your secrets and tuning | | `.nostr/data/` | Created at runtime | No — Postgres data | -| `docker-compose.yml` | Yes (via bootstrap) | **Yes** — re-run bootstrap or PR 2 auto-sync | +| `docker-compose.yml` | Yes (via bootstrap) | **Yes** — re-run bootstrap when release notes say so | | `postgresql.conf` | Yes (via bootstrap) | Rarely | | `.nostr/settings.yaml` | **Optional** | Your overrides only | @@ -129,21 +131,23 @@ Some hosts cannot reach GHCR over IPv4: ## Updating -When a new image is available: +After a new image is on the host (`docker pull`, `docker load`, or the webhook +pull stack), **recreate the relay** so migrations run and the new code starts: ```bash -docker pull ghcr.io/cameri/nostream:main # or: docker load -i nostream-main.tar.gz -docker compose up -d +chmod +x deploy/recreate-relay.sh +./deploy/recreate-relay.sh /opt/nostream ``` -If migrate does not re-run after a load: +The script runs `nostream-migrate`, force-recreates `nostream`, and waits for +`/readyz` to return `200`. -```bash -docker compose up -d --force-recreate nostream-migrate nostream -``` +Full checklist, rollback steps, and troubleshooting: +[`docs/DEPLOY-RUNBOOK.md`](../docs/DEPLOY-RUNBOOK.md). When compose or `postgresql.conf` change in a release, re-run bootstrap against -the new checkout (or copy the updated files). Automated sync is planned separately. +the new checkout before recreating the relay. Automated post-pull recreate and +HAProxy blue/green cutover are planned separately. ## Refresh release-managed files diff --git a/deploy/recreate-relay.sh b/deploy/recreate-relay.sh new file mode 100755 index 00000000..b5a8a287 --- /dev/null +++ b/deploy/recreate-relay.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Recreate the nostream relay after a new image is available on the host. +# +# Usage: +# ./deploy/recreate-relay.sh [/opt/nostream] +# +# Runs migrations from the current image, recreates the relay container, and +# verifies /readyz. Does not pull images — run that step first (manually or via +# the webhook pull stack). + +TARGET="${1:-/opt/nostream}" +RELAY_PORT="${RELAY_PORT:-8008}" +READYZ_URL="http://127.0.0.1:${RELAY_PORT}/readyz" +READYZ_RETRIES="${READYZ_RETRIES:-30}" +READYZ_INTERVAL_SECONDS="${READYZ_INTERVAL_SECONDS:-2}" + +if [[ ! -f "$TARGET/docker-compose.yml" ]]; then + echo "error: $TARGET/docker-compose.yml not found — run deploy/bootstrap.sh first" >&2 + exit 1 +fi + +cd "$TARGET" + +echo "Running migrations (nostream-migrate)..." +docker compose up --no-deps --force-recreate --abort-on-container-exit nostream-migrate + +echo "Recreating relay (nostream)..." +docker compose up -d --force-recreate --no-deps nostream + +echo "Waiting for /readyz..." +for attempt in $(seq 1 "$READYZ_RETRIES"); do + if curl -sf "$READYZ_URL" >/dev/null; then + echo "ready: $READYZ_URL" + curl -s "$READYZ_URL" + echo + exit 0 + fi + + echo "attempt ${attempt}/${READYZ_RETRIES}: not ready yet" + sleep "$READYZ_INTERVAL_SECONDS" +done + +echo "error: relay did not become ready at $READYZ_URL" >&2 +docker compose ps +exit 1 diff --git a/docs/DEPLOY-RUNBOOK.md b/docs/DEPLOY-RUNBOOK.md new file mode 100644 index 00000000..86215195 --- /dev/null +++ b/docs/DEPLOY-RUNBOOK.md @@ -0,0 +1,183 @@ +# Deploy runbook (sandbox / production relay) + +Step-by-step procedure for updating a running nostream host after a new +`ghcr.io/cameri/nostream:main` image is available. Use this **before** automating +deploys or adding HAProxy blue/green cutover. + +For initial host setup see [`deploy/README.md`](../deploy/README.md). For how +images are published see [`docs/DEPLOYMENT.md`](DEPLOYMENT.md). + +## Architecture today + +```text +GitHub push to main + → CI builds and publishes ghcr.io/cameri/nostream:main + → (optional) webhook pull stack downloads the image to the host + → operator runs this runbook to migrate + recreate the relay + → Cloudflare Tunnel / reverse proxy continues pointing at the relay port +``` + +The webhook pull stack **loads new images only**. It does **not** recreate the +relay container or run migrations. That gap is what this runbook closes. + +## Roles of health endpoints + +| Endpoint | Use during deploy | +|----------|-------------------| +| `/healthz` | Liveness — process is up (always `200` while running) | +| `/readyz` | Readiness — Postgres **and** Redis respond (`200` or `503`) | + +Wait for `/readyz` to return `200` before sending user traffic to a new instance. +When HAProxy blue/green is added later, point backend health checks at `/readyz` +with `timeout check 5s` (dependency pings default to 3s inside the relay). + +## Prerequisites + +- Host bootstrapped (`deploy/bootstrap.sh /opt/nostream`) +- New relay image already on the host (`docker pull` or `docker load`) +- Shell access to the host (`cd /opt/nostream`) +- Optional: note the previous image ID for rollback + +```bash +docker image inspect ghcr.io/cameri/nostream:main --format '{{.Id}}' | tee /tmp/nostream-pre-deploy-image-id +``` + +## Standard deploy (manual) + +Run from the repository checkout (for scripts) or directly on the host. + +### 1. Refresh release-managed files (when compose or postgres config changed) + +```bash +./deploy/bootstrap.sh /opt/nostream +``` + +Existing `.env` and `.nostr/settings.yaml` are preserved. + +### 2. Load the new image (skip if webhook pull stack already did) + +```bash +docker pull ghcr.io/cameri/nostream:main +``` + +On hosts that cannot reach GHCR over IPv4, build or pull elsewhere, then +`docker save` → transfer → `docker load`. See +[`deploy/README.md`](../deploy/README.md#image-delivery-on-restricted-networks). + +### 3. Run migrations and recreate the relay + +```bash +chmod +x deploy/recreate-relay.sh +./deploy/recreate-relay.sh /opt/nostream +``` + +Or run the steps manually: + +```bash +cd /opt/nostream +docker compose up --no-deps --force-recreate --abort-on-container-exit nostream-migrate +docker compose up -d --force-recreate --no-deps nostream +``` + +### 4. Verify + +```bash +cd /opt/nostream +docker compose ps +curl -sf http://127.0.0.1:8008/readyz +echo +curl -sf http://127.0.0.1:8008/healthz +curl -s -H 'Accept: application/nostr+json' http://127.0.0.1:8008/ +``` + +Expected: + +- `nostream-migrate` — exited `0` +- `nostream` — `running` +- `/readyz` — HTTP `200`, `"status":"ok"`, database and redis `"ok": true` +- NIP-11 root — JSON with relay metadata + +If admin is enabled, also check `/admin/health` through your normal auth path. + +### 5. Watch logs (first few minutes) + +```bash +cd /opt/nostream +docker compose logs -f --tail=100 nostream +``` + +## Rollback + +When the new relay fails readiness or behaves incorrectly: + +### 1. Stop the bad relay + +```bash +cd /opt/nostream +docker compose stop nostream +``` + +### 2. Restore the previous image + +If you saved the pre-deploy image ID: + +```bash +PREVIOUS_IMAGE="$(cat /tmp/nostream-pre-deploy-image-id)" +docker tag "$PREVIOUS_IMAGE" ghcr.io/cameri/nostream:main +``` + +Or load a known-good tarball: + +```bash +docker load -i /path/to/nostream-main-backup.tar.gz +``` + +### 3. Recreate on the old image + +```bash +./deploy/recreate-relay.sh /opt/nostream +``` + +Do **not** re-run migrations against a downgraded image unless you know the +schema is backward compatible. + +## Troubleshooting + +### `/readyz` stays `503` + +- Check Postgres and Redis: `docker compose ps` +- Inspect relay logs: `docker compose logs nostream --tail=200` +- Confirm `.env` credentials match the running database and cache +- Increase probe patience: readiness pings use `ADMIN_DEPENDENCY_PING_TIMEOUT_MS` + (default 3s) + +### `nostream-migrate` exits non-zero + +- Read migrate logs: `docker compose logs nostream-migrate` +- Do not force-recreate the relay until migrate succeeds +- Escalate if a migration is destructive — expand/contract migrations should be + deployed in compatible order + +### Image pull fails (GHCR / IPv6) + +- Use `docker save` / `docker load` from a machine that can reach GHCR +- Keep `pull_policy: never` on nostream services in compose when using pre-loaded + images + +### Relay up but clients cannot connect + +- Confirm the tunnel or reverse proxy still targets `127.0.0.1:8008` +- Webhook stack changes do not replace this runbook — they only deliver images + +## What comes next (not in this runbook) + +These are planned follow-ups on the zero-downtime track: + +1. **Graceful WebSocket drain** — stop accepting new connections on shutdown, + close existing clients cleanly (SIGTERM handling) +2. **HAProxy blue/green** — two relay backends, cutover via `/readyz`, documented + in a future compose overlay +3. **Automated recreate** — wire post-pull steps into the webhook pipeline after + the manual runbook is proven on sandbox + +Until then, run this procedure after every image update. diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index e90e7659..7d7648fa 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -14,5 +14,6 @@ The publish job runs only for `refs/heads/main` and only when `post-tests` succeeds. Pull requests and manual workflow runs do not publish images. Deployment hosts can pull the `main` image as described in -[`deploy/README.md`](../deploy/README.md). There is no separate manual image -publishing step. +[`deploy/README.md`](../deploy/README.md). After the image is on the host, follow +[`docs/DEPLOY-RUNBOOK.md`](DEPLOY-RUNBOOK.md) to migrate and recreate the relay. +There is no separate manual image publishing step.