Skip to content

fix: size status-light stale threshold to observed prober cadence - #2486

Merged
sr320 merged 1 commit into
masterfrom
server-status-stale-threshold
Aug 9, 2026
Merged

fix: size status-light stale threshold to observed prober cadence#2486
sr320 merged 1 commit into
masterfrom
server-status-stale-threshold

Conversation

@sr320

@sr320 sr320 commented Aug 9, 2026

Copy link
Copy Markdown
Member

Follow-up to #2484.

The problem

The status lights were spending most of each cycle showing unknown even though gannet and klone were reachable the whole time.

I set the 30-minute stale threshold against the interval the GitHub Action requests (*/15) rather than what GitHub actually delivers. Measured across the last 20 runs on this repo:

gaps (min): 57 71 58 57 47 35 46 43 68 57 45 37 37 48 38 44 55 41 30
min=30  median=46  max=71   (the cron expression asks for 15)
gaps exceeding the 30-min threshold: 19/19

Every single interval blew through the threshold. The lights were green for 30 minutes after each run landed, then grey for the remaining ~16 minutes on average, and up to 41 minutes in the worst gap.

The change

  • Stale threshold 30 → 90 minutes, clearing the worst observed gap with headroom.
  • Corrects the page text, which claimed "Checks run every 10–15 minutes" — not true of the current external-only setup.
  • Records the measured cadence in scripts/README.md so the next person to touch the threshold knows what it is sized against.

Verification

Built locally and pointed at the live published status data:

  • At 45 minutes old — the window where the old code showed grey — gannet and klone now render up, with raven correctly unknown since no in-network prober exists yet.
  • A 95-minute-old reading carrying up: true still renders unknown, so a genuinely dead prober remains visible rather than silently reporting everything green.

Caveat

This is a mitigation, not the real fix. It buys tolerance for a slow backstop prober; it does not make the data fresher, and a dead prober now takes 90 minutes to surface instead of 30. Once the in-network cron is running every 10 minutes it becomes the freshest source, the merge logic prefers it, and this ceiling stops mattering in practice. Raven will keep reading unknown until then — setup steps are in scripts/README.md.

🤖 Generated with Claude Code

The 30-minute stale threshold was set against the interval the GitHub
Action *requests* (*/15) rather than what it delivers. Measured across 20
consecutive runs on this repo, the gaps were 30-71 minutes (median 46) --
every single interval exceeded the threshold, so gannet and klone spent
most of each cycle rendering "unknown" despite being reachable the whole
time.

Raises the threshold to 90 minutes, which clears the worst observed gap
with headroom, and corrects the page text, which claimed checks run every
10-15 minutes.

This mainly matters while the GitHub Action is the only prober. Once the
in-network cron is running every 10 minutes it becomes the freshest source
and the ceiling stops mattering in practice -- so the real fix is still to
set that cron up (see scripts/README.md).

Verified against the live published status: at 45 minutes old, gannet and
klone now render up rather than unknown; a 95-minute-old reading with
up:true still correctly renders unknown, so a genuinely dead prober is
still visible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sr320
sr320 merged commit dc92c8e into master Aug 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant