fix(workflow): prevent scheduled scan gaps and starvation

[bmurphy201]

Summary

• drive the scheduler with a fixed ticker and an exact window from the prior successful scan, so scan and execution time cannot create blind cron gaps
• retain the prior successful cursor after a failed or partial database scan, and coalesce a prolonged gap to the latest due cron instant
• keyset-page every enabled scheduled workflow instead of permanently truncating the global scan at 1,000 rows
• freeze each scan at its start time and add a matching partial database index so concurrent inserts cannot extend one scan indefinitely

Root cause

The loop previously slept for 60 seconds after completing each scan but looked back only 60 seconds for due cron instants. Any scan or execution work made the real interval longer than the lookup window. The global query also returned only the oldest 1,000 enabled scheduled workflows on every tick, leaving all later rows permanently unvisited.

This was observed as a hosted scheduled workflow failing to fire while manual execution remained healthy. Hosted relay logs were unavailable, so the production incident is consistent with these defects but does not prove that either was the only contributing cause.

Behavior

The durable (community_id, workflow_id, scheduled_for) claim remains the cross-pod at-most-once boundary. After an extended failed scan, the scheduler claims only the latest due cron instant rather than replaying an unbounded burst of side effects.

Validation

• env -u BUZZ_ACP_SESSION_POLICY bin/just ci
• affected package suites: buzz-db 122 passed / 253 ignored; buzz-workflow 171 passed / 2 ignored
• PostgreSQL test discovery guard passed; the new 1,002-row snapshot-bound pagination regression is included in the ignored PostgreSQL CI lane
• exact head tested: 028762d6ac33bd52a8fd49b96b31d536b738576e

Related work

• Fixes Scheduled (cron) workflows never fire on a hosted multi-tenant relay; manual trigger works #7065
• refreshes the current-main implementation path for the pagination work in draft Fix workflow scheduler pagination #5092
• overlaps the delayed-tick concern in fix(workflows): tolerate delayed cron ticks #3884
• fix(workflow): catch up missed cron fires #7506 proposes stronger persistent per-workflow catch-up across process restarts; this change is compatible in intent but deliberately preserves latest-only recovery semantics

AI-assisted implementation, reviewed and tested against current main before submission.

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is 4cd82f513214aad11c2b742ce7cc7c681e8e32a0...028762d6ac33bd52a8fd49b96b31d536b738576e.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review 028762d6ac33bd52a8fd49b96b31d536b738576e to authorize a new review.
Any previous review applies only to its recorded range.

[bmurphy201]

Maintainers: this fixes two production scheduler gaps we reproduced in a hosted Buzz workflow: elapsed scan time can leave cron blind spots, and the enabled-schedule query stops after 1,000 rows. Bill Murphy has authorized merge and deployment for our affected workspace. The exact PR head passed the full local just ci gate, plus the database and workflow regression suites. GitHub reports the PR mergeable, DCO, Semgrep, and zizmor passing. Please review and merge when the repository gates are satisfied; our account cannot merge or enable auto-merge under the base policy.