feat(consumergate)!: rebuild the gate on the postpone primitive - #492
Draft
behinddwalls wants to merge 1 commit into
Draft
feat(consumergate)!: rebuild the gate on the postpone primitive#492behinddwalls wants to merge 1 commit into
behinddwalls wants to merge 1 commit into
Conversation
## Summary ### Why? The gate parked blocked deliveries in memory: the partition goroutine sat in a watch/extend loop, renewing the parked delivery's visibility until the gate opened. That mechanism only babysat the one delivery it parked — messages already fetched into the per-partition buffer behind it (up to BatchSize) had their visibility lapse, redelivered as duplicates that stuffed the buffer until the topic's routing loop stalled, and burned a retry attempt per lapse until they were spuriously dead-lettered without ever failing. It also held a goroutine and a live lease per blocked partition for the whole close. ### What? The pre-Process gate check now postpones instead of parking: a blocked delivery gets its parked record written (observability) and is postponed for a re-check delay (~1s) — back to the queue as a partition barrier, redelivering without retry cost, re-checking the gate on each redelivery. Every buffered delivery behind a closed gate is postponed in turn as the partition drains, so nothing waits in memory, nothing lapses, and Stop never has parked state to unwind. The admit path unconditionally removes the parked record, which is what releases e2e awaitUnparked. Contract: Entry.Watch and the Wait helper are replaced by Park/Unpark (record ops); Config/DefaultConfig and the never-implemented Factory are deleted — the file store no longer polls, since the re-check cadence is the consumer's postpone delay. Semantics changes: release is quantized to the re-check delay (same order as the old 1s file poll), and a released message is a fresh delivery (Attempt restarts at 1) rather than "the same attempt" — no test asserted either. Consumer-gate RFC, extension READMEs, and the consumer README are updated; in-memory parking moves to the RFC's Rejected list with the buffered-lapse hazard as the reason. ## Test Plan ✅ `make test` (83 targets) — reworked gate tests: blocked deliveries are parked + postponed with the re-check delay and never processed; redelivery after open unparks and processes; partition-scoped gating; Stop is immediate while gated; fail-open on read errors. ✅ `bazel test //test/e2e/submitqueue/...` — the cancel-caught-pre-batch scenario passes unchanged: awaitParked observes the record while closed, awaitUnparked observes its removal after open, the stale check is dropped, the sentinel lands. ✅ `make fmt`, `make mocks`.
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
The gate parked blocked deliveries in memory: the partition goroutine sat in a watch/extend loop, renewing the parked delivery's visibility until the gate opened. That mechanism only babysat the one delivery it parked — messages already fetched into the per-partition buffer behind it (up to BatchSize) had their visibility lapse, redelivered as duplicates that stuffed the buffer until the topic's routing loop stalled, and burned a retry attempt per lapse until they were spuriously dead-lettered without ever failing. It also held a goroutine and a live lease per blocked partition for the whole close.
What?
The pre-Process gate check now postpones instead of parking: a blocked delivery gets its parked record written (observability) and is postponed for a re-check delay (~1s) — back to the queue as a partition barrier, redelivering without retry cost, re-checking the gate on each redelivery. Every buffered delivery behind a closed gate is postponed in turn as the partition drains, so nothing waits in memory, nothing lapses, and Stop never has parked state to unwind. The admit path unconditionally removes the parked record, which is what releases e2e awaitUnparked.
Contract: Entry.Watch and the Wait helper are replaced by Park/Unpark (record ops); Config/DefaultConfig and the never-implemented Factory are deleted — the file store no longer polls, since the re-check cadence is the consumer's postpone delay. Semantics changes: release is quantized to the re-check delay (same order as the old 1s file poll), and a released message is a fresh delivery (Attempt restarts at 1) rather than "the same attempt" — no test asserted either. Consumer-gate RFC, extension READMEs, and the consumer README are updated; in-memory parking moves to the RFC's Rejected list with the buffered-lapse hazard as the reason.
Test Plan
✅
make test(83 targets) — reworked gate tests: blocked deliveries are parked + postponed with the re-check delay and never processed; redelivery after open unparks and processes; partition-scoped gating; Stop is immediate while gated; fail-open on read errors. ✅bazel test //test/e2e/submitqueue/...— the cancel-caught-pre-batch scenario passes unchanged: awaitParked observes the record while closed, awaitUnparked observes its removal after open, the stale check is dropped, the sentinel lands. ✅make fmt,make mocks.Stack