fix(ateapi): drop worker watch events instead of stalling pub/sub - #949
Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
Open
Conversation
A blocking send on the full watch buffer stops the pub/sub socket being drained, so Valkey evicts the subscriber for overrunning its output buffer and every event in flight is lost silently. Overflow now drops events, counts them, and signals an invalidation so the worker cache relists instead of serving a stale snapshot until the periodic relist. Claude-Session: https://claude.ai/code/session_01XuQqkwLf5Zx6CSZFHSC6hb
NekoPunch (orangeCatDeveloper)
force-pushed
the
issue-624-watch-overflow
branch
from
August 15, 2026 09:24
fee3023 to
0cb7738
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #624
Under heavy worker churn the worker cache silently serves a stale worker list
for up to a full relist interval — long enough to schedule actors onto workers
that no longer exist, with nothing in the logs or metrics saying so.
Root cause
WatchWorkersforwards pub/sub messages into a 128-event channel with ablocking send. A slow consumer fills it, the forwarding goroutine parks on
ch <- eventand stops reading the socket, and Valkey evicts the subscriber foroverrunning its pub/sub output buffer:
The eviction never reaches us: go-redis reconnects and resubscribes underneath,
closing its message channel only on
pool.ErrClosed. So there is no closed watchand no resync — queued and in-flight events are simply gone while
readystaystrue. Quieter and worse than #624 assumed: not a scheduling outage, butsilently wrong scheduling input.
The fix
Overflow drops the event, counts it, and raises a coalesced
WorkerWatch.Invalidatedsignal instead of blocking, so the socket keepsdraining and Valkey has no cause to evict.
workercachereads that as "relistnow", first draining events that predate the drop (they would roll the fresh
snapshot back). A failed relist retries on a 100ms→5s backoff instead of waiting
for the periodic ticker; a successful resync or periodic relist cancels a pending
retry. The old snapshot keeps serving throughout — no
ready=falsewindow.The PostgreSQL store keeps its blocking send (nil invalidation channel): its
LISTENconsumer has no equivalent server-side eviction.Evidence
Real Valkey 8,
client-output-buffer-limit pubsub 32kb 8kb 3, stalled consumer,50000 events (~6MB), fresh container per run:
client_output_buffer_limit_disconnectionsate.store.worker_watch.dropped_eventsCache recovery after a lost DELETE, at the production 5-minute relist interval: