Skip to content

fix(ateapi): drop worker watch events instead of stalling pub/sub - #949

Open
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:issue-624-watch-overflow
Open

fix(ateapi): drop worker watch events instead of stalling pub/sub#949
NekoPunch (orangeCatDeveloper) wants to merge 1 commit into
agent-substrate:mainfrom
orangeCatDeveloper:issue-624-watch-overflow

Conversation

@orangeCatDeveloper

@orangeCatDeveloper NekoPunch (orangeCatDeveloper) commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #624

Under heavy worker churn the worker cache silently serves a stale worker list
for up to a full relist interval — long enough to schedule actors onto workers
that no longer exist, with nothing in the logs or metrics saying so.

Root cause

WatchWorkers forwards pub/sub messages into a 128-event channel with a
blocking send. A slow consumer fills it, the forwarding goroutine parks on
ch <- event and stops reading the socket, and Valkey evicts the subscriber for
overrunning its pub/sub output buffer:

Client id=7 ... flags=P ... omem=41008 lib-name=go-redis(,go1.26.3)
  scheduled to be closed ASAP for overcoming of output buffer limits.

The eviction never reaches us: go-redis reconnects and resubscribes underneath,
closing its message channel only on pool.ErrClosed. So there is no closed watch
and no resync — queued and in-flight events are simply gone while ready stays
true. Quieter and worse than #624 assumed: not a scheduling outage, but
silently wrong scheduling input.

The fix

Overflow drops the event, counts it, and raises a coalesced
WorkerWatch.Invalidated signal instead of blocking, so the socket keeps
draining and Valkey has no cause to evict. workercache reads that as "relist
now", first draining events that predate the drop (they would roll the fresh
snapshot back). A failed relist retries on a 100ms→5s backoff instead of waiting
for the periodic ticker; a successful resync or periodic relist cancels a pending
retry. The old snapshot keeps serving throughout — no ready=false window.

The PostgreSQL store keeps its blocking send (nil invalidation channel): its
LISTEN consumer has no equivalent server-side eviction.

Evidence

Real Valkey 8, client-output-buffer-limit pubsub 32kb 8kb 3, stalled consumer,
50000 events (~6MB), fresh container per run:

before after
client_output_buffer_limit_disconnections 1 0
invalidation signal none SIGNALED
ate.store.worker_watch.dropped_events no metric 49872

Cache recovery after a lost DELETE, at the production 5-minute relist interval:

before  corrected_after=NEVER (>10s; waits for the 5m relist)
after   corrected_after=6ms

A blocking send on the full watch buffer stops the pub/sub socket being
drained, so Valkey evicts the subscriber for overrunning its output
buffer and every event in flight is lost silently. Overflow now drops
events, counts them, and signals an invalidation so the worker cache
relists instead of serving a stale snapshot until the periodic relist.

Claude-Session: https://claude.ai/code/session_01XuQqkwLf5Zx6CSZFHSC6hb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P2] Worker cache pub/sub channel (cap 128) silently drops events under high pod churn — forces unscheduled full resync

1 participant