[server] Add off-loop coordinator health cache (mechanism only) - #4061
Draft
affo wants to merge 2 commits into
Draft
[server] Add off-loop coordinator health cache (mechanism only)#4061affo wants to merge 2 commits into
affo wants to merge 2 commits into
Conversation
Adds CoordinatorHealthCache, a copy-on-write cache of cluster/per-tablet-server replica and leader health, published lock-free without going through AccessContextEvent -- the same pattern CoordinatorMetadataCache already uses for server topology. The coalescing/urgency mechanics are extracted into a reusable CoalescingRefreshCache<T>, since the same "callers report facts, this decides when to act" shape applies to more than just this one cache. Wired into real coordinator event processing: every mutation that matters (leader/ISR changes, server death/registration, table/partition create-delete, tag add/remove, reassignment) reports through it, and refreshIfNeeded() runs on every event-loop tick, right next to the existing metrics-timer check. Does not wire any RPC to it yet -- getClusterHealth()/describeTabletServers() are untouched and still compute on demand via AccessContextEvent. The rewrite is confirmed mechanically straightforward (CoordinatorService already derives eventManagerSupplier the same way a healthCacheSupplier would) but intentionally left for a follow-up. Related discussion: apache#1389 (comment) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…e) method Collapses update()/refreshIfNeeded(compute, idle) into a single refresh(compute, force). The caller now computes "force" itself (typically from queue.isEmpty()) instead of the cache interpreting a separate "idle" signal -- same information, smaller interface. The dirty check is now an unconditional first gate that force never bypasses, only the timing/urgency gate does. This matters: without it, "queue empty -> force=true" would trigger a full rescan on every idle tick regardless of whether anything changed, since a healthy coordinator is idle most of the time. A freshly constructed cache starts dirty so warm-up still works without needing force to bypass the dirty check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
Contributor
Author
|
CC @fresh-borzoni if you have some time to review this one related to all others metadata-related PRs 😅 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Companion to the discussion on #1389 about splitting
getServerNodes()/getClusterHealth()/describeTabletServers()by cost and update frequency (comment).Adds
CoordinatorHealthCache: a lock-free, coordinator-thread-published snapshot of cluster-wide and per-tablet-server replica/leader health, kept warm incrementally as mutations happen, instead of recomputed from scratch on every RPC call viaAccessContextEvent. Wired into real mutation handling (leader/ISR changes, server death/registration, topology changes). No RPC reads from it yet —getClusterHealth()/describeTabletServers()are untouched; this PR is the mechanism only.Performance & consistency trade-offs
getClusterHealth()/describeTabletServers()each enqueue a freshO(buckets)scan onto the single coordinator event thread on every call. This cache makes readsO(1)— a lock-free field read — by moving the scan to the write side, where it's decoupled from call frequency and triggered by mutation instead.O(buckets)per recompute, unavoidably — the point isn't to make the scan cheaper, it's to make it run less often. Every mutation is coalesced: a burst of many changes (e.g. a full failover storm re-electing leaders for hundreds of buckets) costs at most one recompute, not one per event, bounded by how long the coordinator's event queue stays busy.Impact on other open PRs
DescribeTabletServers) — once merged, can read this cache directly (it already carries the exact per-server counters that RPC needs) instead of its ownAccessContextEventscan. No new snapshot shape required.getServerNodes/ server tags) — no direct relationship, but confirms the same pattern (this PR'sCoalescingRefreshCache, or the existingCoordinatorMetadataCache) is the right fix for that PR's livezkClient.getServerTags()call, which is doing a ZK round-trip on every call for data that's already cached elsewhere.DescribeBuckets) — same opportunity, but two steps: first stop bypassingCoordinatorContextfor a live ZK read, then a per-bucket cache shaped like this one.UpdateMetadataRequestpropagation) — no relationship, and none expected: that's a coordinator-initiated push to tablet servers, not a client read blocked on the event thread, so this mechanism doesn't apply there.BucketMetadata.javaextensions — this PR doesn't touch that class, so no merge conflict risk from this change.Known, deliberate gap: leader-activation flips (
CoordinatorRequestBatch) aren't wired to this cache yet — bounded staleness, not a correctness issue (the next refresh for any other reason reports it correctly regardless), left for a follow-up.Test Plan
CoordinatorHealthCacheTest(15),CoalescingRefreshCacheTest(6).CoordinatorEventProcessorTest,ClusterHealthTest,CoordinatorEventManagerTest,TableBucketStateMachineTest,CoordinatorContextTest,CoordinatorMetadataCacheTest.mvn spotless:checkclean.🤖 AI-assisted changes - reviewed by human developer