feat(node): hold off RPC queries anchored on just-unseen blocks - #25203
Conversation
| import type { BlockHeader } from '@aztec/stdlib/tx'; | ||
|
|
||
| /** How often a held request re-reads the block source while waiting. */ | ||
| const POLL_INTERVAL_MS = 200; |
There was a problem hiding this comment.
I think this might be too high. This adds a minimum increase in latency of 200ms if a block is missing (which is a lot for an API) even if the block arrives 10ms after the query is first attempted. Maybe we can lower it to 50ms?
There was a problem hiding this comment.
I don't want to hammer the poor archiver. Let's compromise at 100ms :-P
| const arrived = await retryUntil( | ||
| () => read(query), | ||
| `block ${blockParameter}`, | ||
| waitMs / 1000, | ||
| POLL_INTERVAL_MS / 1000, | ||
| ); |
There was a problem hiding this comment.
I'm wondering if a different approach would suit us better where we have a single loop monitoring perpetually monitoring the tips of world state and N=100 queries which get checked against the current tips (sorted by block number so that we cna run binary search on them) vs having 100 individual loops calling the world state.
This is probably something we can only do in v6 where we'll have access to the block number
There was a problem hiding this comment.
Actually, you make a good point. It's very likely that most held-off requests will be asking for exactly the same block hash (the one most recently mined). So we could group together those requests, as opposed to keeping a loop for each.
Alternatively, we can subscribe to the archiver events, so we don't rely in polling. That's probably a better option. I'll have claude sketch something.
| const value = await read(query); | ||
| if (value !== undefined || opts.holdOff === false) { | ||
| return value; | ||
| } |
There was a problem hiding this comment.
In v5 there's no way for us to differentiate between 'the next block hasn't arrived yet' vs 'this block number has been reorged', right?
There was a problem hiding this comment.
If we only have the block hash, nope.
| private readonly blockSource: L2BlockSource, | ||
| private readonly holdOff: UnseenBlockHoldOff, |
There was a problem hiding this comment.
More an interface design than an issue but this class now acts as a router sending some queries to one source and other queries to another. There's no guarantee that both blockSource and holdOff read from the same chain.
Maybe UnseenBlockHoldOff can become a wrapper/proxy implementing the L2BlocKsource interface in v6.
There was a problem hiding this comment.
Fair point on the wrapper. Still, given we're the ones constructing this instance and it's pretty contained, I'm not too worried. I'll give it some thought though.
Behind a load balancer a client can sync to block N+1 through one node and then anchor follow-up queries against another node still at block N, which fails them immediately even though the block lands a moment later. The node now waits a bounded time for the anchor to arrive before answering: a block number exactly one ahead of the proposed tip waits up to RPC_UNSEEN_BLOCK_BY_NUMBER_WAIT_MS (default twice the block duration), an unknown block hash or archive root waits up to RPC_UNSEEN_BLOCK_BY_HASH_WAIT_MS (default 3s), and tags never wait. Setting either to 0 restores the previous fail-fast behavior. Outcomes are unchanged, only delayed: on a miss callers still throw or return undefined exactly as before. A cap of 100 simultaneous holds bounds resource use, beyond which misses fail fast. The world-state retry loop only holds off on its first attempt so a budget is never multiplied by the attempt count.
- Document that a wait budget is approximate: the poll loop sleeps a full interval before re-checking the deadline, so the actual wait can overshoot by up to one poll interval plus block-source read latency. - Replace wall-clock upper bounds with block-source call-count assertions wherever the real signal is "did not hold", and loosen the two upper bounds that are genuinely needed. Lower bounds are deterministic and stay. - Add config-mapping tests covering the unset by-number wait, the by-hash default, and an explicit 0. - Add elapsedMs to the structured context of the arrived / gave-up log lines.
- Add `getBlock` to `UnseenBlockHoldOff`, which polls the cheap block-data read and then reads the full block pinned by the resolved hash, and drop `NodeBlockProvider`'s private hold-off helper. - Add `getBlockNumber` to `UnseenBlockHoldOff` so `NodeWorldStateQueries` resolves a query to a block number through the hold-off instead of unwrapping block data itself. - Resolve the world-state query once before the sync-retry loop instead of threading a first-attempt flag into the resolution: retries re-resolve without holding off, so a resolution miss now propagates without further attempts and a client never waits more than one budget.
- Resolve `getBlockHashMembershipWitness`'s reference block through `#resolveBlockNumberAndHash` like every other world-state query and drop `#resolveBlockNumber`, which leaves `UnseenBlockHoldOff.getBlockNumber` unused: removed. - Give `UnseenBlockHoldOff` a single private read-with-hold-off path parameterized by the read to perform, which `getBlockData` and `getBlock` fill in. A held query is now polled on the read the caller asked for, instead of polling metadata and then reading the block back by the resolved hash.
Adds an automine e2e test where a follower node syncs from L1 only when its archiver is triggered manually, modeling load-balancer skew: queries anchored one block past the follower tip (by number and by hash) are held until the sync is forced, and a query two blocks ahead fails fast instead of consuming the wait budget.
A client anchors on the genesis block before it has synced any block, as a PXE does for its first tagged-log queries. The block is synthetic, so a source that does not answer for it now never will and waiting only delays the answer by a whole by-hash budget.
A lower bound of the same duration as the arrival delay races the clock that delay is scheduled on, and was observed failing at 199.8ms against a 200ms bound. The extra block-source reads prove the wait just as well.
1fdbf27 to
73e7eab
Compare
alexghr
added
the
ci-release-pr
3 participants
Context
When RPC nodes sit behind a load balancer, a client can sync to block N+1 through one node and then issue follow-up queries — anchored on that block — against a different node that has only seen block N. Today the lagging node fails those requests immediately (
block hash not found... possibly a reorg has occurred), even though it would have the block within a second or two. The client then aborts or retries a whole flow (tx construction, note sync) over a transient skew of a single block.Approach
When a query references a block the node has not seen, the node now waits a bounded time for it to arrive instead of failing straight away. Outcomes are unchanged, only delayed: on a miss callers still throw or return
undefinedexactly as before, with the same errors.N: waits only whenN == proposed tip + 1, up toRPC_UNSEEN_BLOCK_BY_NUMBER_WAIT_MS(defaults to twice the block duration, i.e. 6s at the 3s default). A number further ahead, or at/below the tip but missing (pruned or reorged), still fails fast.RPC_UNSEEN_BLOCK_BY_HASH_WAIT_MS(default 3000ms).latest,proven, ...): never waits, since tags resolve against the current tip by definition.A single
UnseenBlockHoldOffinstance is shared by every block-anchored read, so the cap applies across all of them: the world-state witness queries (findLeavesIndexes, membership witnesses,getPublicDataWitness,getPublicStorageAt),getBlock/getBlockData(andgetContractthrough it), and thereferenceBlockanchor ofgetPrivateLogsByTags/getPublicLogsByTags, which fails the same way during PXE note sync. The log store's own in-transaction anchor check stays authoritative — the hold-off only gives the block a chance to land first. The hold-off lives entirely in the RPC-serving layer; internal consumers of the block source (sequencer, validator, archiver sync loops) are untouched.getWorldStateretries a resolution failure three times, so it holds off only on the first attempt — otherwise a 6s budget would become 18s for the client.There is no wire or schema change, so this is fully backwards compatible in both directions and old clients simply benefit.
Notes
BlockParameterso a client can send both the anchor number and hash, which lets the server tell "one ahead" from "reorged" precisely and removes the blind 3s hash wait; that PR will close the issue.Fixes A-1688