Skip to content

meta: serve a shard's replicas nearest-first to the caller that asked - #144

Merged
superhaiou merged 1 commit into
mainfrom
oss/topology-prefers-nearby-replicas
Aug 23, 2026
Merged

meta: serve a shard's replicas nearest-first to the caller that asked#144
superhaiou merged 1 commit into
mainfrom
oss/topology-prefers-nearby-replicas

Conversation

@bjmeetsfo

@bjmeetsfo bjmeetsfo commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

The caller already tells us where it is, and we throw it away

MasterGetTableTopoRequest — the wire type behind POST /MasterService/GetTableTopo — has carried an idc field all along:

#[allow(dead_code)]
struct MasterGetTableTopoRequest {
    namespace: String,
    table_name: String,
    old_topo_version: u64,
    old_topology_version: u64,
    idc: String,      // <- never read
    host: String,
    compress: bool,
}

The route forwards namespace, table_name and the version, and drops the rest. So every caller announces its location on every topology request, and every caller gets back the same answer.

That answer is ordered by placement preference, not by proximity. And placement deliberately spreads replicas as far apart as the topology allows — that is the whole point of the separation ladder in meta/location.rs. Which means most of a shard's replicas are far from any given caller by construction.

The test makes this concrete. Three replicas — east/zone-a, east/zone-b, west/zone-c — and the order every caller currently receives:

["node-a", "node-c", "node-b"]

The replica on the other side of the country is offered second, ahead of the one in the neighbouring zone, to a caller that may well be sitting in east/zone-b.

What this changes

GetTableTopologyRequest gains an optional client_location, the route stops dropping idc, and each shard's replicas are ordered nearest-first using the shared_prefix_len that meta/location.rs already provides for placement.

Three properties hold it in place:

  • The primary never moves. Reordering is about which copy to read. Who owns the shard is a different question, and it is not being answered here.
  • replica_endpoints is reordered with replicas. The two lists are positional; reordering one alone would hand every caller the wrong address for every replica.
  • The sort is stable. Among servers a caller cannot tell apart — same shared prefix — the placement scan's load ordering is still the better answer, so it survives.

Locations are hierarchical, so "no replica in my zone" is not the end of the question: a caller in east/zone-d shares east with two of the three replicas and gets both of them ahead of the western one.

Compatibility

The field is #[serde(default)], and an empty or unrecognised location leaves the order byte-identical to today. A caller that says nothing sees exactly what it saw before — there is a test for precisely that.

Tests

6 new, covering: the caller's own zone coming first, the hierarchical fallback, the primary staying put, endpoints staying aligned, silence changing nothing, and the stable-sort tie-break.

Verification:

  • cargo test -p temporalstore-rust --lib meta -- --test-threads=1 — 267 passed, 2 failed.
  • cargo check -p temporalstore-rust --all-targets — clean.

Correction. I originally wrote that these two fail on clean main. That was wrong. Re-running proxy::tests on its own gives 54 passed, 0 failed both on main and on this branch. The failures I saw were port collisions — the proxy tests bind fixed ports, and another suite was running against the same ports in a second worktree at the time; one of them reported AddrInUse outright, which I should have read as the signal it was. main was not red, and this change does not affect these tests.


Correction to the note above about the two proxy tests.

I wrote that they fail on clean main. That was wrong, and I am striking it. Running proxy::tests on its own gives 54 passed, 0 failed — on main and on this branch alike.

What I actually hit was a port collision: the proxy tests bind fixed ports, and a second worktree was running its own suite against the same ports at the time. One of the failures said AddrInUse outright, which I should have read as the signal it was instead of treating the set as a baseline. main was not red, and this change does not touch these tests.

MasterGetTableTopoRequest, the wire type behind /MasterService/GetTableTopo,
has carried an idc field all along, and the route drops it: only namespace,
table name and version are forwarded. So every caller announces its location on
every topology request, and every caller gets back the same answer.

That answer is ordered by placement preference, not proximity, and placement
deliberately spreads replicas as far apart as the topology allows - that is what
the separation ladder in meta/location.rs is for. So most of a shard's replicas
are far from any given caller by construction. With replicas in east/zone-a,
east/zone-b and west/zone-c, the order every caller receives today is node-a,
node-c, node-b: the replica on the other side of the country offered second,
ahead of the one in the neighbouring zone.

GetTableTopologyRequest gains an optional client_location, the route stops
dropping idc, and each shard's replicas are ordered nearest-first using the
shared_prefix_len that meta/location.rs already provides for placement.

Three properties hold it in place. The primary never moves: reordering is about
which copy to read, not who owns the shard. replica_endpoints is reordered with
replicas, because the two lists are positional and reordering one alone would
hand every caller the wrong address for every replica. And the sort is stable,
so among servers a caller cannot tell apart the placement scan's load ordering
survives.

Locations are hierarchical, so "no replica in my zone" is not the end of the
question: a caller in east/zone-d shares east with two of the three replicas and
gets both ahead of the western one.

The field is serde-default, and an empty or unrecognised location leaves the
order byte-identical to today, with a test for exactly that.

Six tests: the caller's own zone first, the hierarchical fallback, the primary
staying put, endpoints staying aligned, silence changing nothing, and the
stable-sort tie-break.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 22, 2026 23:10
@superhaiou
superhaiou merged commit f8831aa into main Aug 23, 2026
6 checks passed
@superhaiou
superhaiou deleted the oss/topology-prefers-nearby-replicas branch August 23, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants