Skip to content

fix(scoring): scope the value to contested chains, opt-in per bench - #2281

Merged
Flotapponnier merged 1 commit into
devfrom
fix/contested-chain-scoring
Sep 9, 2026
Merged

fix(scoring): scope the value to contested chains, opt-in per bench#2281
Flotapponnier merged 1 commit into
devfrom
fix/contested-chain-scoring

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Supersedes the global chain-wins sort merged in #2280.

Why that one was wrong

#2280 fixed the ordering and broke the reading of it. It sorted on contested-chain wins while still displaying the wide cross-chain aggregate, so the column no longer descended:

wallet-labels-coverage          rpc-capabilities (lower is better)
1. Serialized      77.0 %       1. PublicNode   166.53 ms
2. Mobula         47.11 %       2. dRPC         217.93 ms
3. TonAPI         35.94 %       3. Binance       74.45 ms   <-- best number, rank 3
4. StellarExpert  79.88 %  <--
5. XRPScan        79.84 %

A reader sees that and concludes the page is broken, not that the sort follows another logic. Sorting on a key that is not on screen is the bug.

The actual defect, and where the fix belongs

It was never the sort key. It is that the aggregate includes chains where the provider had no competitor. A cross-chain average is a mix rather than a comparison: it credits a provider for the chains it happens to be measured on.

So narrow the value, not the ordering. With score_scope: contested_chains a bench is scored only on chains where at least two providers reported data, and a provider with none of those leaves the ranked field while staying visible on its own chain tab.

Everything then falls out on its own: one quantity on screen, the ordering follows from it, the column descends, and Serialized reaches first place on 008 without an invisible key.

Scope

Opt-in per bench, named for the property rather than the bench, so any bench whose chain set contains uncontested chains is a candidate and the criterion is readable in the public YAML rather than being a special case someone has to be told about.

Enabled on wallet-labels-coverage only — stellar, xrp and bitcoin carry one measured provider each there.

Four other live benches qualify and are deliberately left untouched pending per-bench review: rpc-capabilities, token-quote-coverage, bridge-fee, perp-liq-rate. Nothing else changes behaviour: without the flag, ranking is by value exactly as before.

Implementation

materialize/load.ts already fetched every provider's per-chain value inside its per-chain loop and threw away all but the leader and trailer. It now keeps them, and applyContestedChainScope rewrites each provider's value to the unweighted mean over contested chains — a chain counts once regardless of sample count.

citation.ts and providers.ts are reverted to sorting by value alone.

Stated limitation

This removes the uncontested win. It does not equalise chain mix: providers are still averaged over the different subsets of contested chains they cover. Documented in /methodology under "Contested-chain scoring" rather than left implicit.

241 tests pass (4 new on the scoping function), typecheck clean, eslint clean on touched files, 218 specs valid.

🤖 Generated with Claude Code

Replaces the global chain-wins sort from #2280, which fixed the ordering but
broke the reading of it. Ranking on a key the reader cannot see in the column
produced tables that no longer descend: 79.88% at rank 4 on 008, and a 74 ms
leader at rank 3 on rpc-capabilities where lower is better.

The defect was never the sort key. It is that the aggregate includes chains
where a provider had no competitor, so the fix belongs on the value: with
`score_scope: contested_chains` a bench is scored only on chains carrying at
least two measured providers, and a provider with none of those leaves the
ranked field (still visible on its own chain tab). One quantity on screen,
ordering follows from it.

Opt-in per bench and named for the property, not the bench, so any bench
whose chain set contains uncontested chains is a candidate and the rule is
readable in the public YAML. Enabled on wallet-labels-coverage only:
stellar, xrp and bitcoin carry one measured provider each there.

load.ts already fetched every provider's per-chain value inside its per-chain
loop and discarded all but the leader; it now keeps them.

Four other live benches qualify and are deliberately left untouched pending
review: rpc-capabilities, token-quote-coverage, bridge-fee, perp-liq-rate.

Residual limitation documented in methodology: providers are still averaged
over the different subsets of contested chains they cover.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Flotapponnier
Flotapponnier merged commit cd3ef81 into dev Sep 9, 2026
1 check failed
@Flotapponnier
Flotapponnier deleted the fix/contested-chain-scoring branch September 9, 2026 14:47
Flotapponnier added a commit that referenced this pull request Sep 9, 2026
…chain scope (#2282)

#2281 changes the provider values on wallet-labels-coverage, but the cached
v62 entries still carry the old cross-chain aggregate, so staging kept
serving StellarExpert first. Both keys bumped in lockstep as usual.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant