fix(scoring): scope the value to contested chains, opt-in per bench - #2281
Merged
Conversation
Replaces the global chain-wins sort from #2280, which fixed the ordering but broke the reading of it. Ranking on a key the reader cannot see in the column produced tables that no longer descend: 79.88% at rank 4 on 008, and a 74 ms leader at rank 3 on rpc-capabilities where lower is better. The defect was never the sort key. It is that the aggregate includes chains where a provider had no competitor, so the fix belongs on the value: with `score_scope: contested_chains` a bench is scored only on chains carrying at least two measured providers, and a provider with none of those leaves the ranked field (still visible on its own chain tab). One quantity on screen, ordering follows from it. Opt-in per bench and named for the property, not the bench, so any bench whose chain set contains uncontested chains is a candidate and the rule is readable in the public YAML. Enabled on wallet-labels-coverage only: stellar, xrp and bitcoin carry one measured provider each there. load.ts already fetched every provider's per-chain value inside its per-chain loop and discarded all but the leader; it now keeps them. Four other live benches qualify and are deliberately left untouched pending review: rpc-capabilities, token-quote-coverage, bridge-fee, perp-liq-rate. Residual limitation documented in methodology: providers are still averaged over the different subsets of contested chains they cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Sep 9, 2026
…chain scope (#2282) #2281 changes the provider values on wallet-labels-coverage, but the cached v62 entries still carry the old cross-chain aggregate, so staging kept serving StellarExpert first. Both keys bumped in lockstep as usual. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes the global chain-wins sort merged in #2280.
Why that one was wrong
#2280 fixed the ordering and broke the reading of it. It sorted on contested-chain wins while still displaying the wide cross-chain aggregate, so the column no longer descended:
A reader sees that and concludes the page is broken, not that the sort follows another logic. Sorting on a key that is not on screen is the bug.
The actual defect, and where the fix belongs
It was never the sort key. It is that the aggregate includes chains where the provider had no competitor. A cross-chain average is a mix rather than a comparison: it credits a provider for the chains it happens to be measured on.
So narrow the value, not the ordering. With
score_scope: contested_chainsa bench is scored only on chains where at least two providers reported data, and a provider with none of those leaves the ranked field while staying visible on its own chain tab.Everything then falls out on its own: one quantity on screen, the ordering follows from it, the column descends, and Serialized reaches first place on 008 without an invisible key.
Scope
Opt-in per bench, named for the property rather than the bench, so any bench whose chain set contains uncontested chains is a candidate and the criterion is readable in the public YAML rather than being a special case someone has to be told about.
Enabled on
wallet-labels-coverageonly —stellar,xrpandbitcoincarry one measured provider each there.Four other live benches qualify and are deliberately left untouched pending per-bench review:
rpc-capabilities,token-quote-coverage,bridge-fee,perp-liq-rate. Nothing else changes behaviour: without the flag, ranking is by value exactly as before.Implementation
materialize/load.tsalready fetched every provider's per-chain value inside its per-chain loop and threw away all but the leader and trailer. It now keeps them, andapplyContestedChainScoperewrites each provider's value to the unweighted mean over contested chains — a chain counts once regardless of sample count.citation.tsandproviders.tsare reverted to sorting by value alone.Stated limitation
This removes the uncontested win. It does not equalise chain mix: providers are still averaged over the different subsets of contested chains they cover. Documented in
/methodologyunder "Contested-chain scoring" rather than left implicit.241 tests pass(4 new on the scoping function), typecheck clean, eslint clean on touched files,218 specs valid.🤖 Generated with Claude Code