fix(ranking): rank multi-chain benches by contested-chain wins, not the mix - #2280
Merged
Conversation
…he mix On a bench with chain dimensions the cross-chain aggregate is a mix, not a comparison. Ranking on it alone let a provider measured on one chain that nobody else reported finish above a provider that led several contested ones. Five live benches shipped that way: rpc-capabilities Binance 1st on 1 chain, PublicNode led 6 wallet-labels-coverage XRPScan 1st on 1 chain, Serialized led 4 token-quote-coverage Jupiter 1st on 1 chain, Mobula led 2 bridge-fee Squid Router 1st on 1 chain perp-liq-rate Lighter 1st on 1 chain rankedCandidates now sorts by contested-chain wins first and uses the aggregate value only to break ties. A chain counts only when at least two providers reported on it, so an uncontested chain awards nothing: you do not win a race you ran alone. Guarded by the per-chain stashes, which materialize/load.ts populates only on the unfiltered view. A chain-filtered variant has none, so ?chain=bnb keeps ranking by value as before. providers.ts reuses the same ordering. The two surfaces disagreeing is what put "#3 of 8" next to five chain-leadership chips on the same bench row. Known and accepted: a provider with one contested win now ranks above one with none and a higher aggregate figure (TonAPI over XRPScan on 008). That is what ranking on head-to-head record means; the win count is on the row. Rule documented in methodology, section II. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
added a commit
that referenced
this pull request
Sep 9, 2026
…2281) Replaces the global chain-wins sort from #2280, which fixed the ordering but broke the reading of it. Ranking on a key the reader cannot see in the column produced tables that no longer descend: 79.88% at rank 4 on 008, and a 74 ms leader at rank 3 on rpc-capabilities where lower is better. The defect was never the sort key. It is that the aggregate includes chains where a provider had no competitor, so the fix belongs on the value: with `score_scope: contested_chains` a bench is scored only on chains carrying at least two measured providers, and a provider with none of those leaves the ranked field (still visible on its own chain tab). One quantity on screen, ordering follows from it. Opt-in per bench and named for the property, not the bench, so any bench whose chain set contains uncontested chains is a candidate and the rule is readable in the public YAML. Enabled on wallet-labels-coverage only: stellar, xrp and bitcoin carry one measured provider each there. load.ts already fetched every provider's per-chain value inside its per-chain loop and discarded all but the leader; it now keeps them. Four other live benches qualify and are deliberately left untouched pending review: rpc-capabilities, token-quote-coverage, bridge-fee, perp-liq-rate. Residual limitation documented in methodology: providers are still averaged over the different subsets of contested chains they cover. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
On a bench that declares chain dimensions, the headline ranking used the cross-chain aggregate. That aggregate is a mix, not a comparison: it rewards a provider for the chains it happens to be measured on. A provider measured on exactly one chain that nobody else reported could top the board without ever beating anyone.
Five live benches shipped that way. Measured against the staging API, all 15 chain-dimensioned live benches:
Bench 008 in detail:
stellar,xrpandbitcoinhave exactly one provider measured on them.The change
rankedCandidatessorts by contested-chain wins first, aggregate value only as a tiebreak. A chain counts toward the total only when at least two providers reported data on it, so an uncontested chain awards nothing.Guarded by the per-chain stashes (
bestPerChain,providersPerChain), whichmaterialize/load.tspopulates only on the unfiltered view. A chain-filtered variant carries neither, so?chain=bnbkeeps ranking by value exactly as before, and a bench with no chain dimensions is untouched.providers.tsreuses the same comparator. The two surfaces disagreeing is what produced#3 of 8sitting next to five chain-leadership chips on one row of/products/serialized.Known trade-off
A provider with one contested win now ranks above a provider with none and a higher aggregate figure — on 008, TonAPI (35.6%, 1 win) above XRPScan (79.8%, 0 contested chains). That is what ranking on head-to-head record means, and the win count is visible on the row. The alternative considered was scoring on the mean over contested chains only, which produces no inversions but drops uncontested providers off the board entirely.
Blast radius
rankedCandidatesfeedsleader(),fieldValue(),/api/stat, MCP, llm-context, JSON-LD, OG images and the templated headline sentences. The published leader changes for the five benches above, and the daily Parquet snapshot'sheadlinestable will follow from the next run.Rule documented in
/methodology, section II.241 tests pass(4 new), typecheck clean, eslint clean on touched files,218 specs valid.🤖 Generated with Claude Code