UK gate adjudication: reference registry, spec-armed thresholds, weight-ratio re-baseline (#630)

[juaristi22]

Closes #630. The campaign's adjudication PR under the #665 master epic, reworked after the finding-2 flip and the #710 upstream fix: the case-2 ruling lands as a reviewed reference registry with per-reference exclusions, the weighted-integrity thresholds become committed spec at post-#710 measured edges, the weight-ratio fence keeps its exact certified June value, and the confirmation build ran as part of this PR — the first national staging build to pass the fully-armed terminal battery, the receipt #630 has been waiting for.

Adjudication history, honestly stated

The original ruling proposed re-baselining the weight-ratio fence to the staged surface's measurement; an earlier revision of this PR implemented that. #705's adversarial review then flipped the adjudication — the 1590.53 excess was 11–12 SPI-synthetic clone rows from two donor lineages, not a structural tail — and #710 fixed the SPI prior allocation upstream (per-stratum mass over sampled quota). This revision drops the re-baseline entirely: the fence stays at 1151.2542195939373, its entry notes record the resolution, and every weight-sensitive threshold below was re-minted on the post-#710 surface. Full history in experiments/630-uk-gate-adjudication-receipts.md.

1. Reference registry for uk_input_mass_parity (#630 case 2)

The comparison target is now an explicit, reviewed, swappable choice declared in uk/gates.json: a reference_registry of named references, each carrying identity pins, the canonical totals digest, and a scope note; parameters.reference names the armed one and the signed report records both the name and the scope note. Reviewed exclusions are scoped per reference (register schema 3): charitable_investment_gifts is excluded on the channel-blind efrs-post-calibration reference (+15,168.8% on the post-#710 surface — the SPI channel functioning as designed against a reference that structurally lacks it), and would be compared normally against any future channel-aware reference with no register change. The runtime holds the declared registry equal to the module registry, and the licensed totals sidecar must resolve to a registry entry — a wrong sidecar fails closed naming the known references.

2. Thresholds into the spec; arming flags retired (#630 tracking note)

uk_input_mass_parity and uk_qrf_tail_concentration now carry their numeric thresholds in gates.json parameters (moving the policy hash), at the measured edges minted from the post-#710 measurement run (m4), no discretionary headroom:

parameter	receipted value	derivation
relative_tolerance	4.521811483823806	worst surviving |drift| at the reviewed floor (is_enhanced_disabled_for_benefits), after the sole breach became the per-reference exclusion
minimum_reference_total	0.0	excludes only arithmetically meaningless zero-mass denominators; at 0.0 the checked set (128 columns) is identical to the floor-1.0 run and the drift edge is an economically meaningful column
top_k	100	measurement-grid anchor
max_top_share	0.9970712395200448	exact measured maximum top-share (charitable_investment_gifts)
min_nonzero_records	274	thinnest measured column (hmrc_spi_taxable_termination_pay); count-based, unchanged by the weight fix as predicted

The five CLI arming flags are deleted. --input-mass-reference-json survives as pure evidence supply (the UKDS-licensed totals cannot be committed; the spec pins their canonical digest, and a missing sidecar now records evidence_absent instead of silently disarming the gate). The --*-exclusions overrides survive as review-time process, self-describing via the evidence payload's exclusions_policy: committed|override content-labelling — and the input-mass evidence payload now seals the exclusions in force, closing the override-smuggle hole the degenerate gate already closed. (The equivalent QRF evidence hook is a named follow-up.)

3. uk_weight_ratio: the fence held

maximum_max_to_median_ratio stays at the exact certified June measurement 1151.2542195939373. The entry notes now record the #630 resolution (the breach was an SPI allocation defect, fixed in #710; post-fix full-scale ratio 681.5534900252635) and relocate the derivation/fence-discipline prose that #691's constant deletion orphaned. The microcosm-data mirrors move in the same reviewed change (active reference, required detail fields, the three vintage digests and the input-mass evidence pin — producer-recomputed over the merged spec, which now includes #703's uk_nonnegative_columns), and a new lockstep test in test_gate_battery_contract_pins.py holds the contract's numeric mirrors equal to the committed spec, restoring the chain #691 broke.

Verification runs (receipts: experiments/630-uk-gate-adjudication-receipts.md)

run	code	scale	verdict
m2 measurement	post-#691 main	f100	blocked as expected; its ratio measurement (fifth bit-exact reproduction) became evidence in the #705 flip
f010 pre/post	pre-#691 vs post-#691	f010	the #691 deferred bit-identity check: PASS — every gate's evidence bit-identical across the typed-weights flip; only the declared policy digests and signature moved; the sole artifact delta is the checkpoint container no longer carrying the stripped household_weight column (content identities bit-equal)
m4 measurement	post-#710 main	f100	weight_ratio passes at the June bound (ratio 681.5534900252635, ESS 0.131, total mass bit-equal); minted the receipted edges above; only failure = the charitable drift the exclusion resolves
m6 confirmation	this branch, rebased onto post-#709 main	f100	all 16 gates green: weight_ratio at the certified June fence, input-mass green with the per-reference exclusion in force (stale/expired/premature all empty), QRF at the measured edges, uk_nonnegative_columns green; staged H5 + schema-3 build record written

Payload receipts for m5: vs the pre-#710 m3 artifact, the differences are exactly household_weight on 200,000 rows — precisely the SPI-synthetic row set #710 repairs — plus capital_gains on 15,170 person rows (the CGT stage's weighted draw following the repaired weights). vs the retained e726abd rebuild (the #635 transitive target; the pinned June staging is local_untracked by charter), the same two changes plus the already-receipted era deltas (CGT family, household column order only, mass-log attr). Bytes moved for named reasons, so the committed national_staging_build_record.json is re-cut verbatim from m5 (schema 1 → 3), per the #635 charter. All runs Logbook-chained (m1 → m2 → f010-pre → f010-post → m3 → m4 → m5). All quoted values are ratios, shares, tolerances, and digests per CD171 §5.2.1.

Not in this PR (named follow-ups)

• QRF-tail evidence hook (the same override-provenance sealing the input-mass gate now has), including pinning the expected QRF surface by manifest-derived column digest (adversarial-review recommendation).
• Tightening the input-mass tolerance by receipting the explained top drifts — filed as Tighten the UK input-mass tolerance by receipting the explained top drifts #713 (review finding: the fence sits at 4.5218 against a population at median 0.24 / p90 0.63; the distribution and the exclusions-versus-tolerance tradeoff are in the receipts doc, and the limitation is recorded in the entry notes).
• Closing the schema-3 legacy report path for non-grandfathered exact-k releases — filed as Close the schema-3 terminal-report path for non-grandfathered exact-k UK releases #712 (adversarial-review finding: pre-existing, gated behind the out-of-band signing key, 35-test fixture migration — the data-shard completion of the Finish the #612 tails: drop the carrier household_weight column; narrow the crosswalk clone entry #654 schema-3 retirement).
• Always-armed QRF on rungs: a 1%/10% rung now evaluates QRF at full-scale edges and can block on a rung-scale artifact — accepted per the UK builds inherit the scale ladder from day one (sample_fraction in the #612 Frame carrier) #627 precedent; note the SPI prior allocation: spread stratum mass over the sampled quota, not donor weights #710 fix already unblocked rungs on weight_ratio.

Merge coordination with #709

#709 (the E4 increment) also edits uk/gates.json and re-cuts the three _UK_GATE_BATTERY_* fingerprints in contract.py. Whoever lands second applies the recipe recorded in #709's "Merge coordination with #706" section: union the disjoint gates.json entries, take #709's policy-string wording for the take-up clause and this PR's for the weighted-integrity sentence, re-run the producer digest recompute once, and mechanically re-cut the build record if needed. (The weight-ratio line of the original recipe is obsolete — the fence keeps its June value.)

Verification

• Full suite (uv run --no-sync pytest -q): green on the final tree; lint (uv run --no-sync ruff check .): clean.
• The m5 confirmation run above is the end-to-end receipt: the first national staging build to pass the fully-armed terminal battery, at the honest June fence, on the SPI prior allocation: spread stratum mass over the sampled quota, not donor weights #710-repaired surface.

Refs #630, #635, #654, #609, #610, #611, #627, #665, #705, #710.

🤖 Generated with Claude Code

[juaristi22]

Status: holding in draft — the weight-ratio re-baseline component of this PR is withdrawn per the #630 adjudication flip (2026-08-17): #705's adversarial review showed the 1590.53 excess is carried by 11–12 SPI-synthetic clone rows from two donor lineages (full-scale ratio ~944.5 without them — inside the June bound), so finding 2 resolves as a data fix upstream of the gate and the fence stays at 1151.2542195939373. "Stable is not structural."

What that means for this PR, component by component:

• Withdrawn: the uk_weight_ratio re-baseline (gates.json value + notes, the contract.py mirror, test literals, its changelog fragment).
• Invalidated as receipts: the m3 confirmation run (it passed only under the withdrawn bound) and the UK national staging onto the outer stage runtime, increment 3 of #612 #635 national_staging_build_record.json re-cut minted from it — both will be redone after the upstream lineage fix lands, when a staging build should pass at the June bound. National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630 closes on that run, not this one.
• To be re-minted post-fix: the weight-sensitive receipted edges (relative_tolerance, max_top_share) were measured on the pathological surface; they will be re-measured on the fixed surface before landing. (top_k and the count-based min_nonzero_records are expected stable.)
• Standing, unaffected by the flip: the reference registry + per-reference schema-3 exclusions (the National staging rebuild fails the current terminal gates: constant source_year release column and SPI weight tail above the reviewed maximum #630 case-2 adjudication — channel-structural, not magnitude-based), the input-mass evidence-payload hardening, the new contract↔spec lockstep test, the arming-flag retirement machinery, the frs_spine fix (byte-duplicate of Port the FRS derived layer as declarative source stages #703's), and the f010 pre/post-#691 bit-identity receipt.

Plan: one surgery pass once the lineage fix PR lands — drop the withdrawn component, re-run the measurement + confirmation on the fixed surface, re-mint the edges, re-cut the record from the genuinely-passing run. The merge-coordination section below re: #709 still applies minus the weight-ratio line.

[juaristi22]

Resolved — the surgery pass is done and the hold is lifted. With #710 merged, this PR was reworked per the flip: the weight-ratio re-baseline is fully removed (the fence keeps its exact certified June value, with the #630 resolution recorded in its entry notes), the weight-sensitive edges were re-minted on the post-#710 surface from a fresh full-scale measurement run (m4: weight_ratio passes at the June bound, ratio 681.5534900252635), and the confirmation run (m5) passes the entire armed battery — the first staging build to do so, and the #630 close-out receipt. Sharpest payload receipt: m5 differs from the pre-fix m3 artifact in household_weight on exactly 200,000 rows — precisely the SPI-synthetic row set #710 repairs. The build record is re-cut from m5. Branch force-pushed with the reworked history; PR body updated; receipts in experiments/630-uk-gate-adjudication-receipts.md.

[juaristi22]

Adversarial review (Codex) run on this branch — verdict needs-attention, both findings dispositioned:

1. [high] Publication verifier trusted report-self-described weighted-integrity thresholds — FIXED in 1ab9c928. The five spec-armed thresholds now have microcosm-data mirrors that passing schema-4 reports must equal exactly (the weight-ratio precedent extended to input-mass tolerance/floor and the QRF trio); the build-shard lockstep test holds the mirrors equal to the committed spec, and re-signed loosened/shrunken reports are regression-tested as refused.
2. [critical] Schema-3 legacy reports still accepted for non-grandfathered exact-k releases — DEFERRED with a named pointer. Pre-existing on main (not introduced here), exploitable only with possession of the out-of-band signing key, and closing it migrates the 35-test schema-3 fixture suite — the data-shard completion of the Finish the #612 tails: drop the carrier household_weight column; narrow the crosswalk clone entry #654 schema-3 retirement, which deserves its own PR rather than a rider here. Named in this PR's follow-ups; happy to file the tracking issue.

The QRF-surface pinning recommendation folds into the already-named QRF evidence-hook follow-up.

[vahid-ahmadi]

The adjudication history section is the best part of this PR: an owner ruling proposed, implemented, adversarially flipped, and withdrawn — with the superseded m3 run kept in the chain as history rather than scrubbed, and the fence holding at its exact certified value because the breach turned out to be an upstream data defect. "Stable is not structural" earned its place in the receipts. The m5 confirmation — first staging build through the fully-armed battery, differing from pre-fix m3 in household_weight on exactly the 200,000-row SPI-synthetic set #710 repairs — is about as tight as a close-out receipt gets.

The mechanics I checked all hold: the reference registry with per-reference exclusions (schema 3) correctly makes the comparison target a reviewed, swappable declaration — the charitable_investment_gifts exclusion scoped to the channel-blind reference, with expiry and the stale-fail semantics keeping the register self-cleaning; the runtime holding declared registry equal to module registry and the sidecar failing closed against unknown references; and the microcosm-data threshold mirrors from the Codex finding-1 fix, with the lockstep test holding them equal to the committed spec. Deferring the schema-3 legacy-report closure with a named pointer is the right scope call — it's pre-existing, key-gated, and deserves the fixture-migration PR it will cost. All the PR's test files pass for me.

One design question I'd want answered in the entry notes before this merges — the shared relative_tolerance of 4.5218 makes the input-mass fence nearly toothless for the other 127 columns.

The value is minted honestly ("worst surviving |drift|, no headroom"), but look at what the edge is: is_enhanced_disabled_for_benefits at 4.52, with sda_reported at 3.06 and jsa_income_reported at 2.17 behind it. The E3 receipt already established that the disability flags match the incumbent bit-for-bit at matching composition — their drift against the frozen reference is entirely the reference's vintage plus SPI-stack composition, the same explained-drift class as the charitable_investment_gifts exclusion, just smaller in magnitude. By folding the worst explained drift into the shared tolerance instead of the register, the gate's unexplained-regression budget for every column becomes ±452%: a genuine 2× mass regression in employment_income — the kind of defect this gate exists to catch — passes silently.

The instrument for this already exists in this very PR: per-reference reviewed exclusions with receipts, expiry, and stale-fail. Moving the explained top drifts (the disability trio, and sda/jsa if their drift traces to the same composition story) into receipted exclusions expiring at the E10 recomposition would let the shared tolerance re-mint at the worst unexplained survivor — plausibly 5–10× tighter — and the stale-fail semantics would force the exclusions to be revisited exactly when E10 changes the composition. The doctrine ("measured edge, no headroom") is honored either way; the difference is whether the edge is set by a column whose deviation you've already explained elsewhere. If there's a reason to prefer the single loose tolerance — e.g. most of the 128 columns sit in the 1.5–4.5 band for compositional reasons and the register would bloat — that distribution belongs in the receipts doc, because right now the receipt names only the top three drifts and they all look explained.

Not blocking on my account — the current value is strictly better than runtime-armed, and the re-mint is a parameter edit under the same mechanism — but I'd rather see the tolerance carry only comparability noise before this fence is what stands between a bad build and a release.

[juaristi22]

Thank you — the tolerance question was the right one to ask, and measuring it changed my view of both the problem and your proposed remedy. The full 128-column surface is now in experiments/630-uk-gate-adjudication-receipts.md; the short version, with the two places the data cuts against the sketch:

Your core point holds, and it's worse than the top-three suggested. The compared population sits at median 0.24, p75 0.42, p90 0.63, p95 1.15 against a 4.5218 fence — so the gate is ~7× looser than the p90 of the distribution it fences. At this value it is a gross mass-loss/explosion fence, not a regression detector: your 2× employment_income regression does pass. That limitation is now written into the entry notes rather than left silent.

But the tightening arithmetic doesn't support 5–10× from three-to-five exclusions. The decay above the bulk is smooth, with no natural cut point:

receipted into the register	re-minted tolerance	tightening
1 (today)	4.5218	—
2	3.0587	1.48×
4	1.5237	2.97×
7	0.9026	5.01×
20	0.5167	8.75×
30	0.4401	10.27×

5–10× needs 7 to 30 receipted columns. Your stated counter-condition is also false, in the helpful direction: the columns do not bank up in a dense 1.5–4.5 band (only four exceed 1.5), so the register wouldn't bloat for a modest gain — it's the deep tightening that gets expensive.

And only one of the top drifts can be receipted today. You were right that the edge column is explained, and the receipt is better than I expected: #703's head-to-head verified the disability categories and flags agree row-for-row at 100.0000% with the incumbent's own create_frs at the pinned revision, attributing the residual to the artifact's vintage plus the ~37% SPI-synthetic composition, with re-measurement scheduled at E10 — a real explanation and exactly the expiry anchor you proposed. sda_reported and jsa_income_reported are plausibly the same compositional story, but neither carries verification of its own, and this register refuses unevidenced reasons. Excluding only the receipted column re-mints at 3.0587 — a 1.48× gain that doesn't change the gate's practical sensitivity, which is why I'd rather not spend the register entry on it in isolation.

So: tolerance stays at the measured edge here, limitation documented, and the triage is filed as #713 with the distribution, the tradeoff table, and the #703 receipt as its methodological template. One thing I'd want that issue to settle rather than assume — an exclusion buys sensitivity on 127 columns by giving up all detection on the excluded one, and the register currently can't express the difference between "explained deviation, still watch the magnitude" and "suppress". That asymmetry seems worth a design answer before receipting seven columns.

Two other changes since your review: the branch is rebased onto post-#709 main (gates.json unioned per the recipe — your take-up sentence, the weighted-integrity sentence from here, digests recomputed once), and #709's two new release-blocking entries aged the confirmation receipt, so it was re-run: m6 evaluates all 16 gates, 13 passed, three parity gates evidence_absent, none failed, and m6's artifact is payload-identical to m5's — the rebase moved policy only, not one byte of built data. The build record is re-cut from m6 with the 16-gate report embedded.

[vahid-ahmadi]

Accepted — you measured what I sketched, and the measurement wins. The distribution (median 0.24, p95 1.15 against a 4.52 fence) confirms the sensitivity problem is real, and the tradeoff table shows my "5–10× from a few exclusions" arithmetic was wrong: the smooth decay means meaningful tightening costs 7–30 register entries, which changes the economics entirely. Holding the tolerance at the measured edge with the limitation written into the entry notes, and filing the triage as #713 with the full distribution attached, is the right disposition — better than spending one register entry for a 1.48× gain that doesn't change what the gate can catch. The register refusing sda_reported/jsa_income_reported for want of their own verification is the evidence discipline doing its job even when it slows the fix I asked for.

Re-verified on the rebased branch: the unioned gates.json, recomputed digests, and contract mirrors all pass locally, and m6's payload-identity to m5 is the right receipt that the rebase moved policy only. The asymmetry question you've posed to #713 — exclusions buy sensitivity on 127 columns by abandoning detection on the excluded one — is the correct thing to settle first; answering there. Nothing further from me on this PR.