Port the UK stochastic layer as declarative source stages

[juaristi22]

Workstream E4 of the UK migration (#145, master epic #665). Closes #680. Stacked on #703 (E3); base flips to main when #703 merges.

What

Ports the incumbent stochastic layer as four declarative source stages appended after frs_education_grant_split, replacing the incumbent's sequential RNG streams with identity-keyed draws (blake2b over seed:variable:entity_id, the US _stable_unit_draws arithmetic) — the standing RNG signed-difference class; parity is weighted-distributional only, as UKDS disclosure rules require anyway:

• frs_take_up (benunit) — reported-receipt anchored take-up for child benefit / pension credit / universal credit (the incumbent assign_takeup_with_reported_anchors semantics exactly: anchors certain, int(rate·n) unweighted full-population target, adjusted residual, overshoot accepted; anchors fail loud on a missing *_reported source column), plain Bernoulli take-up for opts-out/TFC/the three childcare programs, and the clipped-normal extended-childcare-hours draw (inverse-CDF).
• frs_person_draws (person) — marriage allowance, age-banded SCP (0.97 under 6 / 0.85 at 6+), and the persisted private-school uniform.
• frs_household_draws (household) — TV ownership, TV-licence evasion, first-time-buyer, property purchase.
• frs_brma (household) — LHA_category materialized through the rules-engine adapter (consumed, never persisted), benunit-level inverse-CDF sampling from the committed VOA count table, household collapse by identity-keyed uniform member pick.

Supporting: the country-agnostic kernel microcosm/build/stochastic_assignment.py + six generic operation kinds (assign_binary_with_anchored_residual, assign_binary_from_banded_rates, assign_uniform_draw, assign_clipped_normal, aggregate_person_to_benunit, sample_categorical_from_count_table); the cited rate contract build/uk/take_up_contract.json (status vocabulary incl. frozen_by_adjudication for UC 0.55 per U8/uk-data#452 and fitted_offline with receipts; loader refuses unreviewed rates; year≤build-year selection semantics); build/uk/brma_rent_counts.json (60 region×LHA-category cells, 200 BRMAs, 1,257,744 observations — count-weighted sampling ≡ the incumbent's uniform row-sampling incl. the unfiltered 2019+2020 pooling; regenerator tools/build_uk_brma_count_table.py; chronicle provenance registration pending); the delivered take-up gate (take_up_signal added to the shared vocabulary; uk_take_up_signal release-blocking at ±0.05 weighted share vs contract targets — the universal-take-up constant-column landmine catcher) and uk_brma_enum_domain (release-blocking, engine enum domain); the U1 pin fix (rules_engine.version now real via importlib.metadata, was "unknown"); driver sidecar declared_seeds + stochastic_contract_sha256; the household_is_ provenance-flag convention helper (flags themselves land in E7/E8); acceptance instruments tools/verify_uk_identity_stability.py (row-permutation identity at full scale) and tools/emit_uk_brma_distribution.py; and the seeded childcare fitting tool port tools/fit_uk_childcare_takeup.py (explicit default_rng, local H5 input, receipt-emitting, not in the build — U4).

UK-specificity note (required by #680)

Generic contributions: the stochastic-assignment kernel, the six operation kinds, and the take_up_signal gate name live in shared modules/vocabularies, usable by any country. UK-specific: the two build/uk/ resources (program rates and BRMA geography are UK), the four uk_runtime transforms, the gate declarations/bindings, and the childcare fitting tool (UK program semantics). The incumbent package name appears nowhere in the tree.

Signed differences

1. RNG-stream class (standing): every unit-record draw differs from the incumbent by construction — identity-keyed blake2b (seed:variable:entity_id) replaces the sequential default_rng(100)/default_rng(0) streams. Parity is weighted-distributional only. Subsumes the extended-hours inverse-CDF and BRMA count-table variates (distributionally identical). Incumbent seeds (take-up 100, brma 0, property 0) recorded as documentation in the contract.
2. higher_earner_tie_break dropped (adjudicated with María): no consumer in policyengine-uk (the deterministic age tie-break replaced it), absent from both parity instruments; removed from both export registers; a guard test asserts the engine still has no such variable.
3. TV-evasion independence documented: drawn independently of household_owns_tv although the incumbent YAML phrases the rate conditionally; the reference share 0.105666 matches the independent draw at the 2023-applicable 0.1058 rate — ported as-is.
4. Take-up target denominator documented: int(rate × all units), unweighted, full entity population — not the "share of eligible" the incumbent YAML descriptions claim; overshoot when reporters exceed the target accepted silently — ported as-is, recorded in the contract policy string; the int() floor is test-pinned.
5. Parameter literals promoted: the property-purchase rate 0.0385 (HMRC/ONS derivation) and the extended-hours normal (15.019, 4.972, clip [0, 30]) move from code literals to cited contract entries — values unchanged.

Licensed-data acceptance (all data stayed local; disclosure-safe aggregates only)

• Determinism: two independent full builds from the pinned 2023-24 tabs; compare_uk_h5_payload.py verdict payload-identical; shares files byte-identical; both attempts landed hash-chained Logbook rows. A first attempt failed at the configured phase on the driver's pin collector (it assumed every stage artifact was a raw-tab pin and hit frs_brma's resource artifact) — fixed with resource references split into a sidecar resource_pins key, receipted in acceptance_receipt.json.
• Row-permutation identity at full scale (the instrument the e3 dir lacked): every E4 column recomputed from the pure derivations in original and permuted row order, un-permuted by entity id — identical, and equal to the stored artifact columns (permutation_identity_receipt.json, seed 123). The instrument's first version wrote a stub receipt without checking anything; rewritten to verify before sign-off.
• Nonzero shares vs the committed 145-column reference: all 17 E4 columns within ±0.02 (max |Δ| 0.0088, would_claim_targeted_childcare); exact 1.0 for brma and attends_private_school_random_draw.
• Weighted means vs contract rates: max |Δ| 0.0073 absolute across the 13 Bernoulli columns — well inside the uk_take_up_signal ±0.05 band.
• Weighted totals vs the licensed 131-column register: 12 of 16 covered columns beyond ±2% relative (max −14.9%). Investigated, not waved through: our side matches the contract rates tightly (previous bullet), while the register was extracted from the calibrated certified artifact whose weights are heavy-tailed (weight-ratio ≈1590, ESS ≈1%), so an independently redrawn Bernoulli column's weighted total deviates from the incumbent's realization at exactly the observed 3–15% scale. Falls under the standing RNG-stream signed-difference class; per-column deltas recorded in acceptance_checks.json.
• BRMA distribution at the sampler's native margin: benunit-level assignment vs each (region, LHA category) cell's conditional distribution — max |z| 3.66 over ~1,400 unmasked comparisons, within multinomial expectation (SDC min count 3). An earlier region-margin draft misleadingly showed double-digit z on a correct sampler and was rewritten before sign-off. brma has no weighted-register row by design; this is its acceptance instrument.
• Build records carry rules_engine.version: 2.89.0 (the U1 pin, formerly "unknown"), declared_seeds per stage/output, stochastic_contract_sha256, and resource_pins.

Merge coordination with #706

#706 (the #630 close-out) and this PR both edit uk/gates.json and re-cut the _UK_GATE_BATTERY_* fingerprints in microcosm-data/contract.py. The entry edits are disjoint (this PR adds uk_take_up_signal/uk_brma_enum_domain + the tie-break export removal; #706 rewrites the weight-ratio/input-mass/tail-concentration parameters). Whoever lands second: union the entries, take this PR's take-up sentence + #706's weighted-integrity sentence in the policy string, and re-run the fingerprint recompute (test_gate_battery_contract_pins drives it). Recipe agreed between sessions on 2026-08-17.

Implementation notes

Codex implemented under /codex-implement-plan; one recorded deviation, reviewed and kept: the BRMA enum-domain evidence falls back to a frame-derived domain only when the injected engine exposes no _variable("brma") (engine-less test doubles); the production adapter exposes it, and the gate evaluator fails closed when neither source resolves. Post-review fixes on top: the plan-declared nonnegative_outputs for attends_private_school_random_draw restored with its lockstep constant/test; take-up anchor aggregation made fail-loud on a missing *_reported source column (with test); the BRMA household collapse rewritten from an O(households×benunits) scan to a groupby (identical semantics). The licensed run then surfaced three defects PR CI could not see, each fixed and test-covered: the driver's pin collector crashing on frs_brma's non-tab resource artifact (now split into sidecar resource_pins, with the synthetic driver spec mirroring the real artifact so the path stays exercised); the identity-stability instrument writing a stub receipt instead of verifying (now recomputes, permutes, and compares against stored columns, exiting nonzero on mismatch); and the BRMA distribution diagnostic comparing at a misleading region margin (now the native cell margin with per-row z).

🤖 Generated with Claude Code

[vahid-ahmadi]

Checked the same way as E2/E3 — kernel and every transform against the incumbent's code, rates against the incumbent's parameter files — and this is the cleanest of the three ports so far.

Verified faithful: assign_binary_with_anchored_residual reproduces assign_takeup_with_reported_anchors exactly (int(rate·n) full-population target, max(0, …) residual, adjusted rate over non-anchored rows, silent overshoot — all four quirks preserved and the int() floor test-pinned); the benunit anchor aggregation matches the incumbent's _reported_benunit_mask (any member with a positive reported amount); every rate in take_up_contract.json matches the incumbent YAMLs including the 2023-applicable selections (TFC at 0.586 with the 2024-04-06 0.88 correctly not selected, evasion at 0.1058, FTB at 0.384, property purchase 0.0385 with the HMRC/ONS derivation carried over); SCP banding, the clipped-normal parameters, and the BRMA count-table sampling (≡ the incumbent's uniform row-sampling over the pooled observations) all line up; the household collapse is a uniform member pick, same as the incumbent's sample(n=1) modulo the standing RNG class. The signed differences are drawn exactly where they should be — 3 and 4 in particular document incumbent quirks (evasion drawn independently despite the conditional YAML phrasing; the unweighted full-population take-up denominator) instead of silently "fixing" them, which is the right call for a parity-first port. higher_earner_tie_break's removal with an engine guard test is well-warranted. All the PR's test files pass for me with the uk extra.

The acceptance section deserves a specific word: reporting that the permutation-identity instrument's first version wrote a stub receipt without checking anything, and that an earlier BRMA margin draft produced misleading z-scores on a correct sampler — both rewritten before sign-off — is exactly the receipts culture working. An instrument that has never failed is an instrument you haven't tested; documenting the discarded drafts is what makes the surviving receipts credible.

Three observations, none blocking:

• stable_identity_uniforms is a per-id Python loop over blake2b. Fine at FRS scale (~36k persons × ~17 columns), but this kernel is country-agnostic and E7's clone stack — and eventually populace-scale consumers — will multiply it into tens of millions of hash calls per build. Worth a note in the module (or an issue) that the loop is the known hot spot with a straightforward vectorization (hash into a bytes array, np.frombuffer), so the first person who hits it at scale isn't re-deriving the fix under pressure.
• The uk_take_up_signal gate's target for the three anchored columns is the raw contract rate, but the assignment targets an unweighted count with anchor overshoot accepted. If reporters are weight-skewed (plausible for pension credit), the weighted share can drift from the rate legitimately — today's margin is comfortable (max |Δ| 0.0073 against a 0.05 band), but when this gate eventually fails on an anchored column, the first diagnostic question will be "regression or weight-skew?". One sentence in the entry notes distinguishing the two would save that future debugging session.
• The #706 merge-coordination recipe is clear and the entry edits are indeed disjoint — whoever lands second, the fingerprint recompute test makes the union mechanical. No concern.