Port the FRS derived layer as declarative source stages

[juaristi22]

Workstream E3 of the UK migration (#145, master epic #665). Closes #679. Stacked on the merged #692 (E2); based on current main.

What

Ports the incumbent FRS derived layer as six declarative source stages appended after frs_spine in uk/source_stages.json, each a whole-stage callable with its own lockstep, determinism, and hand-computed-fixture tests, run by the (now multi-stage) tools/build_uk_frs_spine.py driver:

• frs_employment — employment_status (exact incumbent semantics incl. the truncated-zip artifact: code 11 → LONG_TERM_DISABLED, never OTHER_INACTIVE), employment_sector, sic_industry_division; deliberately fail-loud on missing mjobsect/sic (incumbent posture preserved).
• frs_council_tax — the (region, band, single-adult) cell-mean imputation via the new generic impute_cell_means operation kind, ported post-uk-data#449 (Scottish Water netted out of CTANNUAL): raw ctannual > 0 donor filter over netted values, dropna=False cells, reported-zero households neither donors nor imputed, no-donor cells → 0. Re-reads househol.tab (byte-identical artifact pin to the spine's) because the spine's council_tax_reported is clipped and the raw −1/NaN missing semantics matter.
• frs_disability — the five categories (thresholds max(0, weekly_rate − £1), ascending overwrite, default NONE, .baseline.gov.dwp rates) and the three flags (plain gov.dwp rates, annualized £1×WEEKS gap, the incumbent's operator asymmetry on dla_sc preserved and pinned). Pure functions over (person, frozen-rates dataclass) so E7's SPI-donor refresh reuses them.
• frs_education — current_education cascade (incl. the incumbent's unreachable POST_SECONDARY branch, pinned dead), the corrected EDUCQUAL_MAP (signed difference, below), QYP flags/entry age/terminal date, EMA fill-with-mean (the impute_cell_means degenerate single-cell case), receives_benefits_in_own_right.
• frs_legacy_proxies — the three ESA/JSA claimant-state proxies (explicitly proxies, not entitlement); state_pension_age materialized through the rules-engine adapter (consumed, never persisted); JSA threshold gov.dwp.JSA.hours.single × WEEKS_IN_YEAR on the spine's signed hours scale (person-set identical to the incumbent's 832-hour cutoff; boundary pinned both sides).
• frs_education_grant_split — the proportional allocator over engine-materialized capacities (childcare_grant, parents_learning_allowance, adult_dependants_grant) with the DSA year gate (zero vector pre-2025, alignment-pinned); rewrites education_grants in place (declared via a rewrites key — frs_spine stays the one canonical producer) and emits disabled_students_allowance_eligible_expenses (reference-documented zero layer at this vintage).

Supporting changes: materialize_rules_engine_predictors declarations with predictor-lockstep tests (first manifest use of the U1 adapter); parameter readers on the CGT deferred-import pattern but resolving through the model system's parameters(year) — the incumbent's exact call shape, including its value-bearing baseline-vs-plain split (signed difference 2)semantics — an April-rates trap a June read would hit); theuk_nonnegative_columnsterminal gate declared (bringing E2's 24nonnegative_outputsdeclarations live;UK_NONNEGATIVE_SOURCE_OUTPUTSmirrors the US pattern; presence stays owned by the coverage contract); microcosm-data mirror digests refreshed; driver sidecar/shares generalized to multi-stage (fixes therecords[0]` single-stage emission).

Two commits here repair main, and can be cherry-picked independently:

• Drop household_weight from the frs_spine output surface — repairs the Retire the UK schema-3 gate path and keep household weights typed in-build (#654) #691×Ingest the full raw FRS table set and assemble the UK spine as a Frame #692 seam (Retire the UK schema-3 gate path and keep household weights typed in-build (#654) #691 moved weights to the typed vector; Ingest the full raw FRS table set and assemble the UK spine as a Frame #692's spine still declared the column as an output; the StagePlan produces-check fails on main today).
• Fix the UK adapter's engine dataset constructor kwarg — UKSingleYearDataset takes fiscal_year, not time_period; E3 is the adapter's first real-engine consumer, so no existing test could see it. Adds a cheap engine-gated construction test.

UK-specificity note (required by #679)

UK-specific: the FRS coding maps (empstati/mjobsect/typeed2/educqual/train), DWP rate structures and the baseline-vs-plain parameter split, QYP legislative approximations, Scottish Water netting, and the value-bearing baseline-vs-plain parameter-tree split. Generic contributions any country can use: the impute_cell_means operation kind (cell means and fill-with-mean as one declarative primitive), the multi-stage single-driver sidecar/shares pattern, and the consumed-not-persisted rules-engine materialization pattern. The incumbent package name appears nowhere in the tree (concatenation idiom preserved).

Signed differences

1. Corrected EDUCQUAL value labels (adjudicated with María: verify-then-fix-and-sign). The incumbent's map is inverted (low codes read as school-level, 17–21 as degrees). Verified label-for-label against the official FRS 2023-24 data dictionary (UK Data Service SN 9367, DOI 10.5255/UKDA-SN-9367-2, adult table) and corroborated by the raw aggregates: code 1 (Doctorate) is 1.8% of adults at the highest mean earnings; codes 18/19 (Welsh/Scottish Baccalaureate) are near-empty; the GCSE band (36–82) carries the mass. Weighted highest_education composition moves from {TERTIARY 9.9%, POST_SECONDARY 21.3%} (inverted map — implausible for the UK) to {TERTIARY 28.1%, POST_SECONDARY 8.4%}. Code 87 (no qualification data) stays unmapped to the UPPER_SECONDARY default.
2. Disability flag shares vs the frozen reference — is_disabled/enhanced/severe unweighted shares run above the 145-column reference. Verified NOT a port defect by the head-to-head receipt: all five categories and all three flags agree row-for-row at 100.0000% with the incumbent's own create_frs at the pinned revision on the same tabs (after the receipt caught and ded274d4 fixed a one-row rate-tree divergence — the incumbent's baseline-vs-plain parameter split is value-bearing: the baseline clone escapes policyengine-uk's fiscal-year conversion, so categories threshold at April-2022-era rates while flags use fiscal-2023-24 values; both readers now reproduce their respective incumbent paths exactly). The residual deltas against the frozen reference are purely the artifact's vintage (rev 655dd07e predates the recent disability-logic fixes) plus composition (the SPI-synthetic/donor stack, ~37% of persons, which E7 adds). Re-measured at matching composition at E10.
3. Council tax per nation (uk-data#448 not silently replicated): our 2023-level totals England £40.34bn / Scotland £3.29bn / Wales £2.32bn vs the frozen incumbent's £44.40bn / £4.36bn / £2.57bn. England/Wales deltas (≈ −9/−10%) are the incumbent's uprating to the calibration year; Scotland's −24.6% additionally reflects the Scottish Water netting — our total sits at actual billed levels while the frozen artifact carries the overstatement. The durable devolved fix remains WS-C/D.
4. JSA hours threshold expressed as 16 × WEEKS_IN_YEAR on E2's signed hours scale (person-set provably identical to the incumbent's 832 cutoff).
5. EMA annualized at WEEKS_IN_YEAR vs the incumbent's bare ×52 in fill_with_mean — the same U8 class as E2's three sites. The rest of fill_with_mean is preserved exactly (participation-gated donor-mean fill, incumbent column pairs with the adema→eduma alias): at the 2023-24 vintage the fill is provably inert — 11 adult / 24 child participants, zero sentinel-negative amounts — so the multiplier is the only live difference.
6. Pre-2025 DSA capacity built as an aligned zero vector without the incumbent's shape-sizing engine read (inert; value-identical).
7. Grant split always runs (incumbent skips when no positive grants exist — inert at 2023-24) and runs pre-BRMA (capacity variables verified brma-free).
8. Dead code not replicated: ct_mean.replace(-1, …) (unreachable by construction).
9. Employment code 11 → LONG_TERM_DISABLED preserved (incumbent artifact, documented).
10. receives_benefits_in_own_right produced in frs_education — one-producer placement settled on #145 (2026-08-17): E3 is the canonical producer; E7 has no rewrite obligation (the incumbent never re-derives the flag on SPI donors).

Licensed-data acceptance (all data stayed local; disclosure-safe aggregates only)

• Determinism: two independent full builds from the pinned 2023-24 tabs; compare_uk_h5_payload.py verdict payload_identical: true. Both attempts landed hash-chained Logbook rows (pipeline uk-frs-spine).
• Nonzero shares vs the committed 145-column reference — every E3 parity column within ±0.002 except the three disability flags (signed, above): council_tax 0.8837 vs 0.8839; employment_status, categories, current/highest education, entry age 1.0000 exact; is_in_non_advanced_education +0.0006; QYP terminal +0.0000; adult/child EMA ±0.0000; receives_benefits_in_own_right +0.0007; education_grants +0.0000. Off-reference outputs (sector 1.0, SIC 0.7314, proxies 0.0096/0.0765/0.0729, DSA expenses 0.0) are candidate-only extras already in both export allowlists or reference-documented zero layers.
• Weighted totals vs the licensed register (context, not a stage gate: the register reflects the final uprated, calibrated artifact while this spine is 2023-level design weights): council_tax −10.5% (≈ two years' uprating), education_grants +49.8% (dominated by the corrected EDUCQUAL map feeding the DSA/maintenance eligibility formulas), adult/child EMA −54%/−17% — examined, not attributed to the port: the columns have 11 and 24 unweighted carriers respectively (EMA survives only in the devolved nations), zero sentinel amounts mean the fill contributes nothing by construction, and the register totals ride on the calibrated, uprated weights of those same few rows.
• Incumbent-code cross-checks on identical inputs: flags exact-share match, categories 100% row agreement (see signed difference 2).

Implementation notes

Codex implemented under /codex-implement-plan; recorded deviations, both reviewed as more faithful than the plan: baseline DWP rates read via CountryTaxBenefitSystem().parameters(year).baseline.gov.dwp (the raw parameter files carry no .baseline subtree — this is the incumbent's exact call shape), and the pre-2025 DSA policy is inert-zero because the 2023 DSA maximum resolves to None (never consumed in the incumbent either). Post-rebase repairs after #691/#692 merged to main: the four gates/contract-pin conflicts re-expressed on main's refactored test structure, and the six stage rebuilds now pass household_weights= explicitly per #691's frame contract.

🤖 Generated with Claude Code

[vahid-ahmadi]

I checked this the same way as #692 — every derivation line-by-line against the incumbent's code — and the port discipline holds almost everywhere. Verified faithful: the disability stage exactly (threshold max(0, rate − 1), ascending overwrite, and the flag operator asymmetry — dla_sc > for enhanced vs dla_sc >= for severe — preserved and pinned, with the parameters(year).baseline.gov.dwp call shape matching the incumbent's _dwp_category_threshold_parameters and your empirical cross-check settling the rest); the council-tax cell means exactly (raw ctannual > 0 donor filter over netted values, dropna=False, no-donor cells → 0, missing = <0 | isna, and the dead ct_mean.replace(-1, …) correctly not replicated — netted donor values are clipped ≥ 0, so a −1 mean is unconstructible); the proxies exactly (the JSA scale-invariance argument is correct: tothours·k < 16·k is the same person set for any k, so signed difference 4 is genuinely inert); the grant allocator near-verbatim against the incumbent's own allocate_reported_education_grants; and the np.select education cascade preserving the incumbent's elif ordering including the pinned-dead POST_SECONDARY branch. Both main-repair commits are right — I confirmed fiscal_year against the incumbent's own UKSingleYearDataset(..., fiscal_year=year) call, and the household_weight drop is the correct resolution of the #691×#692 seam. The driver generalization is clean, and the cross-stage artifact-pin consistency check is a nice fail-closed touch. All 91 tests in this PR's files pass for me with the uk extra.

The EDUCQUAL correction is verified and endorsed. The incumbent's map is the older codeframe misapplied: recent FRS EDUCQUAL is the harmonised highest-qualification frame that starts at 1 = Doctorate and descends, and your decisive checks (code 1 at 1.8% of adults with top mean earnings; 18/19 near-empty baccalaureates; the mass in the 36–82 GCSE band) are exactly the right kind of evidence — this is the #692 lesson applied in the harder direction, catching the incumbent's coded-domain error rather than inheriting it. Two follow-ons: (1) this bug ships in production policyengine-uk-data today — highest_education feeds live models, so it deserves an upstream issue with your aggregates as the evidence, the same way uk-data#448/#449 flowed the other way; (2) the manifest's "swappable constant pending licensed UKDS data-dictionary verification" note is honest — close it by actually citing the 2023-24 data dictionary when you have it open, which also satisfies the checklist rule we adopted after #692.

One thing I'd act on before merge: the EMA port doesn't do what the PR and the manifest say it does.

The incumbent's construction is fill_with_mean(person, "adema", "ademaamt") (child: chema/chemaamt):

needs_fill = (table[code] == 1) & (table[amount] < 0)   # participants with sentinel amounts
fill_mean = table[amount][(table[code] == 1) & (table[amount] >= 0)].mean()
filled_values = np.where(needs_fill, fill_mean, table[amount])
return np.maximum(filled_values, 0) * multiplier

_ema in frs_education.py reads no participation flag at all and computes max(amount, 0) × WEEKS_IN_YEAR, with _num's .fillna(0) upstream — so a participant coded flag=1, amount=−1 gets £0 where the incumbent gives the participant mean. The mean-fill doesn't exist anywhere in the stage. Yet the PR body calls this "EMA fill-with-mean (the impute_cell_means degenerate single-cell case)" and the manifest's operation scope says "EMA cell-mean degenerate fills" (source_stages.json:468) — declared behavior with no implementation behind it, which is exactly the class this repo's own receipts culture exists to prevent. Signed difference 5 covers only the ×52 → WEEKS_IN_YEAR multiplier and reads as if the rest of fill_with_mean were preserved.

Two adjacent problems in the same function:

• Column names. Adult tries ("emaamt", "edumaamt"); the incumbent reads ademaamt, falling back to edumaamt only when adema is absent. "emaamt" looks like a typo for "ademaamt" — if the 2023-24 adult tab carries ademaamt, your first choice never hits and the code silently reads the legacy-era column. The child fallback ("chemaamt", "eduma") names the incumbent's participation flag as an amount — wrong-typed if ever exercised.
• The acceptance can't exonerate this. Adult EMA runs −54% against the register, attributed to "tiny columns" — but the missing mean-fill and a possibly-wrong amount column are unexamined candidate contributors to that deficit. The nonzero-share match at ±0.0000 is four-decimal rounding on a column whose true share is ~0.0003, so it has no power here.

The fix is small either way: port fill_with_mean's semantics with the incumbent's column pairs, or — if the survey genuinely has no fillable rows at this vintage — sign the actual difference (no flag gate, no mean-fill, sentinel→0) and correct the manifest text and PR body to match. What can't stand is the declaration of a fill that isn't there.

Smaller

• Unmapped empstati codes above 11 fall to "CHILD" in your map default; the incumbent's post-map .fillna("LONG_TERM_DISABLED") sends them there instead. Inert if the domain stops at 11, but the truncated-zip artifact you carefully preserved for code 11 shows the domain has surprised people before — one line makes the fallback match.
• derive_frs_employment's docstring says missing mjobsect/sic "fail loudly," and they do — but empstati goes through aligned.get(...), which on a missing column fails as an AttributeError on a float rather than a KeyError naming the column. If empstati's absence should also be loud, direct-index it like its siblings.

The disability signed difference is handled exactly right — running the incumbent's own functions on your spine and getting bit-identical shares is the strongest possible form of "not a port defect," and deferring the composition question to E10 with the reference-vintage explanation on record is the honest disposition.

[juaristi22]

@vahid-ahmadi — all four items addressed in e1f2290; you were right on every count, and the EMA finding was the real thing: declared behavior with no implementation behind it.

EMA. _ema now ports fill_with_mean exactly: participation gate (code == 1), donor mean over participants' non-negative amounts, sentinel rows filled, floor at zero — with the incumbent's column pairs and its alias conditional (adema/ademaamt preferred, eduma/edumaamt when adema is absent; chema/chemaamt for children). Verified against the 2023-24 tab headers: this vintage carries the eduma pair, so the old code's emaamt typo had been silently landing on the right amount column by fallback — with no gate or fill, as you said. The manifest's impute_cell_means entry now declares the true semantics (cells/donor_filter/missing/value) instead of the aspirational text, and the fixture test exercises a sentinel fill plus a dedicated adema-preference test.

On your "the acceptance can't exonerate this" point — it can now, with the fill implemented: the licensed rebuild is share- and payload-identical, and the raw tabs show 11 adult participants and 24 child participants, zero sentinel-negative amounts at this vintage, so the fill is provably inert here and contributes nothing to the register deltas. The −54%/−17% weighted deltas are small-cell sensitivity: those columns have 11/24 unweighted carriers, and the register totals ride on the calibrated, uprated weights of those same few rows. Signed difference 5 and the acceptance section now say exactly this.

empstati fallback. Now the incumbent's post-map fillna(LONG_TERM_DISABLED) for beyond-domain codes (0/NaN still land on CHILD via the map entry). The 2023-24 domain is 1–11 empirically, so it's inert on data — pinned by a code-12 test anyway, per your point that this domain has surprised people before.

empstati loudness. Direct-indexed like its siblings; the parametrized missing-column refusal test now covers all three.

CI. The wheels failure (and the us-extra test lanes) was the Jan-1 instant-pin test constructing the real parameter tree in an engine-less venv — now importorskip-guarded; verified skipping in a locally rebuilt wheel venv.

EDUCQUAL follow-ons. Upstream defect filed with the aggregates as evidence: PolicyEngine/policyengine-uk-data#459. The manifest note now records the completed raw-aggregate verification rather than "pending"; the data-dictionary citation itself will be added when the licensed 2023-24 documentation is staged locally — we don't have it to cite honestly today.

[juaristi22]

Derived-layer receipt: every E3 column compared head-to-head against the incumbent's own code on the same tabs — the analogue of E2's raw-fidelity receipt, with one deliberate method change: instead of the published base artifact, the comparator is the incumbent's create_frs run locally at the pinned revision (ebf733c) over the same 2023-24 tabs — the exact code E3 ports. (E2's receipt showed the published base predates its own repo's mappings; the derived layer is precisely where the incumbent has churned — #417, #433, the EDUCQUAL frame — so a stale artifact would re-import the frozen-reference problem.) All licensed data stayed local; this comment carries counts, agreement rates, and weighted aggregates only.

Axis	Incumbent (ebf733c)	E3 spine	Delta
Persons / households	36,248 / 16,754	36,248 / 16,754	0 / 0, identical id sets
Household weight total	28,782,960	28,782,960	±0
Row-for-row agreement, 24 of 26 columns	—	—	100.0000%
council_tax (wsum)	£45,951.3m	£45,951.3m	exact, every row
education_grants (wsum, engine-materialized split)	£3,133.3m	£3,133.3m	exact, every row
adult_ema / child_ema	—	—	ratio exactly WEEKS_IN_YEAR/52 = 1.003434 on every carrier (11/24 rows); 100% agreement after rescaling — signed difference 5, nothing else
highest_education	TERTIARY 9.9%	TERTIARY 28.1%	38.3% raw agreement — the dictionary-verified EDUCQUAL correction (signed difference 1), by construction

The 100% set includes every category and flag, employment_status/sector/SIC, current_education, all QYP columns, receives_benefits_in_own_right, all three legacy proxies, and the DSA expenses zero layer. The grant split agreeing to the penny also rules out engine-version skew (incumbent ran policyengine-uk 2.89.2, ours 2.89.0).

The receipt caught a real one-row port bug, fixed in ded274d4. The first run had the two higher disability flags at 99.9972% — exactly one person — with all eleven input columns bit-identical. Root cause: the incumbent's baseline-vs-plain parameter split is value-bearing, not just provenance. The baseline clone is created before policyengine-uk's fiscal-year parameter conversion, so the incumbent's category thresholds use April-2022-era weekly rates (92.40) while its flag thresholds use the fiscal-2023-24 values (101.75) — two different rate vintages inside one incumbent function pair. Our flag reader had read the raw dated files at January 1 (92.40); one person sits between the two thresholds. The reader now reads the fiscal-converted plain tree through the model system exactly as the incumbent does, an engine-gated test pins the value-bearing split (92.40 vs 101.75 at 2023), and the re-run receipt above shows both flags at 100.0000%. Signed difference 2's framing is updated accordingly: the port now matches the incumbent bit-for-bit at matching composition; the residual deltas against the frozen 145-column reference are purely the reference artifact's vintage plus the SPI-stack composition, re-measured at E10.

Method note: comparison over the common person/household ids with weights mapped through the household linkage; the incumbent dataset was built in its own pinned checkout and venv, output kept beside the other licensed acceptance artifacts; determinism pair for the E3 spine re-verified payload_identical: true on the same run.