fix(data-preparation): a missing factor value no longer makes a multi-factor study fail

[cnicholas]

Summary

• What: rows with no value in any factor column leave the analysis before the composite subgroup label is built, for one factor or many, and the library says so with a ProcessBehaviorWarning naming the count per factor.
• Why: with one factor such rows were dropped silently by the later rsg dropna. With two or more, _add_composite_column validated for missing values before that drop and raised ValidationError: Cannot build RSG ... missing values in factor columns. The same file formulated with either factor alone and failed with both. Found on a real survey file: 85,101 rows, 7,980 with a blank SURVEY QUESTION, formulated with factors=["SURVEY QUESTION", "HOSPITAL"].
• Scope: one helper in data_preparation.py called before _add_grouping_column; a new test file; CHANGELOG.

Contract / invariants

• Defaults unchanged — files with complete factor columns produce byte-identical results and no warning (pinned). validation/e2e_bishop_report.py exits 0.
• Residuals unaffected — computed on the same rows as before for single-factor studies; multi-factor studies with blanks previously did not formulate at all.
• Row/index alignment preserved — obs_id is stamped before the drop, so lineage to source rows holds.
• Output schema compatible.
• Pinned error strings unchanged — the composite check in _add_composite_column stays as a safety net with its message intact.
• ODS detection on raw data is unchanged; the detector already filtered NA factor rows (test_na_in_factor_is_filtered).

Behaviour changes

• formulate() with two or more factors and a blank factor value now succeeds, dropping those rows, instead of raising.
• One ProcessBehaviorWarning per formulate() when any factor value is missing, e.g. Dropped 7,980 of 85,101 rows with no value in a factor column: SURVEY QUESTION (7,980 missing). A row with no factor value belongs to no cell, so it is excluded from the analysis. Only columns that actually have gaps are named.

Methodology

• No methodology change — a row that names no cell cannot be in the grid; this makes the multi-factor path do what the single-factor path and the detector already did.

Tests

tests/test_na_in_factor_columns.py:

• test_two_factors_formulate_with_missing_factor_values — formulates, drops exactly the blank rows, K = 20 with no phantom level.

• test_one_factor_and_two_factors_drop_the_same_rows.

• test_the_drop_is_reported_with_counts — the warning text, per-column counts, and that a complete column is not named.

• test_no_warning_when_factor_columns_are_complete.

• test_missing_values_in_both_factors_are_named_per_column.

• pytest tests/ — 2365 passed, 10 skipped

• ruff check . — clean

• mypy processbehavior — 17 pre-existing errors, none new

• Golden masters untouched

Notes

• The app's companion issue: its "Formulation failed" message is drawn only on the run in which the click happened, so this error was invisible to the analyst who hit it. Fixed separately in processbehavior-app.
• On the survey file: design state 1, K = 20, T = 12, 77,121 rows after the drop, Xbar/S execute.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BKphCi1LC1hjGYteWEK9JV