From 2016f1756adaf516101a168049be35e94637090a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 20 Aug 2026 22:01:15 -0700 Subject: [PATCH 1/4] docs(release-log): log the never-given vocabulary move #360 landed 6389f06 moved `mc` and `ste` into the never-given half of the particle vocabulary and added `los`, `las` and `das`, which were missing from it entirely. It is on master, in no release tag, and had no Behavior Changes bullet -- so a user-visible change was heading for 2.2 with nothing in the changelog. Five name shapes move: `Mc Donald`, `Ste Marie`, `Los Santos`, `Las Casas` and `Das Silva` each go from a given-plus-family reading to a single folded family name, and the PARTICLE_OR_GIVEN ambiguity they used to report goes with them. Found by the rules-doc corpus (#414): `Mc Donald` is a rules.md#P1 example, and classifying its diff meant finding out which change had moved it. Co-Authored-By: Claude Opus 5 --- docs/release_log.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/release_log.rst b/docs/release_log.rst index de73d8e6..ce02f43c 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -44,6 +44,8 @@ Release Log - Fix a maiden name being lost when a particle stood in front of the marker. ``"Ursula Leyen geb. Albrecht"`` reported maiden ``Albrecht`` correctly, but ``"Ursula von der Leyen geb. Albrecht"`` -- the same words one particle chain apart -- gave family ``von der Leyen geb. Albrecht`` and no maiden name at all, and ``"Jane van der Berg née Jones"`` failed the same way. The particle chain absorbed everything up to the next particle or suffix, and the marker is consumed later in the same grouping stage, by which point the chain had merged the marker away and there was no marker word left standing to find. A suffix already stopped the chain; a marker now does too, so those read family ``von der Leyen`` maiden ``Albrecht`` and family ``van der Berg`` maiden ``Jones``. Only a particle that is not the name's own first word ever reached the marker, which is why a single leading particle always worked -- ``"von Müller geb. Schmidt"`` was already right -- while a leading run of two did not: in ``"von der Müller geb. Schmidt"`` the second particle starts a chain of its own. The bound applies only where the marker is actually taken as a marker. A marker needs a word after it to hand the maiden name to, so a trailing one is just a word and bounds nothing: ``"Jane van der Berg née"`` and ``"Jane van der Berg née PhD"`` both keep family ``van der Berg née``, which is the reading ``"Jones née"`` -> family ``née`` already had. That distinction is what keeps a name like ``"Jane van der Nee"`` intact -- ``Nee`` is an attested surname as well as a marker spelling -- and what keeps ``"St St née"`` reporting its ``PARTICLE_OR_GIVEN`` fork. Two consequences fall out. Under a family-first order the marker used to survive as an ordinary name word and take the leftover given slot, so ``"de la Cruz née Vega"`` read given ``née``, middle ``Vega`` under ``Policy(name_order=FAMILY_FIRST)`` and given ``Vega``, middle ``née`` under ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``; consumed and dropped it never reaches that placement, and both orders now read family ``de la Cruz``, maiden ``Vega``. And stopping the chain can leave a family of nothing but particles, which the #385 rule above already covers: ``"Jane de la née Jones"`` reports family ``de la`` with ``family_base`` ``de la`` rather than an empty base. Two limits remain, and both are recorded in ``rules.md#M2``: the bound reaches only a marker standing as a word of its own, so a conjunction join and a bound given-name join each still absorb one first -- ``"Jane van der Berg née y Jones"`` and ``"van der Berg, abdul née Jones"`` are unchanged. The bracketed form was never affected, since ``"Ursula von der Leyen (geb. Albrecht)"`` is extracted before grouping. No differential corpus name carries a particle ahead of a marker, so this class is invisible to the harness at every baseline and the tests are the whole verification; `#413 `_ tracks the corpus gap (closes #399) + - Move ``mc`` and ``ste`` into the never-given half of the particle vocabulary, and add the Spanish and Portuguese articles ``los``, ``las`` and ``das``, which were absent from it entirely. A never-given particle opening a name folds into the family (``rules.md#P1``) instead of being read as a given name, so ``"Mc Donald"`` was first ``Mc``, last ``Donald`` and is now last ``Mc Donald``; ``"Ste Marie"``, ``"Los Santos"``, ``"Las Casas"`` and ``"Das Silva"`` move the same way, and ``"Mc Donald Smith"`` goes from first ``Mc``, middle ``Donald``, last ``Smith`` to last ``Mc Donald Smith``. The ``PARTICLE_OR_GIVEN`` ambiguity these names used to report goes with it: once the word can never be a given name there is no fork left to report. Names where the particle is not leading are untouched -- ``"Jack Mc Donald"`` already read last ``Mc Donald`` and is byte-identical. Never-given membership goes from 28 words to 33. No differential corpus name moved when this landed; ``"Mc Donald"`` entered the corpora afterwards with ``corpus_rules.jsonl`` and is now the one name that pins it (#360) + **Deprecations** - Rename the four vocabularies whose 1.x names described the fields they feed in v1's words, so the data layer matches the ``Lexicon``: From 7a10689ec07a6e5c96db51c0fc4d055fb4bd030e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 20 Aug 2026 22:01:32 -0700 Subject: [PATCH 2/4] feat(differential): a fourth corpus from the rules doc's own examples MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/design/rules.md's 156 example names fed no differential corpus. 113 of them were in no corpus at all, and 12 had moved during the 2.2 cycle with nothing observing it. The doc tests are not already this. They pin each example against an expectation stored beside it, so a deliberate behaviour change edits both in one commit and the suite stays green -- a doc example cannot warn about the change that edited it. PR #409 is the worked example: it changed M2's "Jane de la née Jones" from family 'de la née Jones' to family 'de la', the doc test followed the edit, and nothing else observed the movement. Parsing the same name against a RELEASED baseline can, because no commit edits 1.4.0. build_rules_corpus.py selects through the doc's own parser rather than a second regex, the way build_cjk_corpus.py selects through the shipped script table, and test_ledger_guards.py pins the checked-in file against the generator. Classifying the 12 is most of this diff, and it found four things: - An unlogged behaviour change, committed separately: #360's never-given vocabulary move, which is what moves `Mc Donald`. - Two ledger rules broad enough to absorb a regression, both harmless until a name of the right shape existed. fix(#274) matched a PARENTHESIZED `Nee` -- a nickname under the default facade the differential parses with, where no marker is active at all -- and stood ready to explain a maiden diff on a name with no marker in it. The acronym rule's `m\.?a\.?` reached the real name `John Smith M.A.`. The vocabulary-presence and member-reach guards caught both; they are the absorption detectors and this is the first corpus that gave them something to detect. - Two arrivals the ledger predicted in writing. fix(#379) said a vd-shaped name "should arrive unexplained and be read once"; fix(#367) named `Sir de Mesnil` as a future arrival. Both arrived, and both are now their own rules -- separate rather than widened literals, because the guards allow one vocabulary alternation per rule and widening both a title and a particle slot needs two. - Two pre-existing behaviours that had never had a corpus name: a nakaguro division meeting a glued hangul honorific, and two typographic quote spans read as one nickname set. All three baselines exit 0 over 864 names. Co-Authored-By: Claude Opus 5 --- tests/v2/test_ledger_guards.py | 104 ++++++++++--- tools/differential/README.md | 21 +++ tools/differential/build_rules_corpus.py | 94 +++++++++++ tools/differential/compare.py | 1 + tools/differential/corpus_rules.jsonl | 156 +++++++++++++++++++ tools/differential/expected_since_1.4.0.toml | 69 +++++++- tools/differential/expected_since_2.0.0.toml | 72 +++++++++ tools/differential/expected_since_2.1.0.toml | 132 ++++++++++++++++ 8 files changed, 629 insertions(+), 20 deletions(-) create mode 100644 tools/differential/build_rules_corpus.py create mode 100644 tools/differential/corpus_rules.jsonl diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index e610e803..1c08c033 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -519,6 +519,29 @@ def test_cjk_corpus_matches_the_case_table() -> None: "`uv run python tools/differential/build_cjk_corpus.py`") +def test_rules_corpus_matches_the_rules_doc() -> None: + """corpus_rules.jsonl is GENERATED from docs/design/rules.md's own + examples (#414), through the doc's parser rather than a second + regex. Same promise as the CJK pin above: an example added to a + rule without regenerating fails HERE, instead of leaving the + normative names outside the differential gate. + + Why the doc's examples need the gate at all, when test_rules_doc + already executes every one of them: those pin against an + expectation stored beside them, so a deliberate behaviour change + edits both in one commit and the test stays green. The corpus + compares against a RELEASED baseline no commit can edit, which is + what forces a moved name to be classified in writing. + """ + module = load_tool("build_rules_corpus") + checked_in = [json.loads(line) for line in + (_TOOLS / "corpus_rules.jsonl") + .read_text(encoding="utf-8").splitlines()] + assert checked_in == module.selected_names(), ( + "corpus_rules.jsonl is stale: regenerate with " + "`uv run python tools/differential/build_rules_corpus.py`") + + #: Which vocabulary constant each ledger rule's alternation is a hand #: copy of. A roster rather than an inference: GLUED_HONORIFICS is a #: SUBSET of SUFFIX_WORDS (asserted at the bottom of @@ -690,6 +713,18 @@ class _LatinCopy(NamedTuple): covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", "du", "la", "le", "los", "mc", "van", "vd", "von", "zu"})), + # The same partial copy as fix(#379), and partial for the same + # reason -- these are the words that actually open a chain in a + # Latin name. The two rules ask different questions of it (#379 + # wants the word that ENDS a comma listing, #399 the word that + # opens a chain running into a maiden marker), so the shared + # snapshot is agreement rather than duplication: a divergence + # would be a fact about one of the rules. + "fix(#399)": _LatinCopy( + vocabulary=PARTICLES, + covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", + "du", "la", "le", "los", "mc", "van", "vd", + "von", "zu"})), } #: Alternations that copy no vocabulary, so discovery must not demand a @@ -698,6 +733,11 @@ class _LatinCopy(NamedTuple): #: question someone answers in writing, not something to skip past. _NOT_A_VOCABULARY_COPY = frozenset({ frozenset({"^", " "}), # the honorific rule's leading anchor + # fix(#400)'s two openings: start-of-name or just after a family + # comma. `abd` joins forward on the given side wherever that side + # begins, and the alternation is over ANCHORS, not over words -- + # there is no vocabulary here to drift from. + frozenset({"^", ",\\s*"}), }) def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: @@ -1022,19 +1062,19 @@ def _claim(rule: dict) -> _Claim: _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('family', 'given', 'middle'), "66e71d60a075"), + _Claim(105, ('family', 'given', 'middle'), "090ac3eef3fb"), "fix(#274) maiden markers consumed": - _Claim(4, ('family', 'maiden', 'middle'), "b31dc2e2bbc4"), + _Claim(16, ('family', 'maiden', 'middle'), "cc672349b500"), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + _Claim(4, ('family', 'given', 'maiden', 'middle'), "27bb9ebc1951"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(2, ('family', 'middle'), "f3a43bebfb91"), + _Claim(8, ('family', 'middle'), "f4d52b062f39"), "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(215, ('given', 'suffix', 'title'), "f16a0e79cba3"), + _Claim(236, ('given', 'suffix', 'title'), "9393109ebf99"), "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(215, ('family', 'given'), "f16a0e79cba3"), + _Claim(236, ('family', 'given'), "9393109ebf99"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(751, ('family', 'given', 'suffix'), "231640fc7535"), + _Claim(864, ('family', 'given', 'suffix'), "7127402a5078"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1042,49 +1082,77 @@ def _claim(rule: dict) -> _Claim: "feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)": _Claim(2, ('family', 'middle'), "3e2b5c6d1f4d"), "feat(#273) typographic nickname delimiters recognized by default": - _Claim(6, ('middle', 'nickname'), "a03c9763c8c4"), + _Claim(8, ('middle', 'nickname'), "968bd4162257"), "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJK order flip": _Claim(6, ('family', 'given', 'nickname'), "ae1dffa01608"), "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7"), "fix(cjk-comma-honorific-peel) glued honorific peels off a post-comma given name": - _Claim(20, ('given', 'suffix'), "b2ea8fa59eea"), + _Claim(21, ('given', 'suffix'), "fdc02562bd15"), "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": - _Claim(20, ('family', 'given', 'middle', 'suffix', 'title'), "b2ea8fa59eea"), + _Claim(21, ('family', 'given', 'middle', 'suffix', 'title'), "fdc02562bd15"), "fix(cjk-glued-honorific-peel) glued honorific peels into suffix": - _Claim(34, ('family', 'given', 'suffix'), "877ab3246d33"), + _Claim(35, ('family', 'given', 'suffix'), "4da08a0090fe"), "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": _Claim(19, ('family', 'given', 'middle', 'suffix'), "aa475ddd4745"), "feat(#269) non-Latin titles/conjunctions recognized": - _Claim(2, ('given', 'middle', 'title'), "c14187bb08f8"), + _Claim(4, ('given', 'middle', 'title'), "e86eeb13eeb2"), "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": _Claim(1, ('given', 'middle', 'suffix', 'title'), "390e7f814d13"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), + "fix(#400) abd joins the word after it as one given name": + _Claim(3, ('given', 'middle'), "6eb6b807523a"), + "fix(#272/#308) nakaguro division and a glued hangul honorific in one name": + _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122"), + "fix(nickname-typographic-pairs) two typographic quote spans read as one nickname set": + _Claim(1, ('family', 'given', 'middle', 'nickname'), "3cf566c78800"), }, "expected_since_2.0.0.toml": { "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(2, ('family', 'middle'), "f3a43bebfb91"), + _Claim(8, ('family', 'middle'), "f4d52b062f39"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('_ambiguities', 'family', 'given', 'middle'), "66e71d60a075"), + _Claim(105, ('_ambiguities', 'family', 'given', 'middle'), "090ac3eef3fb"), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": - _Claim(34, ('family', 'given', 'suffix'), "877ab3246d33"), + _Claim(35, ('family', 'given', 'suffix'), "4da08a0090fe"), "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2"), "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + _Claim(4, ('family', 'given', 'maiden', 'middle'), "27bb9ebc1951"), "fix(#272) nakaguro inside delimited content renders as a space, compounding with the CJK order flip": _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), + "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": + _Claim(1, ('family', 'suffix'), "081ce07f927b"), + "fix(#399) a maiden marker bounds the particle chain that swallowed it": + _Claim(6, ('family', 'maiden'), "b40129435671"), + "fix(#360) mc moved into the never-given particles, so it folds into the family": + _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), + "fix(#400) abd joins the word after it as one given name": + _Claim(3, ('given', 'middle'), "6eb6b807523a"), + "fix(#367) a title no longer displaces a leading never-given particle": + _Claim(1, ('family', 'given'), "db724fb9c779"), + "fix(#272/#308) nakaguro division and a glued hangul honorific in one name": + _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122"), }, "expected_since_2.1.0.toml": { "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(2, ('family', 'middle'), "f3a43bebfb91"), + _Claim(8, ('family', 'middle'), "f4d52b062f39"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), + "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": + _Claim(1, ('family', 'suffix'), "081ce07f927b"), + "fix(#399) a maiden marker bounds the particle chain that swallowed it": + _Claim(6, ('family', 'maiden'), "b40129435671"), + "fix(#360) mc moved into the never-given particles, so it folds into the family": + _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), + "fix(#400) abd joins the word after it as one given name": + _Claim(3, ('given', 'middle'), "6eb6b807523a"), + "fix(#367) a title no longer displaces a leading never-given particle": + _Claim(1, ('family', 'given'), "db724fb9c779"), }, } @@ -1313,7 +1381,7 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(34, "9ea55c4c4382", ()), + _Excluded(42, "e594ae1a2e73", ()), } diff --git a/tools/differential/README.md b/tools/differential/README.md index b4108b59..917e95a6 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -184,6 +184,7 @@ that stops being run. Pass `--corpus PATH` (repeatable) to narrow it. | `corpus.jsonl` | v1's own test suite at a pinned ref | anything 2.0 added — v1's authors had no reason to test a typographic nickname delimiter or a Cyrillic title | | `corpus_issues.jsonl` | name-like strings harvested from the GitHub issue tracker | anything nobody ever reported | | `corpus_cjk.jsonl` | the CJK-bearing rows of `tests/v2/cases.py`, via `build_cjk_corpus.py` (#295) | anything the case table itself missed — it re-witnesses reviewed expectations at the baseline boundary rather than discovering new shapes | +| `corpus_rules.jsonl` | every example in `docs/design/rules.md`, via `build_rules_corpus.py` (#414) | anything the rules doc has no example for — it re-witnesses the normative examples at the baseline boundary | They are deliberately separate rather than merged: `corpus.jsonl` is reproducible forever from an immutable git ref, while the issue @@ -196,12 +197,32 @@ table, and `tests/v2/test_ledger_guards.py` pins the checked-in file against the generator's selection, so a CJK case row added without regenerating fails the suite instead of silently narrowing this gate. +The rules corpus answers a different question from the doc tests that +already execute those examples. Those pin an example against an +expectation stored beside it, so a deliberate behavior change edits +both in one commit and the test stays green — a doc example cannot +warn about the change that edited it. Parsing the same name with a +*released* baseline can, because no commit can edit 1.4.0. It is +generated and pinned the same way the CJK corpus is. + The issue corpus earned its place on the first run — 166 of its 198 names were not in `corpus.jsonl`, and it immediately surfaced five intended-but-unclassified 2.0 behaviors (#273 typographic delimiters, #269 non-Latin vocabulary) plus one shape no test had considered: a **leading** `"Ph. D."`, which v1 split into title `Ph.` + given `D.`. +The rules corpus earned its place the same way — 113 of its 156 names +were in no other corpus, and on its first run 12 of them turned out to +have moved during the 2.2 cycle with nothing observing it. Classifying +them surfaced an unlogged behavior change (`mc` moving into the +never-given particles, which moves `Mc Donald`), two rules broad +enough to absorb a regression once a name of the right shape existed +(`fix(#274)`'s marker matching a *parenthesized* `Nee`, which is a +nickname under the default facade, and the acronym rule's dotted +`m\.?a\.?` reaching the real name `John Smith M.A.`), and two +pre-existing behaviors that had never had a corpus name to exercise +them. + ## Corpus provenance `corpus.jsonl` is checked in as a test fixture. It was built by diff --git a/tools/differential/build_rules_corpus.py b/tools/differential/build_rules_corpus.py new file mode 100644 index 00000000..1009db69 --- /dev/null +++ b/tools/differential/build_rules_corpus.py @@ -0,0 +1,94 @@ +"""Regenerate corpus_rules.jsonl from the examples in docs/design/rules.md. + +The fourth corpus, with the fourth provenance (#414). The other three +each have a blind spot the others cover, and this one closes a blind +spot none of them was built to have: + +- corpus.jsonl comes from v1's own test suite at a pinned git ref, so + it is structurally blind to anything 2.0 added. +- corpus_issues.jsonl comes from what users reported, which is the + adversarial half. +- corpus_cjk.jsonl comes from the CJK rows of the case table, because + neither of the above can carry an unspaced CJK name. +- this file comes from the NORMATIVE RULES -- the names the project + has written down as defining its behavior. + +Why the rules doc needs a corpus when its examples are already tests. +`test_rules_doc.py` executes every example, which pins them harder +than corpus membership pins anything -- but it pins them against an +expectation stored beside them. Change behavior deliberately and the +expected value is edited in the same commit; the test goes green and +says nothing. A doc example cannot warn about the change that edited +it. + +The differential run answers the other question. It parses these names +with a RELEASED baseline, which no commit can edit, so a moved name +arrives unexplained and has to be classified in a ledger in writing. +#409 is the worked example: it changed M2's + + "Jane de la née Jones" → family="de la née Jones" + +to family="de la". The doc test followed the edit, the name was in no +corpus, and nothing independent observed the movement. Twelve names in +this corpus moved during the 2.2 cycle the same way. + +Selection is every distinct example `text`, through the doc's own +parser rather than a second regex -- the same argument that has +build_cjk_corpus.py select through the shipped script table. A rule +gaining an example enters the corpus by being written down where it +already gets reviewed. + +Policy annotations are deliberately ignored. The corpus carries name +STRINGS and the differential run parses them with the default facade, +so a family-first-scoped example is simply one more name to diff -- +build_cjk_corpus.py makes the same call for its zh-scoped rows. + +Boundary strings ("(", ".,", "Anna () Smith") are kept, on the +over-collection principle the other three builders state: a name that +parses to nothing costs one parse and produces no diff. They are more +concentrated here than elsewhere because the doc argues its edges +explicitly, which is a reason to keep them rather than to drop them. + +Regenerate after editing rules.md examples: + + uv run python tools/differential/build_rules_corpus.py + +tests/v2/test_ledger_guards.py pins the checked-in file against this +module's selection, so a stale corpus fails the suite rather than +silently narrowing the differential gate. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[1] +sys.path.insert(0, str(ROOT)) + +from tests.v2.rules_doc import parse_rules_doc # noqa: E402 + +OUT = HERE / "corpus_rules.jsonl" +RULES_DOC = ROOT / "docs" / "design" / "rules.md" + + +def selected_names() -> list[str]: + """Every distinct example text in the rules doc, sorted for a + deterministic file.""" + rules = parse_rules_doc(RULES_DOC.read_text(encoding="utf-8")) + return sorted({example.text + for rule in rules + for example in rule.examples}) + + +def main() -> None: + names = selected_names() + with OUT.open("w", encoding="utf-8") as fh: + for name in names: + fh.write(json.dumps(name, ensure_ascii=False) + "\n") + print(f"wrote {len(names)} names to {OUT.name}") + + +if __name__ == "__main__": + main() diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 28f94380..4d650d2f 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -358,6 +358,7 @@ def _is_latin_only(name: str) -> bool: "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref "corpus_cjk.jsonl": 95, # 97 today, generated from the case table "corpus_issues.jsonl": 190, # 200 today, harvested and append-only + "corpus_rules.jsonl": 150, # 156 today, generated from rules.md } diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl new file mode 100644 index 00000000..bd79671b --- /dev/null +++ b/tools/differential/corpus_rules.jsonl @@ -0,0 +1,156 @@ +"" +"'Smitty' John Jones" +"'Smitty' Jones" +"'Smitty' Jones Jr." +"(" +".," +"Ali Ahmad Vali oglu" +"Ali Ahmad Vali oglu Jr." +"Ali Ahmad oglu" +"Andrew (Andy) Perkins" +"Andrew Perkins (Andy)" +"Andrew Perkins (MBA)" +"Anh Do" +"Anh Van Do" +"Anna () Smith" +"Asst. Vice Chancellor John Smith" +"Beethoven, Ludwig van" +"Berg, Jan van der" +"Berg, Jan vd" +"Del Toro" +"Dr. John van der Berg" +"Dr. Juan Q. Xavier de la Vega III" +"Dr. Smith, John" +"Esq. Smith" +"Gal·la Serra" +"Garcia Juan Carlos" +"Hans „Erster“ und “Zweiter” Müller" +"Hassan Mohamad Ali" +"Hassan, Mohamad Ahmad Ali" +"J. Smith" +"J.R. Smith" +"Jack Wei Ma" +"Jane Smith (Nee)" +"Jane Smith (Nee) (Jones)" +"Jane Smith (née Jones)" +"Jane Smith née Jones" +"Jane Smith née Jones PhD" +"Jane Smith, née Jones" +"Jane de la née Jones" +"Jane née Jones Smith" +"Jane van der Berg née" +"Jane van der Berg née Jones" +"Jane van der Berg née PhD" +"Jane van der Berg née y Jones" +"Jane „JD Smith" +"Jean 'JD' Smith" +"John . Smith" +"John Doctor Smith" +"John Ma" +"John Smith" +"John Smith J.u.n.i.o.r." +"John Smith Jr." +"John Smith M.A." +"John Smith PhD" +"John Smith Q.W.E.R.T." +"John Smith, LEED AP" +"John Smith, MD, Bart" +"John Smith, MD,, Jr." +"John Smith, PhD" +"John Smith, V." +"John van der Berg" +"John van der Berg PhD" +"John van der Berg Smith" +"John van der Berg née Jones" +"John😀Smith" +"Jones née" +"Jong Anke de" +"Jong, Anke de" +"Jose E Maria Santos" +"Juan & Garcia" +"Juan McDonald" +"Juan and Garcia" +"Juan de" +"Juan de la Vega" +"Juan de y" +"Juan van der" +"Juan y Eva Garcia" +"Juan y Garcia" +"Mari' Aube'" +"Marquess of Bath" +"Mary Beth Smith" +"Mc Donald" +"Mesnil de" +"Morse, Det. Insp. Jane" +"Mr. Jack Jill" +"Mr. Jack and Jill" +"Mr. Johnson" +"Mrs. Garcia" +"Nguyen Thi Van" +"Nguyen, Thi Van" +"Nguyen, Van" +"Nguyen, Van Le" +"Nguyễn Thị Minh Khai" +"Nguyễn, Thị Vân" +"Rev. John Smith" +"Salam, abd Allah" +"Sean O'Connor" +"Sidorov Ivan Petrovich Jr." +"Sir John" +"Sir de Mesnil" +"Smith Jr." +"Smith, Abd" +"Smith, John" +"Smith, Major. John" +"Smith, PhD" +"Smith, de Mesnil Juan" +"Van Johnson" +"Vega, Juan de la" +"Vincent van Gogh van Beethoven" +"Xyz. John Smith" +"Xyz. Smith, John" +"abd Allah" +"abd Allah Smith" +"abdul salam ahmed salem" +"de" +"de Mesnil Juan" +"de la Cruz Juan Carlos" +"de la Vega" +"de la Vega y Santos Juan" +"de los Santos" +"ibn Awf abdul Rahman" +"juan mcdonald" +"mohamad ali smith" +"née Jones" +"van Gogh" +"van der Berg, abdul née Jones" +"Иван Петрович Абрамович" +"Сидоров Иван Петрович" +"Хосе И Мария Сантос" +"سلمان، محمد" +"প্রফেসর. Sen" +"マイケル ジャクソン" +"マイケル・ジャクソン" +"威廉·莎士比亚" +"威廉・莎士比亚" +"安东尼·陈志明" +"山田 太郎" +"山田 花子 旧姓：佐藤" +"毛·泽东" +"毛泽东" +"王君" +"田中、太郎" +"田中さん" +"田中さん, V." +"马丁·路德·金씨" +"高橋 みなみ" +"高橋・一郎" +"高橋一郎" +"김 민준" +"김, 민준씨" +"김민준" +"김지양" +"남궁민수" +"남궁민수 지훈" +"남궁민수, 지훈" +"지훈, 남궁민수" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index ad8976f3..1af3409e 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -98,7 +98,18 @@ issue = "fix(#274) maiden markers consumed" # tests/v2/cases.py calls it the spelling English writes most often. # See maiden_markers.py's own "Deliberately absent" note for what has # actually been considered and declined. -name_regex = "(?i)\\b(n[ée]e|geb\\.?|roz\\.?)\\b" +# +# The paren lookarounds are not decoration. corpus_rules.jsonl (#414) +# brought in 'Jane Smith (Nee)' and 'Jane Smith (Nee) (Jones)', which +# are M1's examples of a one-word clause keeping a word that may +# itself be a surname. Under the DEFAULT facade -- which is what the +# differential parses with -- no maiden delimiter is configured, so +# the parenthesized word is a NICKNAME and no marker is active at all. +# Without the lookarounds this rule matched the word anyway and stood +# ready to explain a `maiden` diff on a name that has no marker in it, +# which is absorption waiting to happen. The vocabulary-presence guard +# in test_ledger_guards.py is what caught it. +name_regex = "(?i)(? Date: Thu, 20 Aug 2026 22:50:48 -0700 Subject: [PATCH 3/4] fix(differential): stop the new rules from claiming what they exempt Review found the first cut of #414 doing the thing this ledger exists to prevent, in two rules at once. fix(#399) reached six corpus names and explained three. The other three were exactly the shapes its own comment says it must not claim: the trailing-marker shape, the connective join and the post-comma bound-given join -- M2's accepted limits, two of them tracked at #412. The comment justified the reach with "the consumer needs a non-suffix word after the marker", which `\s+\S` does not express: PhD matches `\S` perfectly well. Anchored now to one word after the marker and no comma in the name, which takes it to four. fix(#274)'s reach went 4 -> 16 when the corpus grew, and nine of the sixteen do not move. Seven arrived with this PR, four of them M2 BOUNDARY examples -- 'Jones nee', 'nee Jones' -- which the vocabulary-presence guard cannot see, because those names really do carry marker vocabulary. It now tests what the rule has always meant: a marker with a name word before it and one after it. Reach 16 -> 11, and all seven names it explains are inside the eleven. Both rules still claim one static name each ('Jane van der Berg nee PhD', 'abd Allah'). Excluding either needs the regex to ask a vocabulary question, which is a second alternation the guards allow only one of per rule -- so they are named in the comments instead, the way fix(#379) names 'Nguyen, Van'. Two new guards, for holes the review demonstrated rather than argued: - _MUST_NOT_MATCH. A rule claiming exactly one corpus name can be widened with its claim count unmoved, because the names it newly reaches are outside the corpora. Six such widenings passed the whole suite: `^mc\s+\S+$` to `^mc`, the vd rule to a bare `\bvd\b`, and four more. The roster is a wall rather than a snapshot -- wrong at recording time too -- seeded from the names the rules' own comments argue about. It catches all six, plus reverting both narrowings. - A rule-count floor and a per-rule contribution check on the rules doc. The staleness pin compares the file against the generator's CURRENT output, so a generator that silently selects less agrees with a regenerated file and both are wrong together. _CORPUS_FLOORS bounded that to a few names; these bound the shapes it cannot see. Also from review: - The builder emitted "" as a name: `[subject]` examples carry no text, and rules.md's three collapsed into one empty string the doc never wrote. Filtered, 156 -> 155. - compare.py returned normally when a corpus file was DELETED -- the glob just found fewer files and the summary read like a full run. The floors already name every corpus that should exist, so main() asks them. - _CROSS_RULE_WINNERS gained the three names that land on the 1.4 catch-all with nothing recording it, and lost a header count that had gone stale twice over. - The #360 bullet claimed non-leading names are untouched. True for mc and ste, which only changed halves; los, las and das were not particles at all, so 'Maria das Neves' moves too -- 6389f06's own headline example. Same for the ambiguity sentence: three of the six names never reported a fork. - The #399 bullet's "no corpus name carries a particle ahead of a marker" stopped being true the moment this corpus landed. Co-Authored-By: Claude Opus 5 --- docs/release_log.rst | 4 +- tests/v2/_differential_fixtures.py | 2 +- tests/v2/test_ledger_guards.py | 145 ++++++++++++++++++- tools/differential/README.md | 2 +- tools/differential/build_rules_corpus.py | 22 ++- tools/differential/compare.py | 15 +- tools/differential/corpus_rules.jsonl | 1 - tools/differential/expected_since_1.4.0.toml | 31 +++- tools/differential/expected_since_2.0.0.toml | 17 ++- tools/differential/expected_since_2.1.0.toml | 33 ++++- 10 files changed, 241 insertions(+), 31 deletions(-) diff --git a/docs/release_log.rst b/docs/release_log.rst index ce02f43c..f952cce1 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -42,9 +42,9 @@ Release Log - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) - - Fix a maiden name being lost when a particle stood in front of the marker. ``"Ursula Leyen geb. Albrecht"`` reported maiden ``Albrecht`` correctly, but ``"Ursula von der Leyen geb. Albrecht"`` -- the same words one particle chain apart -- gave family ``von der Leyen geb. Albrecht`` and no maiden name at all, and ``"Jane van der Berg née Jones"`` failed the same way. The particle chain absorbed everything up to the next particle or suffix, and the marker is consumed later in the same grouping stage, by which point the chain had merged the marker away and there was no marker word left standing to find. A suffix already stopped the chain; a marker now does too, so those read family ``von der Leyen`` maiden ``Albrecht`` and family ``van der Berg`` maiden ``Jones``. Only a particle that is not the name's own first word ever reached the marker, which is why a single leading particle always worked -- ``"von Müller geb. Schmidt"`` was already right -- while a leading run of two did not: in ``"von der Müller geb. Schmidt"`` the second particle starts a chain of its own. The bound applies only where the marker is actually taken as a marker. A marker needs a word after it to hand the maiden name to, so a trailing one is just a word and bounds nothing: ``"Jane van der Berg née"`` and ``"Jane van der Berg née PhD"`` both keep family ``van der Berg née``, which is the reading ``"Jones née"`` -> family ``née`` already had. That distinction is what keeps a name like ``"Jane van der Nee"`` intact -- ``Nee`` is an attested surname as well as a marker spelling -- and what keeps ``"St St née"`` reporting its ``PARTICLE_OR_GIVEN`` fork. Two consequences fall out. Under a family-first order the marker used to survive as an ordinary name word and take the leftover given slot, so ``"de la Cruz née Vega"`` read given ``née``, middle ``Vega`` under ``Policy(name_order=FAMILY_FIRST)`` and given ``Vega``, middle ``née`` under ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``; consumed and dropped it never reaches that placement, and both orders now read family ``de la Cruz``, maiden ``Vega``. And stopping the chain can leave a family of nothing but particles, which the #385 rule above already covers: ``"Jane de la née Jones"`` reports family ``de la`` with ``family_base`` ``de la`` rather than an empty base. Two limits remain, and both are recorded in ``rules.md#M2``: the bound reaches only a marker standing as a word of its own, so a conjunction join and a bound given-name join each still absorb one first -- ``"Jane van der Berg née y Jones"`` and ``"van der Berg, abdul née Jones"`` are unchanged. The bracketed form was never affected, since ``"Ursula von der Leyen (geb. Albrecht)"`` is extracted before grouping. No differential corpus name carries a particle ahead of a marker, so this class is invisible to the harness at every baseline and the tests are the whole verification; `#413 `_ tracks the corpus gap (closes #399) + - Fix a maiden name being lost when a particle stood in front of the marker. ``"Ursula Leyen geb. Albrecht"`` reported maiden ``Albrecht`` correctly, but ``"Ursula von der Leyen geb. Albrecht"`` -- the same words one particle chain apart -- gave family ``von der Leyen geb. Albrecht`` and no maiden name at all, and ``"Jane van der Berg née Jones"`` failed the same way. The particle chain absorbed everything up to the next particle or suffix, and the marker is consumed later in the same grouping stage, by which point the chain had merged the marker away and there was no marker word left standing to find. A suffix already stopped the chain; a marker now does too, so those read family ``von der Leyen`` maiden ``Albrecht`` and family ``van der Berg`` maiden ``Jones``. Only a particle that is not the name's own first word ever reached the marker, which is why a single leading particle always worked -- ``"von Müller geb. Schmidt"`` was already right -- while a leading run of two did not: in ``"von der Müller geb. Schmidt"`` the second particle starts a chain of its own. The bound applies only where the marker is actually taken as a marker. A marker needs a word after it to hand the maiden name to, so a trailing one is just a word and bounds nothing: ``"Jane van der Berg née"`` and ``"Jane van der Berg née PhD"`` both keep family ``van der Berg née``, which is the reading ``"Jones née"`` -> family ``née`` already had. That distinction is what keeps a name like ``"Jane van der Nee"`` intact -- ``Nee`` is an attested surname as well as a marker spelling -- and what keeps ``"St St née"`` reporting its ``PARTICLE_OR_GIVEN`` fork. Two consequences fall out. Under a family-first order the marker used to survive as an ordinary name word and take the leftover given slot, so ``"de la Cruz née Vega"`` read given ``née``, middle ``Vega`` under ``Policy(name_order=FAMILY_FIRST)`` and given ``Vega``, middle ``née`` under ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``; consumed and dropped it never reaches that placement, and both orders now read family ``de la Cruz``, maiden ``Vega``. And stopping the chain can leave a family of nothing but particles, which the #385 rule above already covers: ``"Jane de la née Jones"`` reports family ``de la`` with ``family_base`` ``de la`` rather than an empty base. Two limits remain, and both are recorded in ``rules.md#M2``: the bound reaches only a marker standing as a word of its own, so a conjunction join and a bound given-name join each still absorb one first -- ``"Jane van der Berg née y Jones"`` and ``"van der Berg, abdul née Jones"`` are unchanged. The bracketed form was never affected, since ``"Ursula von der Leyen (geb. Albrecht)"`` is extracted before grouping. When this landed no differential corpus name carried a particle ahead of a marker, so the class was invisible to the harness and the tests were the whole verification. The rules-doc corpus added in the same release closes that gap: three names of this shape are now compared at every baseline (closes #399) - - Move ``mc`` and ``ste`` into the never-given half of the particle vocabulary, and add the Spanish and Portuguese articles ``los``, ``las`` and ``das``, which were absent from it entirely. A never-given particle opening a name folds into the family (``rules.md#P1``) instead of being read as a given name, so ``"Mc Donald"`` was first ``Mc``, last ``Donald`` and is now last ``Mc Donald``; ``"Ste Marie"``, ``"Los Santos"``, ``"Las Casas"`` and ``"Das Silva"`` move the same way, and ``"Mc Donald Smith"`` goes from first ``Mc``, middle ``Donald``, last ``Smith`` to last ``Mc Donald Smith``. The ``PARTICLE_OR_GIVEN`` ambiguity these names used to report goes with it: once the word can never be a given name there is no fork left to report. Names where the particle is not leading are untouched -- ``"Jack Mc Donald"`` already read last ``Mc Donald`` and is byte-identical. Never-given membership goes from 28 words to 33. No differential corpus name moved when this landed; ``"Mc Donald"`` entered the corpora afterwards with ``corpus_rules.jsonl`` and is now the one name that pins it (#360) + - Move ``mc`` and ``ste`` into the never-given half of the particle vocabulary, and add the Spanish and Portuguese articles ``los``, ``las`` and ``das``, which were absent from it entirely. A never-given particle opening a name folds into the family (``rules.md#P1``) instead of being read as a given name, so ``"Mc Donald"`` was first ``Mc``, last ``Donald`` and is now last ``Mc Donald``; ``"Ste Marie"``, ``"Los Santos"``, ``"Las Casas"`` and ``"Das Silva"`` move the same way, and ``"Mc Donald Smith"`` goes from first ``Mc``, middle ``Donald``, last ``Smith`` to last ``Mc Donald Smith``. The ``PARTICLE_OR_GIVEN`` ambiguity goes with it for ``mc`` and ``ste``, the two that were already in the vocabulary: once the word can never be a given name there is no fork left to report. ``los``, ``las`` and ``das`` never reported one, having not been particles at all. ``mc`` and ``ste`` were already particles and only changed halves, so names where they are not leading are untouched -- ``"Jack Mc Donald"`` already read last ``Mc Donald`` and is byte-identical. ``los``, ``las`` and ``das`` were not particles at all, so for those three the ordinary particle join fires from a non-leading position too: ``"Maria das Neves"`` was first ``Maria``, middle ``das``, last ``Neves`` and is now last ``das Neves``. Never-given membership goes from 28 words to 33. No differential corpus name moved when this landed; ``"Mc Donald"`` entered the corpora afterwards with ``corpus_rules.jsonl`` and is now the one name that pins it (closes #360) **Deprecations** diff --git a/tests/v2/_differential_fixtures.py b/tests/v2/_differential_fixtures.py index 2e9da482..8d4e45f2 100644 --- a/tests/v2/_differential_fixtures.py +++ b/tests/v2/_differential_fixtures.py @@ -96,7 +96,7 @@ def load_tool(stem: str) -> ModuleType: wrote the same six lines each: test_differential.py for compare.py, test_ledger_guards.py for build_cjk_corpus.py. - Neither has import-time side effects -- compare.py's main() is + None of them has import-time side effects -- compare.py's main() is behind a __name__ guard, and build_cjk_corpus.py only defines functions -- so importing them to read a constant or call one function is safe. diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 1c08c033..0026701c 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -519,6 +519,80 @@ def test_cjk_corpus_matches_the_case_table() -> None: "`uv run python tools/differential/build_cjk_corpus.py`") +#: Names each rule MUST NOT match, keyed by a substring of its `issue`. +#: +#: A wall, not a change detector, and that is the point. _CORPUS_CLAIMS +#: catches a widening that changes corpus reach or roles -- but a rule +#: whose regex is literal-anchored and claims exactly ONE corpus name +#: can be widened with the count unmoved, because the names it newly +#: reaches are not in the corpora. Six such widenings were demonstrated +#: on this file's own rules with the whole suite green: `^mc\s+\S+$` +#: to `^mc` (every leading Mc*), the vd rule to a bare `\bvd\b`, +#: `^sir\s+de\b` without its anchor, and the nakaguro rule to `·.*씨`. +#: Each of those then stood ready to explain exactly what its own +#: comment promises will arrive UNEXPLAINED. +#: +#: The probes are taken from the names those comments already argue +#: about, so this roster records an answer someone already wrote in +#: prose. Being a wall rather than a snapshot, it is wrong at recording +#: time too -- a probe that matches when it is added fails immediately +#: rather than being blessed as the new normal. +#: +#: A probe must be a name the rule has no business claiming, NOT merely +#: one that does not move today: rules deliberately claim some static +#: names (fix(#399) claims 'Jane van der Berg nee PhD', fix(#400) +#: claims 'abd Allah'), and those are recorded in the rules' comments +#: instead. +_MUST_NOT_MATCH: dict[str, tuple[str, ...]] = { + "fix(#380)": ("vd Berg, Jan", "Jan vd Berg", "Smith vd", + "Berg, Jan mc"), + "fix(#399)": ("Jane van der Berg née", "Jane van der Berg née y Jones", + "van der Berg, abdul née Jones", "Jane Smith née Jones"), + "fix(#360)": ("McDonald, Ronald", "Mcintyre Smith Jr.", "Ste Marie", + "Los Santos"), + "fix(#400)": ("Jane Smith ABD", "Jane Smith A.B.D.", "Abdul Salam"), + "fix(#367) a title no longer displaces a leading never-given particle": + ("John Sir de Mesnil", "Smith, Sir de Vaux", "Sir Smith"), + "fix(#272/#308)": ("田中·太郎 김씨", "A·B 씨", "山田·花子"), + "fix(nickname-typographic-pairs)": ("Hans „Hansi“ Müller", + "John “Jack” Kennedy"), + "fix(#274)": ("Jones née", "née Jones", "Jane van der Berg née", + "Jane Smith (Nee)"), +} + + +def test_no_rule_matches_a_name_it_has_no_business_claiming() -> None: + """The wall _CORPUS_CLAIMS cannot be. + + A claim count moves only when corpus reach moves, so a rule that + claims one corpus name can be widened arbitrarily as long as the + names it newly reaches are outside the corpora -- and every rule + added by #414 is literal-anchored and claims exactly one. This + asserts the boundaries those rules' comments argue for, against + names chosen to sit just outside them. + """ + checked = 0 + for ledger in _LEDGERS: + for rule in _rules(ledger): + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + for key, probes in _MUST_NOT_MATCH.items(): + if key not in rule["issue"]: + continue + for probe in probes: + checked += 1 + assert not re.search(regex, probe), ( + f"{ledger.name}: {rule['issue']!r} matches " + f"{probe!r}, a name its own comment says should " + f"arrive UNEXPLAINED. Either the rule widened " + f"or the boundary moved -- if the latter, the " + f"comment and this roster both need editing") + assert checked, ( + "_MUST_NOT_MATCH matched no rule; the keys are `issue` " + "substrings and a renamed rule silently empties this pin") + + def test_rules_corpus_matches_the_rules_doc() -> None: """corpus_rules.jsonl is GENERATED from docs/design/rules.md's own examples (#414), through the doc's parser rather than a second @@ -541,6 +615,36 @@ def test_rules_corpus_matches_the_rules_doc() -> None: "corpus_rules.jsonl is stale: regenerate with " "`uv run python tools/differential/build_rules_corpus.py`") + # The equality above cannot see a generator that silently selects + # LESS: someone regenerates, the file agrees with the degraded + # selection, and both are wrong together. _CORPUS_FLOORS bounds + # that to a few names; these two bound the shapes it cannot. + # + # A rule whose ID line stops matching, or that gains a heading + # mid-block, loses its examples to the previous rule or to + # nothing. Both are loud today only by luck of a neighbouring + # invariant -- every rule carries `implemented:` and is cited from + # code, so test_doc_citations fails on the unknown ID. That is a + # different file's guarantee and could stop holding. + from .rules_doc import parse_rules_doc + rules = parse_rules_doc( + (_TOOLS.parents[1] / "docs" / "design" / "rules.md") + .read_text(encoding="utf-8")) + assert len(rules) >= 38, ( + f"rules.md parsed to {len(rules)} rules, fewer than the 38 this " + f"corpus was built from. A rule block that stops parsing hands " + f"its examples to its neighbour and shrinks the corpus " + f"silently. Ratchet this floor up when rules are added") + empty = [r.rule_id for r in rules + if not any(e.text for e in r.examples)] + assert empty == ["D1", "D2"], ( + f"rules with no example NAME: {empty}. D1 and D2 are the " + f"expected two -- their examples name a policy rather than a " + f"name string. Any other rule contributing nothing means its " + f"examples stopped being recognized as examples, which the " + f"file-equality check above cannot distinguish from a rule " + f"that never had any") + #: Which vocabulary constant each ledger rule's alternation is a hand #: copy of. A roster rather than an inference: GLUED_HONORIFICS is a @@ -1059,12 +1163,22 @@ def _claim(rule: dict) -> _Claim: #: That is the intended cost: a corpus name added under an existing #: rule is a real change in what that rule explains, and it should be #: read once rather than absorbed silently. +#: Twenty of these moved at once when corpus_rules.jsonl landed +#: (#414), which is a lot of re-recording to review. The jumps are all +#: one cause -- 113 names arriving -- and the ones worth naming are +#: the broad rules: the fields-only catch-alls grew to the whole +#: corpus (751 -> 864), the comma rules 215 -> 236, fix(#274) 4 -> 11 +#: and fix(#379) 2 -> 8. What made each jump reviewable was not this +#: record but the two guards beside it: the vocabulary-presence guard +#: rejected fix(#274)'s first grown reach outright, and the +#: member-reach guard rejected the acronym rule's. A jump that passes +#: both is growth into names the rule genuinely describes. _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": _Claim(105, ('family', 'given', 'middle'), "090ac3eef3fb"), "fix(#274) maiden markers consumed": - _Claim(16, ('family', 'maiden', 'middle'), "cc672349b500"), + _Claim(11, ('family', 'maiden', 'middle'), "16e085499b25"), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": _Claim(4, ('family', 'given', 'maiden', 'middle'), "27bb9ebc1951"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": @@ -1128,7 +1242,7 @@ def _claim(rule: dict) -> _Claim: "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": _Claim(1, ('family', 'suffix'), "081ce07f927b"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": - _Claim(6, ('family', 'maiden'), "b40129435671"), + _Claim(4, ('family', 'maiden'), "32c8178f87c0"), "fix(#360) mc moved into the never-given particles, so it folds into the family": _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), "fix(#400) abd joins the word after it as one given name": @@ -1146,7 +1260,7 @@ def _claim(rule: dict) -> _Claim: "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": _Claim(1, ('family', 'suffix'), "081ce07f927b"), "fix(#399) a maiden marker bounds the particle chain that swallowed it": - _Claim(6, ('family', 'maiden'), "b40129435671"), + _Claim(4, ('family', 'maiden'), "32c8178f87c0"), "fix(#360) mc moved into the never-given particles, so it folds into the family": _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), "fix(#400) abd joins the word after it as one given name": @@ -1229,9 +1343,13 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: #: gate reporting 108/0 throughout. #: #: The comma family is recorded because it is where that went wrong. -#: 42 corpus names are reachable by two or more rules in the same tier; -#: these ten are the ones whose boundaries this file argues about, and -#: pinning the argument is cheaper than re-deriving it. Note +#: Hundreds of corpus names are reachable by two or more rules in the +#: same tier -- 301 in the 1.4 tier as of the rules-doc corpus (#414), +#: up from 42, which is why the count is described rather than pinned: +#: it moves with every corpus addition and pinning it would only ever +#: be re-recorded. The rows below are the ones whose boundaries this +#: file argues about, and pinning the argument is cheaper than +#: re-deriving it. Note #: 'Andrews, M.D.' diffs on the SAME {given, suffix} shape as the seven #: peels -- only the CJK lookahead separates them, so it is the row #: that fails if that lookahead is ever dropped. @@ -1241,6 +1359,21 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: #: a diff shape that shifted is a finding, not a number to update. _CROSS_RULE_WINNERS: dict[str, dict[tuple[str, tuple[str, ...]], str]] = { "expected_since_1.4.0.toml": { + # Three behaviours that got named rules in the 2.0 and 2.1 + # ledgers but not this one, so here they fall to the fields-only + # catch-all. Recorded because the assignment is incidental + # rather than argued: 'Berg, Jan vd' is reachable by three + # named rules and wins none of them (their `fields` do not + # match), while 'Mc Donald' and 'Sir de Mesnil' are reachable + # by NO named rule at all. Being absorbed by the catch-all is + # the recoverable direction, but a rule added later that takes + # one of them silently would change what the 1.4 summary + # attributes -- and AGENTS.md names that summary as the source + # for the release log's Behavior Changes section. #380, #360 + # and #367 respectively. + ("Berg, Jan vd", ("family", "suffix")): "fix(suffix-routing)", + ("Mc Donald", ("family", "given")): "fix(suffix-routing)", + ("Sir de Mesnil", ("family", "given")): "fix(suffix-routing)", ("Andrews, M.D.", ("given", "suffix")): "fix(comma-family)", ("田中, 太郎さん", ("given", "suffix")): "fix(cjk-comma-honorific-peel)", ("김, 민준씨", ("given", "suffix")): "fix(cjk-comma-honorific-peel)", diff --git a/tools/differential/README.md b/tools/differential/README.md index 917e95a6..46abd051 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -211,7 +211,7 @@ intended-but-unclassified 2.0 behaviors (#273 typographic delimiters, #269 non-Latin vocabulary) plus one shape no test had considered: a **leading** `"Ph. D."`, which v1 split into title `Ph.` + given `D.`. -The rules corpus earned its place the same way — 113 of its 156 names +The rules corpus earned its place the same way — 113 of its 155 names were in no other corpus, and on its first run 12 of them turned out to have moved during the 2.2 cycle with nothing observing it. Classifying them surfaced an unlogged behavior change (`mc` moving into the diff --git a/tools/differential/build_rules_corpus.py b/tools/differential/build_rules_corpus.py index 1009db69..87ae3f80 100644 --- a/tools/differential/build_rules_corpus.py +++ b/tools/differential/build_rules_corpus.py @@ -29,8 +29,10 @@ "Jane de la née Jones" → family="de la née Jones" to family="de la". The doc test followed the edit, the name was in no -corpus, and nothing independent observed the movement. Twelve names in -this corpus moved during the 2.2 cycle the same way. +corpus, and nothing independent observed the movement. Twelve names +this corpus CONTRIBUTED moved during the 2.2 cycle the same way -- +thirteen of its names move in total, the thirteenth having already +been in corpus.jsonl. Selection is every distinct example `text`, through the doc's own parser rather than a second regex -- the same argument that has @@ -43,9 +45,18 @@ so a family-first-scoped example is simply one more name to diff -- build_cjk_corpus.py makes the same call for its zh-scoped rows. +The one example form that carries no name is skipped. A `[subject]` +example names a policy or locale rather than a name string, so its +`text` is empty -- rules.md has three, and without the filter all +three collapse into a single "" that the doc never wrote. Skipping +them is also why the promise above is honest: an example enters the +corpus by being written down, and a form with nothing to enter is +better excluded than folded into a shared empty string. + Boundary strings ("(", ".,", "Anna () Smith") are kept, on the -over-collection principle the other three builders state: a name that -parses to nothing costs one parse and produces no diff. They are more +over-collection principle build_corpus.py and build_issues_corpus.py +state: a name that parses to nothing costs one parse and produces no +diff. They are more concentrated here than elsewhere because the doc argues its edges explicitly, which is a reason to keep them rather than to drop them. @@ -79,7 +90,8 @@ def selected_names() -> list[str]: rules = parse_rules_doc(RULES_DOC.read_text(encoding="utf-8")) return sorted({example.text for rule in rules - for example in rule.examples}) + for example in rule.examples + if example.text}) def main() -> None: diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 4d650d2f..2563c56a 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -358,7 +358,7 @@ def _is_latin_only(name: str) -> bool: "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref "corpus_cjk.jsonl": 95, # 97 today, generated from the case table "corpus_issues.jsonl": 190, # 200 today, harvested and append-only - "corpus_rules.jsonl": 150, # 156 today, generated from rules.md + "corpus_rules.jsonl": 150, # 155 today, generated from rules.md } @@ -764,6 +764,19 @@ def main() -> int: if not paths: raise SystemExit( f"no corpus files found in {HERE}: expected corpus*.jsonl") + # A file that DISAPPEARS is the same nightmare one step smaller: + # the glob simply finds fewer files, and the run compares less + # while printing a summary that reads exactly like a full one. The + # floors already name every corpus that is supposed to exist, so + # ask them. Skipped under --corpus, where narrowing is the point. + if not args.corpus: + missing = sorted(set(_CORPUS_FLOORS) - {p.name for p in paths}) + if missing: + raise SystemExit( + f"corpus files named in _CORPUS_FLOORS are not on disk: " + f"{missing}. A corpus that vanishes shrinks the " + f"comparison silently -- restore it, or drop its floor " + f"if it is meant to be gone") per_file = {} corpus = [] for path in paths: diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index bd79671b..7aaf5900 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -1,4 +1,3 @@ -"" "'Smitty' John Jones" "'Smitty' Jones" "'Smitty' Jones Jr." diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 1af3409e..77035568 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -99,7 +99,7 @@ issue = "fix(#274) maiden markers consumed" # See maiden_markers.py's own "Deliberately absent" note for what has # actually been considered and declined. # -# The paren lookarounds are not decoration. corpus_rules.jsonl (#414) +# The lookarounds and the two context tests are not decoration. corpus_rules.jsonl (#414) # brought in 'Jane Smith (Nee)' and 'Jane Smith (Nee) (Jones)', which # are M1's examples of a one-word clause keeping a word that may # itself be a surname. Under the DEFAULT facade -- which is what the @@ -109,7 +109,24 @@ issue = "fix(#274) maiden markers consumed" # ready to explain a `maiden` diff on a name that has no marker in it, # which is absorption waiting to happen. The vocabulary-presence guard # in test_ledger_guards.py is what caught it. -name_regex = "(?i)(? 11, and all seven names it explains are inside the eleven. +# +# Four claimed names still do not move, recorded rather than excluded: +# 'Jane Smith, nee Jones' (post-comma, M2's remaining Accepted limit), +# 'Jane van der Berg nee PhD' (only a suffix follows, so the consumer +# declines), and the connective and bound-given joins (#412). Each +# would need the regex to ask a vocabulary question -- suffix or +# connective membership -- a second alternation this rule cannot +# spend. +name_regex = "(?i)(? Date: Fri, 21 Aug 2026 19:03:35 -0700 Subject: [PATCH 4/4] docs(differential): rewrap a docstring paragraph Left ragged by the [subject]-filter edit. Co-Authored-By: Claude Opus 5 --- tools/differential/build_rules_corpus.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/differential/build_rules_corpus.py b/tools/differential/build_rules_corpus.py index 87ae3f80..6e711026 100644 --- a/tools/differential/build_rules_corpus.py +++ b/tools/differential/build_rules_corpus.py @@ -56,9 +56,9 @@ Boundary strings ("(", ".,", "Anna () Smith") are kept, on the over-collection principle build_corpus.py and build_issues_corpus.py state: a name that parses to nothing costs one parse and produces no -diff. They are more -concentrated here than elsewhere because the doc argues its edges -explicitly, which is a reason to keep them rather than to drop them. +diff. They are more concentrated here than elsewhere because the doc +argues its edges explicitly, which is a reason to keep them rather +than to drop them. Regenerate after editing rules.md examples: