diff --git a/docs/release_log.rst b/docs/release_log.rst index de73d8e6..f952cce1 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -42,7 +42,9 @@ Release Log - Change the ``detail`` text of a ``PARTICLE_OR_GIVEN`` ambiguity to name the role the leading particle was actually given. It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads as family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case, reading the role off the assigned token the way ``SUFFIX_OR_NAME`` already did -- that kind names both parts (``read as a family name rather than a post-nominal``), while this one names only the part it took. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``: the fork really is particle-or-given, and only the human-readable text moved. Default-order output is identical (#355) - - Fix a maiden name being lost when a particle stood in front of the marker. ``"Ursula Leyen geb. Albrecht"`` reported maiden ``Albrecht`` correctly, but ``"Ursula von der Leyen geb. Albrecht"`` -- the same words one particle chain apart -- gave family ``von der Leyen geb. Albrecht`` and no maiden name at all, and ``"Jane van der Berg née Jones"`` failed the same way. The particle chain absorbed everything up to the next particle or suffix, and the marker is consumed later in the same grouping stage, by which point the chain had merged the marker away and there was no marker word left standing to find. A suffix already stopped the chain; a marker now does too, so those read family ``von der Leyen`` maiden ``Albrecht`` and family ``van der Berg`` maiden ``Jones``. Only a particle that is not the name's own first word ever reached the marker, which is why a single leading particle always worked -- ``"von Müller geb. Schmidt"`` was already right -- while a leading run of two did not: in ``"von der Müller geb. Schmidt"`` the second particle starts a chain of its own. The bound applies only where the marker is actually taken as a marker. A marker needs a word after it to hand the maiden name to, so a trailing one is just a word and bounds nothing: ``"Jane van der Berg née"`` and ``"Jane van der Berg née PhD"`` both keep family ``van der Berg née``, which is the reading ``"Jones née"`` -> family ``née`` already had. That distinction is what keeps a name like ``"Jane van der Nee"`` intact -- ``Nee`` is an attested surname as well as a marker spelling -- and what keeps ``"St St née"`` reporting its ``PARTICLE_OR_GIVEN`` fork. Two consequences fall out. Under a family-first order the marker used to survive as an ordinary name word and take the leftover given slot, so ``"de la Cruz née Vega"`` read given ``née``, middle ``Vega`` under ``Policy(name_order=FAMILY_FIRST)`` and given ``Vega``, middle ``née`` under ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``; consumed and dropped it never reaches that placement, and both orders now read family ``de la Cruz``, maiden ``Vega``. And stopping the chain can leave a family of nothing but particles, which the #385 rule above already covers: ``"Jane de la née Jones"`` reports family ``de la`` with ``family_base`` ``de la`` rather than an empty base. Two limits remain, and both are recorded in ``rules.md#M2``: the bound reaches only a marker standing as a word of its own, so a conjunction join and a bound given-name join each still absorb one first -- ``"Jane van der Berg née y Jones"`` and ``"van der Berg, abdul née Jones"`` are unchanged. The bracketed form was never affected, since ``"Ursula von der Leyen (geb. Albrecht)"`` is extracted before grouping. No differential corpus name carries a particle ahead of a marker, so this class is invisible to the harness at every baseline and the tests are the whole verification; `#413 `_ tracks the corpus gap (closes #399) + - Fix a maiden name being lost when a particle stood in front of the marker. ``"Ursula Leyen geb. Albrecht"`` reported maiden ``Albrecht`` correctly, but ``"Ursula von der Leyen geb. Albrecht"`` -- the same words one particle chain apart -- gave family ``von der Leyen geb. Albrecht`` and no maiden name at all, and ``"Jane van der Berg née Jones"`` failed the same way. The particle chain absorbed everything up to the next particle or suffix, and the marker is consumed later in the same grouping stage, by which point the chain had merged the marker away and there was no marker word left standing to find. A suffix already stopped the chain; a marker now does too, so those read family ``von der Leyen`` maiden ``Albrecht`` and family ``van der Berg`` maiden ``Jones``. Only a particle that is not the name's own first word ever reached the marker, which is why a single leading particle always worked -- ``"von Müller geb. Schmidt"`` was already right -- while a leading run of two did not: in ``"von der Müller geb. Schmidt"`` the second particle starts a chain of its own. The bound applies only where the marker is actually taken as a marker. A marker needs a word after it to hand the maiden name to, so a trailing one is just a word and bounds nothing: ``"Jane van der Berg née"`` and ``"Jane van der Berg née PhD"`` both keep family ``van der Berg née``, which is the reading ``"Jones née"`` -> family ``née`` already had. That distinction is what keeps a name like ``"Jane van der Nee"`` intact -- ``Nee`` is an attested surname as well as a marker spelling -- and what keeps ``"St St née"`` reporting its ``PARTICLE_OR_GIVEN`` fork. Two consequences fall out. Under a family-first order the marker used to survive as an ordinary name word and take the leftover given slot, so ``"de la Cruz née Vega"`` read given ``née``, middle ``Vega`` under ``Policy(name_order=FAMILY_FIRST)`` and given ``Vega``, middle ``née`` under ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)``; consumed and dropped it never reaches that placement, and both orders now read family ``de la Cruz``, maiden ``Vega``. And stopping the chain can leave a family of nothing but particles, which the #385 rule above already covers: ``"Jane de la née Jones"`` reports family ``de la`` with ``family_base`` ``de la`` rather than an empty base. Two limits remain, and both are recorded in ``rules.md#M2``: the bound reaches only a marker standing as a word of its own, so a conjunction join and a bound given-name join each still absorb one first -- ``"Jane van der Berg née y Jones"`` and ``"van der Berg, abdul née Jones"`` are unchanged. The bracketed form was never affected, since ``"Ursula von der Leyen (geb. Albrecht)"`` is extracted before grouping. When this landed no differential corpus name carried a particle ahead of a marker, so the class was invisible to the harness and the tests were the whole verification. The rules-doc corpus added in the same release closes that gap: three names of this shape are now compared at every baseline (closes #399) + + - Move ``mc`` and ``ste`` into the never-given half of the particle vocabulary, and add the Spanish and Portuguese articles ``los``, ``las`` and ``das``, which were absent from it entirely. A never-given particle opening a name folds into the family (``rules.md#P1``) instead of being read as a given name, so ``"Mc Donald"`` was first ``Mc``, last ``Donald`` and is now last ``Mc Donald``; ``"Ste Marie"``, ``"Los Santos"``, ``"Las Casas"`` and ``"Das Silva"`` move the same way, and ``"Mc Donald Smith"`` goes from first ``Mc``, middle ``Donald``, last ``Smith`` to last ``Mc Donald Smith``. The ``PARTICLE_OR_GIVEN`` ambiguity goes with it for ``mc`` and ``ste``, the two that were already in the vocabulary: once the word can never be a given name there is no fork left to report. ``los``, ``las`` and ``das`` never reported one, having not been particles at all. ``mc`` and ``ste`` were already particles and only changed halves, so names where they are not leading are untouched -- ``"Jack Mc Donald"`` already read last ``Mc Donald`` and is byte-identical. ``los``, ``las`` and ``das`` were not particles at all, so for those three the ordinary particle join fires from a non-leading position too: ``"Maria das Neves"`` was first ``Maria``, middle ``das``, last ``Neves`` and is now last ``das Neves``. Never-given membership goes from 28 words to 33. No differential corpus name moved when this landed; ``"Mc Donald"`` entered the corpora afterwards with ``corpus_rules.jsonl`` and is now the one name that pins it (closes #360) **Deprecations** diff --git a/tests/v2/_differential_fixtures.py b/tests/v2/_differential_fixtures.py index 2e9da482..8d4e45f2 100644 --- a/tests/v2/_differential_fixtures.py +++ b/tests/v2/_differential_fixtures.py @@ -96,7 +96,7 @@ def load_tool(stem: str) -> ModuleType: wrote the same six lines each: test_differential.py for compare.py, test_ledger_guards.py for build_cjk_corpus.py. - Neither has import-time side effects -- compare.py's main() is + None of them has import-time side effects -- compare.py's main() is behind a __name__ guard, and build_cjk_corpus.py only defines functions -- so importing them to read a constant or call one function is safe. diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index e610e803..0026701c 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -519,6 +519,133 @@ def test_cjk_corpus_matches_the_case_table() -> None: "`uv run python tools/differential/build_cjk_corpus.py`") +#: Names each rule MUST NOT match, keyed by a substring of its `issue`. +#: +#: A wall, not a change detector, and that is the point. _CORPUS_CLAIMS +#: catches a widening that changes corpus reach or roles -- but a rule +#: whose regex is literal-anchored and claims exactly ONE corpus name +#: can be widened with the count unmoved, because the names it newly +#: reaches are not in the corpora. Six such widenings were demonstrated +#: on this file's own rules with the whole suite green: `^mc\s+\S+$` +#: to `^mc` (every leading Mc*), the vd rule to a bare `\bvd\b`, +#: `^sir\s+de\b` without its anchor, and the nakaguro rule to `·.*씨`. +#: Each of those then stood ready to explain exactly what its own +#: comment promises will arrive UNEXPLAINED. +#: +#: The probes are taken from the names those comments already argue +#: about, so this roster records an answer someone already wrote in +#: prose. Being a wall rather than a snapshot, it is wrong at recording +#: time too -- a probe that matches when it is added fails immediately +#: rather than being blessed as the new normal. +#: +#: A probe must be a name the rule has no business claiming, NOT merely +#: one that does not move today: rules deliberately claim some static +#: names (fix(#399) claims 'Jane van der Berg nee PhD', fix(#400) +#: claims 'abd Allah'), and those are recorded in the rules' comments +#: instead. +_MUST_NOT_MATCH: dict[str, tuple[str, ...]] = { + "fix(#380)": ("vd Berg, Jan", "Jan vd Berg", "Smith vd", + "Berg, Jan mc"), + "fix(#399)": ("Jane van der Berg née", "Jane van der Berg née y Jones", + "van der Berg, abdul née Jones", "Jane Smith née Jones"), + "fix(#360)": ("McDonald, Ronald", "Mcintyre Smith Jr.", "Ste Marie", + "Los Santos"), + "fix(#400)": ("Jane Smith ABD", "Jane Smith A.B.D.", "Abdul Salam"), + "fix(#367) a title no longer displaces a leading never-given particle": + ("John Sir de Mesnil", "Smith, Sir de Vaux", "Sir Smith"), + "fix(#272/#308)": ("田中·太郎 김씨", "A·B 씨", "山田·花子"), + "fix(nickname-typographic-pairs)": ("Hans „Hansi“ Müller", + "John “Jack” Kennedy"), + "fix(#274)": ("Jones née", "née Jones", "Jane van der Berg née", + "Jane Smith (Nee)"), +} + + +def test_no_rule_matches_a_name_it_has_no_business_claiming() -> None: + """The wall _CORPUS_CLAIMS cannot be. + + A claim count moves only when corpus reach moves, so a rule that + claims one corpus name can be widened arbitrarily as long as the + names it newly reaches are outside the corpora -- and every rule + added by #414 is literal-anchored and claims exactly one. This + asserts the boundaries those rules' comments argue for, against + names chosen to sit just outside them. + """ + checked = 0 + for ledger in _LEDGERS: + for rule in _rules(ledger): + regex = rule.get("name_regex") + if not isinstance(regex, str): + continue + for key, probes in _MUST_NOT_MATCH.items(): + if key not in rule["issue"]: + continue + for probe in probes: + checked += 1 + assert not re.search(regex, probe), ( + f"{ledger.name}: {rule['issue']!r} matches " + f"{probe!r}, a name its own comment says should " + f"arrive UNEXPLAINED. Either the rule widened " + f"or the boundary moved -- if the latter, the " + f"comment and this roster both need editing") + assert checked, ( + "_MUST_NOT_MATCH matched no rule; the keys are `issue` " + "substrings and a renamed rule silently empties this pin") + + +def test_rules_corpus_matches_the_rules_doc() -> None: + """corpus_rules.jsonl is GENERATED from docs/design/rules.md's own + examples (#414), through the doc's parser rather than a second + regex. Same promise as the CJK pin above: an example added to a + rule without regenerating fails HERE, instead of leaving the + normative names outside the differential gate. + + Why the doc's examples need the gate at all, when test_rules_doc + already executes every one of them: those pin against an + expectation stored beside them, so a deliberate behaviour change + edits both in one commit and the test stays green. The corpus + compares against a RELEASED baseline no commit can edit, which is + what forces a moved name to be classified in writing. + """ + module = load_tool("build_rules_corpus") + checked_in = [json.loads(line) for line in + (_TOOLS / "corpus_rules.jsonl") + .read_text(encoding="utf-8").splitlines()] + assert checked_in == module.selected_names(), ( + "corpus_rules.jsonl is stale: regenerate with " + "`uv run python tools/differential/build_rules_corpus.py`") + + # The equality above cannot see a generator that silently selects + # LESS: someone regenerates, the file agrees with the degraded + # selection, and both are wrong together. _CORPUS_FLOORS bounds + # that to a few names; these two bound the shapes it cannot. + # + # A rule whose ID line stops matching, or that gains a heading + # mid-block, loses its examples to the previous rule or to + # nothing. Both are loud today only by luck of a neighbouring + # invariant -- every rule carries `implemented:` and is cited from + # code, so test_doc_citations fails on the unknown ID. That is a + # different file's guarantee and could stop holding. + from .rules_doc import parse_rules_doc + rules = parse_rules_doc( + (_TOOLS.parents[1] / "docs" / "design" / "rules.md") + .read_text(encoding="utf-8")) + assert len(rules) >= 38, ( + f"rules.md parsed to {len(rules)} rules, fewer than the 38 this " + f"corpus was built from. A rule block that stops parsing hands " + f"its examples to its neighbour and shrinks the corpus " + f"silently. Ratchet this floor up when rules are added") + empty = [r.rule_id for r in rules + if not any(e.text for e in r.examples)] + assert empty == ["D1", "D2"], ( + f"rules with no example NAME: {empty}. D1 and D2 are the " + f"expected two -- their examples name a policy rather than a " + f"name string. Any other rule contributing nothing means its " + f"examples stopped being recognized as examples, which the " + f"file-equality check above cannot distinguish from a rule " + f"that never had any") + + #: Which vocabulary constant each ledger rule's alternation is a hand #: copy of. A roster rather than an inference: GLUED_HONORIFICS is a #: SUBSET of SUFFIX_WORDS (asserted at the bottom of @@ -690,6 +817,18 @@ class _LatinCopy(NamedTuple): covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", "du", "la", "le", "los", "mc", "van", "vd", "von", "zu"})), + # The same partial copy as fix(#379), and partial for the same + # reason -- these are the words that actually open a chain in a + # Latin name. The two rules ask different questions of it (#379 + # wants the word that ENDS a comma listing, #399 the word that + # opens a chain running into a maiden marker), so the shared + # snapshot is agreement rather than duplication: a divergence + # would be a fact about one of the rules. + "fix(#399)": _LatinCopy( + vocabulary=PARTICLES, + covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", + "du", "la", "le", "los", "mc", "van", "vd", + "von", "zu"})), } #: Alternations that copy no vocabulary, so discovery must not demand a @@ -698,6 +837,11 @@ class _LatinCopy(NamedTuple): #: question someone answers in writing, not something to skip past. _NOT_A_VOCABULARY_COPY = frozenset({ frozenset({"^", " "}), # the honorific rule's leading anchor + # fix(#400)'s two openings: start-of-name or just after a family + # comma. `abd` joins forward on the given side wherever that side + # begins, and the alternation is over ANCHORS, not over words -- + # there is no vocabulary here to drift from. + frozenset({"^", ",\\s*"}), }) def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: @@ -1019,22 +1163,32 @@ def _claim(rule: dict) -> _Claim: #: That is the intended cost: a corpus name added under an existing #: rule is a real change in what that rule explains, and it should be #: read once rather than absorbed silently. +#: Twenty of these moved at once when corpus_rules.jsonl landed +#: (#414), which is a lot of re-recording to review. The jumps are all +#: one cause -- 113 names arriving -- and the ones worth naming are +#: the broad rules: the fields-only catch-alls grew to the whole +#: corpus (751 -> 864), the comma rules 215 -> 236, fix(#274) 4 -> 11 +#: and fix(#379) 2 -> 8. What made each jump reviewable was not this +#: record but the two guards beside it: the vocabulary-presence guard +#: rejected fix(#274)'s first grown reach outright, and the +#: member-reach guard rejected the acronym rule's. A jump that passes +#: both is growth into names the rule genuinely describes. _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('family', 'given', 'middle'), "66e71d60a075"), + _Claim(105, ('family', 'given', 'middle'), "090ac3eef3fb"), "fix(#274) maiden markers consumed": - _Claim(4, ('family', 'maiden', 'middle'), "b31dc2e2bbc4"), + _Claim(11, ('family', 'maiden', 'middle'), "16e085499b25"), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + _Claim(4, ('family', 'given', 'maiden', 'middle'), "27bb9ebc1951"), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(2, ('family', 'middle'), "f3a43bebfb91"), + _Claim(8, ('family', 'middle'), "f4d52b062f39"), "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(215, ('given', 'suffix', 'title'), "f16a0e79cba3"), + _Claim(236, ('given', 'suffix', 'title'), "9393109ebf99"), "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(215, ('family', 'given'), "f16a0e79cba3"), + _Claim(236, ('family', 'given'), "9393109ebf99"), "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix": - _Claim(751, ('family', 'given', 'suffix'), "231640fc7535"), + _Claim(864, ('family', 'given', 'suffix'), "7127402a5078"), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": _Claim(0, ('suffix',), "e3b0c44298fc"), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": @@ -1042,49 +1196,77 @@ def _claim(rule: dict) -> _Claim: "feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)": _Claim(2, ('family', 'middle'), "3e2b5c6d1f4d"), "feat(#273) typographic nickname delimiters recognized by default": - _Claim(6, ('middle', 'nickname'), "a03c9763c8c4"), + _Claim(8, ('middle', 'nickname'), "968bd4162257"), "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJK order flip": _Claim(6, ('family', 'given', 'nickname'), "ae1dffa01608"), "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7"), "fix(cjk-comma-honorific-peel) glued honorific peels off a post-comma given name": - _Claim(20, ('given', 'suffix'), "b2ea8fa59eea"), + _Claim(21, ('given', 'suffix'), "fdc02562bd15"), "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": - _Claim(20, ('family', 'given', 'middle', 'suffix', 'title'), "b2ea8fa59eea"), + _Claim(21, ('family', 'given', 'middle', 'suffix', 'title'), "fdc02562bd15"), "fix(cjk-glued-honorific-peel) glued honorific peels into suffix": - _Claim(34, ('family', 'given', 'suffix'), "877ab3246d33"), + _Claim(35, ('family', 'given', 'suffix'), "4da08a0090fe"), "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": _Claim(19, ('family', 'given', 'middle', 'suffix'), "aa475ddd4745"), "feat(#269) non-Latin titles/conjunctions recognized": - _Claim(2, ('given', 'middle', 'title'), "c14187bb08f8"), + _Claim(4, ('given', 'middle', 'title'), "e86eeb13eeb2"), "fix(leading-credential) a split 'Ph. D.' before the name stays one unit": _Claim(1, ('given', 'middle', 'suffix', 'title'), "390e7f814d13"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), + "fix(#400) abd joins the word after it as one given name": + _Claim(3, ('given', 'middle'), "6eb6b807523a"), + "fix(#272/#308) nakaguro division and a glued hangul honorific in one name": + _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122"), + "fix(nickname-typographic-pairs) two typographic quote spans read as one nickname set": + _Claim(1, ('family', 'given', 'middle', 'nickname'), "3cf566c78800"), }, "expected_since_2.0.0.toml": { "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(2, ('family', 'middle'), "f3a43bebfb91"), + _Claim(8, ('family', 'middle'), "f4d52b062f39"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(97, ('_ambiguities', 'family', 'given', 'middle'), "66e71d60a075"), + _Claim(105, ('_ambiguities', 'family', 'given', 'middle'), "090ac3eef3fb"), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": - _Claim(34, ('family', 'given', 'suffix'), "877ab3246d33"), + _Claim(35, ('family', 'given', 'suffix'), "4da08a0090fe"), "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2"), "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": - _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + _Claim(4, ('family', 'given', 'maiden', 'middle'), "27bb9ebc1951"), "fix(#272) nakaguro inside delimited content renders as a space, compounding with the CJK order flip": _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), + "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": + _Claim(1, ('family', 'suffix'), "081ce07f927b"), + "fix(#399) a maiden marker bounds the particle chain that swallowed it": + _Claim(4, ('family', 'maiden'), "32c8178f87c0"), + "fix(#360) mc moved into the never-given particles, so it folds into the family": + _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), + "fix(#400) abd joins the word after it as one given name": + _Claim(3, ('given', 'middle'), "6eb6b807523a"), + "fix(#367) a title no longer displaces a leading never-given particle": + _Claim(1, ('family', 'given'), "db724fb9c779"), + "fix(#272/#308) nakaguro division and a glued hangul honorific in one name": + _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122"), }, "expected_since_2.1.0.toml": { "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(2, ('family', 'middle'), "f3a43bebfb91"), + _Claim(8, ('family', 'middle'), "f4d52b062f39"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), + "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": + _Claim(1, ('family', 'suffix'), "081ce07f927b"), + "fix(#399) a maiden marker bounds the particle chain that swallowed it": + _Claim(4, ('family', 'maiden'), "32c8178f87c0"), + "fix(#360) mc moved into the never-given particles, so it folds into the family": + _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), + "fix(#400) abd joins the word after it as one given name": + _Claim(3, ('given', 'middle'), "6eb6b807523a"), + "fix(#367) a title no longer displaces a leading never-given particle": + _Claim(1, ('family', 'given'), "db724fb9c779"), }, } @@ -1161,9 +1343,13 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: #: gate reporting 108/0 throughout. #: #: The comma family is recorded because it is where that went wrong. -#: 42 corpus names are reachable by two or more rules in the same tier; -#: these ten are the ones whose boundaries this file argues about, and -#: pinning the argument is cheaper than re-deriving it. Note +#: Hundreds of corpus names are reachable by two or more rules in the +#: same tier -- 301 in the 1.4 tier as of the rules-doc corpus (#414), +#: up from 42, which is why the count is described rather than pinned: +#: it moves with every corpus addition and pinning it would only ever +#: be re-recorded. The rows below are the ones whose boundaries this +#: file argues about, and pinning the argument is cheaper than +#: re-deriving it. Note #: 'Andrews, M.D.' diffs on the SAME {given, suffix} shape as the seven #: peels -- only the CJK lookahead separates them, so it is the row #: that fails if that lookahead is ever dropped. @@ -1173,6 +1359,21 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: #: a diff shape that shifted is a finding, not a number to update. _CROSS_RULE_WINNERS: dict[str, dict[tuple[str, tuple[str, ...]], str]] = { "expected_since_1.4.0.toml": { + # Three behaviours that got named rules in the 2.0 and 2.1 + # ledgers but not this one, so here they fall to the fields-only + # catch-all. Recorded because the assignment is incidental + # rather than argued: 'Berg, Jan vd' is reachable by three + # named rules and wins none of them (their `fields` do not + # match), while 'Mc Donald' and 'Sir de Mesnil' are reachable + # by NO named rule at all. Being absorbed by the catch-all is + # the recoverable direction, but a rule added later that takes + # one of them silently would change what the 1.4 summary + # attributes -- and AGENTS.md names that summary as the source + # for the release log's Behavior Changes section. #380, #360 + # and #367 respectively. + ("Berg, Jan vd", ("family", "suffix")): "fix(suffix-routing)", + ("Mc Donald", ("family", "given")): "fix(suffix-routing)", + ("Sir de Mesnil", ("family", "given")): "fix(suffix-routing)", ("Andrews, M.D.", ("given", "suffix")): "fix(comma-family)", ("田中, 太郎さん", ("given", "suffix")): "fix(cjk-comma-honorific-peel)", ("김, 민준씨", ("given", "suffix")): "fix(cjk-comma-honorific-peel)", @@ -1313,7 +1514,7 @@ class _Excluded(NamedTuple): ("fix(comma-family)", "fix(comma-precomma-family)", "fix(suffix-routing)")), '(^|[\\w.]\\s+)[("\'][^)"\']+[)"\'](\\s+\\w|\\s*$)': - _Excluded(34, "9ea55c4c4382", ()), + _Excluded(42, "e594ae1a2e73", ()), } diff --git a/tools/differential/README.md b/tools/differential/README.md index b4108b59..46abd051 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -184,6 +184,7 @@ that stops being run. Pass `--corpus PATH` (repeatable) to narrow it. | `corpus.jsonl` | v1's own test suite at a pinned ref | anything 2.0 added — v1's authors had no reason to test a typographic nickname delimiter or a Cyrillic title | | `corpus_issues.jsonl` | name-like strings harvested from the GitHub issue tracker | anything nobody ever reported | | `corpus_cjk.jsonl` | the CJK-bearing rows of `tests/v2/cases.py`, via `build_cjk_corpus.py` (#295) | anything the case table itself missed — it re-witnesses reviewed expectations at the baseline boundary rather than discovering new shapes | +| `corpus_rules.jsonl` | every example in `docs/design/rules.md`, via `build_rules_corpus.py` (#414) | anything the rules doc has no example for — it re-witnesses the normative examples at the baseline boundary | They are deliberately separate rather than merged: `corpus.jsonl` is reproducible forever from an immutable git ref, while the issue @@ -196,12 +197,32 @@ table, and `tests/v2/test_ledger_guards.py` pins the checked-in file against the generator's selection, so a CJK case row added without regenerating fails the suite instead of silently narrowing this gate. +The rules corpus answers a different question from the doc tests that +already execute those examples. Those pin an example against an +expectation stored beside it, so a deliberate behavior change edits +both in one commit and the test stays green — a doc example cannot +warn about the change that edited it. Parsing the same name with a +*released* baseline can, because no commit can edit 1.4.0. It is +generated and pinned the same way the CJK corpus is. + The issue corpus earned its place on the first run — 166 of its 198 names were not in `corpus.jsonl`, and it immediately surfaced five intended-but-unclassified 2.0 behaviors (#273 typographic delimiters, #269 non-Latin vocabulary) plus one shape no test had considered: a **leading** `"Ph. D."`, which v1 split into title `Ph.` + given `D.`. +The rules corpus earned its place the same way — 113 of its 155 names +were in no other corpus, and on its first run 12 of them turned out to +have moved during the 2.2 cycle with nothing observing it. Classifying +them surfaced an unlogged behavior change (`mc` moving into the +never-given particles, which moves `Mc Donald`), two rules broad +enough to absorb a regression once a name of the right shape existed +(`fix(#274)`'s marker matching a *parenthesized* `Nee`, which is a +nickname under the default facade, and the acronym rule's dotted +`m\.?a\.?` reaching the real name `John Smith M.A.`), and two +pre-existing behaviors that had never had a corpus name to exercise +them. + ## Corpus provenance `corpus.jsonl` is checked in as a test fixture. It was built by diff --git a/tools/differential/build_rules_corpus.py b/tools/differential/build_rules_corpus.py new file mode 100644 index 00000000..6e711026 --- /dev/null +++ b/tools/differential/build_rules_corpus.py @@ -0,0 +1,106 @@ +"""Regenerate corpus_rules.jsonl from the examples in docs/design/rules.md. + +The fourth corpus, with the fourth provenance (#414). The other three +each have a blind spot the others cover, and this one closes a blind +spot none of them was built to have: + +- corpus.jsonl comes from v1's own test suite at a pinned git ref, so + it is structurally blind to anything 2.0 added. +- corpus_issues.jsonl comes from what users reported, which is the + adversarial half. +- corpus_cjk.jsonl comes from the CJK rows of the case table, because + neither of the above can carry an unspaced CJK name. +- this file comes from the NORMATIVE RULES -- the names the project + has written down as defining its behavior. + +Why the rules doc needs a corpus when its examples are already tests. +`test_rules_doc.py` executes every example, which pins them harder +than corpus membership pins anything -- but it pins them against an +expectation stored beside them. Change behavior deliberately and the +expected value is edited in the same commit; the test goes green and +says nothing. A doc example cannot warn about the change that edited +it. + +The differential run answers the other question. It parses these names +with a RELEASED baseline, which no commit can edit, so a moved name +arrives unexplained and has to be classified in a ledger in writing. +#409 is the worked example: it changed M2's + + "Jane de la née Jones" → family="de la née Jones" + +to family="de la". The doc test followed the edit, the name was in no +corpus, and nothing independent observed the movement. Twelve names +this corpus CONTRIBUTED moved during the 2.2 cycle the same way -- +thirteen of its names move in total, the thirteenth having already +been in corpus.jsonl. + +Selection is every distinct example `text`, through the doc's own +parser rather than a second regex -- the same argument that has +build_cjk_corpus.py select through the shipped script table. A rule +gaining an example enters the corpus by being written down where it +already gets reviewed. + +Policy annotations are deliberately ignored. The corpus carries name +STRINGS and the differential run parses them with the default facade, +so a family-first-scoped example is simply one more name to diff -- +build_cjk_corpus.py makes the same call for its zh-scoped rows. + +The one example form that carries no name is skipped. A `[subject]` +example names a policy or locale rather than a name string, so its +`text` is empty -- rules.md has three, and without the filter all +three collapse into a single "" that the doc never wrote. Skipping +them is also why the promise above is honest: an example enters the +corpus by being written down, and a form with nothing to enter is +better excluded than folded into a shared empty string. + +Boundary strings ("(", ".,", "Anna () Smith") are kept, on the +over-collection principle build_corpus.py and build_issues_corpus.py +state: a name that parses to nothing costs one parse and produces no +diff. They are more concentrated here than elsewhere because the doc +argues its edges explicitly, which is a reason to keep them rather +than to drop them. + +Regenerate after editing rules.md examples: + + uv run python tools/differential/build_rules_corpus.py + +tests/v2/test_ledger_guards.py pins the checked-in file against this +module's selection, so a stale corpus fails the suite rather than +silently narrowing the differential gate. +""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[1] +sys.path.insert(0, str(ROOT)) + +from tests.v2.rules_doc import parse_rules_doc # noqa: E402 + +OUT = HERE / "corpus_rules.jsonl" +RULES_DOC = ROOT / "docs" / "design" / "rules.md" + + +def selected_names() -> list[str]: + """Every distinct example text in the rules doc, sorted for a + deterministic file.""" + rules = parse_rules_doc(RULES_DOC.read_text(encoding="utf-8")) + return sorted({example.text + for rule in rules + for example in rule.examples + if example.text}) + + +def main() -> None: + names = selected_names() + with OUT.open("w", encoding="utf-8") as fh: + for name in names: + fh.write(json.dumps(name, ensure_ascii=False) + "\n") + print(f"wrote {len(names)} names to {OUT.name}") + + +if __name__ == "__main__": + main() diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 28f94380..2563c56a 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -358,6 +358,7 @@ def _is_latin_only(name: str) -> bool: "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref "corpus_cjk.jsonl": 95, # 97 today, generated from the case table "corpus_issues.jsonl": 190, # 200 today, harvested and append-only + "corpus_rules.jsonl": 150, # 155 today, generated from rules.md } @@ -763,6 +764,19 @@ def main() -> int: if not paths: raise SystemExit( f"no corpus files found in {HERE}: expected corpus*.jsonl") + # A file that DISAPPEARS is the same nightmare one step smaller: + # the glob simply finds fewer files, and the run compares less + # while printing a summary that reads exactly like a full one. The + # floors already name every corpus that is supposed to exist, so + # ask them. Skipped under --corpus, where narrowing is the point. + if not args.corpus: + missing = sorted(set(_CORPUS_FLOORS) - {p.name for p in paths}) + if missing: + raise SystemExit( + f"corpus files named in _CORPUS_FLOORS are not on disk: " + f"{missing}. A corpus that vanishes shrinks the " + f"comparison silently -- restore it, or drop its floor " + f"if it is meant to be gone") per_file = {} corpus = [] for path in paths: diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl new file mode 100644 index 00000000..7aaf5900 --- /dev/null +++ b/tools/differential/corpus_rules.jsonl @@ -0,0 +1,155 @@ +"'Smitty' John Jones" +"'Smitty' Jones" +"'Smitty' Jones Jr." +"(" +".," +"Ali Ahmad Vali oglu" +"Ali Ahmad Vali oglu Jr." +"Ali Ahmad oglu" +"Andrew (Andy) Perkins" +"Andrew Perkins (Andy)" +"Andrew Perkins (MBA)" +"Anh Do" +"Anh Van Do" +"Anna () Smith" +"Asst. Vice Chancellor John Smith" +"Beethoven, Ludwig van" +"Berg, Jan van der" +"Berg, Jan vd" +"Del Toro" +"Dr. John van der Berg" +"Dr. Juan Q. Xavier de la Vega III" +"Dr. Smith, John" +"Esq. Smith" +"Gal·la Serra" +"Garcia Juan Carlos" +"Hans „Erster“ und “Zweiter” Müller" +"Hassan Mohamad Ali" +"Hassan, Mohamad Ahmad Ali" +"J. Smith" +"J.R. Smith" +"Jack Wei Ma" +"Jane Smith (Nee)" +"Jane Smith (Nee) (Jones)" +"Jane Smith (née Jones)" +"Jane Smith née Jones" +"Jane Smith née Jones PhD" +"Jane Smith, née Jones" +"Jane de la née Jones" +"Jane née Jones Smith" +"Jane van der Berg née" +"Jane van der Berg née Jones" +"Jane van der Berg née PhD" +"Jane van der Berg née y Jones" +"Jane „JD Smith" +"Jean 'JD' Smith" +"John . Smith" +"John Doctor Smith" +"John Ma" +"John Smith" +"John Smith J.u.n.i.o.r." +"John Smith Jr." +"John Smith M.A." +"John Smith PhD" +"John Smith Q.W.E.R.T." +"John Smith, LEED AP" +"John Smith, MD, Bart" +"John Smith, MD,, Jr." +"John Smith, PhD" +"John Smith, V." +"John van der Berg" +"John van der Berg PhD" +"John van der Berg Smith" +"John van der Berg née Jones" +"John😀Smith" +"Jones née" +"Jong Anke de" +"Jong, Anke de" +"Jose E Maria Santos" +"Juan & Garcia" +"Juan McDonald" +"Juan and Garcia" +"Juan de" +"Juan de la Vega" +"Juan de y" +"Juan van der" +"Juan y Eva Garcia" +"Juan y Garcia" +"Mari' Aube'" +"Marquess of Bath" +"Mary Beth Smith" +"Mc Donald" +"Mesnil de" +"Morse, Det. Insp. Jane" +"Mr. Jack Jill" +"Mr. Jack and Jill" +"Mr. Johnson" +"Mrs. Garcia" +"Nguyen Thi Van" +"Nguyen, Thi Van" +"Nguyen, Van" +"Nguyen, Van Le" +"Nguyễn Thị Minh Khai" +"Nguyễn, Thị Vân" +"Rev. John Smith" +"Salam, abd Allah" +"Sean O'Connor" +"Sidorov Ivan Petrovich Jr." +"Sir John" +"Sir de Mesnil" +"Smith Jr." +"Smith, Abd" +"Smith, John" +"Smith, Major. John" +"Smith, PhD" +"Smith, de Mesnil Juan" +"Van Johnson" +"Vega, Juan de la" +"Vincent van Gogh van Beethoven" +"Xyz. John Smith" +"Xyz. Smith, John" +"abd Allah" +"abd Allah Smith" +"abdul salam ahmed salem" +"de" +"de Mesnil Juan" +"de la Cruz Juan Carlos" +"de la Vega" +"de la Vega y Santos Juan" +"de los Santos" +"ibn Awf abdul Rahman" +"juan mcdonald" +"mohamad ali smith" +"née Jones" +"van Gogh" +"van der Berg, abdul née Jones" +"Иван Петрович Абрамович" +"Сидоров Иван Петрович" +"Хосе И Мария Сантос" +"سلمان، محمد" +"প্রফেসর. Sen" +"マイケル ジャクソン" +"マイケル・ジャクソン" +"威廉·莎士比亚" +"威廉・莎士比亚" +"安东尼·陈志明" +"山田 太郎" +"山田 花子 旧姓：佐藤" +"毛·泽东" +"毛泽东" +"王君" +"田中、太郎" +"田中さん" +"田中さん, V." +"马丁·路德·金씨" +"高橋 みなみ" +"高橋・一郎" +"高橋一郎" +"김 민준" +"김, 민준씨" +"김민준" +"김지양" +"남궁민수" +"남궁민수 지훈" +"남궁민수, 지훈" +"지훈, 남궁민수" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index ad8976f3..77035568 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -98,7 +98,35 @@ issue = "fix(#274) maiden markers consumed" # tests/v2/cases.py calls it the spelling English writes most often. # See maiden_markers.py's own "Deliberately absent" note for what has # actually been considered and declined. -name_regex = "(?i)\\b(n[ée]e|geb\\.?|roz\\.?)\\b" +# +# The lookarounds and the two context tests are not decoration. corpus_rules.jsonl (#414) +# brought in 'Jane Smith (Nee)' and 'Jane Smith (Nee) (Jones)', which +# are M1's examples of a one-word clause keeping a word that may +# itself be a surname. Under the DEFAULT facade -- which is what the +# differential parses with -- no maiden delimiter is configured, so +# the parenthesized word is a NICKNAME and no marker is active at all. +# Without the lookarounds this rule matched the word anyway and stood +# ready to explain a `maiden` diff on a name that has no marker in it, +# which is absorption waiting to happen. The vocabulary-presence guard +# in test_ledger_guards.py is what caught it. +# +# The same corpus addition also put M2's BOUNDARY examples in reach -- +# 'Jones nee', 'nee Jones', 'Jane van der Berg nee', 'Muller geb.' -- +# names M2 states as unchanged, because a marker with nothing before +# or after it is just a word. Nothing guards that: those names really +# do carry marker vocabulary, so the presence guard is satisfied. +# Hence the two context tests, which say what this rule has always +# meant: a marker with a name word BEFORE it and one AFTER it. Reach +# 16 -> 11, and all seven names it explains are inside the eleven. +# +# Four claimed names still do not move, recorded rather than excluded: +# 'Jane Smith, nee Jones' (post-comma, M2's remaining Accepted limit), +# 'Jane van der Berg nee PhD' (only a suffix follows, so the consumer +# declines), and the connective and bound-given joins (#412). Each +# would need the regex to ask a vocabulary question -- suffix or +# connective membership -- a second alternation this rule cannot +# spend. +name_regex = "(?i)(?