From 38ed3feb3617fe2c29a9737dc8e4b92a73d6e4d6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 8 Sep 2026 21:49:14 -0700 Subject: [PATCH 01/12] fix(pieces): the leading title peel leaves a name word a suffix cannot be The leading peel runs before the trailing suffix peel and its only floor was "leave one piece", so `Dr King Jr` peeled `Dr King` and left `Jr` to be the name -- title 'Dr King', family 'Jr', no suffix. Derek's question put it plainly: Jr is a recognized suffix, so it could not count as a following name. leading_titles gains a second floor. Where everything behind the run is suffix pieces and the run's last word is not itself one, the run gives that word back. The word given back must be a name CANDIDATE, which is what leaves the all-suffix inputs alone -- `MD DDS` and `Jr. Ph. D.` are unchanged because `md` and `jr` are suffix vocabulary -- and the all-title ones too, there being no rest for the floor to read. It lands in the leaf rather than in assign because leading_titles is the one predicate for the leading run (mechanisms.md#ONE-PREDICATE-PER-QUESTION): assign sets the roles and group's chain reads the same count. `Dr King Jr` now reads title 'Dr', family 'King', suffix 'Jr' -- what v1 wanted, what rules.md#S2's descriptive note predicted, and what the comma form `King, Dr Jr` has always given. `Dr. King MD` moves the same way. Both report `title-or-name`, `king` being title vocabulary and now being the word left standing. One corpus name moves, measured over all 1123. Accepted edges: `Dr Jr` reads given 'Dr', suffix 'Jr' and `Sir Jr` given 'Sir', suffix 'Jr' -- with the title given back there is no title left to make either reading H1's. Two things the measurement forced, beyond the shape above. The floor's gate is an inline tag read rather than is_suffix_piece: leading_titles runs four times per parse, and asking the predicate first cost 8 frames against a band with room for two, so the two tag tests -- the cheapest NECESSARY condition for the next piece to be a suffix piece at all -- keep the ordinary titled name out of the branch entirely. Frame delta is now zero on both entry points. And under test_parser's overlap lexicon `Dr. Do Jr.` moves the same way `Dr. King MD` does, so the no-op-prefix-chain test names the fork it is about (PARTICLE_OR_GIVEN) instead of asserting silence: the report it now sees is H4's, about the word the floor left standing. Review round. The give-back is gated to a ONE-WORD piece: without it `Prince of Wales Jr` gave up its title entirely (given `Prince of Wales`, no title), where 1.4.0 and the pre-commit tree both read title `Prince of Wales`, family `Jr` -- a new parity row, `title_run_floor_keeps_a_joined_title_unit`, and a leaf test pin it, and the frame count is unmoved at 416 parse / 453 facade on 3.11. The docstring and the comment now say "its last piece, when that piece is one word" rather than "its last word", claim only that the piece given back is a name candidate, and name two residuals: a run whose last word IS suffix vocabulary is not given back (`Dr King MD PhD` keeps title `Dr King MD`), and the floor asks is_suffix_piece, which vetoes a bare initial-shaped numeral, so `Dr King V` still reads family `V`. The 8-call history is marked as measured on the plan's ordering, the committed shape having since hoisted the pieces[n-1] test into the guard. Three test notes were wrong: the `Sir Jr` row is renamed `title_run_floor_gives_back_a_given_name_title_run` and says it carries rules.md#S2's provenance rather than a second mechanism, the `MD DDS` row no longer calls itself the bare-suffix carve-out (that is `DDS MD`; `MD DDS` reads title `MD` plus H1's fold and reports nothing), and test_king drops its duplicated "v1 aspired" sentence for what `king` being title vocabulary still buys. test_parser's no-op chain test pins full silence on five rows and the exact `TITLE_OR_NAME` tuple on `Dr. Do Jr.` -- superseding the paragraph above, which describes the filter-one-kind-out shape it replaced. Co-Authored-By: Claude Fable 5.1 --- nameparser/_pipeline/_pieces.py | 53 +++++++++++++++++++-- tests/test_suffixes.py | 36 +++++++------- tests/v2/cases.py | 82 ++++++++++++++++++++++++++------ tests/v2/pipeline/test_pieces.py | 53 ++++++++++++++++++++- tests/v2/test_parser.py | 13 ++++- 5 files changed, 201 insertions(+), 36 deletions(-) diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 001d3abc..a4672293 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -85,10 +85,17 @@ def leading_titles(pieces: Sequence[Sequence[int]], tokens: Sequence[WorkToken]) -> int: """How many leading pieces assign peels as titles: the first non-title index. A title needs a following piece, unless the whole - segment is one title (v1 parity). One definition, read by assign - (which sets the roles) and by the chain's trailing-run walk; the - leading-particle scan shares the predicate, is_leading_title, - but stops at a title-and-particle word (P4, #367, #424).""" + segment is one title (v1 parity). And the run gives back its last + piece when that piece is a name candidate: where everything behind + the run is suffix pieces, the run gives back its last piece, when + that piece is one word and is not itself suffix vocabulary -- the + one-word half is what leaves 'Prince of Wales Jr' alone and the + vocabulary half what leaves 'MD DDS' and 'Jr. Ph. D.' alone + (rules.md#H3, decisions.md#H3). + One definition, read by assign (which sets the roles) and by the + chain's trailing-run walk; the leading-particle scan shares the + predicate, is_leading_title, but stops at a title-and-particle + word (P4, #367, #424).""" n = 0 while n < len(pieces): if ((n + 1 < len(pieces) or len(pieces) == 1) @@ -96,6 +103,44 @@ def leading_titles(pieces: Sequence[Sequence[int]], n += 1 continue break + # rules.md#H3 -- the run gives back its last piece when that piece + # is a name candidate: where everything behind the run is a suffix + # piece, the run hands its last piece back, provided that piece is + # one word and is not itself suffix vocabulary. + # + # ONE WORD, because a joined unit led by a title is a title run and + # handing it back would lose the title: 'Prince of Wales Jr' reads + # title 'Prince of Wales', family 'Jr', not given 'Prince of Wales' + # with no title at all. + # + # Two residuals. A run whose last word IS suffix vocabulary is not + # given back, so 'Dr King MD PhD' still reads title 'Dr King MD', + # family 'PhD'. And the floor asks is_suffix_piece, which vetoes a + # bare initial-shaped numeral, so 'Dr King V' still reads family + # 'V' -- that numeral fork is outside this floor. + # + # The two inline tag reads are the cheapest NECESSARY condition for + # the piece behind the run to be a suffix piece at all -- + # is_suffix_piece cannot answer yes without one of them -- so the + # ordinary titled name, whose next piece is no kind of suffix, + # leaves this branch without entering a frame. Measured on the + # plan's ordering rather than the shape below: asking the + # authoritative predicate first cost 8 calls per parse of the + # reference name (leading_titles runs four times), against a band + # with room for two (decisions.md#parse-cost). is_suffix_piece + # stays the predicate that ANSWERS, here and in the walk. + if (n and n < len(pieces) + and ("suffix" in ptags[n] + or "vocab:suffix" in tokens[pieces[n][0]].tags) + and len(pieces[n - 1]) == 1 + and not is_suffix_piece(pieces[n - 1], ptags[n - 1], + tokens)): + k = n + while k < len(pieces) and is_suffix_piece(pieces[k], ptags[k], + tokens): + k += 1 + if k == len(pieces): + n -= 1 return n diff --git a/tests/test_suffixes.py b/tests/test_suffixes.py index f8f81104..f97f3e83 100644 --- a/tests/test_suffixes.py +++ b/tests/test_suffixes.py @@ -152,27 +152,31 @@ def test_king(self) -> None: # 'king' stays in the titles vocabulary, because removing it breaks # the addressing forms it is there for ("King Charles"). # decisions.md#vocabulary-collisions cuts toward keeping it, so the - # title chain takes 'Dr King'. The comma format is the road to the - # surname reading, pinned below. - # RECORDED, NOT ENDORSED: what becomes of the leftover 'Jr'. - # rules.md#S2 has a trailing suffix-vocabulary word read as a suffix, - # and its Accepted clause consumes one even when that leaves no family - # at all, so S2 predicts suffix 'Jr', family ''. Once the title - # chain has eaten two words, H1 claims the one that remains and it - # reads family 'Jr' instead. Compare 'Dr Smith Jr', which reads - # family 'Smith', suffix 'Jr' exactly as S2 states. A change moving - # this toward S2's prediction is an IMPROVEMENT that updates this - # pin, not a regression. + # title chain reaches 'King' at all. The comma format reads the + # surname without any of that, and is pinned below as the contrast. + # WHAT BECOMES OF THE LEFTOVER 'Jr': the improvement this pin + # anticipated, taken 2026-09-08 (#489, decisions.md#H3). The title + # chain used to eat two words and leave 'Jr' to be the name, against + # rules.md#S2, which reads a trailing suffix-vocabulary word as a + # suffix -- and against 'Dr Smith Jr', which reads family 'Smith', + # suffix 'Jr' exactly as S2 states. The leading peel now leaves a + # name word a suffix cannot be: everything behind the run is a + # suffix piece and 'King' is not one, so the run gives it back. + # Every field below is now what 'Dr Smith Jr' reads, so what + # 'king' being title vocabulary still buys this row is the + # 'title-or-name' report on the word left standing -- which is a + # 2.0 surface this v1 facade test does not assert. hn = HumanName("Dr King Jr") - self.m(hn.title, "Dr King", hn) + self.m(hn.title, "Dr", hn) self.m(hn.first, "", hn) self.m(hn.middle, "", hn) - self.m(hn.last, "Jr", hn) - self.m(hn.suffix, "", hn) + self.m(hn.last, "King", hn) + self.m(hn.suffix, "Jr", hn) def test_king_as_a_family_name_via_the_comma_format(self) -> None: - # The workaround test_king's decision rests on: writing the family - # first defeats the title chain and reads 'King' as the family. + # The contrast: writing the family first defeats the title chain + # outright, and reads 'King' as the family without needing the peel + # floor above. Unchanged by #489 -- both roads now arrive together. hn = HumanName("King, Dr Jr") self.m(hn.title, "Dr", hn) self.m(hn.last, "King", hn) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 587e02aa..de5c0955 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -2322,21 +2322,75 @@ def _check_cjk_shape_purity(self) -> None: "gives the same reading and the same kind. The site was " "H1's retag until the family-first orders were measured " "silent there"), - Case("title_run_then_a_credential_reports_nothing", "Dr. King MD", - {"title": "Dr. King", "family": "MD"}, classification="parity", - notes="rules.md#H4's boundary, both halves -- the title peel " - "took `Dr. King` and left no name word at all, so the " - "credential is the name by the bare-suffix carve-out -- " - "which is scoped to a run no title preceded, and the " - "title half needs a name word left standing. Neither " - "claims it"), - Case("title_run_then_a_bare_generational_reports_nothing", - "Dr King Jr", {"title": "Dr King", "family": "Jr"}, + Case("title_run_then_a_credential_reports_the_name_word", + "Dr. King MD", + {"title": "Dr.", "family": "King", "suffix": "MD"}, + ambiguities=("title-or-name",), classification="fix(#489)", + notes="rules.md#H3's floor: everything behind the run is a " + "suffix piece and `King` is not one, so the run gives " + "it back and the credential is a credential. `king` " + "being title vocabulary, the word left standing is " + "H4's title half -- the same reading `Dr. King, Jr.` " + "has always had"), + Case("title_run_then_a_bare_generational_reports_the_name_word", + "Dr King Jr", {"title": "Dr", "family": "King", "suffix": "Jr"}, + ambiguities=("title-or-name",), classification="fix(#489)", + notes="v1 wanted exactly this reading and the 2026-09-01 " + "triage pinned the old one as NOT FIXED; rules.md#S2's " + "descriptive note said a change toward S2's prediction " + "would be an improvement, and this is it"), + Case("title_run_floor_gives_back_a_given_name_title", + "Dr Jr", {"given": "Dr", "suffix": "Jr"}, + ambiguities=("title-or-name",), classification="fix(#489)", + notes="the accepted edge: with `Dr` given back and `Jr` " + "peeled as the suffix, no title is left to make the " + "reading H1's, so the lone name word is H4's -- " + "`dr` is title vocabulary and the word standing is it"), + Case("title_run_floor_gives_back_the_last_of_a_run", + "Lord Chancellor Jr", + {"title": "Lord", "family": "Chancellor", "suffix": "Jr"}, + ambiguities=("title-or-name",), classification="fix(#489)", + notes="the floor takes back ONE word, the run's last, and " + "`Lord Chancellor` is the input decisions.md#H4 already " + "uses for the all-titles convention"), + Case("title_run_floor_keeps_a_joined_title_unit", + "Prince of Wales Jr", + {"title": "Prince of Wales", "family": "Jr"}, + classification="parity", + notes="the floor gives back its last piece only when that " + "piece is ONE WORD: a joined unit led by a title is a " + "title run, and handing it back would turn a title into " + "a given name and leave the name with no title at all"), + Case("title_run_floor_declines_an_all_suffix_input", "MD DDS", + {"title": "MD", "family": "DDS"}, classification="parity", + notes="negative control: the word the run would give back is " + "`MD`, which IS suffix vocabulary, so the floor " + "declines. `md` is title vocabulary too, so the run is " + "`MD` and H1's fold claims `DDS` -- not the bare-suffix " + "carve-out, which reads the FIRST word as the name and " + "reports `suffix-or-name` (`DDS MD`). Nothing reports " + "here"), + Case("title_run_floor_declines_a_split_credential", "Jr. Ph. D.", + {"title": "Jr.", "suffix": "Ph. D."}, classification="parity", + notes="negative control: `Jr.` is suffix vocabulary wearing " + "H2's opening-abbreviation shape, so the floor declines " + "there too"), + Case("title_run_floor_declines_an_all_title_input", + "Marquess of Bath", {"title": "Marquess of Bath"}, classification="parity", - notes="the same boundary in the shape the assign comment " - "names: `Dr King` peels whole, `Jr` is read as the name " - "for want of another, and after a title that reading is " - "H1's rather than either of H4's conventions"), + notes="negative control: nothing stands behind the run, so " + "there is no all-suffix rest for the floor to see"), + Case("title_run_floor_gives_back_a_given_name_title_run", "Sir Jr", + {"given": "Sir", "suffix": "Jr"}, + ambiguities=("title-or-name",), classification="fix(#489)", + notes="`Sir Jr` is the input rules.md#S2's descriptive note " + "names -- it read given `Jr` and now reads given `Sir`, " + "suffix `Jr`. The row is here for that provenance and " + "not for a second mechanism: the reading is `Dr Jr`'s " + "in every part -- same roles, same kind, the detail " + "being the word left standing -- because once the floor " + "empties the run no branch reads `vocab:given-title` " + "at all"), Case("all_suffix_input_reports_suffix_or_name", "Rinpoche", {"given": "Rinpoche"}, ambiguities=("suffix-or-name",), classification="feat(#491)", diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index 0f9a8db6..08b26708 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -10,7 +10,8 @@ from nameparser._pipeline._classify import classify from nameparser._pipeline._group import group from nameparser._pipeline._pieces import ( - _numeral_behind_the_initial_veto, segment_suffix_reading, + _numeral_behind_the_initial_veto, leading_titles, + segment_suffix_reading, ) from nameparser._pipeline._segment import segment from nameparser._pipeline._state import ParseState @@ -102,3 +103,53 @@ def test_strict_ends_the_run_at_the_initial_shaped_numeral() -> None: args = (state.pieces[1], state.piece_tags[1], list(state.tokens)) assert segment_suffix_reading(*args, True) == (True, True) assert segment_suffix_reading(*args, False) is None + + +def _leading(text: str) -> int: + state = _through_group(text) + return leading_titles(state.pieces[0], state.piece_tags[0], + list(state.tokens)) + + +def test_the_leading_run_gives_back_the_name_word_a_suffix_cannot_be() -> None: + """The peel's second floor, at the predicate. + + The run is counted before the trailing suffix peel runs, so its + only floor was "leave one piece" and a run in front of nothing but + suffix pieces took the last name word with it. The floor gives one + word back -- and only where the rest really is all suffix pieces, + which is what separates the first two readings below. + """ + assert _leading("Dr King Jr") == 1 # the floor fired: 2 -> 1 + assert _leading("Dr King") == 1 # no all-suffix rest to see + assert _leading("Lord Chancellor Jr") == 1 + assert _leading("Lord Chancellor") == 1 + + +def test_the_leading_run_declines_to_give_back_a_suffix_word() -> None: + """The three shapes the floor must not touch. + + The word given back has to be a name CANDIDATE, so a run whose + last word is itself suffix vocabulary keeps it; and a run with + nothing behind it has no all-suffix rest to read at all, which is + what leaves the whole-segment carve-out and the all-title inputs + exactly as they were. + """ + assert _leading("MD DDS") == 1 # 'MD' is suffix vocabulary + assert _leading("Jr. Ph. D.") == 1 # so is 'Jr.' + assert _leading("Marquess of Bath") == 1 # nothing behind the run + assert _leading("Dr.") == 1 # the whole-segment carve-out + + +def test_the_leading_run_keeps_a_joined_unit_it_cannot_give_back() -> None: + """The piece given back has to be ONE WORD. + + A joined unit led by a title is a title run in its own right, so + handing it back would turn the title into a given name and leave + the name with no title at all -- 'Prince of Wales Jr' reads title + 'Prince of Wales', family 'Jr'. The run behind it is all suffix + pieces and its last piece is no kind of suffix, so every other + condition of the floor is met and only the length gate declines. + """ + assert _leading("Prince of Wales Jr") == 1 + assert _leading("Prince of Wales") == 1 diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 30fd64e1..2b093d11 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -697,7 +697,18 @@ def test_the_p5_licence_and_h1_read_a_title_run_the_same_way( "Dr. Van Jr.", "Dr. Van MD", "Dr. Do Jr.", ]) def test_no_op_prefix_chain_is_not_a_fork(text: str) -> None: - assert _overlap_parser().parse(text).ambiguities == () + # Five rows stay fully silent. The sixth is pinned to its exact + # report instead, because since #489 it reports from somewhere else + # entirely: the leading peel's floor gives 'Do' back as the name + # word (the run stood in front of nothing but 'Jr.'), so + # 'Dr. Do Jr.' reads family 'Do', suffix 'Jr.' and H4's title half + # claims the lone name word, which happens to be title vocabulary + # here. That report is about the word left standing, not about a + # chain that never chained -- PARTICLE_OR_GIVEN is still absent. + expected = ((AmbiguityKind.TITLE_OR_NAME,) + if text == "Dr. Do Jr." else ()) + assert tuple(a.kind for a in + _overlap_parser().parse(text).ambiguities) == expected def test_a_fork_is_reported_by_exactly_one_stage() -> None: From fdf9b86d6496a4e27b7f08c41bbd54617b4b0438 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 8 Sep 2026 22:17:41 -0700 Subject: [PATCH 02/12] fix(lexicon): a title run addresses as its last title does `Her Majesty Queen Elizabeth` read family 'Elizabeth' because the given-name-title lookup was keyed on the whole run -- 'her majesty queen' is no shipped entry. A run is written as several titles and addresses the way its final one does, so the run's folded key is now matched whole OR by its last word. The whole-run ARM is what keeps a caller's multi-word phrase entry working: `lt col` is stored and matched as one key. Its order carries nothing -- the `or` short-circuits but decides nothing, since for a one-word run the key IS its last word. Every shipped given-name title is a single word, now asserted in test_lexicon.py, so that arm is dead for the default set: it can only ever match a one-word run, which the last-word arm reads the same way. Both sites move together, through one predicate in _lexicon beside _title_key, because the 2026-08-22 #369 decision rests on H1 and P5's licence never reading one run two ways. That invariant holds under last-word keying, and 'mr sir' -- #369's own example of a run neither site treats as a given-name title -- now IS one to both. The leaf test that pinned the old reading of that example, test_group's test_the_title_run_is_one_key_as_h1_reads_it, is renamed test_the_title_run_is_read_as_h1_reads_it and pins the invariant rather than the key's shape: 'mr sir abdul rahman' now joins. `Reverend Mother Teresa`, `Dr. Sir John`, `Mr Sir John` and `Xyz. Sir John` move with it, and through P5's licence `Sir Sheikh abdul rahman` reads given 'abdul rahman' with no family. What does not move: `His Excellency Lord Duncan` and `Her Royal Highness Princess Anne` -- `lord` and `princess` are not given-name titles, and whether they should be is a vocabulary question with its own frequency argument, deliberately left alone. One corpus name moves, measured over all 1123. The v1 suite's #489 xfail is dropped; xfail_strict would fail on the XPASS otherwise. Frame delta is zero on both entry points (416 parse / 453 facade on 3.11, unmoved): each site's read sits inside the branch it serves, and the reference name `Dr. Juan0000 de la Vega III` enters neither -- H1's guard wants an unoccupied family, P5's a bound given-name word. Review round. The predicate is one fold and one pair of lookups over that key, no list and no empty-run guard, measured unmoved on both counts. The docstrings drop the ordering language and say what the two arms reach instead: the last word is the last word of the FOLDED key, so a run token that folds away (the lone '.' the conjunction merge can leave, `Sir and . John`) cannot empty that arm; and an unlisted abbreviation can never MATCH as a last-word key, but may sit inside a caller's phrase entry, since given_name_titles is deliberately not validated against `titles`. _title_key's own claim that a lone '.' is not a title token was false and now says the fold drops it. post_rules' two-level `if` collapses to one condition and cites decisions.md#P5, where the #369 entry lives; group's comment is re-wrapped; test_lexicon's phrase-entry test folds into the per-word fold test it duplicated; and test_group's conjunction test is renamed for the mutation it catches, keyed over every token of the piece. Co-Authored-By: Claude Fable 5.1 --- nameparser/_lexicon.py | 73 +++++++++++++++++++++++------ nameparser/_pipeline/_group.py | 33 +++++++------ nameparser/_pipeline/_post_rules.py | 32 ++++++++----- tests/test_conjunctions.py | 1 - tests/v2/cases.py | 61 ++++++++++++++++++++++++ tests/v2/pipeline/test_group.py | 18 ++++--- tests/v2/test_lexicon.py | 24 ++++++++++ tests/v2/test_parser.py | 9 ++-- 8 files changed, 199 insertions(+), 52 deletions(-) diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index a95a873a..c225ad2a 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -11,7 +11,7 @@ import functools import sys import warnings -from collections.abc import Iterable, Mapping +from collections.abc import Iterable, Mapping, Set from dataclasses import dataclass, field from types import FrameType, MappingProxyType from typing import cast @@ -83,7 +83,11 @@ #: The two are not the same mechanism, and the difference is why the #: exemption is a list rather than a rule. A given_name_titles run is #: identified per word FIRST -- 'lt' and 'col' are each title -#: vocabulary -- and only then joined and looked up. A maiden marker +#: vocabulary -- and only then looked up, as the whole run's key or as +#: its LAST word's (_run_addresses_by_given, #489). The whole-run arm +#: is the one a phrase entry is matched by; the last-word arm is a +#: single word and cannot reach a phrase, so storage stays what it was +#: (the two are asked together, not in precedence). A maiden marker #: phrase has no such per-word foothold: 'z' and 'domu' are not markers #: individually, and adding them separately (which this warning used to #: advise) reads 'Maria Kowalska z domu Nowak' as maiden 'domu Nowak' @@ -129,22 +133,62 @@ def _title_key(words: Iterable[str]) -> str: A multi-word title is matched as one key ('lt col'), so the fold has to run per word and rejoin -- _normalize on the whole phrase would leave interior periods. Defined once because it is built at match - time (post_rules for H1, group for the P5 licence -- which must - agree, see #369) and at translation time (the v1 facade's - first_name_titles), and a divergence between them fails silently: - the entry simply stops matching. + time (_run_addresses_by_given, which is how post_rules' H1 and + group's P5 licence both reach it) and at translation time + (_config_shim's first_name_titles), and a divergence between them + fails silently: the entry simply stops matching. Words that fold away are DROPPED, not joined as empty. Keeping the gap makes the fold non-idempotent -- 'lt .' would store 'lt ', which - match time can never build (post_rules joins token texts, and a lone - '.' is not a title token), so the entry is inert. Storage re-runs - this fold on unpickle and on every dataclasses.replace, so a value - that changes under a second pass is one Lexicon later rejects as + match time can never build: a run CAN carry a lone '.' (the + conjunction merge puts one there), but the fold drops the empty + word, so the key is 'lt' and the stored entry is inert. Storage + re-runs this fold on unpickle and on every dataclasses.replace, so a + value that changes under a second pass is one Lexicon later rejects as "not written by this version". _normalize converges for the same reason; so must anything built on top of it.""" return " ".join(filter(None, (_normalize(w) for w in words))) +def _run_addresses_by_given(words: Iterable[str], + vocabulary: Set[str]) -> bool: + """Whether a run of title words addresses by the GIVEN name. + + Two sites ask it and they must agree -- post_rules for H1, group + for the P5 licence -- because a run read two ways is a rule + contradicting itself (decisions.md#P5, the 2026-08-22 #369 entry). + One predicate, so they cannot drift. + + The WHOLE run's key, or that key's LAST word (#489): a run is + written as several titles and addresses the way its final one does, + so 'Her Majesty Queen' addresses by given name because 'queen' + does. The two arms are asked of one key and neither is the other's + fallback -- the `or` short-circuits but decides nothing, since for + a one-word run the key IS its last word. The last word is the last + word of the FOLDED key, not of the raw run, so a run token that + folds away cannot empty that arm: the conjunction merge can put a + lone '.' in the run ('Sir and . John'), and the fold drops it. + + The whole-run arm is what keeps a caller's multi-word phrase entry + working: 'lt col' is stored as one key and matched as one run. Over + the SHIPPED vocabulary it is dead -- every shipped entry is a single + word (asserted in test_lexicon.py), so it can only ever match a + one-word run, which is a run the last-word arm reads the same way. + + H2's unlisted abbreviations ride in the run. One can never match as + the last-word key, being in no vocabulary by definition: 'Xyz. Sir' + keys 'sir' and matches, 'Sir Xyz.' keys 'xyz' and does not. It CAN + sit inside a whole-run key that matches, because given_name_titles + is deliberately not validated against titles: a caller may store + 'sir xyz', and 'Sir Xyz. John' then reads given. + + The vocabulary is passed in rather than read off a default: a + caller's own Lexicon is the one that has to be consulted, and this + module is where Lexicon is defined.""" + key = _title_key(words) + return key in vocabulary or key.rpartition(" ")[2] in vocabulary + + def _reject_buffer(value: object, label: str, plural: str) -> None: """Binary sequences iterate to INTS, so every downstream entry check reports a byte value -- "must be strings, got 100" for b'dr', where @@ -327,10 +371,11 @@ class Lexicon: stripped -- so matching is case-insensitive. Vocabulary entries are single words -- a multi-word entry warns at construction and can never match. Two fields are exempt, and they differ in HOW they - match: ``given_name_titles`` is looked up as the space-joined run - of words the parse has ALREADY read as titles, while - ``maiden_markers`` is matched by lookahead, longest first, over - words that need not be markers on their own (``"z domu"``). + match: ``given_name_titles`` is looked up against the run of words + the parse has ALREADY read as titles -- the whole run space-joined, + or that run's last word -- while ``maiden_markers`` is matched by + lookahead, longest first, over words that need not be markers on + their own (``"z domu"``). Field docs below show examples, not full contents; inspect any field's shipped vocabulary directly, e.g. ``Lexicon.default().conjunctions``.""" diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 75895e6e..1e1bdbce 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -42,7 +42,7 @@ from collections.abc import Iterable, Sequence, Set from enum import IntEnum -from nameparser._lexicon import _title_key +from nameparser._lexicon import _run_addresses_by_given from nameparser._pipeline._pieces import ( is_leading_title, is_suffix_piece, is_title_piece, leading_titles, peel_trailing, peel_walk, trailing_start, @@ -738,21 +738,26 @@ def chain(tail: int) -> None: # A given-name title ahead of the bound word asserts # that a given name follows -- the assertion H1 reads # when it keeps "Sir John" a given name -- so behind - # one there is no family to spare (#369). Keyed on the - # WHOLE title run exactly as post_rules keys H1, so the - # two rules cannot disagree about what one run asserts - # (post_rules' run also takes H2's unlisted - # abbreviations, which no given-name title key can - # contain, so the runs match whenever the key does). - # The licence lifts the reserve for two name WORDS: the - # piece the join would take must be one word -- a - # particle chain is the family name P2 built ('Sir + # one there is no family to spare (#369). Asked of the + # title run through the ONE predicate post_rules asks + # for H1, so the two rules cannot disagree about what + # one run asserts; what it reads is the whole run's key + # or that key's LAST word (#489). H2's unlisted + # abbreviations ride in the run either way. One is in + # no vocabulary by definition, so it never matches as + # the last-word key -- 'Xyz. Sir' keys 'sir' and 'Sir + # Xyz.' keys 'xyz' -- but a caller's phrase entry may + # contain one, and the whole-run arm is what matches + # that. The licence lifts the reserve for two name + # WORDS: the piece the join would take must be one word + # -- a particle chain is the family name P2 built ('Sir # abdul van der Berg' keeps family 'van der Berg'). licensed = (fk > 0 and len(pieces[fk + 1]) == 1 - and _title_key(tokens[i].text - for k in range(fk) - for i in pieces[k]) - in given_name_titles) + and _run_addresses_by_given( + (tokens[i].text + for k in range(fk) + for i in pieces[k]), + given_name_titles)) reserve = BoundJoin.LENIENT if licensed else BoundJoin.STRICT if same_suffixes and after.names >= reserve: # the pair is a given name whatever tag the word diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 42d417b7..37b119a7 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -23,7 +23,7 @@ import dataclasses import re -from nameparser._lexicon import _title_key +from nameparser._lexicon import _run_addresses_by_given from nameparser._pipeline._assign import _name_positions from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, _NEVER_FLIPPED, @@ -359,17 +359,25 @@ def post_rules(state: ParseState) -> ParseState: # further name words is what emptied the family (#410) # (known gap: the guard tests which roles are unoccupied, it does # not count units -- decisions.md#H1) (v1 handle_firstnames) - if titles and givens and not middles and not families: - joined = _title_key(tokens[i].text for i in titles) - if joined not in state.lexicon.given_name_titles: - for i in givens: - _retag(tokens, i, Role.FAMILY) - # every rule below reads these lists; recompute after any - # retag so no guard can inspect a name that has already - # moved -- a stale index list is the bug shape #359 fixed - givens = _idx(tokens, Role.GIVEN) - middles = _idx(tokens, Role.MIDDLE) - families = _idx(tokens, Role.FAMILY) + # + # rules.md#H1 -- a RUN of several titles addresses as its last + # title does (#489), so 'Her Majesty Queen Elizabeth' reads given + # 'Elizabeth': the run is not a given-name title but 'queen' is. + # The predicate lives beside _title_key because the P5 licence in + # group asks the same question of the same run, and a run read two + # ways is a rule contradicting itself (decisions.md#P5, #369). + if (titles and givens and not middles and not families + and not _run_addresses_by_given( + (tokens[i].text for i in titles), + state.lexicon.given_name_titles)): + for i in givens: + _retag(tokens, i, Role.FAMILY) + # every rule below reads these lists; recompute after any + # retag so no guard can inspect a name that has already + # moved -- a stale index list is the bug shape #359 fixed + givens = _idx(tokens, Role.GIVEN) + middles = _idx(tokens, Role.MIDDLE) + families = _idx(tokens, Role.FAMILY) # rules.md#M4: "a maiden name standing beside exactly one name # word makes that word the family name, whatever suffix or diff --git a/tests/test_conjunctions.py b/tests/test_conjunctions.py index 8133d622..9fccc3d9 100644 --- a/tests/test_conjunctions.py +++ b/tests/test_conjunctions.py @@ -213,7 +213,6 @@ def test_conjunction_in_an_address_with_a_title(self) -> None: self.m(hn.title, "His Excellency Lord", hn) self.m(hn.last, "Duncan", hn) - @pytest.mark.xfail(reason="#489") def test_conjunction_in_an_address_with_a_first_name_title(self) -> None: hn = HumanName("Her Majesty Queen Elizabeth") self.m(hn.title, "Her Majesty Queen", hn) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index de5c0955..3efe1204 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -2391,6 +2391,67 @@ def _check_cjk_shape_purity(self) -> None: "being the word left standing -- because once the floor " "empties the run no branch reads `vocab:given-title` " "at all"), + Case("title_run_addresses_by_its_last_title", + "Her Majesty Queen Elizabeth", + {"title": "Her Majesty Queen", "given": "Elizabeth"}, + classification="fix(#489)", + notes="rules.md#H1 -- `queen` is a given-name title and it is " + "the last word of the run, so the run addresses as it " + "does; the empty family is H1's Accepted outcome"), + Case("title_run_addresses_by_its_last_title_with_a_suffix", + "Her Majesty Queen Elizabeth II", + {"title": "Her Majesty Queen", "given": "Elizabeth", + "suffix": "II"}, classification="fix(#489)", + notes="the suffix peel runs first and does not decide H1"), + Case("title_run_addresses_by_its_last_title_clerical", + "Reverend Mother Teresa", + {"title": "Reverend Mother", "given": "Teresa"}, + classification="fix(#489)", + notes="`mother` is a given-name title, `reverend` is not"), + Case("title_run_addresses_by_its_last_title_dotted", + "Dr. Sir John", {"title": "Dr. Sir", "given": "John"}, + classification="fix(#489)", + notes="the whole run keyed 'dr sir', which the shipped " + "vocabulary has no entry for; the last word is `sir`"), + Case("title_run_addresses_by_its_last_title_bare", "Mr Sir John", + {"title": "Mr Sir", "given": "John"}, + classification="fix(#489)", + notes="the 2026-08-22 #369 entry's own example -- 'mr sir' is " + "not a given-name title to either site, and now the " + "LAST word is what both sites read"), + Case("title_run_addresses_by_its_last_title_unlisted_first", + "Xyz. Sir John", {"title": "Xyz. Sir", "given": "John"}, + classification="fix(#489)", + notes="H2's unlisted abbreviation joins the run: it sits " + "inside the whole-run key, which the shipped " + "vocabulary has no entry for, and an unlisted word can " + "never match as a last-word key either -- `sir` is " + "what this run is read by"), + Case("title_run_licences_the_bound_join_by_its_last_title", + "Sir Sheikh abdul rahman", + {"title": "Sir Sheikh", "given": "abdul rahman"}, + classification="fix(#489)", + notes="rules.md#P5's licence keys the same way H1 does, the " + "invariant the #369 entry set: behind a given-name " + "title there is no family to spare"), + Case("title_run_does_not_address_by_a_non_given_name_title", + "His Excellency Lord Duncan", + {"title": "His Excellency Lord", "family": "Duncan"}, + classification="parity", + notes="negative control: `lord` is not a given-name title, so " + "the run's last word does not address by given name"), + Case("title_run_princess_is_vocabulary_scope", + "Her Royal Highness Princess Anne", + {"title": "Her Royal Highness Princess", "family": "Anne"}, + classification="parity", + notes="negative control: `princess` is not a given-name title " + "either -- a vocabulary question with its own frequency " + "argument (Prince Harry, Lady Gaga), deliberately out"), + Case("title_and_two_name_words_is_not_h1s", "Sir John Smith", + {"title": "Sir", "given": "John", "family": "Smith"}, + classification="parity", + notes="negative control: H1 never fires with two name words, " + "whatever the run keys to"), Case("all_suffix_input_reports_suffix_or_name", "Rinpoche", {"given": "Rinpoche"}, ambiguities=("suffix-or-name",), classification="feat(#491)", diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index 44de7365..4d049942 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -604,13 +604,16 @@ def test_the_licence_still_needs_two_name_words() -> None: assert _piece_texts(out) == [["sir", "abdul", "jr"]] -def test_the_title_run_is_one_key_as_h1_reads_it() -> None: - # post_rules looks the WHOLE title run up as one key, so "mr sir" - # is not "sir". P5 has to read the run the same way: if it joined - # here, H1 would then read the joined piece as the family name -- - # the two rules would disagree about what the same run asserts. +def test_the_title_run_is_read_as_h1_reads_it() -> None: + # P5 and post_rules ask ONE predicate of the run + # (_run_addresses_by_given), so they cannot disagree about what the + # same run asserts: if P5 joined where H1 then read the joined + # piece as the family name, one run would have two readings. The + # run "mr sir" is #369's own example and it moved with #489 -- the + # whole run is no entry, its LAST word is, so the run addresses by + # given name and the licence fires. out = _grouped("mr sir abdul rahman", lexicon=_GIVEN_NAME_TITLE_LEX) - assert _piece_texts(out) == [["mr", "sir", "abdul", "rahman"]] + assert _piece_texts(out) == [["mr", "sir", "abdul rahman"]] def test_the_licence_joins_a_word_not_a_particle_chain() -> None: @@ -636,7 +639,8 @@ def test_the_licence_takes_a_name_word_not_a_suffix() -> None: assert _piece_texts(out) == [["sir", "abdul", "jr", "rahman"]] -def test_a_conjunction_joined_title_run_is_keyed_whole() -> None: +def test_a_conjunction_joined_title_run_is_keyed_over_every_token_of_the_piece( +) -> None: # A conjunction-merged title is one multi-token PIECE. The key is # built from every token of every title piece, as post_rules # builds it from every title token; keyed on first tokens alone, diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index f4765574..239ff708 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -256,15 +256,39 @@ def test_given_name_titles_folds_per_word_so_abbreviations_match( # 'lt. col' verbatim kept the interior period and matched nothing -- # a silent no-op on the config surface, the failure this field is # most prone to (it has no validation by design). + # + # It is the whole-run arm that matches here, and only that arm: + # `col` alone is not a given-name title, so #489's last-word arm + # must not have replaced the phrase lookup. base = Lexicon.default() lex = dataclasses.replace( base, titles=base.titles | {"lt", "col"}, given_name_titles=base.given_name_titles | {spelling}) assert "lt col" in lex.given_name_titles + assert "col" not in lex.given_name_titles assert Parser(lexicon=lex).parse("Lt. Col. Smith").given == "Smith" +def test_the_shipped_given_name_titles_are_every_one_a_single_word() -> None: + # The invariant that makes the whole-run arm dead for the shipped + # vocabulary: it can only ever match a one-word run, which the + # last-word arm reads the same way (#489). A phrase entry here + # would give that arm reach the docstring says it has not. + assert not any(" " in t for t in Lexicon.default().given_name_titles) + + +def test_a_run_matches_by_its_last_word_when_the_whole_run_does_not() -> None: + # The other arm, on the shipped vocabulary: 'dr sir' is no entry + # and never could be, and `sir` is the run's last word (#489). + lex = Lexicon.default() + assert "dr sir" not in lex.given_name_titles + assert "sir" in lex.given_name_titles + parsed = Parser(lexicon=lex).parse("Dr. Sir John") + assert (parsed.title, parsed.given, parsed.family) == \ + ("Dr. Sir", "John", "") + + @pytest.mark.parametrize("entry", ["lt .", "lt . col", ". col", ". ."]) def test_given_name_titles_fold_is_a_fixed_point(entry: str) -> None: # Storage re-runs the fold on unpickle and on every diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 2b093d11..32d39446 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -675,10 +675,11 @@ def test_the_p5_licence_and_h1_read_a_title_run_the_same_way( title: str) -> None: # The licence's one invariant, as a contract: P5 lifts the reserve # behind a title run exactly when H1 keeps the one word after that - # run a given name. Both key the run through _title_key; if either - # side's key construction drifted, a run P5 licensed that H1 then - # read as title-plus-family would hand the joined pair to the - # family. So "no family" must agree, run by run. + # run a given name. Both ask _run_addresses_by_given -- the whole + # run's key, or the run's last word's (#489); if either side's read + # drifted, a run P5 licensed that H1 then read as title-plus-family + # would hand the joined pair to the family. So "no family" must + # agree, run by run. assert (parse(f"{title} John").family == "") == \ (parse(f"{title} abdul rahman").family == "") From 5568fdcced76de9ce5b2036cdd22709056180013 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 8 Sep 2026 23:00:11 -0700 Subject: [PATCH 03/12] fix(assign): a trailing run of period-marked title words is a title `John Smith Prof.` read family 'Prof.' and lost the surname, while `Smith, Prof.` read title 'Prof.' -- the two comma paths disagreed, and #316 is the question. They agree now: after the trailing suffix peel has taken its run, successive single words that wear the period-abbreviation shape AND are title vocabulary chain into the title from the end, leaving one name word standing. A RUN, mirroring H3, so `John Smith Prof. Dr.` reads title 'Prof. Dr.' rather than making `Prof.` a name word. The title view joins TITLE tokens in token order, so `Dr. John Smith Prof.` reads title 'Dr. Prof.' with no rendering change. The doctrine, stated rather than left to be inferred: the LEADING slot has a shape rule that outranks vocabulary (H2 -- `Esq. Smith` is a title though `esq` is post-nominal vocabulary, and that is unchanged); the TRAILING slot has no shape rule and reads vocabulary only. So `John Smith Xyz.` keeps family 'Xyz.', and a BARE trailing title word is a name word -- `John Smith Sir` and `Mary Jane King` are untouched, TITLES holding ordinary surnames being exactly why. The walk uses is_title_piece, never is_leading_title: that predicate carries H2's inference and would take `Xyz.`. No ambiguity is reported: under the input-is-a-name premise a period-marked title word is not a fork a reader would hesitate over. Two decisions taken with the spec's questions open. A trailing title is TRANSPARENT to the suffix peel -- `X Prof. Y` reads exactly as `X Y` reads, plus the title. The peel ahead of the walk is PROVISIONAL: all it settles is where the walk starts, because the walk can remove the very word that stopped it. So the pieces the walk takes are spliced out and ONE peel then runs over what stands, in original order, and that peel alone places a piece or reports a fork. `John Smith Jr. Prof.` reads suffix 'Jr.' where a single pass promoted a generational suffix to the family name; `John Prof. MA` reads the family 'MA' that `John MA` reads, S2's reserve keeping a bare ambiguous acronym the family of a two-word name, where a second peel laid over a first read family 'John', suffix 'MA'; and `John Smith V Prof. VI` reads what `John Smith V VI` reads -- middle 'Smith V', family 'VI', nothing reported -- where two peels each reporting their own last piece read suffix 'V VI' and reported the numeral fork twice. And the family-comma segment-1 walk ships: the comma gate does NOT route `Smith, John Prof.`, which read middle 'Prof.' at every baseline -- the `Smith, Dr.` family of rows that already route go through the no-name gate, a different mechanism. The same principle governs its candidates, which are the pieces that segment does not read as a SUFFIX, asked of the segment's own predicate rather than of a copy of half of it: `Smith, John Prof. V` is `Smith, John V` plus a title because the #144 lenient tail claims the numeral, and `Smith, John V Prof.` is too, because the walk moved where this segment's name ends and the lenient test follows it. Both are corpus-neutral, measured. Five corpus names move, measured over all 1123 -- the four `John Smith .` rows the issue planted, and `Andrew Perkins (Mgr.)`, which measurement found: its trailing period keeps the parenthetical out of nickname parsing and `mgr` is title vocabulary. One PINNED row moves that is not a corpus name, and is named here rather than left to be found: `Smith, PSM Dr. I` (family_comma_title_resets_the_credential_run) reads title 'Dr.' where it read middle 'Dr.', which is `Smith, PSM I` plus a title. The reset that row exists to pin still fires and the row still fails without it. Its parity claim went with the move and had outlived the audit anyway -- 1.4.0 reads suffix 'Dr., I', `dr` still being post-nominal vocabulary before #296. Two consequences worth naming. `titled` in the lone-name-word emitter is now true for a name whose only title is a trailing one, so `Smith Prof.` reports nothing where an untitled lone name word reports `given-or-family` -- H1 decides the field, the same answer `Dr. Smith` gets. And H4's JOIN clause gains one reach: `John of Prince Prof.` reports `title-or-name` because the walk left the unit `John of Prince` standing alone, which is what that unit already reports on its own. H4's single-word half cannot be reached this way -- a lone title-vocabulary word in front of a trailing title is taken by the LEADING run first, so `King Prof.` still reads title 'King'. Neither call site is gated. The question is asked by its predicate and by nothing else (mechanisms.md#ONE-PREDICATE-PER-QUESTION), so the walk's floor and its vocabulary read are the only things that answer it and a mutation removing either is visible. That costs the reference name one frame on each entry point -- inside the plan's design target of two -- and one frame comes back beside it: the group-flagged suffix scan collected a list it never read again, and a comprehension is a frame of its own on 3.11. Measured delta is 0 on both entry points, cold (416 parse / 453 facade on 3.11) and in a full pytest session (416 / 455). The give-back is not optional: 455 of the 455.94 allowed facade calls is what MASTER already costs in a full session, so the walk's frame alone measured 456 and failed the band -- the 453/454 pair a module-only run reports is not the number the test sees. Review round: both inline necessary conditions are gone, the walk is called directly at both sites, the mixed-provenance `Peel._replace` and the double numeral report with it, and `_report_numeral` is inlined back at the one site left. Six rows pin the transparency readings and each fails under its own mutation of the two peels or of the comma candidates; the seven-name corpus sweep against a0b93f0 is unchanged by any of it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- nameparser/_pipeline/_assign.py | 149 ++++++++++++++--- nameparser/_pipeline/_pieces.py | 48 +++++- tests/v2/cases.py | 268 +++++++++++++++++++++++++++---- tests/v2/pipeline/test_assign.py | 105 ++++++++++++ tests/v2/pipeline/test_pieces.py | 67 +++++++- 5 files changed, 577 insertions(+), 60 deletions(-) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 536612d3..8475f6de 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -16,17 +16,24 @@ (a title needs a following piece, unless the whole name is one title); then positional assignment per name_order with the trailing-suffix rule: the piece from which everything after is a strict suffix is the -last name-position piece, the rest are suffixes. The v1 single-name+ -nickname rule lives here (decisions.md#N3): a nonempty nickname -beside exactly one piece in total puts that piece in FAMILY. +last name-position piece, the rest are suffixes. That peel is only +provisional: behind it a trailing run of period-marked title words +chains into the title from the end, leaving one name piece standing, +and the peel then runs ONCE over the pieces with the titled ones +spliced out, so a trailing title is transparent to it. +The v1 single-name+nickname rule lives here (decisions.md#N3): a +nonempty nickname beside exactly one piece in total puts that piece +in FAMILY. FAMILY_COMMA: segment 0 wholly FAMILY (v1 parity) UNLESS segment 1 holds no name word (titles and suffixes only), which fixed no family boundary -- there segment 0 takes the NO_COMMA positional read instead, order and all ('John Smith, Dr.', 'John Smith, Mr. Jr.'); segment 1 is wholly SUFFIX when it is nothing but suffix pieces ('Smith, Jr.', 'Smith, Ph. D. Jr.' -- the credential run C1 describes, in the listing -form), else gets leading titles, then given, then middles with -strict-suffix pieces to suffix; segments 2+ are suffixes (lenient -- +form), else gets leading titles, then the same trailing title run +over the pieces this segment does not read as suffixes ('Smith, John +Prof.'), then given, then middles with those suffix-reading pieces to +suffix -- one predicate for both; segments 2+ are suffixes (lenient -- segment already flagged non-suffixy ones COMMA_STRUCTURE). SUFFIX_COMMA: segment 0 as NO_COMMA; segments 1+ wholly SUFFIX. Emits PARTICLE_OR_GIVEN when the leading name piece is a lone @@ -55,7 +62,7 @@ ) from nameparser._pipeline._pieces import ( is_suffix_piece, leading_titles, peel_trailing, peel_walk, - segment_suffix_reading, + segment_suffix_reading, trailing_titles, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, _NEVER_FLIPPED, @@ -221,15 +228,18 @@ def _assign_main(seg_idx: int, state: ParseState, ptags = state.piece_tags[seg_idx] has_nickname = any(t.role is Role.NICKNAME for t in tokens) n = _peel_leading_titles(pieces, ptags, tokens) - rest = list(range(n, len(pieces))) - if not rest: + if n == len(pieces): return None # group-flagged suffix pieces (the ph-d merge) are suffixes at ANY # position -- v1's fix_phd extracted the credential from the string - # before parsing, so position never mattered (PR review I3) - flagged = [k for k in rest if "suffix" in ptags[k]] - for k in flagged: - _set_roles(tokens, pieces[k], Role.SUFFIX) + # before parsing, so position never mattered (PR review I3). + # Walked rather than collected first: a comprehension is a frame of + # its own on 3.11 and the list was never read again, which is one + # frame back against the one the H5 walk below costs + # (decisions.md#parse-cost). + for k in range(n, len(pieces)): + if "suffix" in ptags[k]: + _set_roles(tokens, pieces[k], Role.SUFFIX) rest = peel_walk(n, ptags) if not rest: return None @@ -251,7 +261,38 @@ def _assign_main(seg_idx: int, state: ParseState, # because the wording reads the role back, and which role "not # peeled" means depends on name_order. (The roman-numeral fork # needs no such deferral and is reported here.) + # + # This first peel is PROVISIONAL: all it settles is where the H5 + # walk below starts. Its roles and its reports are never used -- + # the walk can remove the very word that stopped it, so when the + # walk takes anything the peel is asked again over the pieces as + # they then stand, and that second answer is the only one that + # places a piece or reports a fork. peeled = peel_trailing(rest, pieces, ptags, tokens) + # rules.md#H5 -- the trailing title run, read over what the suffix + # peel left and set BEFORE _name_positions, so the shortened list is + # what the positional read and the script test both see (a trailing + # Latin title must not make a wholly-CJK name look mixed-script, the + # same reason the leading peel runs first). Placed ahead of the + # bare-suffix carve-out because with no name piece left there is + # nothing for the walk to read: its floor returns 0 on an empty + # list, so the branch below is reached exactly as before. + titled_tail = trailing_titles(rest[:peeled.names], pieces, ptags, + tokens) + if titled_tail: + cut = peeled.names - titled_tail + for piece_idx in rest[cut:peeled.names]: + _set_roles(tokens, pieces[piece_idx], Role.TITLE) + # The title is TRANSPARENT to the suffix peel: the pieces the + # walk took are spliced out and the peel runs once over what + # is left -- the name pieces the walk kept, then the pieces + # the provisional peel had taken, in original order -- so + # 'X Prof. Y' reads exactly as 'X Y' reads, plus the title. + # 'John Smith Jr. Prof.' reads suffix 'Jr.' where a single + # pass promoted a generational suffix to the family name, and + # 'John Prof. MA' reads the family 'MA' that 'John MA' reads. + rest = rest[:cut] + rest[peeled.names:] + peeled = peel_trailing(rest, pieces, ptags, tokens) if peeled.numeral is not None: # a trailing single letter is a name part unless it happens # to be a roman numeral -- and V/X/I are ordinary middle @@ -498,6 +539,42 @@ def assign(state: ParseState) -> ParseState: if len(state.segments) > 1: pieces = state.pieces[1] ptags = state.piece_tags[1] + titled_idx: tuple[int, ...] = () + + def reads_as_a_suffix(m: int, last: int) -> bool: + """Does this segment's walk read piece `m` as a suffix? + + Asked twice, and by one predicate rather than by two + conditions written to match + (mechanisms.md#ONE-PREDICATE-PER-QUESTION): once to + find the pieces the H5 title walk must not reach past, + and once by the walk order below, which is the site + that places them. `last` is where this segment's name + ends -- provisionally the segment's last piece, and + after the title walk the last piece the walk left, + since a title it took is not where a name ends. + + `titled_idx` is read at CALL time and is empty on the + first pass, which is what makes the second reading the + one 'as if the titled pieces were absent': the lenient + test's preceding piece skips them too. + """ + if is_suffix_piece(pieces[m], ptags[m], tokens): + return True + prev = m - 1 + while prev in titled_idx: + prev -= 1 + # trailing piece of a two-part name is unambiguously + # positioned: v1 accepts the lenient test there + # ('Smith, John V' -> suffix='V', #144); with a third + # comma part the trailing token is more likely a middle + # initial, so strict only + return (m == last and len(state.segments) == 2 + and len(pieces[m]) == 1 + and _reads_as_a_trailing_suffix( + pieces[m], pieces[prev], ptags[prev], + tokens, state.lexicon)) + # rules.md#C1: "a credential run after the comma means the # name is in natural order with suffixes appended" -- and # with one word before the comma the listing form holds, @@ -526,6 +603,32 @@ def assign(state: ParseState) -> ParseState: n = len(pieces) else: n = _peel_leading_titles(pieces, ptags, tokens) + # rules.md#H5 -- the trailing title run, on this walk + # too. A name word in segment 1 is what keeps the gate + # above from reading the segment as a credential run, + # so 'Smith, John Prof.' had no route to title at all + # and read middle 'Prof.' at every baseline; the + # 'Smith, Dr.' family of rows that already route are + # the gate's doing, a different mechanism. + # + # The candidates are the pieces this walk would NOT + # read as a suffix, which is this path's answer to the + # peel the no-comma path runs first -- 'Smith, John + # Prof. Jr.' must reach `Prof.` past the postnominal + # behind it, and 'Smith, John Prof. V' past the + # numeral the lenient tail test claims (#144), which + # is why the filter is the walk's own predicate and + # not the strict suffix test alone. Piece `n` is + # always the given below, whatever that predicate + # would say of it, so it is always a candidate. + walkable = [k for k in range(n, len(pieces)) + if k == n or not reads_as_a_suffix( + k, len(pieces) - 1)] + taken = trailing_titles(walkable, pieces, ptags, tokens) + if taken: + titled_idx = tuple(walkable[len(walkable) - taken:]) + for k in titled_idx: + _set_roles(tokens, pieces[k], Role.TITLE) # v1 walk order: the first non-title piece is ALWAYS the # given, before any suffix check -- 'Hardman, RN - CRNA' # keeps first='RN'. The one deliberate 2.0 deviation, @@ -537,19 +640,17 @@ def assign(state: ParseState) -> ParseState: # so the walk here never meets the case. if n < len(pieces): _set_roles(tokens, pieces[n], Role.GIVEN) + # the walk's floor leaves a name piece standing, so the + # given above is never one of the pieces it took; what the + # walk DOES move is where this segment's name ends, and the + # lenient tail test below turns on that (#144) + last_kept = len(pieces) - 1 + while last_kept in titled_idx: + last_kept -= 1 for m in range(n + 1, len(pieces)): - # trailing piece of a two-part name is unambiguously - # positioned: v1 accepts the lenient test there - # ('Smith, John V' -> suffix='V', #144); with a third - # comma part the trailing token is more likely a middle - # initial, so strict only - last_of_two = (m == len(pieces) - 1 - and len(state.segments) == 2) - if is_suffix_piece(pieces[m], ptags[m], tokens) or ( - last_of_two and len(pieces[m]) == 1 - and _reads_as_a_trailing_suffix( - pieces[m], pieces[m - 1], ptags[m - 1], - tokens, state.lexicon)): + if m in titled_idx: + continue + if reads_as_a_suffix(m, last_kept): _set_roles(tokens, pieces[m], Role.SUFFIX) else: _set_roles(tokens, pieces[m], Role.MIDDLE) diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index a4672293..4e14a68e 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -27,7 +27,9 @@ The S2 trailing peel travels as the unit decisions.md describes -- peel_walk, peel_trailing and trailing_start together -- though only -the first two cross a stage boundary. +the first two cross a stage boundary. trailing_titles joins them +because it reads what that peel left: the two answer one question +between them, where the tail of a name stops being the name. Layering: imports _state and _vocab only; _group and _assign import it, and neither of the two it imports imports it back. @@ -359,3 +361,47 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], continue break return Peel(k, numeral, tuple(picks)) + + +# rules.md#H5 -- the trailing run's own predicate, and a forward note +# rather than a citation until H5 is written. NOT is_leading_title: +# that predicate carries H2's unlisted-abbreviation inference, which is +# the LEADING slot's shape rule and has no trailing counterpart, so +# with it 'John Smith Xyz.' would lose its family name to a title +# (decisions.md#H5). The vocabulary read is is_title_piece's, shared +# with the leading run so the two cannot disagree about what a title +# WORD is while disagreeing, deliberately, about what a title SHAPE is. +def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], + ptags: Sequence[Set[str]], + tokens: Sequence[WorkToken]) -> int: + """How many pieces at the END of `rest` are period-marked title + words. `rest` is the NAME pieces the S2 peel left, in piece order. + Floor: one name piece stands, so a name is never all title -- and + an empty `rest` returns 0, which is what leaves assign's + bare-suffix carve-out reached exactly as before. + + ONE WORD per piece, the same gate the leading peel's give-back + uses: a joined unit is not the shape this reads, and the tokens of + one are not each a title word. + + Every parse enters this frame -- assign asks the question here + rather than answering a cheaper version of it inline + (mechanisms.md#ONE-PREDICATE-PER-QUESTION) -- so what it costs an + ordinary name is one frame and one regex match. The shape test + runs BEFORE the vocabulary one to keep it at that: _PERIOD_ABBREV + is a compiled regex (a C call, no Python frame) where + is_title_piece is a call, and almost no name ends in a + period-marked word, so the ordinary parse pays the one match and + stops (decisions.md#parse-cost). + """ + n = 0 + while len(rest) - n > 1: + idx = rest[len(rest) - n - 1] + piece = pieces[idx] + if (len(piece) == 1 + and _PERIOD_ABBREV.match(tokens[piece[0]].text) + and is_title_piece(piece, ptags[idx], tokens)): + n += 1 + continue + break + return n diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 3efe1204..b4126248 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -3173,24 +3173,25 @@ def _check_cjk_shape_purity(self) -> None: # the ONLY audit words that lose SUFFIX membership; every other # audit word keeps its suffix membership, so trailing position is # untouched for them. - Case("audit_dr_trailing_joins_the_title_word_gap", "John Smith Dr.", - {"given": "John", "middle": "Smith", "family": "Dr."}, - classification="fix(#296)", - notes="'dr' left SUFFIX_WORDS, so a trailing 'Dr.' is no " - "longer suffix vocabulary and falls to the positional " - "read, taking the family name with it. NOT a new defect " - "class -- no trailing title word routes to title on the " - "no-comma path, so 'John Smith Prof.' and 'John Smith " - "Mr.' already read this way (both pinned below; #316 is " - "the open question). The v1-residue suffix entry was the " - "only thing making 'dr' behave unlike every other " - "title-only word. This row records that 'dr' JOINED the " - "existing behavior, not that the behavior is right"), + Case("audit_dr_trailing_is_a_title", "John Smith Dr.", + {"title": "Dr.", "given": "John", "family": "Smith"}, + classification="fix(#316)", + notes="'dr' left SUFFIX_WORDS, so a trailing 'Dr.' stopped " + "being suffix vocabulary and fell to the positional " + "read, taking the family name with it -- the reading " + "every other title-only word already had, which this " + "row recorded 'dr' JOINING rather than endorsing. #316 " + "answers the class, so it leaves that reading with " + "them: the trailing walk takes the word to the title " + "and 'Smith' is the family again"), Case("audit_sra_trailing_joins_the_title_word_gap", "John Smith Sra", {"given": "John", "middle": "Smith", "family": "Sra"}, classification="fix(#296)", notes="the same move for the other word losing suffix " - "membership"), + "membership -- and, since #316, the pair's contrast: " + "'Dr.' wears the period the trailing walk reads and has " + "gone to the title, 'Sra' bare wears no shape at all " + "and a bare trailing title word is a name word"), Case("family_comma_lone_generational_suffix", "Smith, Jr.", {"family": "Smith", "suffix": "Jr."}, classification="fix(#296)", @@ -3376,16 +3377,24 @@ def _check_cjk_shape_purity(self) -> None: "generation it looks like. This row is what makes " "#432's fix a period test rather than a numeral test"), Case("family_comma_title_resets_the_credential_run", "Smith, PSM Dr. I", - {"given": "PSM", "middle": "Dr.", "family": "Smith", + {"title": "Dr.", "given": "PSM", "family": "Smith", "suffix": "I"}, - notes="THE RESET, and unchanged since 1.4.0. A title ends the " - "run: what follows a bare title is not continuing a " - "credential, so the numeral behind it does not join and " - "the segment is no run at all. Removing that one line " - "left the whole suite green while this became title " - "'Dr.' + suffix 'PSM I' -- the reset fires 60 times " - "across the suite and until this row no input observed " - "it, which is the inert-measurement shape"), + classification="fix(#316)", + notes="THE RESET. A title ends the run: what follows a bare " + "title is not continuing a credential, so the numeral " + "behind it does not join and the segment is no run at " + "all. Removing that one line leaves the whole suite " + "green while this becomes title 'Dr.' + suffix 'PSM " + "I' -- the reset fires across the suite and until this " + "row no input observed it, which is the " + "inert-measurement shape. Since #316 the word it " + "resets ON is a title here rather than a middle name: " + "'I' is what this segment reads as its suffix, so " + "'Dr.' is the trailing piece and the walk takes it, " + "and 'Smith, PSM Dr. I' is 'Smith, PSM I' plus a " + "title. 1.4.0 read suffix 'Dr., I' -- 'dr' was still " + "postnominal vocabulary before #296's audit, so the " + "row's old parity claim had outlived it"), Case("family_comma_run_numeral_after_a_split_credential", "Smith, Ph. D. I", {"family": "Smith", "suffix": "Ph. D. I"}, @@ -3523,14 +3532,13 @@ def _check_cjk_shape_purity(self) -> None: "applies to the pre-comma name as it does to 'John " "Smith' alone -- deliberate", shape=4), - Case("title_word_trailing_is_not_a_title", "John Smith Prof.", - {"given": "John", "middle": "Smith", "family": "Prof."}, - notes="the pre-existing behavior the audit_dr_trailing and " - "audit_sra_trailing rows join, pinned so the pair reads " - "as consistency rather than as damage -- and so the " - "general fix (#316) has a row to flip when it lands. " - "Contrast 'Smith, Prof.', which the comma path DOES " - "route to title: the two paths disagree today"), + Case("title_word_trailing_is_a_title", "John Smith Prof.", + {"title": "Prof.", "given": "John", "family": "Smith"}, + classification="fix(#316)", + notes="rules.md#H5 -- after the trailing suffix run, a " + "period-marked title word chains into the title from " + "the end. The comma path already read it this way " + "('Smith, Prof.'); the two paths agree now"), Case("ja_honorific_glued_family_comma_title_only", "田中さん, Dr.", {"title": "Dr.", "family": "田中さん"}, classification="fix(#296)", @@ -3541,8 +3549,202 @@ def _check_cjk_shape_purity(self) -> None: "the honorific stays glued, joining master's '田中さん, " "Mr.'. Master peeled it through the suffix-comma route", tolerated=True), - Case("title_word_trailing_is_not_a_title_mr", "John Smith Mr.", - {"given": "John", "middle": "Smith", "family": "Mr."}), + Case("title_word_trailing_is_a_title_mr", "John Smith Mr.", + {"title": "Mr.", "given": "John", "family": "Smith"}, + classification="fix(#316)"), + + # -- #316(a): the trailing title RUN. The leading slot has a SHAPE + # rule that outranks vocabulary (H2); the trailing slot has no + # shape rule and reads vocabulary only. #316(b), the bare-safe + # subset ('Smith Dr'), is deliberately out. + Case("title_word_trailing_run_chains", "John Smith Prof. Dr.", + {"title": "Prof. Dr.", "given": "John", "family": "Smith"}, + classification="fix(#316)", + notes="a RUN, mirroring H3 -- one word only would leave " + "'Prof.' a name word"), + Case("title_word_trailing_joins_the_leading_run_in_input_order", + "Dr. John Smith Prof.", + {"title": "Dr. Prof.", "given": "John", "family": "Smith"}, + classification="fix(#316)", + notes="the title view joins TITLE tokens in token order, so " + "leading then trailing needs no rendering change"), + Case("title_word_trailing_leaves_one_name_word", "Smith Prof.", + {"title": "Prof.", "family": "Smith"}, + classification="fix(#316)", + notes="the floor: the walk stops with one name piece " + "standing, and H1 then names it the family. Nothing is " + "reported -- the trailing title is a Role.TITLE by the " + "time the lone-name-word emitter looks, so H1 decided " + "the field and O5's convention did not, the same answer " + "'Dr. Smith' gets"), + Case("title_word_trailing_behind_a_peeled_suffix", + "John Smith Prof. Jr.", + {"title": "Prof.", "given": "John", "family": "Smith", + "suffix": "Jr."}, classification="fix(#316)", + notes="the suffix peel takes its run first and the title " + "walk reads what it left"), + Case("title_word_trailing_ahead_of_a_suffix_word", + "John Smith Jr. Prof.", + {"title": "Prof.", "given": "John", "family": "Smith", + "suffix": "Jr."}, classification="fix(#316)", + notes="the same reading with the two words swapped, and the " + "reason the first peel is PROVISIONAL: 'Prof.' stood " + "behind 'Jr.', so that peel halted there, and the walk " + "then removed the very word that had stopped it. The " + "walk's piece is spliced out and one peel runs over " + "what stands, reading 'Jr.' the suffix it is; a single " + "pass promoted a generational suffix to the family " + "name"), + Case("title_word_trailing_is_transparent_to_the_suffix_peel", + "John Smith MA Prof.", + {"title": "Prof.", "given": "John", "family": "Smith", + "suffix": "MA"}, ambiguities=("suffix-or-name",), + classification="fix(#316)", + notes="the principle the two peels serve: 'X Prof. Y' reads " + "exactly as 'X Y' reads, plus the title. This is " + "'John Smith MA' -- suffix 'MA', ONE report -- because " + "the reports come from the single peel over the " + "spliced pieces. Collecting both peels' picks instead " + "reported the same coin flip twice"), + Case("title_word_trailing_keeps_the_bare_acronym_reserve", + "John Prof. MA", + {"title": "Prof.", "given": "John", "family": "MA"}, + ambiguities=("suffix-or-name",), classification="fix(#316)", + notes="the same principle where the peel's answer DEPENDS on " + "the spliced list: S2's reserve keeps a bare ambiguous " + "acronym the family of a two-word name, so this is " + "'John MA' plus a title and reads exactly as 'Prof. " + "John MA' does. Laying a second peel's roles over the " + "first peel's read family 'John', suffix 'MA'"), + Case("title_word_trailing_between_two_numerals", + "John Smith V Prof. VI", + {"title": "Prof.", "given": "John", "middle": "Smith V", + "family": "VI"}, classification="fix(#316)", + notes="'John Smith V VI' plus a title, reports and all. The " + "numeral fork reads the piece BEFORE the numeral, and " + "over the spliced list that piece is 'V', an initial " + "shape, so the fork declines and nothing is reported. " + "Two peels each reporting their own last piece read " + "suffix 'V VI' and reported the fork twice"), + Case("title_word_trailing_unlisted_abbreviation_is_a_name_word", + "John Smith Xyz.", + {"given": "John", "middle": "Smith", "family": "Xyz."}, + classification="parity", + notes="negative control, and the doctrine: the leading slot " + "has a SHAPE rule that outranks vocabulary (H2), the " + "trailing slot reads vocabulary only"), + Case("title_word_trailing_bare_is_a_name_word", "John Smith Sir", + {"given": "John", "middle": "Smith", "family": "Sir"}, + classification="parity", + notes="negative control: no period, so no claim -- TITLES " + "holds ordinary surnames and a bare trailing title word " + "is a name word (#316(b), deliberately out)"), + Case("title_word_trailing_ordinary_surname_is_a_name_word", + "Mary Jane King", {"given": "Mary", "middle": "Jane", + "family": "King"}, + classification="parity", + notes="negative control: decisions.md's trailing-position " + "rule that must NOT be adopted, still not adopted"), + Case("title_word_trailing_credential_is_a_suffix", + "John Smith Esq.", + {"given": "John", "family": "Smith", "suffix": "Esq."}, + classification="parity", + notes="negative control: the suffix peel runs first, so a " + "period-marked post-nominal never reaches the walk"), + Case("title_word_trailing_leading_slot_is_unchanged", "Esq. Smith", + {"title": "Esq.", "family": "Smith"}, + classification="parity", + notes="negative control: H2 stays unconditional and S2's 'a " + "suffix never opens the string' stands (#316 open " + "question 2 declined)"), + Case("title_word_trailing_in_a_parenthetical", + "Andrew Perkins (Mgr.)", + {"title": "Mgr.", "given": "Andrew", "family": "Perkins"}, + classification="fix(#316)", + notes="the fifth corpus name the walk reaches, found by " + "measurement: the trailing period keeps the " + "parenthetical out of nickname parsing and 'mgr' is " + "title vocabulary, so the word arrives in the trailing " + "slot as any other piece would"), + Case("title_word_trailing_leaves_a_joined_title_unit_standing", + "John of Prince Prof.", + {"title": "Prof.", "family": "John of Prince"}, + ambiguities=("title-or-name",), classification="fix(#316)", + notes="H4's join clause, reached through the trailing walk: " + "the unit left standing is the one H4 already reports " + "for on its own ('John of Prince'), and the walk is " + "what makes it the only one. The single-word half of " + "H4 cannot be reached this way -- a lone title-" + "vocabulary word in front of a trailing title is taken " + "by the LEADING run first ('King Prof.' reads title " + "'King')"), + Case("title_word_trailing_after_a_family_comma", + "Smith, John Prof.", + {"title": "Prof.", "given": "John", "family": "Smith"}, + classification="fix(#316)", + notes="the comma path's walk gets the same rule: a name word " + "in segment 1 keeps the gate from reading the segment " + "as a credential run, so the trailing title had no " + "route to 'title' there either. Contrast the no-name " + "segment ('Smith, Dr.', pinned above as " + "family_comma_lone_title): that shape routes through " + "the no-name gate, a different mechanism, untouched"), + Case("title_word_trailing_after_a_family_comma_run", + "Smith, John Prof. Dr.", + {"title": "Prof. Dr.", "given": "John", "family": "Smith"}, + classification="fix(#316)"), + Case("title_word_trailing_after_a_family_comma_ahead_of_a_suffix", + "Smith, John Prof. Jr.", + {"title": "Prof.", "given": "John", "family": "Smith", + "suffix": "Jr."}, classification="fix(#316)", + notes="'Smith, John Jr.' plus a title. The walk's candidates " + "are the pieces this segment does NOT read as a " + "suffix, which is this path's answer to the peel the " + "no-comma path runs first, so the walk reaches 'Prof.' " + "past the postnominal behind it; over every piece it " + "found no title at all"), + Case("title_word_trailing_after_a_family_comma_ahead_of_a_numeral", + "Smith, John Prof. V", + {"title": "Prof.", "given": "John", "family": "Smith", + "suffix": "V"}, classification="fix(#316)", + notes="the same, through the LENIENT tail test (#144) rather " + "than the strict one: 'V' is what this segment reads " + "as its suffix, so it is not a candidate either and " + "'Smith, John Prof. V' is 'Smith, John V' plus a " + "title. Filtering the candidates on the strict suffix " + "test alone left this a middle 'Prof.'"), + Case("title_word_trailing_after_a_family_comma_behind_a_numeral", + "Smith, John V Prof.", + {"title": "Prof.", "given": "John", "family": "Smith", + "suffix": "V"}, classification="fix(#316)", + notes="the mirror of the row above, and what makes the " + "lenient test read the pieces as if the title were " + "absent: the walk took the last piece, so 'V' is where " + "this segment's name ends and the test applies to it. " + "Against the segment's literal last piece, 'V' was a " + "middle initial here and a suffix one word earlier"), + Case("title_word_trailing_after_a_family_comma_behind_an_initial", + "Smith, John V. Prof.", + {"title": "Prof.", "given": "John", "middle": "V.", + "family": "Smith"}, classification="fix(#316)", + notes="the boundary of the row above, kept: #432 reads the " + "period as the abbreviation mark it is, so 'V.' is a " + "middle initial -- 'Smith, John V.' plus a title"), + Case("title_word_trailing_ahead_of_a_reserved_acronym", + "John Smith Prof. MA", + {"title": "Prof.", "given": "John", "family": "Smith", + "suffix": "MA"}, ambiguities=("suffix-or-name",), + classification="fix(#316)", + notes="'John Smith MA' plus a title, from the other " + "direction: S2's peel takes the acronym BEFORE the " + "walk here, so the title is the trailing piece and the " + "spliced peel reads the same suffix it reads without " + "it. The comma path parts company here and is left " + "alone -- after a family comma a bare ambiguous " + "acronym has been a MIDDLE name since 2.0 ('Smith, " + "John MA'), so in 'Smith, John Prof. MA' a name word " + "stands behind 'Prof.' and no title is in trailing " + "position at all"), # -- #271: script-scoped order + segmentation (amendment 2026-07-27) Case("ko_unspaced_default", "김민준", diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index 010402c0..ce73a9ca 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -212,6 +212,111 @@ def test_trailing_suffix_run_no_comma() -> None: assert _by_role(out, Role.SUFFIX) == "PhD MD" +def test_trailing_title_run_is_set_before_the_positional_read() -> None: + """The walk shortens the name-piece list, and does it early. + + Two TITLE tokens from opposite ends of the input, and the piece + between them read as the family name -- which is only true if the + walk ran before _name_positions did. `order` is the default + because the positional read still happened: the walk removes + pieces from it, it does not replace it. + """ + out = _assigned("Dr. John Smith Mr.") + assert _by_role(out, Role.TITLE) == "Dr. Mr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "Smith" + assert out.order == Policy().name_order + + +def test_the_suffix_peel_runs_over_the_pieces_the_walk_left() -> None: + """The trailing title can stand BEHIND a suffix word. + + 'Mr.' stopped the first peel, so a single pass would have left + 'Jr.' as the last name piece and made a generational suffix the + family name. The first peel is provisional: the walk's piece is + spliced out and the peel runs over what stands, reading 'Jr.' as + the suffix it is. + """ + out = _assigned("John Smith Jr. Mr.") + assert _by_role(out, Role.TITLE) == "Mr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "Smith" + assert _by_role(out, Role.SUFFIX) == "Jr." + + +def test_the_trailing_title_is_transparent_to_the_suffix_peel() -> None: + """The principle the provisional peel serves. + + 'X Mr. Y' reads exactly as 'X Y' reads, plus the title -- so the + peel decides over the pieces with the title spliced OUT, in + original order, rather than over the two halves separately. Both + readings below are what the same input without 'Mr.' gives: the + reserve keeps a bare ambiguous acronym the family of a two-word + name, and takes it as a credential when a full name remains. + """ + lex = _LEX.add(suffix_acronyms={"ma"}, + suffix_acronyms_ambiguous={"ma"}) + out = _assigned("John Mr. MA", lexicon=lex) + assert _by_role(out, Role.TITLE) == "Mr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "MA" + assert not _by_role(out, Role.SUFFIX) + out = _assigned("John Smith Mr. MA", lexicon=lex) + assert _by_role(out, Role.FAMILY) == "Smith" + assert _by_role(out, Role.SUFFIX) == "MA" + + +def test_the_walk_reports_only_the_peel_that_decided() -> None: + """One peel decides, so one peel reports. + + The numeral fork reads the piece before the numeral. Over the + input as written that piece is 'Mr.' and 'VI' is taken; over the + spliced pieces it is 'V', an initial shape, and the fork declines + -- which is the answer 'John Smith V VI' gets, with no report. + Reporting from the provisional peel as well said suffix 'V VI' + and reported the fork twice. + """ + out = _assigned("John Smith V Mr. VI") + assert _by_role(out, Role.TITLE) == "Mr." + assert _by_role(out, Role.MIDDLE) == "Smith V" + assert _by_role(out, Role.FAMILY) == "VI" + assert not _by_role(out, Role.SUFFIX) + assert not out.ambiguities + + +def test_the_family_comma_walk_reads_past_its_own_suffix_tail() -> None: + """Segment 1's candidates are what its walk does not read as a + suffix -- the strict test AND the lenient one (#144). + + 'V' after the comma is this segment's suffix, so the title behind + it is still the trailing piece; and with the two words swapped + 'V' is where the name ends once the walk has taken the title, so + the lenient test still reaches it. Both are 'Smith, John V' plus + a title. + """ + for text in ("Smith, John Mr. V", "Smith, John V Mr."): + out = _assigned(text) + assert _by_role(out, Role.TITLE) == "Mr.", text + assert _by_role(out, Role.GIVEN) == "John", text + assert _by_role(out, Role.FAMILY) == "Smith", text + assert _by_role(out, Role.SUFFIX) == "V", text + assert not _by_role(out, Role.MIDDLE), text + + +def test_trailing_title_run_after_a_family_comma() -> None: + """The same rule on segment 1's own walk. + + A name word after the comma keeps the no-name gate from reading + the segment as a credential run, so this shape had no route to + TITLE at all and read the word as a middle name. + """ + out = _assigned("Smith, John Mr.") + assert _by_role(out, Role.TITLE) == "Mr." + assert _by_role(out, Role.GIVEN) == "John" + assert _by_role(out, Role.FAMILY) == "Smith" + assert not _by_role(out, Role.MIDDLE) + + def test_initial_veto_keeps_v_in_middle() -> None: out = _assigned("John V. Smith") assert _by_role(out, Role.MIDDLE) == "V." diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index 08b26708..4a83b2a3 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -10,8 +10,8 @@ from nameparser._pipeline._classify import classify from nameparser._pipeline._group import group from nameparser._pipeline._pieces import ( - _numeral_behind_the_initial_veto, leading_titles, - segment_suffix_reading, + _numeral_behind_the_initial_veto, leading_titles, peel_trailing, + peel_walk, segment_suffix_reading, trailing_titles, ) from nameparser._pipeline._segment import segment from nameparser._pipeline._state import ParseState @@ -153,3 +153,66 @@ def test_the_leading_run_keeps_a_joined_unit_it_cannot_give_back() -> None: """ assert _leading("Prince of Wales Jr") == 1 assert _leading("Prince of Wales") == 1 + + +def _trailing(text: str) -> int: + """trailing_titles over the rest assign hands it: the name pieces + the S2 peel left, after the leading run is counted off.""" + state = _through_group(text) + pieces, ptags = state.pieces[0], state.piece_tags[0] + tokens = list(state.tokens) + rest = peel_walk(leading_titles(pieces, ptags, tokens), ptags) + peeled = peel_trailing(rest, pieces, ptags, tokens) + return trailing_titles(rest[:peeled.names], pieces, ptags, tokens) + + +def test_the_trailing_run_chains_period_marked_title_words() -> None: + """A RUN, mirroring the leading one. + + One word only would leave 'Prof.' a name word in the first + reading below, which is the reason the walk chains rather than + taking the last piece and stopping. + """ + assert _trailing("John Smith Prof. Dr.") == 2 + assert _trailing("John Smith Prof.") == 1 + assert _trailing("Dr. John Smith Prof.") == 1 # leading run too + + +def test_the_trailing_run_leaves_one_name_piece_standing() -> None: + """The floor, and the empty rest assign's carve-out needs. + + A name is never all title: the walk stops with one name piece + left. An input that IS all title never reaches the walk at all -- + the leading peel took the whole segment and the rest is empty, + where the same floor returns 0 and leaves assign's bare-suffix + carve-out reached exactly as before. + """ + assert _trailing("Smith Prof.") == 1 + assert _trailing("Dr. Prof.") == 0 # the floor, on a rest the + # walk would otherwise take + assert _trailing("Smith") == 0 # one-piece rest + assert _trailing("Prof.") == 0 # empty rest + + +def test_the_trailing_run_reads_vocabulary_and_not_shape() -> None: + """The whole difference from is_leading_title. + + That predicate carries H2's unlisted-abbreviation inference, so + with it the first reading below would lose its family name to a + title. The trailing slot has no shape rule: a period-marked word + is claimed there only when the vocabulary claims it, and a bare + title word is claimed not at all. + """ + assert _trailing("John Smith Xyz.") == 0 # unlisted abbreviation + assert _trailing("John Smith Sir") == 0 # no period + assert _trailing("John Smith Esq.") == 0 # the peel took it first + + +def test_the_trailing_run_refuses_a_joined_piece() -> None: + """ONE WORD per piece, the gate the leading give-back shares. + + 'de la Prof.' is one piece of three tokens, and the tokens of a + joined unit are not each a title word -- the particle chain made + that unit a name. + """ + assert _trailing("John de la Prof.") == 0 From 71fb4583bc5bd657dff639c8dd80a3f279f5755f Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Tue, 8 Sep 2026 23:58:03 -0700 Subject: [PATCH 04/12] change(suffixes): esq leaves the acronym set, and the two suffix sets are asserted disjoint Esquire is a contraction, not an initialism. The SUFFIX_ACRONYMS entry arrived in the 2019 bulk Wikipedia post-nominal import (af5bdab, #93) and was never reviewed; its only unique coverage is the spelling "E.S.Q.", which nobody writes. Same criterion as the rai/cha decision: does the entry describe the WORD or the machinery. `John Smith E.S.Q.` now reads family 'E.S.Q.' where every release since 1.4.0 read suffix. `Esq` and `Esq.` are untouched -- the SUFFIX_WORDS membership carries them, and it stops being inert -- and so are `Smith, Esq.` and `Esq. Smith`. One corpus name moves. What that buys is an invariant: SUFFIX_ACRONYMS and SUFFIX_WORDS are now asserted disjoint at import. The two sets normalize differently, so a word in both is matched by two rules and which one fired is unreadable from outside. 'esq' was the only overlap and the reason the assert could not exist; the two comment blocks defending it, the AGENTS.md gotcha and the `deliberately not asserted disjoint` note all retire with it. A behavior change, not a fix: a 2.x parity break, classified on all four ledgers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- nameparser/config/suffixes.py | 55 +++++++++++++++++------------------ tests/v2/cases.py | 38 +++++++++++++++++++----- tests/v2/test_lexicon.py | 17 +++++++++++ 3 files changed, 74 insertions(+), 36 deletions(-) diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index 6d1ebec8..791a487c 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -129,12 +129,14 @@ numeral listed above. So membership here is not the last word on a dotted token; the sentence is about this set's lookup alone. :data:`SUFFIX_ACRONYMS` is the set matched -with every period removed, so it alone covers the multi-dot spelling -"E.S.Q." -- and, having no interior period to lose, "Esq" as well. 'esq' -is listed here too (v1 data): inert against the shipped acronym set, since -dropping it changes no parse, but what keeps "Esq" matching for a caller -who removes it from :data:`SUFFIX_ACRONYMS`. That is why the two sets are -deliberately not asserted disjoint -- see the guard block at the bottom. +with every period removed, so it alone covers a multi-dot spelling: both +"P.H.D." and the bare "PhD" reach its `phd` entry, the two normalizing to +the same string once the periods come off. The two sets are asserted +DISJOINT (see the guard block at the bottom): a post-nominal belongs to +one of them or the other, and which one holds it is what decides whether +its multi-dot spelling reaches a whole-token lookup at all -- +``period_joined_vocab`` may still claim the token chunk by chunk, as the +"J.u.n.i.o.r." example above shows. """ GLUED_HONORIFICS = frozenset({ @@ -603,14 +605,6 @@ 'emt-p', 'enp', 'erd', - # The load-bearing membership: the acronym test strips every - # period, so this entry is the only thing matching the multi-dot - # spelling, and removing it costs the family name ("John Smith - # E.S.Q." -> family='E.S.Q.'). 'esq' is in SUFFIX_WORDS as well, - # which against this set is inert -- "Esq" has no interior period, - # so it matches here too -- but that is not a duplicate to clean - # up: it is what still matches "Esq" if this entry ever goes. - 'esq', 'evp', 'faafp', 'faan', @@ -929,21 +923,24 @@ # construction, which is what protects a caller's own vocabulary. assert SUFFIX_ACRONYMS_AMBIGUOUS <= SUFFIX_ACRONYMS, \ "SUFFIX_ACRONYMS_AMBIGUOUS must stay a subset of SUFFIX_ACRONYMS" -# NOT asserted: disjointness of SUFFIX_ACRONYMS and SUFFIX_WORDS. -# The two are matched with different normalization -- the word test strips -# only edge periods, the acronym test strips all of them -- and no -# SUFFIX_WORDS entry carries an interior period, so for a word in both -# sets the acronym branch fires wherever the word branch does (the assert -# just below keeps such a word out of the period-gated ambiguous subset). -# The single overlap, 'esq', is therefore inert as shipped rather than a -# second spelling: SUFFIX_ACRONYMS covers "E.S.Q." AND "Esq", and dropping -# 'esq' from SUFFIX_WORDS changes no parse. It stays because these sets -# are caller-editable -- it is what still matches "Esq" once 'esq' leaves -# SUFFIX_ACRONYMS -- and an inert overlap is not worth an assert that -# would reject a working config. -# DO assert that an ambiguous acronym is not also a plain suffix word: -# suffix_as_written ORs the two branches, so the word membership would -# bypass the period gate the ambiguous set exists to impose. +# The two sets normalize differently -- the word test strips only edge +# periods, the acronym test strips all of them -- so a word in both is +# matched twice by two rules, and which one fired is unreadable from the +# outside. The single overlap was 'esq', dropped 2026-09-08 +# (decisions.md#suffix-acronym-collisions), and the assert is what keeps +# a bulk import from quietly re-creating one. It guards the SHIPPED sets +# only: Lexicon has no matching invariant, so a caller who wants the +# overlap in their own vocabulary may still have it. +assert not (SUFFIX_ACRONYMS & SUFFIX_WORDS), \ + "a post-nominal belongs to one set or the other, never both (the " \ + "two normalize differently): " \ + f"{sorted(SUFFIX_ACRONYMS & SUFFIX_WORDS)}" +# The narrower claim, kept for the message it prints: an ambiguous +# acronym must not also be a plain suffix word, because suffix_as_written +# ORs the two branches, so the word membership would bypass the period +# gate the ambiguous set exists to impose. Implied by the disjointness +# above for as long as the ambiguous set stays a subset of the acronyms, +# which is what the first assert holds. assert not (SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_WORDS), \ "an ambiguous acronym must not also be a suffix word (the word " \ "branch bypasses its period gate): " \ diff --git a/tests/v2/cases.py b/tests/v2/cases.py index b4126248..adc4aefe 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -358,13 +358,37 @@ def _check_cjk_shape_purity(self) -> None: {"given": "John", "family": "Smith", "suffix": "Esq"}, notes="the suffix_words branch of the delimited-content " "escape (v1 parity, pinned live 2026-07-17)"), - Case("suffix_acronym_multidot_spelling", "John Smith E.S.Q.", - {"given": "John", "family": "Smith", "suffix": "E.S.Q."}, - notes="'esq' is in BOTH suffix_acronyms and suffix_words on " - "purpose, and the two are not redundant: the word test " - "strips only EDGE periods, the acronym test strips all " - "of them, so only the acronym membership matches the " - "multi-dot spelling (v1 parity, pinned live 2026-07-19)"), + Case("suffix_acronym_multidot_spelling_is_a_name_word", + "John Smith E.S.Q.", + {"given": "John", "middle": "Smith", "family": "E.S.Q."}, + classification="fix(suffix-acronym-collisions)", + notes="a BEHAVIOR CHANGE rather than a fix -- a deliberate " + "2.x parity break, every release from 1.4.0 read " + "suffix 'E.S.Q.' here. 'esq' left SUFFIX_ACRONYMS " + "2026-09-08: Esquire is a contraction, not an " + "initialism, the entry arrived in the 2019 bulk " + "post-nominal import (af5bdab, #93), and the multi-dot " + "spelling was its only unique coverage. Same criterion " + "as the rai/cha rows above -- " + "decisions.md#suffix-acronym-collisions -- asked of the " + "machinery instead of a surname: does the entry " + "describe the WORD or the set's normalization. The " + "classification is a slug and not an issue number " + "because no issue asked for it; the bundle that " + "carried it is #489/#316"), + Case("suffix_word_esq_still_reads_as_a_suffix", "John Smith Esq", + {"given": "John", "family": "Smith", "suffix": "Esq"}, + notes="the other half of the row above, and what the removal " + "rests on: the SUFFIX_WORDS membership carries every " + "single-token spelling on its own, so it stops being " + "inert rather than becoming dead. Deleting 'esq' there " + "too is what this row refuses. 'Esquire' rides the " + "same membership and is deliberately unpinned, being " + "one more word in a set rather than a fork " + "(mechanisms.md#VOCABULARY-EXERCISES-FORKS); the " + "dotted 'John Smith Esq.', the comma form 'Smith, " + "Esq.' and the leading 'Esq. Smith' each have a row of " + "their own below"), Case("bound_given_whole_segment", "salem, abdul salam", {"given": "abdul salam", "family": "salem"}, notes="v1 joins bound given names freely in the post-comma " diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index 239ff708..a79a7779 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -11,6 +11,7 @@ _title_key, ) from nameparser._policy import Script, _SCRIPT_RANGES +from nameparser.config.suffixes import SUFFIX_ACRONYMS, SUFFIX_WORDS def test_entries_are_normalized_at_construction() -> None: @@ -237,6 +238,22 @@ def test_suffix_ambiguous_must_be_subset_of_acronyms() -> None: Lexicon(suffix_acronyms_ambiguous=frozenset({"ma"})) +def test_the_shipped_suffix_sets_are_disjoint() -> None: + # suffixes.py asserts this at import; the assert is stripped under + # `python -O`, so the invariant gets a named test as well. The two + # sets normalize differently -- the word test strips only edge + # periods, the acronym test strips all of them -- so a word in both + # is matched by two rules and which one fired is unreadable from + # outside. The last overlap, 'esq', left the acronyms 2026-09-08 + # (decisions.md#suffix-acronym-collisions). + # + # The SHIPPED sets only: Lexicon has no such invariant, and a + # caller who wants the overlap in their own vocabulary keeps it. + assert SUFFIX_ACRONYMS & SUFFIX_WORDS == frozenset() + default = Lexicon.default() + assert default.suffix_acronyms & default.suffix_words == frozenset() + + def test_given_name_titles_may_hold_a_multi_word_phrase() -> None: # given_name_titles is looked up against the SPACE-JOINED title run # (_pipeline/_post_rules.py), not per token, so "grand duke" is a From beec200f3b07bbe8ae46d210428cbcbe62351127 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 00:55:45 -0700 Subject: [PATCH 05/12] =?UTF-8?q?docs(design+release):=20the=20trailing=20?= =?UTF-8?q?slot=20reads=20vocabulary=20=E2=80=94=20#489=20and=20#316=20rec?= =?UTF-8?q?orded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit rules.md gains H5, the trailing title run: after the suffix peel, successive single period-marked title-vocabulary words chain into the title from the end, floor one name piece, and the title is TRANSPARENT to the suffix reading. The H Background states the doctrine the two slots split on — shape at the front, vocabulary at the back, a bare title word a name word because TITLES holds surnames. H1 gains the run clause (a run addresses as its last title does), H3 the peel floor, H4's Accepted clause is rewritten now that `Dr. King MD` reports, P5's "one key" sentence and S2's descriptive note are replaced by the argument each asked for, and the code citations are upgraded to verbatim excerpts. decisions.md gains `### H3` and `### H5` and amends five entries: H1 (the last-word keying, #489), P5's 2026-08-22 #369 entry ('mr sir' IS a given-name-title run to both sites now, the invariant kept), H2's `Open: #316` hook closed, H4's #491 population corrected from six to ten with seven contract and three radar, the trailing-position prohibition scoped to BARE words, the esq Excluded block replaced by the drop and its criterion, and the v1-xfail triage's `Dr King Jr` bullet moved from NOT FIXED to FIXED with #489 out of FIX CANDIDATES. Each measured population is scoped to the tree it was taken on, with today's figure beside it, since this commit's own examples grow the corpus. mechanisms.md re-checks TWO-LAYER-ASSIGN's one named exception against H5 (still one) and adds `trailing_titles` to ONE-PREDICATE-PER- QUESTION's census. release_log.rst gains three fix bullets and the esq behavior change. AGENTS.md's "runs in one direction only" line becomes the doctrine, the esq gotcha retires with the assert that replaced it, and 694 → 746, measured. usage.rst and customize.rst follow. corpus_rules.jsonl regenerated: 23 rows, 15 new names. Design-docs review of this commit (2026-09-09), six findings folded in by fixup. decisions.md#P5's trailing-position bullet RETRACTS half its own argument: a period-marked trailing word is not one nobody writes as a surname, and the collision it was said to prevent is simply accepted there -- `Mary Jane King.` reads title 'King.', given 'Mary', family 'Jane', `John Smith Judge.` reads title 'Judge.', and 627 of the 746 titles in no suffix set read as a trailing title once a period is written behind them, the 119 that do not being held out by the abbreviation SHAPE alone (a digit, a hyphen, an apostrophe, a script with combining marks), no plain ASCII-letter title missing. rules.md#H5 carries the cost as an Accepted clause with `Mary Jane King.` as its executable example. H5's transparency sentence is SCOPED: it claims what it claims where two or more name words stand, and where the chain leaves ONE, H1 decides that word's field -- `Smith Prof.` reads family 'Smith' where `Smith` alone reads given 'Smith' and reports given-or-family. `_assign.py`'s verbatim excerpt follows the new wording. H1 gains the TRAILING REACH it always had in the code: a title run standing BEHIND the one name word decides its field the same way, so `Smith Sir.` and `Smith Queen.` read given 'Smith' while `Smith Dr.` reads family. `"Smith Sir." -> given="Smith"` is an example line, H1 and H5 cross-link in `interacts:`, and no code moved for it. Three counts corrected. release_log's "Ten names ... gain a report" is EIGHT distinct names in ten corpus rows, measured at a0b93f0, and decisions.md#H4's "these ten names" says eight, twelve after this bundle, with the recompute beside it, so the entry's two TENs stop naming two quantities. decisions.md#P5's renamed group test pins P5's side alone, the two-site agreement resting on the shared `_run_addresses_by_given` and on test_parser.py::test_the_p5_licence_and_h1_read_a_title_run_the_same_way rather than on that leaf. And #H5's A2 "moves NO corpus name" is scoped to the fix: `Smith, John Prof.` enters the corpus here, and the rule's population is FIVE at the fix and FOURTEEN here, re-measured with the walk stubbed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- AGENTS.md | 8 +- docs/customize.rst | 11 +- docs/design/decisions.md | 61 +++++++-- docs/design/mechanisms.md | 4 +- docs/design/rules.md | 173 +++++++++++++++++++++----- docs/release_log.rst | 10 +- docs/usage.rst | 22 +++- nameparser/_pipeline/_assign.py | 13 +- nameparser/_pipeline/_pieces.py | 19 +-- nameparser/_pipeline/_post_rules.py | 9 +- tools/differential/corpus_rules.jsonl | 23 ++++ 11 files changed, 277 insertions(+), 76 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d34c1521..a811e883 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -356,7 +356,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is intentionally absent from the default titles (see `docs/customize.rst` — users who need it add it via opt-in `Constants`). Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. -**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` → title, which the shape rule claims even once `jr` leaves `TITLES`). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. **And it runs in one direction only**: a trailing abbreviation has no structural counterpart and is matched against the suffix vocabulary alone, so a trailing TITLE word is not a title (`"John Smith Prof."` → `family='Prof.'`, and the comma path disagrees — `"Smith, Prof."` → `title='Prof.'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four trailing-period behaviors, four different resolutions; unifying them is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") +**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` → title, which the shape rule claims even once `jr` leaves `TITLES`). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules.md#H5`) — the comma path always read it that way and the two agree now — and a trailing period-marked word the SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane King"`), which is the doctrine line below. Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") **Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185) @@ -366,9 +366,9 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **A comparison that runs both sides in one tree reports 0 differences, which is what success looks like.** `tools/differential/` has been hardened against this — it generates its baseline worker into a temp dir, strips `PYTHONPATH` from the child, and aborts unless both the version AND the resolved path check out on each side; its README's three invocation traps are the analysis behind that design, worth reading before changing the harness. **The exposure is ad-hoc comparisons you write yourself**, where the same collapse has a cause the harness cannot disarm for you: **the shell's working directory persists between tool calls**, so a two-tree comparison written as two `cd`s silently runs both halves in whichever tree it landed in. Pin absolute paths, and assert `nameparser.__file__` on both sides the way `compare.py` does. The general rule outlives any particular trap: before believing a null result, prove the harness can report a difference — a clean run and a broken harness are the same output. -**`esq` is in both `SUFFIX_ACRONYMS` and `SUFFIX_WORDS` on purpose — do not "deduplicate" it, and do not describe it as two spellings** — the two branches normalize differently (see the asymmetry gotcha above): the word test strips only edge periods, the acronym test strips all of them. The load-bearing membership is the ACRONYM one, and it carries *both* spellings: `"Esq"` also survives `.replace(".","")` unchanged, so it is in `SUFFIX_ACRONYMS` too. The word membership is therefore inert as shipped — **provably**, not just on a sample: the intersection of the two sets is exactly `{esq}`, no `SUFFIX_WORDS` entry carries an interior period, and `suffixes.py` asserts `SUFFIX_ACRONYMS_AMBIGUOUS ∩ SUFFIX_WORDS == ∅`, so for a word in both, the acronym branch fires wherever the word branch does. Measured to match: `SUFFIX_WORDS − {esq}` changes **no** parse on either API, while `SUFFIX_ACRONYMS − {esq}` changes exactly the multi-dot spellings — `E.S.Q.`/`E.S.Q` move, losing the family name there (`"John Smith E.S.Q."` → `family='E.S.Q.'`), while every single-token spelling (`Esq`, `Esq.`, `ESQ`, `esq`) is untouched — and that half is a PROOF, not a sample, from the algebra just above: each normalizes to `esq` under the word test's edge-period strip, so `SUFFIX_WORDS` catches it whatever the acronym set holds. A 26-frame sweep found no exception, but the sweep only corroborates; the argument is what makes it safe to rely on. Deliberately no changed-parse COUNT here — four people built four grids of this shape and got four different numbers (12, 15, 18, 10); the count is a property of the grid, while the zero, the direction, and *which spellings move* are properties of the code. "Changes many" is what this line said before, and it is the shape of claim to avoid: it sounds measured, survives any grid, and tells the next person nothing about where to look. The word membership is still not junk: it is v1 data parity, and it is what keeps `"Esq"` matching for a caller who removes `esq` from `SUFFIX_ACRONYMS` themselves (verified — after `C.suffix_acronyms.remove('esq')`, `"John Smith Esq"` still parses `suffix='Esq'`, and dropping both memberships gives `family='Esq'`). Pinned by the `suffix_acronym_multidot_spelling` case-table row. The disjointness that IS asserted is `SUFFIX_ACRONYMS_AMBIGUOUS ∩ SUFFIX_WORDS`, which is a different claim: `suffix_as_written` ORs the branches, so a word membership bypasses the period gate the ambiguous set exists to impose. `suffixes.py`'s `# NOT asserted:` block states the same reasoning at the code — keep the two in step. +**`SUFFIX_ACRONYMS` and `SUFFIX_WORDS` are asserted DISJOINT at import (since 2026-09-08) — a bulk post-nominal import that re-creates an overlap fails there, and the fix is to choose a set, not to relax the assert** — the two branches normalize differently (the word test strips only edge periods, the acronym test strips all of them), so a word in both is matched by two rules and which one fired is unreadable from outside. `esq` was the single overlap and left `SUFFIX_ACRONYMS`, which is what made the assert possible; the whole algebra that had made the overlap safe, and the criterion that decided which set a word belongs to, are in the `suffix-acronym-collisions` entry of `docs/design/decisions.md` if anyone wants them. -**`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse, and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `GIVEN_NAME_TITLES ⊆ TITLES` puts the entry in `titles` too, which is per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). +**`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse — matched whole OR by that key's LAST word since #489, a run addressing the way its final title does — and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `GIVEN_NAME_TITLES ⊆ TITLES` puts the entry in `titles` too, which is per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). **`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_title_key` joins per-word `_normalize` and DROPS words that fold away; keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". @@ -402,7 +402,7 @@ Don't use the bare `python3 -m doctest <file>.rst` CLI (no `optionflags`) to che **Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the twelve words still in both (`md`, `ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, both measured 2026-08-01: a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and trailing position has no such rule at all, so a title word there is neither title nor suffix but a NAME part (`"John Smith Prof."` → `family='Prof.'`) — which is what the comma path already disagrees with. Why it is not simply inverted to "vocabulary decides": `TITLES` holds 694 words that are in no suffix set, and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first trailing rule would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one — `King` is a surname, `King.` is not. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the twelve words still in both (`md`, `ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules.md#H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`), there being no trailing shape rule. The bare word is where "positional" still holds whole, and the reason it must: `TITLES` holds words that are in no suffix set — 746 of them, measured on this tree with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (decisions.md#P5's trailing-position bullet, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. ### Tests (`tests/`) diff --git a/docs/customize.rst b/docs/customize.rst index 70bd2475..b3231724 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -55,7 +55,10 @@ exceptions, so a multi-word entry like ``titles={"grand moff"}`` can never match; the constructor warns when it sees one (``capitalization_exceptions`` keys included — they are looked up per word too). The exceptions are ``given_name_titles``, looked up as the -space-joined run of words already read as titles, and +space-joined run of words already read as titles or as that run's last +word — several titles written together are one form of address and the +last one does the addressing, so ``"Her Majesty Queen Elizabeth"`` is +read by ``queen`` — and ``maiden_markers``, matched by lookahead over the words as written: ``maiden_markers={"z domu"}`` matches the pair and neither word alone, which is how the shipped Polish entry works. The words have to stand @@ -102,9 +105,9 @@ field too, so add to both and remove from the marker first. The last three enforce that: anything else raises ``ValueError`` naming the orphans rather than leaving a marker entry that no rule will ever consult. ``given_name_titles`` is deliberately unchecked — a title run -is matched as one space-joined string, so a legitimate entry like -``"sir and dame"`` is no single word in ``titles`` — and an orphan -there is inert rather than harmful. +is matched as one space-joined string, or by that run's last word, so a +legitimate entry like ``"sir and dame"`` is no single word in +``titles`` — and an orphan there is inert rather than harmful. Turning title detection off ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/design/decisions.md b/docs/design/decisions.md index dba003b2..c6aa64d1 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -260,7 +260,7 @@ Recomputed 2026-09-07 with that recipe, after this section's own #342 decision l Open: [#348](https://github.com/derek73/python-nameparser/issues/348) applying C-i to the 711 title entries, then titles_ambiguous plus a TITLE_OR_GIVEN kind. Blocked on data, not on judgement — the census needs a given-name frequency corpus this repo does not have, which is why the criterion is recorded here and the census is not attempted. -### v1-xfail-triage — the eight inherited aspirations, four retired and four kept +### v1-xfail-triage — the eight inherited aspirations, four retired and four kept (two of those eight since fixed, 2026-09-08) The reconciled v1-style banks (`tests/test_*.py`) carried eight `@pytest.mark.xfail` tests inherited from v1, each an aspiration nobody had judged since. `xfail_strict = true`, so all eight were live claims that 2.0 still does not do the thing — but a bare marker says only that, never whether the thing is WANTED, and that is the gap this triage closes. After it, every surviving marker cites an issue and every retired one is a deliberate pin of current behavior. The parses quoted below are this entry's evidence and its own recompute: @@ -268,15 +268,15 @@ The reconciled v1-style banks (`tests/test_*.py`) carried eight `@pytest.mark.xf - 2026-09-01 (Derek's triage) — NOT FIXED, four. The aspiration is rejected, the marker is gone, and the test now pins what the parser does with the grounds recorded beside it so nobody re-derives the rejected proposal: - `Maier, Amy I, Jr.` — v1 wanted suffix "I, Jr."; it reads given Amy, middle I, family Maier, suffix "Jr.". A middle initial `I` is far more common than an ordinal I borne without a Sr./Jr.-style companion, so where an explicit suffix comma has already named the suffix, the trailing I stays a middle initial. That is #vocabulary-collisions' commonality reasoning applied to a SHAPE rather than to a word — the criterion is written per-word, and the extension is named here rather than smuggled in. **This amends rules.md#C1 in the same diff.** C1's one-character clause said a bare single letter behind a name word is the generation, full stop, which predicts exactly the suffix "I, Jr." being rejected here. The discriminator the parser has always applied and C1 omitted is the FURTHER COMMA: `Maier, Amy I` reads suffix 'I' while `Maier, Amy I, Jr.` reads middle 'I', and `Smith, John V` against `Smith, John V, Jr.` is the same pair on a name C1 already exemplifies — all four measured. So the rule was incomplete, not the behavior: a rules.md defect rather than a parser one, which is the call this file already made for `'Donald mc'` in the fields-only arc entry (2026-08-27, #451). The qualifier is normative prose carrying no example line of its own; C1's Accepted block records why and names the bank pair as its executable witness. - - `Dr King Jr` — v1 wanted title 'Dr', family 'King', suffix 'Jr'; it reads title 'Dr King', family 'Jr'. `king` stays in the titles vocabulary: it is there for the addressing forms ("King Charles"), and taking it out to serve the surname reading trades a common use for a rarer one, which is the direction #vocabulary-collisions cuts. TITLES has no ambiguous subset and no AmbiguityKind, so — as with MAIDEN_MARKERS and `roz` — the only two expressions of the criterion available here are ship and do not ship; #348 is the open work that would give this set a third answer. The comma format is the road to the surname reading, and is now pinned alongside rather than left as prose — `King, Dr Jr` reads title 'Dr', family 'King', suffix 'Jr'. The test cites [#27](https://github.com/derek73/python-nameparser/issues/27), which is closed; this is the decision it never got. - Two halves, and only one is decided. DECIDED: `king` stays in TITLES. RECORDED, not endorsed: what becomes of the leftover `Jr`. rules.md#S2 predicts suffix 'Jr' with an empty family — its Accepted clause consumes an unambiguous suffix even when nothing is left to be the family (`Smith Jr.` → family "") — but once the title chain has taken `Dr King`, H1 claims the one remaining word and it reads family 'Jr', suffix ''. `Dr Smith Jr` isolates the cause: family 'Smith', suffix 'Jr', exactly as S2 states. S2 now carries a descriptive note saying so. A future change moving `Dr King Jr` toward S2's prediction is an IMPROVEMENT and updates the pin; it is not a regression, and the test says as much so nobody reads the pin as an endorsement. + - `Dr King Jr` — **FIXED 2026-09-08 (the #316/#489 bundle), and this is the one bullet of the four that moved out of NOT FIXED.** v1 wanted title 'Dr', family 'King', suffix 'Jr', and that is what the parse gives; the leading title peel now leaves a name word a suffix cannot be (#H3). The bullet is amended in place rather than moved, because the half it DECIDED is untouched and is still the reason the shape exists at all. What follows is the 2026-09-01 record, with the reading it describes corrected where the fix falsified it. It read title 'Dr King', family 'Jr' then. `king` stays in the titles vocabulary: it is there for the addressing forms ("King Charles"), and taking it out to serve the surname reading trades a common use for a rarer one, which is the direction #vocabulary-collisions cuts. TITLES has no ambiguous subset and no AmbiguityKind, so — as with MAIDEN_MARKERS and `roz` — the only two expressions of the criterion available here are ship and do not ship; #348 is the open work that would give this set a third answer. The comma format is the road to the surname reading, and is now pinned alongside rather than left as prose — `King, Dr Jr` reads title 'Dr', family 'King', suffix 'Jr'. The test cites [#27](https://github.com/derek73/python-nameparser/issues/27), which is closed; this is the decision it never got. + Two halves, and only one was decided. DECIDED: `king` stays in TITLES, which is unchanged and is what still puts `King` inside the title run in the first place. RECORDED, not endorsed: what becomes of the leftover `Jr`. rules.md#S2 predicts suffix 'Jr' with an empty family — its Accepted clause consumes an unambiguous suffix even when nothing is left to be the family (`Smith Jr.` → family "") — but once the title chain had taken `Dr King`, H1 claimed the one remaining word and it read family 'Jr', suffix ''. `Dr Smith Jr` isolated the cause: family 'Smith', suffix 'Jr', exactly as S2 states. S2 carried a descriptive note saying so. This entry then said that a future change moving `Dr King Jr` toward S2's prediction is an IMPROVEMENT and updates the pin — that change is #H3's peel floor and it landed 2026-09-08. The reading is now title 'Dr', family 'King', suffix 'Jr': the run gives back its last word where everything behind it is post-nominal and that word is not itself suffix vocabulary, so `Dr King Jr` reads like `Dr Smith Jr` rather than against it, and S2's note is replaced by the argument it asked for. The pin in `tests/test_suffixes.py::test_king` moved with it, by decision rather than by drift, and `king` being title vocabulary now buys the row a `title-or-name` report (#H4) rather than a different reading. - `Ahmad ben Husain` — v1 wanted family "ben Husain"; it reads given Ahmad, middle ben, family Husain. Already decided in v0.2.5, when `ben` came out of the prefixes, and for the reason that still holds: `ben` collides with the given name Ben, in the position the particle claim would act on — `Ahmad Ben Husain` reads middle 'Ben' today, which is exactly the token a forward-joining particle claim would take. That is C-i's position test, and it keeps `ben` out. Recorded a second time as a standing keep-out in this file's Excluded block for the particle set, because that is where a wordlist sweep meets it: a keep-out that lives only in a triage entry is one the next Arabic/Hebrew patronymic-particle sweep never reads. Worth naming as a failure mode of its own — the marker was an aspiration that outlived its own resolution, and nothing in a bare xfail says which of the eight were like that. - `The Right Hon. the President of the Queen's Bench Division` — v1 wanted the whole string as one title; it reads title "The Right Hon. the President of the Queen's Bench", family 'Division'. This is a name parser, not a title parser: handed an input that is all titles it assumes the last title-word is the name. Accepted as convention rather than defended as correct — and since 2026-09-07 the guess is no longer silent: rules.md#H4 states the convention and the parse reports `title-or-name`, which is what [#491](https://github.com/derek73/python-nameparser/issues/491) asked for and is a report rather than a change of reading. See decisions.md#H4. -- 2026-09-01 — FIX CANDIDATES, four. The marker stays and now carries its issue, so `pytest -rx` names the work instead of listing anonymous aspirations: - - [#489](https://github.com/derek73/python-nameparser/issues/489) — `Her Majesty Queen Elizabeth` should address by given name (`tests/test_conjunctions.py::test_conjunction_in_an_address_with_a_first_name_title`). +- 2026-09-01 — FIX CANDIDATES, four. Each carries its issue, so `pytest -rx` names the work instead of listing anonymous aspirations. THREE of the four still carry a marker; #489 was fixed 2026-09-08 and its marker is gone, which is why this line no longer says the marker stays: + - [#489](https://github.com/derek73/python-nameparser/issues/489) — **FIXED 2026-09-08 (the #316/#489 bundle).** `Her Majesty Queen Elizabeth` should address by given name (`tests/test_conjunctions.py::test_conjunction_in_an_address_with_a_first_name_title`), and it does: a title RUN addresses as its LAST title does, `queen` being a given-name title (#H1). The xfail marker is removed and the test is an ordinary passing pin. - [#490](https://github.com/derek73/python-nameparser/issues/490) — `E.T. Smith` (`tests/test_conjunctions.py::test_two_initials_conflict_with_conjunction`) and `U.S. District Judge Marc Thomas Treadwell` (`tests/test_titles.py::test_chained_title_first_name_title_is_initials`). One issue for two tests deliberately: each test's own comment names the other's shape as what blocks a fix — dotted initials against dotted title and credential vocabulary — so they are one question, and fixing either alone is what has failed before. - [#492](https://github.com/derek73/python-nameparser/issues/492) — `capitalize()` leaves `juan garcia III` lowercase (`tests/test_capitalization.py::test_capitalization_exception_for_already_capitalized_III_KNOWN_FAILURE`). The `_KNOWN_FAILURE` suffix is kept: it is still true, and beside the annotated reason it reads as redundant rather than misleading. -- [#485](https://github.com/derek73/python-nameparser/issues/485) is superseded. It proposed prefixing corpus labels with `xfail:` so a radar diff on a name like `Dr King Jr` reads as a known-bad parse improving rather than a regression. The premise was that a v1 xfail marker is a usable triage signal; after this triage it is not, because the marker no longer tracks the disposition — `Dr King Jr` is now a PIN, so a radar diff on it means a decided reading moved, which is precisely the signal the prefix would have suppressed. The four that remain carry issue numbers, which is the same information in a place that cannot go stale against the pinned historical ref. +- [#485](https://github.com/derek73/python-nameparser/issues/485) is superseded. It proposed prefixing corpus labels with `xfail:` so a radar diff on a name like `Dr King Jr` reads as a known-bad parse improving rather than a regression. The premise was that a v1 xfail marker is a usable triage signal; after this triage it is not, because the marker no longer tracks the disposition — `Dr King Jr` is now a PIN, so a radar diff on it means a decided reading moved, which is precisely the signal the prefix would have suppressed. The four that remain carry issue numbers, which is the same information in a place that cannot go stale against the pinned historical ref. The 2026-09-08 fix STRENGTHENS this rather than unsettling it, and the argument is worth stating because the fix looks at first like the counterexample: `Dr King Jr` moved, so a diff on it is exactly what #485's prefix predicted — but it moved by DECISION, with a ledger rule and a rewritten pin, which is a classified change and not a known-bad parse quietly improving. Had the prefix shipped, that diff would have read as the aspiration finally coming true and nobody would have been asked to classify it. Three markers remain rather than four, and they still carry issue numbers. ### suffix-field-composition — three kinds of thing in one field @@ -393,27 +393,59 @@ Declined (ambiguity kinds for script-resolved names, 2026-07-27): - 2026-08-25 (#410) — H1's "and nothing else" counted a suffix, a nickname and a maiden name as further name words, so a title-plus-surname name reported no family the moment any of them stood beside it: `Dr. Smith` reads family 'Smith' and `Dr. Smith née Jones` read given 'Smith' with the family empty. #410 reports the maiden flavor; measured, all three roles suppress the rule identically and the fix is one term in one guard, so the term went rather than the maiden role being special-cased. What decided the width is H1's own rationale — a title addresses by surname — which says nothing about what stands BESIDE the name. The rationale is stated for H1 rather than as a doc-wide principle about what a suffix is, deliberately: N3 counts a suffix the other way (`'Smitty' Jones Jr.` reads given 'Jones', family ''), and which of the two readings is right for a nickname-led name is not decided here. #399 is what made this urgent rather than latent: stopping the particle chain at the marker routed the canonical title-and-particle shape (`Freiherr von Richthofen geb. Albrecht`) into H1 for the first time. N3 moves with it, without N3 changing: `'Smitty' Dr. Jones` declines N3's one-piece count as it always did, and H1 now names the family behind it, so it reads family 'Jones' where it read given 'Jones' through 2.1. Five corpus names change reading at every baseline, of which four arrive as new diffs and are classified with the fix. The fifth, `Freiherr von Richthofen V`, was already classified under the `fix(#424)` rule: #410 narrows its diff from {given, family, suffix} to {family, suffix}, and `classify()` takes the first rule whose declared `fields` are a SUPERSET of the observed diff, so a shrinking diff kept matching and no run ever named it. A real movement behind a green gate, and the second of that kind here: the fields-only `fix(suffix-routing)` catch-all was absorbing three names at 1.4.0 the same way, its heading growing with nothing to announce it. Both are one shape — a rule broader than the diff it explains — and that is the lesson worth keeping rather than either fix. It is also, being a title, one name word and a suffix, the suffix flavor of this very shape, which two ledger comments had claimed no corpus carries. At 1.4.0 three of the four were being absorbed by the fields-only `fix(suffix-routing)` rule (its heading went 14 -> 17) — invisible to `_CORPUS_CLAIMS`, which records the whole corpus for a rule with no `name_regex`, so the guard could not see them arrive. They have their own rule now, as #372's names got one off the same catch-all. The v1 suite shipped the nickname reading as a strict xfail (`tests/test_nicknames.py::test_nickname_and_last_name_with_title`), which passes now. Known gap between the statement and the guard: H1 says "exactly one name word" and "that word", but the guard never counts units. It tests which ROLES are unoccupied — no middle, no family — and then retags every given token, so a name whose given run is two name WORDS (units, not tokens) reaches it and moves both: `Freiherr de V Jr` reads family 'de V'. The distinction matters for reading the claim that follows: a multi-token given run is ordinary — the rules.md example `Freiherr von Richthofen V` has one, its two tokens being a single particle unit — and it is a run of two UNITS that no real input produces. A 58,338-name sweep found no corpus name and no rules.md example with one — but it is recorded because decisions.md elsewhere leans on H1's "exactly one name word" as a settled scope when narrowing P5's licence, and that reading of H1 is stronger than the code behind it. Not fixed here: the widening this entry records is about which roles disqualify the rule, not about how it counts. +- 2026-09-08 [#489](https://github.com/derek73/python-nameparser/issues/489) (the #316/#489 bundle) — a title RUN addresses as its LAST title does. `Her Majesty Queen Elizabeth` read family `Elizabeth` because the `given_name_titles` lookup was keyed on the whole run, and `her majesty queen` is no shipped entry. A run is written as several titles and is addressed the way its final one is, so the run's folded key is now matched whole OR by its last word. **This AMENDS the 2026-08-22 #369 entry under #P5**, which keyed both sites on the whole run so they could never disagree; the amendment keys both on the LAST word and the invariant is untouched — H1's retag and P5's licence still cannot read one run two ways, through one predicate beside `_title_key`. What the amendment reopens is that entry's EXAMPLE and not its reasoning: `'mr sir'` was #369's own instance of a run neither site treats as a given-name title, and it now IS one to both, `sir` being the last word. + The whole-run arm is kept for a caller's multi-word phrase entry — `lt col` is stored and matched as one key — and its order carries nothing: the `or` short-circuits but decides nothing, since for a ONE-WORD run the key IS its last word. Every shipped given-name title is a single word, asserted in tests/v2/test_lexicon.py rather than counted here, so under the default vocabulary that arm can only ever match a one-word run, which the last-word arm reads the same way. Recompute the single-word property with `L = Parser().lexicon; sorted(t for t in L.given_name_titles if " " in t)`, which gives `[]` on 2026-09-08; the set holds 47 entries that day, and the argument survives the digit moving. + Measured over the corpus glob as it stood at the fix: ONE name moves, `Her Majesty Queen Elizabeth`. TWO after the docs commit that follows, which puts `Dr. Sir John` in corpus_rules.jsonl as an H1 example — re-measured there with the predicate's last-word arm dropped to a whole-run key, so the diff read forwards is this amendment's. Probes that move with it: `Her Majesty Queen Elizabeth II`, `Reverend Mother Teresa` (`mother` is a given-name title), `Dr. Sir John`, `Mr Sir John`, `Xyz. Sir John` — H2's unlisted word in the run does not change the run's last word — and, through P5's licence, `Sir Sheikh abdul rahman`, which reads given `abdul rahman` with an EMPTY FAMILY, the Accepted outcome H1 already records for `Sir John`. + VOCABULARY SCOPE, recorded so the boundary is not read as a gap this bundle left. `prince`, `princess`, `lord` and `lady` are not given-name titles, so `His Excellency Lord Duncan` still reads family `Duncan` and `Her Royal Highness Princess Anne` family `Anne`; `king`, `queen` and `mother` are. Whether the first four should join is a vocabulary question with its own frequency argument — it would move `Prince Harry` and `Lady Gaga` — and is deliberately out of this bundle, filed as [#519](https://github.com/derek73/python-nameparser/issues/519). Recompute the membership with `sorted(Parser().lexicon.given_name_titles)`. + Frame delta measured at zero on both entry points: each site's read sits inside the branch it serves, and the reference name `Dr. Juan0000 de la Vega III` enters neither — H1's guard wants an unoccupied family, P5's a bound given-name word. + TRAILING REACH, added 2026-09-09 in review of the docs commit. H1's site asks which ROLES are unoccupied and never where the title stands, so a run that #H5 chained in BEHIND the one name word decides that word's field exactly as a run in front of it does. Measured on this tree: `Smith Sir.` reads given `Smith` with an empty family and `Smith Queen.` reads given `Smith`, both runs ending in a given-name title, while `Smith Dr.` reads family `Smith`. No code moved for this — the reading has been the shipped one since the trailing walk landed — but the STATEMENT said "a title followed by exactly one name word" and described only the front slot, so what the review changed is the statement, an example line (`"Smith Sir." → given="Smith"`, which puts the name in corpus_rules.jsonl), and the `interacts:` lines, H1 gaining H3 and H5 and H5 gaining H1. The trailing half is where H1 and H5 meet: H5 decides which words leave the name, H1 decides the field of the one left standing. + ### H2 — the leading-abbreviation title - 2026-06-30 (leading-period-title design; v2 core, PR #288) — the shape test is v1 parity (period_abbreviation): two-plus letters then a period, bare initials exempt. Its site is the head of the part CARRYING THE GIVEN NAME — the whole name, or the post-comma part under a family comma — not "the head of the name"; that scope correction is PR #315 (2026-08-01, docs-only), verified against 1.4.0 from PyPI, so the parity claim is real and the narrower description never was. The extraction litmus (2026-08-15): the spec drafted this rule as "recognized by vocabulary, not by written shape" and the live parser falsified that framing — the shape heuristic is real, and what the abugida gap (#343/#344) shows is its LIMIT, not its absence. Recorded as the rule's Accepted consequence — and the 2026-08-15 landing initially re-narrowed the scope to "name- opening", correcting one error while preserving another; the eighth review round fixed it. -Open: [#316](https://github.com/derek73/python-nameparser/issues/316) what a trailing title-vocabulary word should do (the comma paths disagree today). +Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparser/issues/316), what a trailing title-vocabulary word should do while the comma paths disagreed): the trailing slot reads VOCABULARY, not shape (#H5), and the leading inference here stays unconditional — #316's open question 2, a symmetric leading rule that would read "Esq. Smith" as a suffix, is DECLINED. #109 shipped the leading inference on purpose, the family comma already carries the credential case ("Smith, Esq." → suffix, this rule's own Accepted clause), and reversing S2's "a suffix never opens the string" for period-marked words for the sake of one word is the wrong trade. The two comma paths agree now: "John Smith Prof." and "Smith, Prof." both read title Prof. See #H5. + +### H3 — the title run's floor + +- 2026-09-08 (the #316/#489 bundle) — the leading title run leaves a name word standing, and a post-nominal is not one. Derek's question is what opened it, verbatim: "Why does `Dr King Jr` need to parse `king` differently than `Dr King`? Jr is a recognized suffix, so it could not count as a following name." The mechanism behind the old reading is an ORDER: the leading peel runs before the trailing suffix peel and its only floor was "a title needs a following piece", so it took `Dr King` whole and left `Jr` to be the name — title `Dr King`, family `Jr`, no suffix at all. The floor added here is a second one, asked of the run's last word: where everything behind the run is a suffix piece and that word is not itself suffix vocabulary, the run gives it back. `Dr King Jr` now reads title `Dr`, family `King`, suffix `Jr` — what v1 wanted (#v1-xfail-triage), what rules.md#S2's descriptive note predicted, and what the comma spelling `King, Dr Jr` has always given. +- Measured over the differential corpus glob as it stood at the fix — 1123 distinct names, 1263 rows, before this bundle's own rules.md examples entered corpus_rules.jsonl and made it 1136 and 1284 — tree against the same tree with the floor removed: exactly one name changes, `Dr King Jr`. FOUR after the docs commit, which puts `Dr. King MD`, `Dr Jr` and `Sir Jr` in corpus_rules.jsonl as examples of the floor and its edge — re-measured there, and `Prince of Wales Jr` enters with them and does NOT move, which is the one-word gate witnessed in the corpus. The case row `Dr. King MD` moves the same way. Recompute by parsing every name in `tools/differential/corpus*.jsonl` twice, once on the tree and once with `_pieces.leading_titles` monkeypatched back to the bare "a title needs a following piece" loop, and diffing the seven name fields plus `ambiguities`; 1.4.0 gives the degenerate reading for both names, so the change diffs at every baseline. +- The word given back must be a NAME CANDIDATE, and that is what leaves the degenerate inputs alone. All-suffix inputs are untouched because the run's last word is itself suffix vocabulary — `MD DDS` keeps title `MD`, family `DDS`, and `Jr. Ph. D.` keeps title `Jr.`, suffix `Ph. D.` — and all-title inputs are untouched because there is no rest for the floor to read: `Dr.`, `Marquess of Bath` and `Coach` are unchanged. +- ONE WORD is the gate, and it is a Task 1 review outcome rather than the drafting's shape. A joined title unit given back would lose the title entirely, which is worse than the reading it replaces: `Prince of Wales Jr` keeps title `Prince of Wales`, family `Jr` rather than becoming given `Prince of Wales` with no title at all. `Lord Chancellor Jr`, whose run is two separate words, does move — title `Lord`, family `Chancellor`, suffix `Jr`. +- The edge accepted: where the run's whole content is the word given back, no title is left to make the reading H1's, so the word stands as the NAME rather than as the family. `Dr Jr` reads given `Dr`, suffix `Jr` and `Sir Jr` given `Sir`, suffix `Jr`, both reporting `title-or-name` by H4's lone-title-word convention. That is the residue rules.md#S2's note is now scoped to. +- Two residuals the floor does not reach, recorded because they look like misses and are not. A run whose LAST word is itself suffix vocabulary is not given back, so `Dr King MD PhD` still reads title `Dr King MD`, family `PhD` — the floor asks about one word, not about the run's contents. And the floor asks `is_suffix_piece`, which vetoes a bare initial-shaped numeral, so `Dr King V` keeps the whole run as the title and reads given `V` (`king` being a given-name title and the run's last word, which is #H1's 2026-09-08 amendment), where `Dr Smith V` reads family `Smith`, suffix `V`. That numeral fork is #401/#421's territory and is deliberately outside this floor. +- The floor lands in `leading_titles` rather than in assign because that predicate is the ONE answer to where the leading run ends (mechanisms.md#ONE-PREDICATE-PER-QUESTION): assign sets the roles from it and group's chain guard reads the same count, so a floor in assign would have given the two sites different runs. One cost measured and one shape forced by it: the branch's entry gate is two inline tag reads rather than `is_suffix_piece`, because `leading_titles` runs four times per parse and asking the authoritative predicate first cost 8 calls per parse of the reference name against a band with room for two (#parse-cost). The tag reads are the cheapest NECESSARY condition for the next piece to be a suffix piece at all, so an ordinary titled name leaves the branch without entering a frame; `is_suffix_piece` stays the predicate that ANSWERS. Frame delta measured at zero on both entry points. ### H4 — an input that is nothing but vocabulary still has to name somebody - 2026-09-07 #491 — the reading is unchanged and the silence is what ends. Handed a string the title peel eats down to one last word which is itself title vocabulary, the parser reads that word as the name; decisions.md#v1-xfail-triage recorded it in the fourth of its NOT FIXED entries — "this is a name parser, not a title parser" — and said in the same breath that what is actually wrong is the guess being silent. It now reports `title-or-name`, with `detail` naming the word that was made into the name. - Six corpus names gain it: the Queen's Bench string, `Lord Chancellor`, `Dr. King`, `The Rt Hon`, `His Holiness` and `His Holiness the Dalai Lama`. Three read contract-tier and three radar, and the tier split is an artifact of the documentation rather than of the shape — the three contract ones are contract because this bundle made them rules.md examples, which puts them in corpus_rules.jsonl. `Dr. King` is the one worth arguing about and it is deliberate: `king` is in the titles vocabulary for the addressing forms, which the triage entry above decided and did not reopen, so `Dr. King` IS an input whose last standing word is title vocabulary and the rule claims it. Reporting there is honest rather than noisy — the reading came from a convention, not from anything the input says — and a caller who wants only the exotic cases has `detail` to filter on. + Six corpus names gain it when this is written: the Queen's Bench string, `Lord Chancellor`, `Dr. King`, `The Rt Hon`, `His Holiness` and `His Holiness the Dalai Lama`. Three read contract-tier and three radar, and the tier split is an artifact of the documentation rather than of the shape — the three contract ones are contract because this bundle made them rules.md examples, which puts them in corpus_rules.jsonl. (2026-09-08, the #316/#489 bundle: the peel half's population is TEN, and the count moved for two different reasons. `Dr King Jr` and `Dr. King MD` gained the report because the title run's floor now leaves a name word standing where the run used to swallow it (#H3), and `Dr Jr` and `Sir Jr` gained it as the same floor's residue; three of the four also ENTERED the corpus in that commit, as rules.md examples of the floor. Seven contract and three radar now. Recompute rather than trusting either number: parse every name in the `tools/differential/corpus*.jsonl` glob and collect the ones whose `ambiguities` carry the kind, keeping the tier the file it first appears in declares — the README's table is where the tiers are.) `Dr. King` is the one worth arguing about and it is deliberate: `king` is in the titles vocabulary for the addressing forms, which the triage entry above decided and did not reopen, so `Dr. King` IS an input whose last standing word is title vocabulary and the rule claims it. Reporting there is honest rather than noisy — the reading came from a convention, not from anything the input says — and a caller who wants only the exotic cases has `detail` to filter on. ONE emitter, and it is at assign's lone-name-word site rather than at H1's retag, which is where the drafting put it. H1 is not the site: under a declared family-first order the assignment places the word in the family directly and H1 never runs, so an emitter there would report under one order and not the other for a reading that is the same either way. Measured under the default order, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, the six report identically. That placement is also why the `detail` names no field, unlike O5's: under the default order H1 retags the word after assign, so a field named at the emitter would be the one the word was placed in and not the one it ends in — and the fork the kind reports is title-versus-name, which no field answers either way. - What is silent, all measured 2026-09-08. A lone title word: `Dr.`, `Sir`, `King` and the chained `Prince of Wales` are a title run with nothing behind it, the peel takes the whole string, no word is left standing to be read as a name, and nothing was chosen — mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE's "a branch that runs but changes nothing is not a decision". A title with an ordinary word behind it: `Dr. Smith` and `King Charles` leave a word standing, but not a title-vocabulary one, so H1 alone explains them. And a title followed by post-nominal vocabulary: `Dr King Jr` and `Dr. King MD` peel `Dr King` and `Dr. King` WHOLE, leaving a credential that the bare-suffix carve-out makes the name — a different convention, and its report is scoped to inputs no title stands in (spelled `n == 0` when this was written, one clause of the `field_undecided` predicate since the 2026-09-08 consolidation recorded under O5), so a title in front of the run takes the input out of it and leaves the reading H1's. The peeled titles are never tested for anything: H2 makes an unlisted abbreviation a title by SHAPE, and `Xyz. Smith` is not this input. What the rule turns on is the word left standing. + What is silent, all measured 2026-09-08. A lone title word: `Dr.`, `Sir`, `King` and the chained `Prince of Wales` are a title run with nothing behind it, the peel takes the whole string, no word is left standing to be read as a name, and nothing was chosen — mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE's "a branch that runs but changes nothing is not a decision". A title with an ordinary word behind it: `Dr. Smith` and `King Charles` leave a word standing, but not a title-vocabulary one, so H1 alone explains them. And a title followed by post-nominal vocabulary: `Dr King Jr` and `Dr. King MD` peel `Dr King` and `Dr. King` WHOLE, leaving a credential that the bare-suffix carve-out makes the name — a different convention, and its report is scoped to inputs no title stands in (spelled `n == 0` when this was written, one clause of the `field_undecided` predicate since the 2026-09-08 consolidation recorded under O5), so a title in front of the run takes the input out of it and leaves the reading H1's. **Corrected 2026-09-08 (the #316/#489 bundle):** the two example names no longer read that way and the sentence is wrong about them, though the SCOPE it describes is unchanged and still true of the suffix half. The title run's floor gives back a one-word last piece where everything behind the run is post-nominal (#H3), so `Dr King Jr` reads title `Dr`, family `King`, suffix `Jr` and `Dr. King MD` title `Dr.`, family `King`, suffix `MD`. Both now leave a NAME word standing, and both report through this rule's TITLE half rather than being silent — `king` being title vocabulary is what makes them the peel shape. What is still true, and is what the sentence was written to say, is that the SUFFIX half is reached only where no title was peeled first: `MD DDS` has a title peeled and reports nothing, `DDS` being no title. rules.md#H4's Accepted clause is rewritten to match, and `Dr. King MD` is an example line there now. The peeled titles are never tested for anything: H2 makes an unlisted abbreviation a title by SHAPE, and `Xyz. Smith` is not this input. What the rule turns on is the word left standing. A refinement of Derek's, made after the population was measured and widening the rule past the all-titles shape it was drafted for: a lone name word that is a JOIN (P3) carrying title vocabulary reports `title-or-name` too, the fork there being whether the title word inside the unit is a title at all rather than which field the unit takes. `John of Prince` and `Smith and Prince` are the measured inputs; no corpus name reaches that branch, because a join LED by a title word is chained into a title run by H3 (`Prince of Wales`), so the two are pinned as case rows rather than as rules.md examples. It sits on O5's branch and takes precedence there, which is why O5's statement says a title silences THIS kind and not every report at the site. The suffix half is the same argument on the other vocabulary and needed no new kind. An input whose every word is post-nominal vocabulary reads its first word as a name — assign's "everything suffix-shaped after titles: first one is the name" carve-out — and that is the doubt SUFFIX_OR_NAME already names, so it reports that. `Rinpoche` and `QC MP` are the corpus names; `PhD`, `MBA` and `III` are the same shape. `Jr.` alone is NOT the shape at all: H2's opening-abbreviation rule reads it as a title before the suffix vocabulary is consulted, which is that rule's stated precedence and is recorded here because the expectation going in was that `Jr.` alone read as a name. The guard carries two exclusions of its own. A maiden name beside the credential says the input is not post-nominal vocabulary and nothing else, so `abd née Jones` is out for the reason M4 keeps it out of O5's report. And the report is scoped to names no script order placed, so a lone glued CJK honorific — さん, 씨, 선생님 — reports nothing: that is the same shape read through the glued-honorific rules (W2, #271/#308) and the script's own order, and whether those readings should report is left to the arc that revisits them rather than settled here. An asymmetry on the boundary between this rule and O5, noted and deliberately not fixed. `MA` and `Ma` alone report `given-or-family`; `PhD` alone reports `suffix-or-name`. Both are a bare credential with nothing beside it, and what separates them is which gate reads them: `ma` is an AMBIGUOUS acronym, so S2's gate declines to peel it and the word stands as the one name word, which is O5's branch; `phd` is unambiguous, so it peels to suffix, leaves no name piece, and reaches the bare-suffix carve-out, which is this rule's. Two conventions, two kinds, and the reading each name gets is the same either way — the caller sees a report in both cases and only the kind differs. Fixing it would mean one of the two gates changing what it reads, which moves fields for a bundle that moves none. - Nothing moves but the report, and the rules go in the three 2.x ledgers only — two of them at first and a third after the 2026-09-08 round below, which is where the ten names and the join clause's own rule come from. None is a copy of a wordlist and all three are declared in `_NOT_A_VOCABULARY_COPY`: a member copying TITLES would reach `Dr. Smith`, `Rabbi Cohen` and every other title-plus-surname name, and a member copying the suffix sets would reach `John Smith MD` and `Smith Jr.`, none of which moves. What selects these ten names is a SHAPE — the peel leaving one unit, and vocabulary standing in it — which no wordlist expresses. + Nothing moves but the report, and the rules go in the three 2.x ledgers only — two of them at first and a third after the 2026-09-08 round below, which is where the ten names and the join clause's own rule come from. None is a copy of a wordlist and all three are declared in `_NOT_A_VOCABULARY_COPY`: a member copying TITLES would reach `Dr. Smith`, `Rabbi Cohen` and every other title-plus-surname name, and a member copying the suffix sets would reach `John Smith MD` and `Smith Jr.`, none of which moves. What selects these names is a SHAPE — the peel leaving one unit, and vocabulary standing in it — which no wordlist expresses. The population is EIGHT distinct names when this is written and TWELVE after the #316/#489 bundle; the ten this entry says twice above are two OTHER quantities — the ten corpus ROWS the eight names occupy, `Dr. King` and the Queen's Bench string each appearing in two corpus files, and the peel half's ten after the bundle. Recompute all three by parsing every name in the `tools/differential/corpus*.jsonl` glob, collecting the ones whose `ambiguities` carry the kind, and counting rows and distinct names separately (2026-09-09: eight names in ten rows at a0b93f0, twelve names in fifteen rows on this branch). - 2026-09-08 #518 review round — the join clause is HOISTED beside the peel clause instead of sitting under O5's. It had been written inside O5's `n == 0` leg, so a leading title, a maiden marker or a vocabulary claim on the word silenced it — and every one of those decides which FIELD the unit takes, which is not what this clause asks. Measured before and after on this branch: `Lord Chancellor née Jones` and `Dr. King née Jones` (maiden), `Dr. Smith and Prince` and `Mr. John and King` (a peeled title), `van and Prince` and `J. and Prince` (a claimed word) were all silent and all now report `title-or-name`, with no role moving on any of them. The maiden pair takes the peel half and the other four the join half. TWO CORPUS NAMES MOVE, which the drafting expected to be zero: `Attorney General of Minnesota` and `Deputy Secretary of State`, both a title peeled in front of a joined unit that stands last — and a title needs a following piece, so H3 cannot chain the join into a title run the way it chains `Prince of Wales`. "No corpus name reaches the branch" was true only of the guarded version, and the claim is corrected in rules.md#H4 as well. They take a third `feat(#491)` ledger rule of their own rather than joining the all-titles alternation, whose issue line describes an all-titles input, which neither of these is. Also this round: the emitter comment's carve-out list no longer names "a group-flagged credential" (no such condition is in the code, and `phd_split` pins the opposite), and the `Order` NamedTuple is `EffectiveOrder` with field `order`, `roles` having been the name of two different things three lines apart. +### H5 — the trailing title run + +- 2026-09-08 (the #316/#489 bundle; closes #316(a)) — a trailing run of period-marked title words is a title. `John Smith Prof.` read family `Prof.` and lost the surname while `Smith, Prof.` read title `Prof.`; the two comma paths disagreed and #316 is that question. They agree now. +- **The doctrine, stated rather than left to be inferred.** A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (#H2) and the shape outranks the vocabulary, which is why `Esq. Smith` reads title though `esq` is post-nominal vocabulary. At the back there is no shape rule, so only a listed title word chains into the title and `John Smith Xyz.` keeps family `Xyz.`. A BARE title word at the back is a name word — `John Smith Sir` and `Mary Jane King` are untouched — TITLES holding ordinary surnames being exactly why, which is the Excluded block under #P5 ("the trailing-position rule that must NOT be adopted"). The asymmetry is honest rather than an oversight: #109 shipped the leading inference on purpose, and no blanket trailing vocabulary rule is safe while king, judge and bishop are titles. +- **Q3 declined: H2 stays unconditional.** #316's open question 2 asked for the symmetric leading rule, `Esq. Smith` → suffix. Declined: it reverses rules.md#S2's "a suffix never opens the string" for period-marked words for the sake of one word, and the family comma already carries the credential case (`Smith, Esq.` → suffix). Recorded at #H2, where the `Open: #316` hook was. +- **Q2 decided: a RUN, mirroring #H3.** Successive single words that wear the abbreviation shape AND are title vocabulary chain into the title from the end, floor one name piece, so `John Smith Prof. Dr.` reads title `Prof. Dr.` rather than making `Prof.` a name word. Not chosen: one word only, which would have left `Prof.` standing as a name word in exactly the input the rule is about. The title field joins in INPUT order and the rendering needed no change — the title view joins TITLE tokens in token order already, so `Dr. John Smith Prof.` reads title `Dr. Prof.` for free. +- **The predicate refuses `is_leading_title` and that refusal is the rule.** The walk uses `is_title_piece`, the vocabulary read, shared with the leading run so the two cannot disagree about what a title WORD is while disagreeing, deliberately, about what a title SHAPE is. `is_leading_title` carries H2's unlisted-abbreviation inference; with it, `John Smith Xyz.` would lose its family name to a title. That is the plan's mutation check for this commit. +- **No fork is reported**, which is #316's open question 4. Under the input-is-a-name premise a period-marked title word is not a reading a reader would hesitate over; where the doubt is real it is the word left STANDING that carries it, and #H4 already reports that. No `AmbiguityKind` is added by this bundle and `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` is unchanged. +- **A1, the peel order, decided by Derek: the first peel is PROVISIONAL and a trailing title is TRANSPARENT to the suffix reading.** `X Prof. Y` reads exactly as `X Y` reads, plus the title. SCOPED 2026-09-09, in review of the docs commit: that is a claim about inputs where a name word still stands on both sides of the chain — two or more name words. Where the chain leaves ONE name word there is no second reading for it to be transparent to, and #H1 decides the field instead: `Smith Prof.` reads family `Smith` where `Smith` alone reads given `Smith` and reports `given-or-family`, and `Smith Sir.` reads given `Smith` with an empty family, `sir` being a given-name title. All three measured on this tree; rules.md#H5's statement carries the scope and `_assign.py`'s excerpt of it was updated in the same edit. The problem is order: with ONE peel, `Prof.` standing behind `Jr.` stopped the suffix peel before `Jr.`, and the title walk then removed the very word that had been blocking it — `John Smith Jr. Prof.` read family `Jr.`, a generational suffix promoted to the family name, which is worse output than the reading it replaced. So the pieces the walk takes are spliced out and ONE peel runs over what stands, in original order, and that second answer alone places a piece or reports a fork. Measured: `John Smith Jr. Prof.` reads suffix `Jr.`; `John Prof. MA` reads the family `MA` that `John MA` reads, S2's reserve keeping a bare ambiguous acronym the family of a two-word name where a second peel laid over a first read family `John`, suffix `MA`; and `John Smith V Prof. VI` reads what `John Smith V VI` reads — middle `Smith V`, family `VI`, nothing reported — where two peels each reporting their own last piece reported twice. The choice is CORPUS-NEUTRAL: both variants move the same eight names, no more and no fewer, so it is a question about output quality alone. +- **A2, the family-comma segment-1 path, decided: it gets the walk.** The spec's condition was "if the comma segment gate already routes those, say so and leave it". Measured, it does not: `Smith, John Prof.` read middle `Prof.` at 1.4.0, 2.0.0, 2.1.0, 2.2.0 and at this branch's parent — all five measured — while `Smith, John Prof. Dr.` read middle `Prof.`, suffix `Dr.` through 2.1.0 and middle `Prof. Dr.` from 2.2.0, `dr` having left the suffix vocabulary in #296. The eleven comma rows the spec calls "already routed" are the `Smith, Prof.` shape, where segment 1 holds NO name word and `segment_suffix_reading` reads it piece by piece — a different gate and a different mechanism. So the segment-1 walk was genuinely missing; with it, `Smith, John Prof.` reads title `Prof.`, given `John`, family `Smith`. It moved NO corpus name at the fix; `Smith, John Prof.` is a rules.md#H5 example, so it ENTERS the corpus in the docs commit that follows and is one of the fourteen in the population bullet below. Case rows pin it either way. The segment's walk follows the same transparency principle with its own lenient loop, which is why the candidates are the pieces that walk would not read as a suffix rather than the strict suffix test alone (`Smith, John Prof. Jr.` must reach past the post-nominal, `Smith, John Prof. V` past the numeral the lenient tail test claims, #144). +- **What A2 does NOT reach, and why that is right.** `Smith, John Prof. MA` still reads middle `Prof. MA`: after a family comma a bare ambiguous acronym is a MIDDLE name and has been since 2.0 (the `Smith, Ed` cost S2 already accepted), so `MA` stands at the end of the segment, is not a period-marked title word, and stops the walk before it starts. The walk reads from the end; it does not hunt. +- **Measured population: FIVE corpus names at the fix, FOURTEEN after the docs commit that follows it.** The five are the four planted no-comma rows in corpus_issues.jsonl (`John Smith Dr.`, `John Smith Mr.`, `John Smith Prof.`, `John Smith Rev.`) plus `Andrew Perkins (Mgr.)`, which the drafting did not expect. The other nine are rules.md examples entering corpus_rules.jsonl in the docs commit, eight of them this rule's own — `Smith Prof.`, `Dr. John Smith Prof.`, `John Smith Prof. Dr.`, `John Smith Prof. Jr.`, `John Smith Jr. Prof.`, `John Prof. MA`, `Smith, John Prof.` and `Mary Jane King.`, the last of those added by the 2026-09-09 review of that commit — plus `Smith Sir.`, which is #H1's example and reaches this walk because the run it puts behind the name word is what the walk takes. So the growth is documentation rather than reach, and both numbers come off the same recipe run on the two trees. Re-measured on the docs tree with `_pieces.trailing_titles` stubbed to return 0, which disables the walk at both sites. That fifth is a genuine hit rather than a misfire: rules.md#S1 drops the brackets and reads the content exactly as if written bare, so `(Mgr.)` is a trailing period-marked title word and reads as one; its own test stays green. Recompute by parsing every name in `tools/differential/corpus*.jsonl` on the tree and on the branch's third commit's parent and diffing the seven name fields plus `ambiguities`. Baselines differ per name — `John Smith Dr.` read suffix `Dr.` at 1.4.0, 2.0.0 and 2.1.0 (`dr` left the suffix vocabulary in 2.2, #296) and family `Dr.` at 2.2.0 — so the ledger entries are per-baseline. +- **The inline frame-free gates at the two walk sites were REMOVED in review**, and this is the ONE-PREDICATE-PER-QUESTION half of the entry. Each site had a cheap inline test written to match the walk's own first condition; that is a second implementation of the question, and the measurement that justified it did not survive re-running. The band test runs early in a session where the facade sits at 453, so the "one frame of headroom" claim did not reproduce. With the gates gone the walk costs +1 frame on each entry point, inside the plan's target of two and inside `test_facade_cost_stays_within_its_band`. The walk's own cheapness is where the saving lives instead: the abbreviation-shape test is a compiled regex (a C call, no Python frame) and runs BEFORE the vocabulary call, and almost no name ends in a period-marked word, so the ordinary parse pays one match and stops. +- **A pre-existing detail mismatch, recorded and NOT fixed.** #H4's join shape reports `title-or-name` with a `detail` that says the unit was "read as a given name by convention", while under the default order H1 retags the unit to the family — so `Dr. John of Prince` reports that text with the unit in `family`. This rule adds a second input with the same mismatch, `John of Prince Prof.`, and fixes neither: the wording predates this bundle, the fork the kind reports is title-versus-name which no field answers either way, and #H4 already records why the detail names no field for the peel shape. + ### W1 — unspaced CJK division - 2026-07-27 #271 (decision; shipped in 2.1.0 via PR #294) — Korean division ships as a default: the census surname list is closed, hangul is self-selecting (a hangul entry can only match hangul text), and being unsplit is recoverable while a wrong split is not — which is also why an unrecognized name stays whole. The filed proposal (#271, 2026-07-07) asked for OPT-IN segmentation for Korean too, "like all localization"; default-on is the later refinement, and the census/self-selecting argument above is what justified promoting Korean past the blanket opt-in stance. @@ -449,9 +481,9 @@ Excluded (the never-given / ambiguous particle line, nameparser/config/particles Open (contested vocabulary memberships — the rule is right, the word's set is questioned; the issue is canonical): none. The block held three entries and all three were answered inside a fortnight — #346, #343 and #344 together on 2026-09-06 (see #indic-honorifics) and #342 on 2026-09-07 (see #suffix-acronym-collisions). The heading stays with nothing under it on purpose, the same reason an empty ledger roster section is a statement and a missing one is nobody having looked. sa, se and om are named as the next candidates by #342's own comment and carry no issue, so they are not entries here. -Excluded (SUFFIX_ACRONYMS / SUFFIX_WORDS — the esq dual membership, deliberate; AGENTS.md's gotcha carries the full algebra): +Excluded (SUFFIX_ACRONYMS — esq, removed 2026-09-08, #316/#489 bundle): -- esq is in BOTH sets and must not be "deduplicated". The load-bearing membership is the acronym one (it carries the multi-dot spellings: removing it costs "John Smith E.S.Q." its family name); the word membership is inert as shipped but is what keeps "Esq" matching for a caller who edits suffix_acronyms themselves. esq is the ONLY member of SUFFIX_ACRONYMS ∩ SUFFIX_WORDS — that singleton is why the two sets cannot carry a disjointness assert, which is the standing cost this entry defends. Deliberately no changed-parse count — the count is a property of the measuring grid, not of the code. +- esq is OUT of SUFFIX_ACRONYMS and must not be put back by a sweep that finds "Esq." parsing and assumes the acronym set is what carries it. The SUFFIX_WORDS membership carries every single-token spelling — Esq, Esq., ESQ, esq — and stops being inert with the acronym entry gone. What the acronym entry uniquely covered is the multi-dot spelling, so "John Smith E.S.Q." reads family E.S.Q. where every release since 1.4.0 read suffix; that is the whole of the cost and it is a spelling nobody writes. The criterion is #suffix-acronym-collisions', asked of the WORD rather than of the frequency: does the entry describe the word or the machinery. Esquire is a contraction, not an initialism, so the initialism set was never its home; it arrived in the 2019-12-11 bulk Wikipedia post-nominal import (af5bdab, #93) and was never reviewed. This SUPERSEDES the dual-membership entry that stood here from the 2.2 cycle, which called the acronym membership load-bearing and the word membership inert and concluded that the singleton "is why the two sets cannot carry a disjointness assert". The assert now exists — `not (SUFFIX_ACRONYMS & SUFFIX_WORDS)` in suffixes.py's guard block — and it is what the removal buys: the two sets normalize differently (the word test strips edge periods, the acronym test strips all of them), so a word in both is matched by two rules and which one fired is unreadable from outside. AGENTS.md's esq gotcha, the two defending comment blocks in suffixes.py, the `# NOT asserted:` note and the `suffix_acronym_multidot_spelling` case row all retire with it; a row pinning `John Smith E.S.Q.` → family replaces the last of those. Measured over the corpus glob as it stood at the change (1123 distinct names, 1263 rows; the docs commit that follows adds this bundle's rules.md examples and makes it 1136 and 1284): exactly one moves, `John Smith E.S.Q.` itself, and it diffs at every baseline. Classified a behavior change, a 2.x parity break, on all four ledgers. ### suffix-acronym-collisions — the trailing-position collision class, decided (2026-09-07, #342/#454) @@ -465,6 +497,7 @@ Closes #342 (a wordlist question) and #454 (a rules.md question) together, becau - **#454 is BY DESIGN, and the words are mc and vd.** mc has no vowel and is not a borne name — which reverses #342's comment, where mc is listed among the entries "borne as surnames" and is the one member of that list its model-recall hedge does not cover; #360 measured it: mc and ste are contractions of Mac and Sainte, borne in neither position, which is why both left the ambiguous particle half. The Scottish prefix never detaches from the name it belongs to, so "Donald Mc" is not a name anyone writes and its suffix reading costs nothing real. vd needs a surname after it, so a bare trailing vd is the decoration. Neither joins the ambiguous subset and the `fix(suffix-routing)` ledger rule — a two-token name ending in a credential acronym keeps it in `suffix` — stays. What #454's example actually turns on is a different membership: "Mc Donald" reads family "Mc Donald" and "John van Mc" reads family "van Mc" because mc is a never-given PARTICLE (#360), which is why the trailing bare shape is the only one at issue. - **What #454 got right, and it is a rules.md defect that is now fixed.** rules.md#P6's Accepted clause said that under the default order a comma-less name keeps its positional reading, with "Jong Anke de" → family "de" as its only example. Measured, that holds for de and for do and fails for exactly the two words that are in BOTH the particle vocabulary and the UNAMBIGUOUS suffix vocabulary: "Donald mc" and "Smith vd" lose the family entirely to `suffix`. The clause is repaired to say so, and the two places that already conceded it in prose — the `'Donald mc'` bullet of #differential-ledger's fields-only arc and the corresponding comment in expected_since_1.4.0.toml — are amended from "recorded here rather than fixed" to point at the repair. No example line moves: the clause's own example is de-shaped and stays true, which is why no test caught the defect and why regenerating corpus_rules.jsonl is a no-op here. Recompute the membership with `L = Parser().lexicon; sorted(L.particles & (L.suffix_acronyms - L.suffix_acronyms_ambiguous))`, which gives ['mc', 'vd'] on 2026-09-07, and `sorted(L.particles & L.suffix_acronyms_ambiguous)`, which gives ['do']. - **Parking lot: the shape-plus-position credential heuristic.** #490's idea without the "collides with vocabulary" part, and what would recover "John Smith RAI" without any wordlist at all. Two shapes: an ALL-CAPS acronym standing in the suffix position of a MIXED-CASE name ("John Smith XYZ", "John Smith, XYZ"), where the case contrast is the signal; and a DOTTED acronym in that position regardless of case ("John Smith X.Y.Z.", "john smith x.y.z."), where the periods are. Measured 2026-09-07, none of the four is read as a suffix today — the bare form gives family XYZ, the comma form given XYZ with family "John Smith", and the second two family X.Y.Z. — while the roman-numeral accident above reads three dotted forms as suffixes on a fork that is about generations, not credentials. That contrast is the evidence the dotted shape is unhandled: the parser already produces the wanted answer for X.Y.I. and R.A.V. by coincidence and the unwanted one for R.A.X. A follow-up issue carries it; nothing in this entry depends on it. +- **esq left SUFFIX_ACRONYMS (2026-09-08, the #316/#489 bundle).** The same criterion asked of a word the frequency argument cannot reach: does the entry describe the WORD or the machinery. Esquire is a contraction, not an initialism, so the initialism set was the wrong home for it whatever its frequency — this is the criterion's third kind of answer, and it is the one rai and cha did not need. The entry arrived with the same 2019-12-11 bulk import (af5bdab, #93) that brought rai and cha and was never reviewed either; its only unique coverage was the multi-dot spelling `E.S.Q.`, the SUFFIX_WORDS membership carrying every single-token spelling. Measured over the corpus glob: one name moves, `John Smith E.S.Q.`, from suffix to family, at every baseline. What the removal buys is the invariant this entry's neighbours could not have: `SUFFIX_ACRONYMS ∩ SUFFIX_WORDS == ∅` is asserted at import, esq having been the only member of that intersection and the reason the assert could not exist. Recompute the intersection with `L = Parser().lexicon; sorted(L.suffix_acronyms & L.suffix_words)`, which gives `[]` on 2026-09-08. The standing keep-out is in the Excluded block above. - **Not decided here:** sa stays where #296's audit put it (title and suffix dual, position decides); se and om stay unambiguous, there being no surname evidence worth standing behind and OM being the Order of Merit; the S2 comma-form boundary ("Smith, Ed" → given Ed) is not reopened. #342's own comment lists sa, se and om beside ba and cha as model recall rather than corpus-attested, and per the #360 lesson they need a human before they move. - **Measurement (2026-09-07).** Five corpus names move, every one from the radar-tier corpus_issues.jsonl, in three diff shapes with one cause: Aishwarya Rai moves {family, suffix}, Lala Lajpat Rai and John Smith RAI move {middle, family, suffix}, and John Smith, RAI and Ahmad Jayadi, CHA move {given, family, suffix}. Recompute by parsing every name in the tools/differential/corpus*.jsonl glob twice — once with the shipped lexicon, once with `Lexicon.default().add(suffix_acronyms={"rai","cha"})`, which RESTORES the two entries this bundle removed so the diff read forwards is the removal's — and diffing the seven name fields plus `ambiguities`; the ba step of the same sweep moves nothing. Aishwarya Rai is the one name whose diff DISAPPEARS at 1.4.0, that release having read family Rai too, which is why the `fix(#342) NOT WANTED` ledger rule was deleted rather than rewritten and why the 1.4.0 gate lists four names under the replacement where the three 2.x ledgers list five. Read today's intentional counts off the `corpus:` line of `uv run python tools/differential/compare.py --baseline X`; they rose by three at 1.4.0 and by five at each 2.x baseline. @@ -545,6 +578,8 @@ Closes #469, and continues the corpus-tier arc below rather than standing apart - 2026-08-22 #369 — a given-name title licenses the join with one word to spare. #367 regressed `Sheik Abu Bakar` from given 'Abu Bakar' to given 'Abu', family 'Bakar', and its release note recorded why the old reading had been an accident: `abu` is a particle as well as a bound word, and the title used to displace it out of the leading position so the particle CHAIN took 'Bakar'. `Sheik abdul salam` showed that a bound word alone never joined behind a title — the reserve wanted three name words and a title is not one. The question the issue put was whether `Sheik abdul salam` is a two-word given name with no surname, or given 'abdul' plus family 'salam'. Decided on the signal the title already carries: `sheik` is a GIVEN-NAME title, and H1 reads that membership as the assertion that the one word after it is the given name, family empty. P5 now reads it the same way — behind a given-name title the family comma's LENIENT reserve applies, 'abdul salam' joins, and `family=''` is the Accepted outcome H1 records for "Sir John". Not "after any title": `Dr. abdul salam` keeps given 'abdul', family 'salam', because `dr` addresses by family and the reserve exists for exactly that word. Keyed on the whole title run through `_title_key`, exactly as post_rules keys H1, so the two rules cannot read one run two ways: 'mr sir' is not a given-name title to either. Only STRICT relaxes — the family comma's DISABLED segment has no given name to join, and its post-comma segment is LENIENT already. + AMENDED 2026-09-08 (#489, the #316/#489 bundle): the key is no longer the whole run. Both sites now match the run's folded key WHOLE or by its LAST WORD, and the invariant this entry rests on — that H1 and P5 cannot read one run two ways — is what the amendment preserves rather than what it costs, both sites still reading through one predicate beside `_title_key`. What moves is this entry's own example: `'mr sir'` IS a given-name-title run to both sites now, `sir` being its last word, so `mr sir abdul rahman` joins where the sentence above says it does not. The leaf test that pinned the old reading, test_group's `test_the_title_run_is_one_key_as_h1_reads_it`, is renamed `test_the_title_run_is_read_as_h1_reads_it`; it runs the GROUP stage alone, so what it pins is P5's side — that `mr sir abdul rahman` joins — and not the agreement between the two sites. The agreement rests on the shared predicate, `_run_addresses_by_given` beside `_title_key`, which is the one place either site asks the question, and it is ASSERTED end to end by `tests/v2/test_parser.py::test_the_p5_licence_and_h1_read_a_title_run_the_same_way`: eleven title runs, each parsed in both spellings, requiring `parse(f'{title} John').family == ''` and `parse(f'{title} abdul rahman').family == ''` to agree run by run. Nothing else in this entry changes: `Dr. abdul salam` still keeps given `abdul`, family `salam`, `dr` addressing by family. See #H1 for the measurement and the vocabulary scope. + Measured at all three baselines: `Sheik abdul salam` moves (given, family), and with it the `Sheik abdul salam Jr` rules example — the same move with the suffix standing — and no other corpus name's fields do; `Sheik Abu Bakar` returns to the fields it had at 1.4.0, 2.0.0 and 2.1.0 — a restoration, which the ledgers record as a non-event — with one v2-surface change: the PARTICLE_OR_GIVEN report it carried through 2.1 is gone, because the join that takes 'Bakar' is P5's rather than the chain's, and a bound word read as the bound word is not a fork. The Arabic-script spelling comes along: `الشيخ عبد الله` read given 'عبد', family 'الله' and now reads given 'عبد الله', which is the name. The previous #369 ledger rules at all three baselines classified the regression "because the cause is known and recorded, NOT because the reading is wanted"; they are rewritten for the fix, and the 1.4.0 one is what keeps `Sheik abdul salam` explained at all: it landed on the fields-only suffix-routing catch-all before #451 deleted that rule, and with no fields-only rule left in any ledger the name now arrives UNEXPLAINED without it — measured by driving `classify()` over the 1.4.0 ledger minus this rule, which returns None for the name and for the `Sheik abdul salam Jr` example. A stronger reason to keep the rule than the one this entry first gave: the alternative is a red gate, not a mislabel. Two things review added before merge. The licence lifts the reserve for two WORDS, so the piece it lets the join take must be a single word: a particle chain is one piece, but it is the family name P2 built, and without that clause `Sir abdul van der Berg` read given 'abdul van der Berg', family '' — where the untitled name keeps family 'van der Berg', and so does `Sir John van der Berg`. H1's own scope is "exactly one name word", and the licence now matches it. `Sheik abdul al Rahman` therefore stays given 'abdul', family 'al Rahman', the same limit the `abd` entry above records for the three-token spelling. `Sir abdul van der Berg` is a rules.md example and so sits in the rules corpus, where it reads byte-identical to every baseline — the gate witnesses the clause; `Sheik abdul al Rahman` is in no corpus, and the group test is its only witness. And P5 now states the precedence `Sheik Abu Bakar` only exercised: a bound word that is also a particle is read as the bound word, the join outranking P4's leading-position reading, with no fork reported. That is the reading `Abu Bakar Salim` has always had — P5's join and P4's "the words stay separate" were never reconciled in writing — and the two rules now point at each other. @@ -601,7 +636,7 @@ Excluded (TITLES): - शेख / শেখ (Sheikh) — a clan name and a family name in both scripts ("শেখ হাসিনা"). The divergence from Latin is deliberate and is the sri/shri case run backwards: the Latin sheikh/sheik/shaykh/shaikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family (2026-09-06, #344/#343). - आचार्य (Acharya) — a Brahmin surname (2026-09-06, #344). - राजा / रानी (Raja/Rani) — common given names (2026-09-06, #344). -- The trailing-position rule that must NOT be adopted: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). +- The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules.md#H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules.md#H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5. Excluded (Policy.script_orders defaults): Script.KATAKANA is deliberately absent — a pure-katakana token is predominantly a transcribed foreign name kept in its source order, so nothing defaults on it (rule W4's boundary). Noted 2026-08-15: of the three Script-keyed axes, this is the one with no force-a-decision guard (mechanisms.md#FORCE-A-DECISION-TABLE), so a new Script member silently gets no order. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 42a3671d..2e2094cb 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -43,7 +43,7 @@ Problem shape. A rule should fire only under one comma convention. Contract stat Problem shape. Where should a new "recognize X" behavior live? Contract statement. A vocabulary layer first claims words for what they ARE, wherever they sit; a positional layer then reads every unclaimed word by where it STANDS. Every rule belongs to exactly one layer — with one named exception: the leading-abbreviation shape (rule H2) fires before the suffix vocabulary is consulted, so "Esq. Smith" reads title, not suffix. -How it works. The two layers compose without ordering bugs because the positional layer never overrides a vocabulary claim (rule O4 is the positional layer's contract). Lives in. _classify/_group (vocabulary side), _assign (positional side). Reach for it when. A proposed rule wants a word's identity AND its position at once — split it, or it will fight both layers. +How it works. The two layers compose without ordering bugs because the positional layer never overrides a vocabulary claim (rule O4 is the positional layer's contract). The exception is still ONE, re-checked 2026-09-08 against the trailing title run (rule H5), which reads title vocabulary in the trailing slot and is NOT a second one: it runs after the suffix peel rather than before it, so a post-nominal keeps its claim — `John Smith Esq.` reads suffix and `John Smith Prof. Jr.` reads suffix `Jr.`, both measured. That rule is also the worked answer to the Reach-for-it line below: it wanted a word's identity and its position at once and was split by SLOT, the front asking about shape (H2) and the back about vocabulary (H5), two questions with two criteria rather than one rule fighting both layers. Lives in. _classify/_group (vocabulary side), _assign (positional side). Reach for it when. A proposed rule wants a word's identity AND its position at once — split it, or it will fight both layers. ## STATE-OFFSET-CHANNELS — early facts ride the state @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles, peel_walk and peel_trailing are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles, peel_walk and peel_trailing are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` joins that last shape (2026-09-08, the #316/#489 bundle, rules.md#H5): assign alone calls it, at TWO sites — the main walk and the family-comma segment-1 walk — and it is in the leaf rather than inline because each site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. Re-measured 2026-09-08 by call site over `_pipeline/*.py`, the whole census above holds unchanged: is_suffix_piece, leading_titles, peel_walk and peel_trailing shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it diff --git a/docs/design/rules.md b/docs/design/rules.md index 304999d9..4e7a3814 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -29,27 +29,48 @@ The marker's unit is the whole rule, because that is the unit `tools/differentia ## Titles & honorifics (H) -Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. What a TRAILING title-vocabulary word should do is unresolved (#316): today "John Smith Prof." keeps Prof. a name word while "Smith, Prof." reads it as a title — the two comma paths disagree, and TITLES holding ordinary surnames (king, judge, bishop) is what bars the blanket vocabulary-wins answer. An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break +Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (H2), the shape outranking the vocabulary; at the back there is no shape rule, so only a listed title word chains into the title (H5) and an unlisted abbreviation stays a name word. A BARE title word at the back is a name word too, TITLES holding ordinary surnames — king, judge, bishop — which is what bars the blanket vocabulary-wins answer (#316, decided 2026-09-08). An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break "Rabbi Cohen". H1. Rationale: a title normally addresses by surname, so a title followed by a single name word usually names the family; but a - given-name title addresses by given name. What stands beside - that word — a suffix, a nickname, a maiden name — does not make - the name any longer, so it does not decide this reading. + given-name title addresses by given name. Several titles written + together are one form of address, and the one that does the + addressing is the last — the words in front of it rank the + person rather than name them. What stands beside the name word — + a suffix, a nickname, a maiden name — does not make the name any + longer, so it does not decide this reading. And a run written + BEHIND the one name word is the same form of address written on + the other side of it, so it decides the same reading. A title followed by exactly one name word makes that word the family name, whatever suffix, nickname or maiden name stands beside it, unless the title is a given-name title, which keeps - it the given name. + it the given name; a run of several titles addresses as its last + title does, and a title run standing BEHIND the one name word + decides that word's field the same way. "Mr. Johnson" → family="Johnson" "Mrs. Garcia" → family="Garcia" "Dr. Smith née Jones" → family="Smith" + "Her Majesty Queen Elizabeth" → given="Elizabeth" + "Dr. Sir John" → given="John" + "Smith Sir." → given="Smith" + "His Excellency Lord Duncan" → family="Duncan" "Sir John" → given="John" · boundary Accepted: a given-name title plus one name word leaves the family empty — the input names no family, and inventing one - would be worse. + would be worse. A run whose last title is a given-name title + reads the same way, `queen` doing the addressing where `her + majesty` only ranks. "Sir John" → family="" - history: decisions.md#H1 · interacts: P2, P3, P5, M2, S1, S2, N1, N3 · implemented: nameparser/_pipeline/_post_rules.py + "Her Majesty Queen Elizabeth" → family="" + "Smith Sir." → family="" + Accepted: a title word in the run that no vocabulary knows does + not change what the run addresses by, the last word being the + one asked — `His Excellency Lord Duncan` reads family `Duncan` + because `lord` is not a given-name title, not because the run is + long, and `Her Royal Highness Princess Anne` reads family `Anne` + for the same reason. + history: decisions.md#H1 · interacts: H3, H5, P2, P3, P5, M2, S1, S2, N1, N3 · implemented: nameparser/_pipeline/_post_rules.py H2. Rationale: before a name, an abbreviation is almost always a title — "Rev.", "Ing.", "Mag." — and no vocabulary can list @@ -86,22 +107,42 @@ H2. Rationale: before a name, an abbreviation is almost always a not open: the vocabulary decides, and "Esq." is the postnominal it is. "Smith, Esq." → suffix="Esq." - history: decisions.md#H2 · interacts: C1, P4 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py + history: decisions.md#H2 · interacts: C1, P4, H5 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py H3. Rationale: compound titles are written as a run of title words, connectives included; a title word standing inside the name is - just a name word. + just a name word. And a title addresses somebody, so the run + leaves somebody there to address: a post-nominal is written + about a name rather than being one, so it cannot be the name the + run left. Successive title words at the start of the part carrying the given name chain into one title; a title word elsewhere in the - name does not. + name does not. The run leaves one NAME word standing, and a word + of the suffix vocabulary is not that word — where everything + behind the run is post-nominal, the run gives its last word back + to the name, provided that word stands alone and is not itself + suffix vocabulary. "Asst. Vice Chancellor John Smith" → title="Asst. Vice Chancellor" "Marquess of Bath" → title="Marquess of Bath" "Morse, Det. Insp. Jane" → title="Det. Insp." + "Dr King Jr" → family="King" + "Dr King Jr" → suffix="Jr" + "MD DDS" → family="DDS" "John Doctor Smith" → middle="Doctor" · boundary Accepted: before a family comma the pre-comma text is wholly the family name (C1), title words included. "Dr. Smith, John" → family="Dr. Smith" - interacts: C1 · implemented: nameparser/_pipeline/_pieces.py + Accepted: the word given back must stand alone, so a title + written as one joined unit stays whole and the post-nominal + behind it is the name — turning the unit into a name and leaving + no title at all is the worse of the two readings. + "Prince of Wales Jr" → title="Prince of Wales" + Accepted: where the run's whole content is the word given back, + no title is left to make the reading H1's, so the word stands as + the name and the parse reports the doubt (H4). + "Dr Jr" → given="Dr" + "Dr Jr" → ambiguities=("title-or-name",) + history: decisions.md#H3 · interacts: C1, H1, H4, H5, S2 · implemented: nameparser/_pipeline/_pieces.py H4. Rationale: this is a name parser, not a title parser. Handed a string the title peel eats down to one last word which is itself @@ -123,6 +164,7 @@ H4. Rationale: this is a name parser, not a title parser. Handed a "The Right Hon. the President of the Queen's Bench Division" → family="Division" "The Right Hon. the President of the Queen's Bench Division" → ambiguities=("title-or-name",) "Dr. King" → ambiguities=("title-or-name",) + "Dr. King MD" → ambiguities=("title-or-name",) "Dr." → title="Dr." · boundary "Dr." → ambiguities=() "Dr. Smith" → ambiguities=() @@ -135,9 +177,12 @@ H4. Rationale: this is a name parser, not a title parser. Handed a kind is reported either way. Accepted: the suffix half is reached only where no title was peeled first, so a title in front of the run takes the input out - of this rule and leaves it H1's — "Dr. King MD" reports nothing, - the credential having been read as the name after a title peel - rather than for want of one. + of that half. What it does NOT do is take the input out of the + rule: the title run leaves a name word standing (H3), and where + that word is title vocabulary the TITLE half claims it — + `Dr. King MD` reports `title-or-name` on `King`, while `MD DDS` + reports nothing, `DDS` being no title. + "MD DDS" → ambiguities=() "Rinpoche" → ambiguities=("suffix-or-name",) "QC MP" → given="QC" "Jr." → title="Jr." @@ -185,7 +230,65 @@ H4. Rationale: this is a name parser, not a title parser. Handed a which FIELD the unit takes and none of them whether the word inside it is a title. The other measured inputs are pinned in the case table rather than here. - history: decisions.md#H4 · interacts: H1, H2, H3, S2, O5 · implemented: nameparser/_pipeline/_assign.py + history: decisions.md#H4 · interacts: H1, H2, H3, H5, S2, O5 · implemented: nameparser/_pipeline/_assign.py + +H5. Rationale: a word abbreviated with a period at the END of a name + is standing where a post-nominal stands, and the parts of a name + that get abbreviated are the ones outside it. There is no shape + rule there — H2's belongs to the front slot — so only a word the + vocabulary knows as a title is one, and a bare title word is a + name word, TITLES holding ordinary surnames. + After the trailing suffix run has been taken, successive single + words that wear the abbreviation shape and are title vocabulary + chain into the title from the end, leaving one name word + standing. A bare title word there is a name word, and an + unlisted abbreviation there is a name word. The title is + TRANSPARENT to the suffix reading: where two or more name words + stand, what stands once the chain is taken reads exactly as it + would read written without the title, plus the title. Where the + chain leaves ONE name word, there is no second reading for it to + be transparent to, and the title behind that word decides its + field as a title in front of it would (H1). + "John Smith Prof." → title="Prof." + "John Smith Prof." → family="Smith" + "John Smith Prof. Dr." → title="Prof. Dr." + "Dr. John Smith Prof." → title="Dr. Prof." + "Smith, John Prof." → title="Prof." + "Smith Prof." → family="Smith" + "John Smith Sir" → family="Sir" + "Mary Jane King" → family="King" + "John Smith Esq." → suffix="Esq." + "John Smith Xyz." → family="Xyz." · boundary + Accepted: the trailing title words join the title field in input + order after any leading run, because the title view reads the + tokens in the order they were written. + "Dr. John Smith Prof." → family="Smith" + Accepted: transparency is what makes the suffix reading proof + against a title standing among the post-nominals — a title AHEAD + of a suffix word is taken all the same, and the suffix reading + is then taken over what stands, not over what stood. + "John Smith Prof. Jr." → suffix="Jr." + "John Smith Jr. Prof." → suffix="Jr." + "John Prof. MA" → family="MA" + Accepted: the reach is the whole title vocabulary, the ordinary + surnames in it included. TITLES holds king, judge and bishop, + and a period written behind one of them is enough to make it the + title and take it out of the name: `Mary Jane King.` reads title + `King.` with family `Jane`, where the bare spelling keeps family + `King`. That is a cost accepted under the premise that the input + is a name (H Background), not a case this rule prevents — the + period is a writing convention rather than evidence about the + word, so every title whose spelling wears the abbreviation shape + is reachable this way, and the BARE spelling is what the + trailing slot is protected from. + "Mary Jane King." → title="King." + "Mary Jane King." → family="Jane" + Accepted: no fork is reported. Under the premise that the input + is a name, a period-marked word the title vocabulary knows is + not a reading a reader would hesitate over — where the doubt is + real it is the word left STANDING that carries it, which is H4's + report and not this rule's. + history: decisions.md#H5 · interacts: H1, H2, H3, H4, S2, C1 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py ## Particles & surname prefixes (P) @@ -385,8 +488,9 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" given-name title, which asserts that a given name follows (H1): there the two words join and the name has no family — two name WORDS, so neither a particle chain (P2), which is the family - name, nor a suffix word is joined. The title run is read as one - key, as H1 reads it. Where the join fires, a bound word that is + name, nor a suffix word is joined. A run of several titles is + read as H1 reads it, by the title that does the addressing, + which is its last. Where the join fires, a bound word that is also a particle is read as the bound word: the join outranks the leading-position reading (P4) and no fork is reported; where the reserve blocks the join, P4's reading and its fork stand. Where @@ -685,22 +789,25 @@ S2. Rationale: generational suffixes and credentials are recognized "Jack Wei Ma" → suffix="Ma" "Jack Wei Ma" → ambiguities=("suffix-or-name",) "Smith Jr." → family="" - Note, DESCRIPTIVE and not promised: where a title chain consumes - every word but one, the word left over is claimed by H1's - one-word reading before this rule's trailing-suffix reading - reaches it — as the family ordinarily, as the given name behind - a given-name title (`Sir Jr` reads given `Jr`). `Dr King - Jr` reads title `Dr King`, family `Jr`, empty suffix — not the - suffix `Jr` with an empty family the Accepted clause above - predicts — because `king` is title vocabulary and H1 then takes - the one remaining word. The contrast that isolates the cause is - `Dr Smith Jr`, which reads family `Smith`, suffix `Jr` as - stated. Only the vocabulary half is decided - (decisions.md#v1-xfail-triage: `king` stays a title, for the - addressing forms); the leftover reading is recorded as today's - rather than endorsed, and a change moving it toward this rule's - prediction is an improvement, to be argued here. - interacts: H1, H2, C1 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py + Accepted: the title chain no longer takes the word this rule + needs, and the argument a descriptive note here asked for is + made. A title run leaves one NAME word standing and a + post-nominal cannot be it (H3, decided 2026-09-08), so `Dr King + Jr` reads family `King`, suffix `Jr` exactly as this rule + states, where it read title `Dr King`, family `Jr` before — + `Dr Smith Jr`, which always read as stated, was the contrast + that isolated the cause and now reads like its neighbour rather + than against it. What the floor cannot reach stays descriptive + and is small: where the run's whole content is the word given + back, no title is left to name anybody, so the word stands as + the name rather than as the family — `Dr Jr` reads given `Dr`, + suffix `Jr` and `Sir Jr` given `Sir`, suffix `Jr`, both + reporting `title-or-name` (H4). The vocabulary half is decided + and unchanged (decisions.md#v1-xfail-triage: `king` stays a + title, for the addressing forms). + "Dr Jr" → suffix="Jr" + "Sir Jr" → suffix="Jr" + interacts: H1, H2, H3, H5, C1 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py S3. Rationale: credentials are often written run together with periods; the chunks between the periods are what carry the diff --git a/docs/release_log.rst b/docs/release_log.rst index 11dfb45f..cbb63b16 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -18,6 +18,14 @@ Release Log - **Mark ba as an acronym that is also an ordinary name, so a bare trailing Ba keeps the family name.** ``HumanName("Anna Ba")`` gives last ``Ba`` and reports a suffix-or-name ambiguity, where 2.0.0 through 2.2.0 gave suffix ``Ba`` and no last name. The SPACED full-name form keeps the credential reading: ``John Smith BA`` still gives suffix ``BA``, now flagged, and the dotted ``John Smith B.A.`` is an unflagged suffix, the periods settling it. The COMMA forms move, and this is the marking's real cost: ``Smith, BA`` gives first ``BA``, and ``John Smith, BA`` gives first ``BA``, last ``John Smith``, where 2.0.0 through 2.2.0 gave suffix ``BA`` for both -- what ``Smith, Ed`` costs, which S2 already accepted for the other ambiguous acronyms. A bracketed or quoted ``John Smith (BA)`` falls through to nickname parsing, as the 2.0 note for ``ma``/``do`` below recorded for that pair. Write ``B.A.`` to keep the credential reading. BA is a common credential and Ba a real surname in Vietnamese and Senegalese Fula, which is the ``ma``/``Ma`` shape exactly. No corpus name moves (#342) + - **Fix a title run addressing by its first title rather than its last.** ``HumanName("Her Majesty Queen Elizabeth")`` gives first ``Elizabeth`` with an empty last name, where every release since 1.4.0 gave last ``Elizabeth``. Several titles written together are one form of address and the one that does the addressing is the last, so the run is now matched whole or by its last word: ``Reverend Mother Teresa``, ``Dr. Sir John`` and ``Mr Sir John`` move the same way, and ``Sir Sheikh abdul rahman`` gives first ``abdul rahman``. What does NOT move is a run whose last word addresses by surname: ``His Excellency Lord Duncan`` still gives last ``Duncan`` and ``Her Royal Highness Princess Anne`` last ``Anne``, ``lord`` and ``princess`` not being given-name titles -- a vocabulary question with its own argument, filed separately (#519). A caller's multi-word entry still matches as a phrase. Two names in the differential corpora read differently for this rule. See the ``H1`` entry of ``docs/design/decisions.md`` (closes #489) + + - **Fix the leading title peel taking a name word and leaving a post-nominal to be the name.** ``HumanName("Dr King Jr")`` gives title ``Dr``, last ``King``, suffix ``Jr``, where every release since 1.4.0 gave title ``Dr King``, last ``Jr`` and no suffix at all; ``Dr. King MD`` moves the same way, and both now read as the comma spelling ``King, Dr Jr`` always has. A title addresses somebody, so the run leaves a name word standing and a post-nominal is not one. A name that is nothing but titles or nothing but post-nominals is untouched, the word given back having to be a name candidate: ``Marquess of Bath``, ``MD DDS`` and ``Jr. Ph. D.`` are unchanged, and so is a title written as one joined unit -- ``Prince of Wales Jr`` keeps title ``Prince of Wales`` rather than losing the title to make a name. Where the run's whole content is the word given back there is no title left, so ``Dr Jr`` gives first ``Dr``, suffix ``Jr`` and reports a title-or-name ambiguity. Four names in the differential corpora read differently for this rule. See the ``H3`` entry of ``docs/design/decisions.md`` + + - **Fix a trailing abbreviated title reading as a name word.** ``HumanName("John Smith Prof.")`` gives title ``Prof.``, first ``John``, last ``Smith``, where every release since 1.4.0 gave last ``Prof.`` and lost the surname; ``John Smith Mr.``, ``John Smith Rev.``, ``John Smith Dr.`` and ``Andrew Perkins (Mgr.)`` move the same way. A run chains from the end (``John Smith Prof. Dr.`` gives title ``Prof. Dr.``), a leading title keeps its place (``Dr. John Smith Prof.`` gives title ``Dr. Prof.``), and the comma forms agree with the bare ones now -- ``Smith, John Prof.`` gives title ``Prof.``, first ``John``, last ``Smith`` where it gave middle ``Prof.`` at every release. The trailing title is transparent to the post-nominal reading, so ``John Smith Jr. Prof.`` gives suffix ``Jr.`` rather than promoting the generational suffix to the last name. What does NOT move: an unlisted abbreviation (``John Smith Xyz.`` keeps last ``Xyz.``), a bare title word (``John Smith Sir``, ``Mary Jane King``) and a post-nominal (``John Smith Esq.``). Only a listed title word wearing the abbreviation period is claimed -- the leading slot infers a title from the shape alone, the trailing slot never does. Twelve names in the differential corpora read differently for this rule. See the ``H5`` entry of ``docs/design/decisions.md`` (closes #316) + + - **Remove esq from the default post-nominal acronyms, and assert the two post-nominal sets disjoint.** ``HumanName("John Smith E.S.Q.")`` gives middle ``Smith``, last ``E.S.Q.``, where every release since 1.4.0 gave suffix ``E.S.Q.``. ``Esq``, ``Esq.``, ``ESQ`` and ``esq`` are unchanged, the post-nominal word list carrying every single-token spelling; the acronym entry's only unique coverage was the multi-dot spelling. Esquire is a contraction rather than an initialism, so the initialism set was never its home, and it was the one word in both post-nominal sets -- which is why the sets can now assert they do not overlap, a word in both being matched by two rules that normalize differently. A caller who needs it back adds it: ``Lexicon.default().add(suffix_acronyms={"esq"})``. One name moves in the differential corpora. See the ``suffix-acronym-collisions`` entry of ``docs/design/decisions.md`` + - **Fix the East Slavic and Turkic patronymic rotations overriding a declared family-first name order.** With ``patronymic_rules`` opted in and ``Policy(name_order=FAMILY_FIRST)``, ``Мицкевич Адам Юзеф`` gave last ``Адам`` through 2.2.0 and now gives last ``Мицкевич`` -- the reading the declaration asks for -- and ``oglu Ahmad Vali Ali`` with Turkic handling gave last ``Ahmad`` and now ``oglu``. The rotations exist to restore the given-first reading a family-first listing hides, so under a declared family-first order the declaration decides. No corpus name moves. See the ``O1`` entry of ``docs/design/decisions.md`` (closes #384) **Additions** @@ -32,7 +40,7 @@ Release Log - **Add AmbiguityKind.GIVEN_OR_FAMILY, reported when a name of one name word had nothing to decide which field it is:** ``parse("Andrew")`` still gives given ``Andrew`` and now says that field was a convention rather than a reading -- one word gives the positional rule nothing to compare, so the library picks the given name under the default order and the family name under a declared family-first one, and ``detail`` names the field it picked. A trailing suffix does not decide it either: ``parse("Smith Jr.")`` reports it too, the suffix being peeled and the convention placing the one name word left. A name something DID decide stays silent -- ``"Dr. Smith"``, ``"Smith née Jones"``, ``"'Smitty' Jones"`` and ``"Smith, Andrew"`` -- and so do ``"abdul"`` and ``"de"``, where the bound given-name and particle vocabularies claimed the word, and ``"J."``, claimed by the initial's own shape. A name whose script settles the order is silent too: ``"毛泽东"`` reads family by convention of the writing system, not of this rule. Twenty-seven names in the differential corpora gain the report, and no field moves anywhere. See the ``O5`` entry of ``docs/design/decisions.md`` (closes #449) - - **Add AmbiguityKind.TITLE_OR_NAME, reported when an input that is nothing but honorifics had its last word read as the name:** ``parse("Lord Chancellor")`` still gives title ``Lord``, family ``Chancellor``, and now says so -- this is a name parser, not a title parser, so handed a string with no name in it, it reads the last title word as one. ``"The Right Hon. the President of the Queen's Bench Division"`` and ``"His Holiness"`` move the same way, and so does ``"Dr. King"``, ``king`` being title vocabulary for the addressing forms. A title with an ordinary word behind it is silent (``"Dr. Smith"``, ``"King Charles"``), and so is a lone title word: ``parse("Dr.")`` is a title with no name beside it, the peel having taken the whole string, so no word was left standing to be read as a name and nothing was chosen. The same convention on the other vocabulary reports the existing suffix-or-name -- ``parse("Rinpoche")`` gives given ``Rinpoche`` and flags it, as does ``"QC MP"`` -- while ``"Jr."`` is unchanged and unflagged, reading as a title on its shape. The same doubt inside a joined unit reports too -- ``parse("Attorney General of Minnesota")`` reads title ``Attorney``, family ``General of Minnesota``, and whether ``General`` is a title is the fork. Ten names in the differential corpora gain a report, and no field moves. See the ``H4`` entry of ``docs/design/decisions.md`` (closes #491) + - **Add AmbiguityKind.TITLE_OR_NAME, reported when an input that is nothing but honorifics had its last word read as the name:** ``parse("Lord Chancellor")`` still gives title ``Lord``, family ``Chancellor``, and now says so -- this is a name parser, not a title parser, so handed a string with no name in it, it reads the last title word as one. ``"The Right Hon. the President of the Queen's Bench Division"`` and ``"His Holiness"`` move the same way, and so does ``"Dr. King"``, ``king`` being title vocabulary for the addressing forms. A title with an ordinary word behind it is silent (``"Dr. Smith"``, ``"King Charles"``), and so is a lone title word: ``parse("Dr.")`` is a title with no name beside it, the peel having taken the whole string, so no word was left standing to be read as a name and nothing was chosen. The same convention on the other vocabulary reports the existing suffix-or-name -- ``parse("Rinpoche")`` gives given ``Rinpoche`` and flags it, as does ``"QC MP"`` -- while ``"Jr."`` is unchanged and unflagged, reading as a title on its shape. The same doubt inside a joined unit reports too -- ``parse("Attorney General of Minnesota")`` reads title ``Attorney``, family ``General of Minnesota``, and whether ``General`` is a title is the fork. Eight names in the differential corpora gain a report from this change -- ten rows, two of those names sitting in two corpus files each -- and no field moves. Four more inputs join the kind later in this cycle, from the title-peel fix above -- ``Dr King Jr``, ``Dr. King MD``, ``Dr Jr`` and ``Sir Jr``, where the run now leaves a title-vocabulary word standing; those move fields, for the reasons that bullet gives. See the ``H4`` entry of ``docs/design/decisions.md`` (closes #491) * 2.2.0 - August 31, 2026 diff --git a/docs/usage.rst b/docs/usage.rst index 026e2ced..17faa2a4 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -739,10 +739,24 @@ word after the given name is a middle name: Because this is structural rather than vocabulary-driven, emptying ``titles`` does not switch it off; see :doc:`customize`. -Only the title direction is inferred this way. A *trailing* -abbreviation is matched against the suffix vocabulary and nothing more, -so an abbreviated post-nominal is recognized only if it is a word the -parser already knows. +Only the leading slot INFERS. At the back of a name a period-marked +word is read by vocabulary alone: a word the parser already knows as a +post-nominal or as a title reads as one, and an unfamiliar abbreviation +stays a name word. + +.. doctest:: + + >>> parse("John Smith Prof.").title + 'Prof.' + >>> parse("John Smith Xyz.").family + 'Xyz.' + +That asymmetry is deliberate. The title vocabulary holds hundreds of +words that are ordinary surnames — ``king``, ``judge``, ``bishop`` — +so inferring a trailing title from the shape alone would cost real +names their family field. The period is what separates the two: a +trailing title word written *without* one is a name word, so +``parse("Mary Jane King").family`` is ``'King'``. Comparing names ---------------- diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 8475f6de..c079ef6c 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -9,7 +9,7 @@ across pieces); token/piece tags; Lexicon only through tags already applied by classify (plus the leading-title period rule). -Implements rules H2, H4, N3, O4, O5 and W4 of docs/design/rules.md, +Implements rules H2, H4, H5, N3, O4, O5 and W4 of docs/design/rules.md, each cited at its code below. Ports v1's assignment loops. NO_COMMA (per name_order): leading title pieces chain while no given-position name has been seen @@ -269,7 +269,10 @@ def _assign_main(seg_idx: int, state: ParseState, # they then stand, and that second answer is the only one that # places a piece or reports a fork. peeled = peel_trailing(rest, pieces, ptags, tokens) - # rules.md#H5 -- the trailing title run, read over what the suffix + # rules.md#H5: "successive single words that wear the abbreviation + # shape and are title vocabulary chain into the title from the end, + # leaving one name word standing" + # -- read over what the suffix # peel left and set BEFORE _name_positions, so the shortened list is # what the positional read and the script test both see (a trailing # Latin title must not make a wholly-CJK name look mixed-script, the @@ -603,7 +606,11 @@ def reads_as_a_suffix(m: int, last: int) -> bool: n = len(pieces) else: n = _peel_leading_titles(pieces, ptags, tokens) - # rules.md#H5 -- the trailing title run, on this walk + # rules.md#H5: "the title is TRANSPARENT to the suffix + # reading: where two or more name words stand, what + # stands once the chain is taken reads exactly as it + # would read written without the title, plus the title" + # -- the trailing title run, on this walk # too. A name word in segment 1 is what keeps the gate # above from reading the segment as a credential run, # so 'Smith, John Prof.' had no route to title at all diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 4e14a68e..ba5086d6 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -105,10 +105,9 @@ def leading_titles(pieces: Sequence[Sequence[int]], n += 1 continue break - # rules.md#H3 -- the run gives back its last piece when that piece - # is a name candidate: where everything behind the run is a suffix - # piece, the run hands its last piece back, provided that piece is - # one word and is not itself suffix vocabulary. + # rules.md#H3: "where everything behind the run is post-nominal, + # the run gives its last word back to the name, provided that word + # stands alone and is not itself suffix vocabulary" # # ONE WORD, because a joined unit led by a title is a title run and # handing it back would lose the title: 'Prince of Wales Jr' reads @@ -118,8 +117,11 @@ def leading_titles(pieces: Sequence[Sequence[int]], # Two residuals. A run whose last word IS suffix vocabulary is not # given back, so 'Dr King MD PhD' still reads title 'Dr King MD', # family 'PhD'. And the floor asks is_suffix_piece, which vetoes a - # bare initial-shaped numeral, so 'Dr King V' still reads family - # 'V' -- that numeral fork is outside this floor. + # bare initial-shaped numeral, so 'Dr King V' keeps the whole run + # as the title and reads the numeral as the name -- given 'V', + # 'king' being a given-name title and the run's last word, where + # 'Dr Smith V' reads suffix 'V'. That numeral fork is outside this + # floor (decisions.md#H3). # # The two inline tag reads are the cheapest NECESSARY condition for # the piece behind the run to be a suffix piece at all -- @@ -363,8 +365,9 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], return Peel(k, numeral, tuple(picks)) -# rules.md#H5 -- the trailing run's own predicate, and a forward note -# rather than a citation until H5 is written. NOT is_leading_title: +# rules.md#H5: "only a word the vocabulary knows as a title is one, +# and a bare title word is a name word" +# -- the trailing run's own predicate. NOT is_leading_title: # that predicate carries H2's unlisted-abbreviation inference, which is # the LEADING slot's shape rule and has no trailing counterpart, so # with it 'John Smith Xyz.' would lose its family name to a title diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 37b119a7..810369ff 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -355,13 +355,14 @@ def post_rules(state: ParseState) -> ParseState: # rules.md#H1: "a title followed by exactly one name word makes # that word the family name, whatever suffix, nickname or maiden # name stands beside it, unless the title is a given-name title, - # which keeps it the given name" -- counting those three as - # further name words is what emptied the family (#410) + # which keeps it the given name; a run of several titles addresses + # as its last title does" -- counting suffix, nickname and maiden + # as further name words is what emptied the family (#410) # (known gap: the guard tests which roles are unoccupied, it does # not count units -- decisions.md#H1) (v1 handle_firstnames) # - # rules.md#H1 -- a RUN of several titles addresses as its last - # title does (#489), so 'Her Majesty Queen Elizabeth' reads given + # rules.md#H1: "a run of several titles addresses as its last + # title does" -- #489, so 'Her Majesty Queen Elizabeth' reads given # 'Elizabeth': the run is not a given-name title but 'queen' is. # The predicate lives beside _title_key because the P5 licence in # group asks the same question of the same run, and a run read two diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 123f24dd..b2231511 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -31,10 +31,15 @@ "Berg, abdul van" "Berg, abdul vd" "Del Toro" +"Dr Jr" +"Dr King Jr" "Dr." +"Dr. John Smith Prof." "Dr. John van der Berg" "Dr. Juan Q. Xavier de la Vega III" "Dr. King" +"Dr. King MD" +"Dr. Sir John" "Dr. Smith" "Dr. Smith née Jones" "Dr. Smith, John" @@ -48,6 +53,8 @@ "Hans „Erster“ und “Zweiter” Müller" "Hassan Mohamad Ali" "Hassan, Mohamad Ahmad Ali" +"Her Majesty Queen Elizabeth" +"His Excellency Lord Duncan" "II Van Johnson" "J. Smith" "J. née Jones Smith V" @@ -78,14 +85,22 @@ "John . Smith" "John Doctor Smith" "John Ma" +"John Prof. MA" "John Smith" +"John Smith Esq." "John Smith J.u.n.i.o.r." "John Smith Jr." +"John Smith Jr. Prof." "John Smith M.A." "John Smith MD PhD" "John Smith Mc V" "John Smith PhD" +"John Smith Prof." +"John Smith Prof. Dr." +"John Smith Prof. Jr." "John Smith Q.W.E.R.T." +"John Smith Sir" +"John Smith Xyz." "John Smith, Jones" "John Smith, LEED AP" "John Smith, MD, Bart" @@ -122,6 +137,7 @@ "Juan y Garcia" "Juan y Garcia née Jones" "Lord Chancellor" +"MD DDS" "Mari' Aube'" "Maria Kowalska (z domu Nowak)" "Maria Kowalska (z domu)" @@ -129,6 +145,8 @@ "Maria Luisa y de la Cruz" "Marquess of Bath" "Mary Beth Smith" +"Mary Jane King" +"Mary Jane King." "Mc Donald" "Mesnil de" "Morse, Det. Insp. Jane" @@ -146,6 +164,7 @@ "Nguyễn, Thị Vân" "Ph. D. Van Johnson" "Ph. D., John" +"Prince of Wales Jr" "QC MP" "Rev. John Smith" "Rinpoche" @@ -159,6 +178,7 @@ "Shirley Maclaine" "Sidorov Ivan Petrovich Jr." "Sir John" +"Sir Jr" "Sir Ph. D. Van Johnson" "Sir abdul van der Berg" "Sir de Mesnil" @@ -167,6 +187,8 @@ "Smith (née Jones)" "Smith Jr." "Smith Jr., Mr." +"Smith Prof." +"Smith Sir." "Smith née Jones" "Smith née Jones PhD" "Smith, Abd" @@ -175,6 +197,7 @@ "Smith, Esq." "Smith, John" "Smith, John PhD I." +"Smith, John Prof." "Smith, John V" "Smith, John V." "Smith, Jr." From 610a30093a8d8374af9cb7b5f8adb4672e07c17b Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 01:16:49 -0700 Subject: [PATCH 06/12] chore(differential): classify the title-run bundle at all four baselines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Twenty-one corpus names move: the eight the parser changes reach, plus the thirteen the rules-doc corpus gained when Task 5 regenerated it from rules.md's own examples -- eleven at first and two more from the 2026-09-09 design-docs review of that commit, `Mary Jane King.` and `Smith Sir.`. Before, on this branch with the corpora already regenerated: corpus: 1135 names; intentional diffs: 367; unexplained: 16; radar unclassified: 5 corpus: 1142 names; intentional diffs: 297; unexplained: 16; radar unclassified: 5 corpus: 1142 names; intentional diffs: 211; unexplained: 16; radar unclassified: 5 corpus: 1142 names; intentional diffs: 74; unexplained: 16; radar unclassified: 5 After: corpus: 1135 names; intentional diffs: 388; unexplained: 0; radar unclassified: 0; 283 of 388 changed names are Latin-only corpus: 1142 names; intentional diffs: 318; unexplained: 0; radar unclassified: 0; 218 of 318 changed names are Latin-only corpus: 1142 names; intentional diffs: 232; unexplained: 0; radar unclassified: 0; 218 of 232 changed names are Latin-only corpus: 1142 names; intentional diffs: 95; unexplained: 0; radar unclassified: 0; 85 of 95 changed names are Latin-only Exit 0 at all four. Every baseline rises by the same 21, which is the before-run's unexplained plus its radar unclassified: at the older baselines the names do not arrive, they move from unclassified to explained. 'John Smith Dr.' is the one that changed RULES rather than arriving there, which the fix(#296) paragraph below records. FOUR rules, one per argument, in all four ledgers and last in each file. Anchored alternations of names, so the reach IS the mover list: fix(#489) a title run addresses by its last title 2 names change(suffix-acronym-collisions) esq leaves the acronym set 1 name fix(#489) the title peel leaves a name word a suffix cannot be 4 names fix(#316) a trailing period-marked title word reads as a title 14 names Last in the file is narrow-first rather than a preference: every rule already present that REACHES one of the nineteen declares a strict subset of the bundle rule claiming it, and a wide rule ahead of a narrower one it shares a name with is an order-decided contest the run refuses (#382). Measured: at 1.4.0 fix(#296)'s trailing `dr` reaches two of the trailing-title names, the two comma routings reach 'Smith, John Prof.', and the two-token `jr` rule reaches two of the floor's four; at 2.0.0 and 2.1.0 only fix(#296) reaches any; at 2.2.0 none does. None of them ADMITS the diff it reaches, so no rule lost a name it was explaining and no [[change.precedes_narrower]] block was needed for the bundle itself. `_initials` is in none of the four field lists. The plan predicted it on six of the eight movers; the run puts it on none, because the derived view enters a diff only where every role and ambiguity kind agrees, and all nineteen move roles. validate_rules refuses it beside another field in any case. THE fix(#296) `dr` RULE, reconciled by widening nothing. 'John Smith Dr.' moves {title, suffix} at the three older baselines and {title, middle, family} at 2.2.0, neither a subset of that rule's fields, so it falls through to the #316 rule below it -- narrow-first placement means the DIFF decides the handover, not file order. The rule's closing sentence, which said no trailing title-only word routes to title on the no-comma path and called #316 the open question for the class, is retracted in the same edit; the bare spelling ('Smith Dr', the 'dr ... dr' names) stays with it, #316(b) being out of scope. The gate then called that rule OVER-DECLARED: with 'John Smith Dr.' gone, none of the three names it still explains moves a middle name, so `fields` narrows {middle, family, suffix} -> {family, suffix}. That narrowing made it a strict subset of the glued-CJK-honorific rule's three roles in the 2.0.0 ledger, where both regexes reach '田中さん, Dr.' -- a name that produces no diff at that baseline at all. The honorific rule keeps its place with a [[change.precedes_narrower]] block declaring the LATENT pair: it describes the compound the dr rule knows nothing about. ROSTERS in tests/v2/test_ledger_guards.py, every number taken from `_claim` rather than typed. _CORPUS_CLAIMS gains the four rules per ledger (2 / 1 / 4 / 14, identical digests across the four files, only the floor rule's roles differing -- `_ambiguities` at the 2.x baselines, absent at 1.4.0). Five existing claims moved with the corpus and each says which name arrived: fix(#296) `dr` 11 -> 12 ('John Smith Prof. Dr.'), the two comma routings 288 -> 289 ('Smith, John Prof.'), the two-token `jr` rule 5 -> 7 ('Dr Jr', 'Sir Jr') and the connective-run initials rule 96 -> 97 ('Prince of Wales Jr'). _NOT_A_VOCABULARY_COPY gains three alternations, _MUST_NOT_MATCH four keys with eighteen probes, and _ORDER_EXEMPTION_EFFECT its first 2.0.0 row. The last two members of the #316 rule, 'Mary Jane King.' and 'Smith Sir.', arrived with the 2026-09-09 review of the docs commit. Both are new rules.md examples and both diff identically at all four baselines. 'Smith Sir.' sits HERE rather than on the fix(#489) run rule because the DIFF decides: it moves {title, family}, the walk taking 'Sir.' out of the name, while its given/family reading is what every baseline already gave -- and the #489 rule declares {family, given}, which does not reach a title role at all. Neither takes a _WATCHED_DIFFS row: a rules.md example line is a test literal, executed by test_rules_doc.py, so neither is that roster's population shape. The #316 rule spells 'Andrew Perkins (Mgr.)' with \x28 and \x29. A literal parenthesis inside an alternation body, escaped or not, hides the whole group from test_ledger_guards._alternations, and a twelve-member alternation should not be invisible to the pass that demands every alternation declare what it copies. _WATCHED_DIFFS: the two rows this bundle could have moved, 'Smith, Prof.' at 1.4.0 and 'Esq. van Gogh' at 1.4.0 and 2.0.0, are unchanged and no run reports a moved shape. None of the nineteen sits at a shape two rules could admit, so none belongs there on that ground. One does belong on the roster's own POPULATION clause: 'John Smith Rev.' is the only one of the nineteen named by no test literal anywhere, so a ledger rule is its only watcher, and it takes a row at each baseline. That addition moves the row counts and nothing else -- the population paragraph was not re-derived. Full suite: 7536 passed, 177 skipped, 4 xfailed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- tests/v2/test_ledger_guards.py | 248 ++++++++++++++++- tools/differential/compare.py | 53 +++- tools/differential/expected_since_1.4.0.toml | 239 +++++++++++++++- tools/differential/expected_since_2.0.0.toml | 270 ++++++++++++++++++- tools/differential/expected_since_2.1.0.toml | 239 +++++++++++++++- tools/differential/expected_since_2.2.0.toml | 199 ++++++++++++++ 6 files changed, 1203 insertions(+), 45 deletions(-) diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 392c693f..07ff1dd3 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -962,6 +962,39 @@ def test_case_shape_ids_exist_in_the_inventory() -> None: ("Attorney General of Minnesota Smith", "Deputy Secretary of State Jones", "Prince of Wales", "Duke of Edinburgh"), + # The 2.3 title-run bundle's four rules, all literal-anchored, so + # these probes are the wall _CORPUS_CLAIMS cannot be: a reach this + # small can be widened into names the corpora lack without moving + # a claim count. + # + # The run rule's boundary is the vocabulary the SHAPE needs: a + # one-word run addresses nobody by given name, and a run whose + # last word is title vocabulary that is NOT given-name vocabulary + # keeps its family name -- which is the question filed as #519 and + # deliberately not answered here. + "fix(#489) a title run addresses by its last title": + ("Sir John Smith", "His Excellency Lord Duncan", + "Her Royal Highness Princess Anne", "Dr. Smith"), + # The floor's boundary is the word it gives back having to be a + # NAME candidate: 'MD DDS' and 'Jr. Ph. D.' are the two all-suffix + # rests it declines, 'Marquess of Bath' the all-title input with + # no rest to read at all, and 'Dr Smith Jr' the ordinary titled + # name where the piece behind the run was never a suffix. + "fix(#489) the title peel leaves a name word a suffix cannot be": + ("Dr Smith Jr", "MD DDS", "Marquess of Bath", "Prince of Wales Jr"), + # The trailing rule's boundaries are the three readings that did + # NOT move -- an unlisted abbreviation, a bare title word, a + # post-nominal -- plus the comma shape whose segment holds no name + # word and which segment_suffix_reading has always routed by a + # different gate. + "fix(#316) a trailing period-marked title word reads as a title": + ("John Smith Xyz.", "John Smith Sir", "Mary Jane King", + "John Smith Esq.", "Smith, Prof."), + # The esq boundary is every spelling SUFFIX_WORDS still carries, + # in each of the three positions the corpora write it in. + "change(suffix-acronym-collisions) esq leaves the acronym set": + ("John Smith Esq", "John Smith Esq.", "Esq. Smith", "Smith, Esq.", + "Esq. van Gogh"), } @@ -1721,6 +1754,48 @@ class _LatinCopy(NamedTuple): # CONJUNCTIONS -- "e." is no entry, and matching it against the # vocabulary would be a false claim of correspondence. frozenset({"[EY]", "[EeYy]\\."}), + # #489's run movers, one corpus name per alternative -- a list of + # names, not a copy of GIVEN_NAME_TITLES. The rule's subject is a + # SHAPE the vocabulary participates in (a multi-word title run + # whose LAST word is a given-name title), and a member copying the + # wordlist would reach 'Sir John Smith' and 'Dr. Smith', which do + # not move -- the run has to be multi-word before the last word is + # asked about at all. One set, identical in all four ledgers. + frozenset({r"Dr\. Sir John", "Her Majesty Queen Elizabeth"}), + # #489's peel-floor movers, one corpus name per alternative -- a + # list of names, and NOT a copy of either wordlist the floor + # consults. A member copying TITLES would reach 'Marquess of Bath' + # and every titled name besides; a member copying SUFFIX_WORDS + # would reach 'MD DDS' and 'Jr. Ph. D.', which are the inputs the + # floor DECLINES. What selects these four is the shape: a title + # run with nothing but post-nominal pieces behind it, whose own + # last word is not one of those. One set, identical in all four + # ledgers. + frozenset({"Dr Jr", "Dr King Jr", r"Dr\. King MD", "Sir Jr"}), + # #316's trailing-title movers, one corpus name per alternative -- + # a list of names, not a copy of TITLES. The rule's subject is a + # SHAPE the vocabulary participates in (a trailing run of + # period-marked LISTED title words), and a member copying the + # wordlist would reach the BARE 'Mary Jane King' and 'John Smith + # Sir', which do not move, while a member spelled as the shape -- + # a trailing dotted word -- would reach 'John Smith Xyz.', the + # unlisted abbreviation this slot deliberately does not infer + # from. 'Mary Jane King\.' and 'Smith Sir\.' joined in the + # 2026-09-09 review of the docs commit, as rules.md examples: the + # first is the period-marked spelling of the very name the + # wordlist test above uses, which is the point of it -- the bare + # word is protected and the abbreviated one is not, and the member + # says so by carrying the period. 'Andrew Perkins \x28Mgr\.\x29' + # carries the ledger's \x28/\x29 spelling of the parentheses, + # without which this whole alternation would be invisible to + # _alternations above. One set, identical in all four ledgers. + frozenset({r"Andrew Perkins \x28Mgr\.\x29", r"Dr\. John Smith Prof\.", + r"John Prof\. MA", r"John Smith Dr\.", + r"John Smith Jr\. Prof\.", r"John Smith Mr\.", + r"John Smith Prof\.", r"John Smith Prof\. Dr\.", + r"John Smith Prof\. Jr\.", r"John Smith Rev\.", + r"Mary Jane King\.", r"Smith Prof\.", r"Smith Sir\.", + r"Smith, John Prof\."}), }) def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: @@ -2196,16 +2271,32 @@ def _claim(rule: dict) -> _Claim: # #486 had to AUTHOR -- 'John Smith Jr., PhD' and # 'Kennedy, John (Jack)' -- and neither diffs at this baseline # either, so all four names are reach without absorption. + # 288 -> 289 on 2026-09-08: 'Smith, John Prof.' joined the + # rules corpus with the 2.3 title-run bundle and matches on + # the bare comma, like the 288 before it. It does not diff + # on this rule's roles at any baseline -- {title, middle} -- + # so this is reach without absorption, as the paragraph + # above records for the four names before it. "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(288, ('given', 'suffix', 'title'), "10c78dd0f2d2", None), + _Claim(289, ('given', 'suffix', 'title'), "837dc1177415", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": _Claim(2, ('family', 'given'), "5bd9c6d96c38", None), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": _Claim(2, ('family', 'given', 'suffix'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": _Claim(3, ('given', 'middle', 'title'), "263d5957cfc1", None), + # `middle` left the ROLES in the same edit, at the gate's + # own OVER-DECLARED insistence: with 'John Smith Dr.' gone, + # no name the rule still explains moves a middle name. + # 11 -> 12 on 2026-09-08: the 2.3 title-run bundle put + # 'John Smith Prof. Dr.' in the rules corpus, and the + # trailing-`dr` regex reaches any name ending in " Dr.". + # Reach, not explanation -- the rule explains NEITHER of + # the two 'Dr.' names now, both having moved to the #316 + # rule at the end of the ledger, and the comment there + # records the handover. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(11, ('family', 'middle', 'suffix'), "b9cfc0d88bf6", None), + _Claim(12, ('family', 'suffix'), "c3446b32e8bd", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": @@ -2216,8 +2307,10 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('suffix', 'title'), "f025c5f70a4e", None), "fix(#367) an inferred title no longer displaces a leading particle either": _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), + # 288 -> 289 with fix(comma-family) above and for the same + # one name, both rules matching on the bare comma. "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(288, ('family', 'given'), "10c78dd0f2d2", None), + _Claim(289, ('family', 'given'), "837dc1177415", None), "fix(#397) NOT WANTED: a trailing Catalan/Polish linking 'i' is read as a generation marker and the family is lost": _Claim(1, ('family', 'suffix'), "498602f3cfd0", None), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": @@ -2316,8 +2409,15 @@ def _claim(rule: dict) -> _Claim: # the acronym and M.A. rules reach exactly what they explain. "fix(suffix-routing) a two-token name ending in a roman numeral keeps it in `suffix`": _Claim(4, ('family', 'suffix'), "fc52089dfa8e", None), + # 5 -> 7 on 2026-09-08: 'Dr Jr' and 'Sir Jr' joined the + # rules corpus with the 2.3 title-run bundle. Both are + # two-token names ending in `jr`, so the regex reaches + # them; neither diff fits {family, suffix} -- each moves + # `title` and `given` too -- so both fall through to the + # peel-floor rule at the end of the ledger, and the surplus + # is the one the paragraph above says these four carry. "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`": - _Claim(5, ('family', 'suffix'), "602e2d83a23b", None), + _Claim(7, ('family', 'suffix'), "4cd8e7fbd20d", None), "fix(suffix-routing) a two-token name ending in a credential acronym keeps it in `suffix`": _Claim(2, ('family', 'suffix'), "ed72c9672214", None), "fix(suffix-routing) the dotted M.A. spelling reads as a credential (ma-do)": @@ -2343,14 +2443,42 @@ def _claim(rule: dict) -> _Claim: _Claim(27, ('_initials',), "6b242c287db8", ('DEFAULT',)), "fix(#360) los joined the particles, so it no longer initials": _Claim(1, ('_initials',), "cd721215f463", ('DEFAULT',)), + # 96 -> 97 on 2026-09-08: 'Prince of Wales Jr' joined the + # rules corpus with the 2.3 title-run bundle -- a parity + # row, kept as the boundary the peel floor declines -- and + # `of` is a connective. Reach, not explanation. "fix(initials-per-word) a connective run initials each word (facade, since 2.0.0)": - _Claim(96, ('_initials',), "fa69850d2cd4", ('DEFAULT',)), + _Claim(97, ('_initials',), "6af5338ad4d5", ('DEFAULT',)), "fix(initials-per-word) a bound-given run initials each word (facade, since 2.0.0)": _Claim(41, ('_initials',), "e99f56c955d5", ('DEFAULT',)), "fix(initials-per-word) a particle chain inside a name part initials each word (facade, since 2.0.0)": _Claim(108, ('_initials',), "45f0b2c1a7d4", ('DEFAULT',)), "fix(initials-per-word) the Ph. D. merge initials each word (facade, since 2.0.0)": _Claim(18, ('_initials',), "f67d8ebddd56", ('DEFAULT',)), + # The 2.3 title-run bundle's four rules, last in every + # ledger. All four are anchored alternations of NAMES, so the + # reach IS the mover list and the four numbers are the four + # release-log bullets one for one: 2 names for the run keying, + # 1 for the esq drop, 4 for the peel floor, 12 for the + # trailing title. Nineteen in all, and every one of them is + # explained by the rule that names it -- these are the rare + # rows where reach and explanation coincide, which is what an + # anchored name list buys. A widening past those names moves + # the digest here before it can reach the gate. + "fix(#489) a title run addresses by its last title": + _Claim(2, ('family', 'given'), "e14159a4d48f", None), + "change(suffix-acronym-collisions) esq leaves the acronym set": + _Claim(1, ('family', 'middle', 'suffix'), "ef9c8cfc56d8", None), + # `_ambiguities` is in the ROLES at the three 2.x baselines and + # not at 1.4.0, which has no ambiguity surface to compare + # (compare.py's _RULE_FIELDS). Same regex, so the reach and the + # digest are identical in all four ledgers and only the roles + # move. + "fix(#489) the title peel leaves a name word a suffix cannot be": + _Claim(4, ('family', 'given', 'suffix', 'title'), "ac7318881b28", None), + "fix(#316) a trailing period-marked title word reads as a title": + _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), + "4130dc8bbf40", None), }, "expected_since_2.0.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -2510,8 +2638,18 @@ def _claim(rule: dict) -> _Claim: _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), + # `middle` left the ROLES in the same edit, at the gate's + # own OVER-DECLARED insistence: with 'John Smith Dr.' gone, + # no name the rule still explains moves a middle name. + # 11 -> 12 on 2026-09-08: the 2.3 title-run bundle put + # 'John Smith Prof. Dr.' in the rules corpus, and the + # trailing-`dr` regex reaches any name ending in " Dr.". + # Reach, not explanation -- the rule explains NEITHER of + # the two 'Dr.' names now, both having moved to the #316 + # rule at the end of the ledger, and the comment there + # records the handover. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(11, ('family', 'middle', 'suffix'), "b9cfc0d88bf6", None), + _Claim(12, ('family', 'suffix'), "c3446b32e8bd", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": @@ -2579,6 +2717,30 @@ def _claim(rule: dict) -> _Claim: # regex is the same string in each. "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), + # The 2.3 title-run bundle's four rules, last in every + # ledger. All four are anchored alternations of NAMES, so the + # reach IS the mover list and the four numbers are the four + # release-log bullets one for one: 2 names for the run keying, + # 1 for the esq drop, 4 for the peel floor, 12 for the + # trailing title. Nineteen in all, and every one of them is + # explained by the rule that names it -- these are the rare + # rows where reach and explanation coincide, which is what an + # anchored name list buys. A widening past those names moves + # the digest here before it can reach the gate. + "fix(#489) a title run addresses by its last title": + _Claim(2, ('family', 'given'), "e14159a4d48f", None), + "change(suffix-acronym-collisions) esq leaves the acronym set": + _Claim(1, ('family', 'middle', 'suffix'), "ef9c8cfc56d8", None), + # `_ambiguities` is in the ROLES at the three 2.x baselines and + # not at 1.4.0, which has no ambiguity surface to compare + # (compare.py's _RULE_FIELDS). Same regex, so the reach and the + # digest are identical in all four ledgers and only the roles + # move. + "fix(#489) the title peel leaves a name word a suffix cannot be": + _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), + "fix(#316) a trailing period-marked title word reads as a title": + _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), + "4130dc8bbf40", None), }, # The 2.3 cycle's first rule, and a facade-only render fix: every # role is identical, so `_initials` alone. Reach and digest as in @@ -2664,6 +2826,30 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('suffix',), "1b67339cf744", None), "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), + # The 2.3 title-run bundle's four rules, last in every + # ledger. All four are anchored alternations of NAMES, so the + # reach IS the mover list and the four numbers are the four + # release-log bullets one for one: 2 names for the run keying, + # 1 for the esq drop, 4 for the peel floor, 12 for the + # trailing title. Nineteen in all, and every one of them is + # explained by the rule that names it -- these are the rare + # rows where reach and explanation coincide, which is what an + # anchored name list buys. A widening past those names moves + # the digest here before it can reach the gate. + "fix(#489) a title run addresses by its last title": + _Claim(2, ('family', 'given'), "e14159a4d48f", None), + "change(suffix-acronym-collisions) esq leaves the acronym set": + _Claim(1, ('family', 'middle', 'suffix'), "ef9c8cfc56d8", None), + # `_ambiguities` is in the ROLES at the three 2.x baselines and + # not at 1.4.0, which has no ambiguity surface to compare + # (compare.py's _RULE_FIELDS). Same regex, so the reach and the + # digest are identical in all four ledgers and only the roles + # move. + "fix(#489) the title peel leaves a name word a suffix cannot be": + _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), + "fix(#316) a trailing period-marked title word reads as a title": + _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), + "4130dc8bbf40", None), }, "expected_since_2.1.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -2813,8 +2999,18 @@ def _claim(rule: dict) -> _Claim: _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), + # `middle` left the ROLES in the same edit, at the gate's + # own OVER-DECLARED insistence: with 'John Smith Dr.' gone, + # no name the rule still explains moves a middle name. + # 11 -> 12 on 2026-09-08: the 2.3 title-run bundle put + # 'John Smith Prof. Dr.' in the rules corpus, and the + # trailing-`dr` regex reaches any name ending in " Dr.". + # Reach, not explanation -- the rule explains NEITHER of + # the two 'Dr.' names now, both having moved to the #316 + # rule at the end of the ledger, and the comment there + # records the handover. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(11, ('family', 'middle', 'suffix'), "b9cfc0d88bf6", None), + _Claim(12, ('family', 'suffix'), "c3446b32e8bd", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": @@ -2874,6 +3070,30 @@ def _claim(rule: dict) -> _Claim: # in every 2.x wheel, so the baseline makes no difference. "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), + # The 2.3 title-run bundle's four rules, last in every + # ledger. All four are anchored alternations of NAMES, so the + # reach IS the mover list and the four numbers are the four + # release-log bullets one for one: 2 names for the run keying, + # 1 for the esq drop, 4 for the peel floor, 12 for the + # trailing title. Nineteen in all, and every one of them is + # explained by the rule that names it -- these are the rare + # rows where reach and explanation coincide, which is what an + # anchored name list buys. A widening past those names moves + # the digest here before it can reach the gate. + "fix(#489) a title run addresses by its last title": + _Claim(2, ('family', 'given'), "e14159a4d48f", None), + "change(suffix-acronym-collisions) esq leaves the acronym set": + _Claim(1, ('family', 'middle', 'suffix'), "ef9c8cfc56d8", None), + # `_ambiguities` is in the ROLES at the three 2.x baselines and + # not at 1.4.0, which has no ambiguity surface to compare + # (compare.py's _RULE_FIELDS). Same regex, so the reach and the + # digest are identical in all four ledgers and only the roles + # move. + "fix(#489) the title peel leaves a name word a suffix cannot be": + _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), + "fix(#316) a trailing period-marked title word reads as a title": + _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), + "4130dc8bbf40", None), }, } @@ -4464,7 +4684,19 @@ def test_a_rule_reaching_no_corpus_name_says_why_it_is_kept() -> None: ("fix(cjk-glued-honorific-peel) glued honorific peels into suffix", "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`", 1), ], - "expected_since_2.0.0.toml": [], + # The 2.0.0 list was EMPTY until 2026-09-08, and what put a row in + # it was a narrowing rather than a widening: #316 took 'John Smith + # Dr.' off the dr rule, the gate's OVER-DECLARED check then took + # `middle` out of that rule's `fields`, and {family, suffix} is a + # strict subset of the glued-honorific rule's three roles where + # {middle, family, suffix} was not. The one contested name, + # '田中さん, Dr.', produces no diff at this baseline at all, so the + # pair is latent; the ledger's [[change.precedes_narrower]] block + # carries the argument. + "expected_since_2.0.0.toml": [ + ("fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix", + "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word", 1), + ], "expected_since_2.1.0.toml": [], "expected_since_2.2.0.toml": [], } diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 72e2f467..691f640a 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -1948,10 +1948,19 @@ class _ShapeMismatch(NamedTuple): #: the population, while 'Lala Lajpat Rai' and 'John Smith, RAI' are #: named nowhere under tests/ outside test_ledger_guards.py and #: entered it, so the scan went 52 -> 53 and the roster 50 -> 51. -#: The counts: 37 / 33 / 32 / 7 rows, 109 in all, over those 51 names -#: -- and the roster is now exactly the population, the five contest -#: rows beyond it having gone to _RECORDED_DIFFS with #501 and five -#: more with #498, which left the population by gaining a +#: Recounted 2026-09-08 with the title-run bundle: one name entered the +#: population, 'John Smith Rev.', the only one of that bundle's nineteen +#: movers no test literal names, and it took a row at each of the four +#: baselines. It is named NOWHERE under tests/, so it counts in both +#: scans -- the every-file figures in the RECOMPUTE paragraph below +#: rise by one each too. That commit moved the ROW counts and nothing +#: else: it did not re-derive the population clause above, so the +#: equality sentence that follows is dated 2026-09-07 and is not +#: restated for today. +#: The counts: 38 / 34 / 33 / 8 rows, 113 in all, over those 52 names +#: -- and as of 2026-09-07 the roster was exactly the population, the +#: five contest rows beyond it having gone to _RECORDED_DIFFS with #501 +#: and five more with #498, which left the population by gaining a #: _RECORDED_DIFFS key rather than by ceasing to be watched anywhere. #: 48 of the 50 sit in corpus_issues.jsonl and 3 in corpus.jsonl, with #: 'dr Vincent van Gogh dr' in both, so the per-file counts overlap by @@ -2010,6 +2019,15 @@ class _ShapeMismatch(NamedTuple): "Jack M.A.": ("family", "suffix"), "Jane van der Berg 旧姓 Jones": ("family", "maiden"), "Janey née Jones": ("family", "given", "maiden", "middle"), + # The one name of the 2.3 title-run bundle's nineteen + # movers that no test literal watches: the other four on + # radar carry a cases.py row or a v1 test. Shape measured + # by the run, 2026-09-08, and the same at all four + # baselines. The POPULATION paragraph above was last + # re-derived 2026-09-07 and this addition did not re-derive + # it -- the row counts below are updated by the four rows + # added, nothing else. + "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "John V": ("family", "suffix"), "John of the Doe": ("_initials",), @@ -2053,6 +2071,15 @@ class _ShapeMismatch(NamedTuple): "Jane van der Berg 旧姓 Jones": ("family", "maiden"), "Janey née Jones": ("family", "given"), "Joe E. Smith": ("_initials",), + # The one name of the 2.3 title-run bundle's nineteen + # movers that no test literal watches: the other four on + # radar carry a cases.py row or a v1 test. Shape measured + # by the run, 2026-09-08, and the same at all four + # baselines. The POPULATION paragraph above was last + # re-derived 2026-09-07 and this addition did not re-derive + # it -- the row counts below are updated by the four rows + # added, nothing else. + "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "John, Smith, Dr.": ("_ambiguities",), "Jong van der": ("_initials",), @@ -2094,6 +2121,15 @@ class _ShapeMismatch(NamedTuple): "Jane van der Berg 旧姓 Jones": ("family", "maiden"), "Janey née Jones": ("family", "given"), "Joe E. Smith": ("_initials",), + # The one name of the 2.3 title-run bundle's nineteen + # movers that no test literal watches: the other four on + # radar carry a cases.py row or a v1 test. Shape measured + # by the run, 2026-09-08, and the same at all four + # baselines. The POPULATION paragraph above was last + # re-derived 2026-09-07 and this addition did not re-derive + # it -- the row counts below are updated by the four rows + # added, nothing else. + "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "John, Smith, Dr.": ("_ambiguities",), "Jong van der": ("_initials",), @@ -2119,6 +2155,15 @@ class _ShapeMismatch(NamedTuple): "E Anne D,Leonardo": ("_initials",), "JOSE E MARIA SANTOS": ("_initials",), "Joe E. Smith": ("_initials",), + # The one name of the 2.3 title-run bundle's nineteen + # movers that no test literal watches: the other four on + # radar carry a cases.py row or a v1 test. Shape measured + # by the run, 2026-09-08, and the same at all four + # baselines. The POPULATION paragraph above was last + # re-derived 2026-09-07 and this addition did not re-derive + # it -- the row counts below are updated by the four rows + # added, nothing else. + "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "Jose E. Maria Santos": ("_initials",), "Lala Lajpat Rai": ("family", "middle", "suffix"), diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index a7f49a90..887b97f7 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -454,18 +454,36 @@ fields = ["title", "given", "middle"] [[change]] issue = "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word" -# 'John Smith Dr.' and 'dr Vincent van Gogh dr' with its #100 siblings -# (and, at 2.x only, 'Smith Dr', which 1.4.0 already read as the -# family): 'dr' left SUFFIX_WORDS, where it was v1 -# residue -- 'Dr.' is not a postnominal in any tradition -- so a -# trailing bare 'Dr' is no longer suffix vocabulary and falls to the -# positional read, taking the family name with it. NOT a new class: -# no trailing title-only word routes to title on the no-comma path -# ('John Smith Prof.' reads family 'Prof.' at every baseline), and -# the suffix entry was the only thing making 'dr' the exception. #316 -# is the open question for the class. +# 'dr Vincent van Gogh dr' with its #100 siblings (and, at 2.x only, +# 'Smith Dr', which 1.4.0 already read as the family): 'dr' left +# SUFFIX_WORDS, where it was v1 residue -- 'Dr.' is not a postnominal +# in any tradition -- so a trailing bare 'Dr' is no longer suffix +# vocabulary and falls to the positional read, taking the family name +# with it. +# +# 2026-09-08: 'John Smith Dr.' has LEFT this rule, and the sentence +# that used to close this comment has been retracted with it. It read +# "NOT a new class: no trailing title-only word routes to title on +# the no-comma path ('John Smith Prof.' reads family 'Prof.' at every +# baseline) ... #316 is the open question for the class". #316 is +# answered: a trailing PERIOD-MARKED listed title word now routes to +# title, so 'John Smith Dr.' moves {title, suffix} against this +# baseline and {title, middle, family} against 2.2.0, neither of them +# a subset of the three roles below, and the bundle rule `fix(#316) a +# trailing period-marked title word reads as a title` at the end of +# this file claims it. What stays here is the BARE spelling, which +# #316(b) leaves open: 'Smith Dr' and the 'dr ... dr' names wear no +# abbreviation period and the trailing walk never sees them. This +# rule is narrower than that one and stands first, so the DIFF +# decides which of the two claims a name, not file order. +# +# `middle` left the field list in the same edit: it was there for +# 'John Smith Dr.', whose middle name moved, and the three names the +# rule still explains move `family` and `suffix` alone. The gate said +# so itself -- OVER-DECLARED, "classify() matches by SUBSET, so the +# excess is not inert" (#452). name_regex = "(?i)\\sdr\\.?$" -fields = ["middle", "family", "suffix"] +fields = ["family", "suffix"] [[change]] issue = "fix(#296) a credential-only comma string reads a name and its postnominal" @@ -2882,3 +2900,202 @@ issue = "fix(initials-per-word) the Ph. D. merge initials each word (facade, sin name_regex = "(?i)\\bph\\. d\\." fields = ["_initials"] orders = ["DEFAULT"] + +# --------------------------------------------------------------- +# THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last +# title), #316 (a trailing period-marked title word reads as a +# title), the leading peel's name-word floor, and the `esq` acronym +# drop that shipped beside them. Four rules, one per ARGUMENT rather +# than one per name, over nineteen corpus names. +# +# LAST in the file, and that is narrow-first rather than a +# preference: every rule already here that REACHES one of the +# nineteen declares `fields` that are a strict subset of the bundle +# rule claiming that name, and a wide rule sitting AHEAD of a +# narrower one it shares a corpus name with is an order-decided +# contest the run refuses (#382). Measured 2026-09-08, all four +# ledgers: at 1.4.0 fix(#296)'s trailing `dr` reaches 'John Smith +# Dr.' and 'John Smith Prof. Dr.', the two comma routings reach +# 'Smith, John Prof.', and the two-token `jr` rule reaches 'Dr Jr' +# and 'Sir Jr'; at 2.0.0 and 2.1.0 only fix(#296) reaches any of +# them; at 2.2.0 none does. Not one of those rules ADMITS the diff +# it reaches -- every shape is outside its field list, measured -- +# so nothing below takes a name off a rule that was explaining it, +# and no [[change.precedes_narrower]] block is needed anywhere. +# +# `_initials` appears in none of the four field lists although the +# initials of several of the nineteen do move. The derived view +# enters a diff only where every ROLE and every ambiguity kind +# agrees (compare.py's _RULE_FIELDS, main()'s roles-identical +# guard), and all nineteen move roles -- so no run can produce a +# shape here carrying it, and validate_rules refuses it beside +# another field in any case. +# --------------------------------------------------------------- + +[[change]] +issue = "fix(#489) a title run addresses by its last title" +# 'Her Majesty Queen Elizabeth' and 'Dr. Sir John': the +# given_name_titles lookup was keyed on the WHOLE run, and 'her +# majesty queen' is in no vocabulary and never could be, so the one +# name word behind the run stayed the family. Both sites that ask -- +# rules.md#H1's retag and rules.md#P5's licence -- now test the whole +# run's key OR the last word's, and 'queen' and 'sir' are +# GIVEN_NAME_TITLES entries. The name word behind the run becomes the +# GIVEN name and the family empties, which is H1's Accepted outcome. +# Both roles move together and `fields` declares both. `title` is +# deliberately OUT: the run itself is unchanged, only what it +# addresses as, so a name whose TITLE moved arrives UNEXPLAINED, as +# it should. +# +# 'Reverend Mother Teresa', 'Mr Sir John', 'Xyz. Sir John' and 'Sir +# Sheikh abdul rahman' move the same way and are NOT members: none of +# them is in any corpus file, so a member for one would be a rule +# reaching nothing. +# +# An anchored alternation of the two NAMES rather than a shape. What +# selects them is a SHAPE -- a multi-word title run whose LAST word +# is a given-name title -- and a member copying GIVEN_NAME_TITLES +# would reach 'Sir John Smith' and 'Dr. Smith', which do not move. +# The members are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY in tests/v2/test_ledger_guards.py records; +# _CORPUS_CLAIMS pins the reach at 2 with its digest, and +# _MUST_NOT_MATCH carries three boundaries: a ONE-word run ('Sir John +# Smith'), and two runs whose last word is title vocabulary that is +# not given-name vocabulary ('His Excellency Lord Duncan', 'Her Royal +# Highness Princess Anne' -- the open vocabulary question, #519). One +# set, identical in all four ledgers. +name_regex = "^(?:Dr\\. Sir John|Her Majesty Queen Elizabeth)$" +fields = ["family", "given"] + +[[change]] +issue = "change(suffix-acronym-collisions) esq leaves the acronym set" +# 'John Smith E.S.Q.': 'esq' left SUFFIX_ACRONYMS, where the 2019 bulk +# post-nominal import put it (af5bdab, #93) -- esquire is a +# contraction, not an initialism -- so the multi-dot spelling stops +# being credential vocabulary and falls to the positional read: given +# 'John', middle 'Smith', family 'E.S.Q.', where every release since +# 1.4.0 read family 'Smith', suffix 'E.S.Q.'. +# +# A BEHAVIOR CHANGE rather than a fix, which is what the `change` tag +# says: this is a deliberate parity break at all four baselines, and +# the entry that argues it is decisions.md#suffix-acronym-collisions +# -- there is no issue number to cite, and the release-log bullet +# carries none either. It is not NOT WANTED: that spelling is for a +# reading nobody wants, and this reading is the one the decision +# chose. +# +# Literal, no alternation, and exactly as wide as the diff: 'esq' +# keeps its SUFFIX_WORDS membership, so the acronym entry's only +# unique coverage was this one spelling and this one corpus name. +# _MUST_NOT_MATCH carries the three spellings that did NOT move -- +# 'John Smith Esq' (the word list still reads it), 'Esq. Smith' +# (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone +# post-comma credential). +name_regex = "^John Smith E\\.S\\.Q\\.$" +fields = ["family", "middle", "suffix"] + +[[change]] +issue = "fix(#489) the title peel leaves a name word a suffix cannot be" +# 'Dr King Jr', 'Dr. King MD', 'Dr Jr' and 'Sir Jr': the leading title +# peel's only floor was "leave one piece", so it peeled 'Dr King' +# whole and left the post-nominal to be the name -- title 'Dr King', +# family 'Jr', no suffix at all. leading_titles now also refuses to +# leave a word that is nothing but suffix vocabulary standing as the +# name and gives the run's last word back (rules.md#H3, +# decisions.md#H3), so 'Dr King Jr' reads title 'Dr', family 'King', +# suffix 'Jr' -- what the comma spelling 'King, Dr Jr' always read. +# +# FOUR roles, and no single name moves all four. The two three-word +# names move `title`, `family` and `suffix`; 'Dr Jr' and 'Sir Jr' +# move `given` as well, or instead of `family`, the word given back +# standing alone with the credential peeled off it. At the 2.x +# baselines `_ambiguities` joins the list -- with the title given +# back there is no title left to make the reading H1's, so all four +# report title-or-name through rules.md#H4 -- and it is absent at +# 1.4.0, which has no ambiguity surface to compare (compare.py's +# _RULE_FIELDS). +# +# An anchored alternation of the four NAMES rather than a shape. What +# selects them is a SHAPE -- a title run with nothing but post-nominal +# pieces behind it, whose own last word is not one of those -- and a +# member copying TITLES or SUFFIX_WORDS would reach 'MD DDS', 'Jr. +# Ph. D.' and 'Marquess of Bath', every one of which the floor +# declines and every one of which is a _MUST_NOT_MATCH probe, with +# 'Dr Smith Jr' beside them for the ordinary titled name the floor +# never sees. The members are a list of names and copy no wordlist, +# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins +# the reach at 4 with its digest. One set, identical in all four +# ledgers; only the `fields` differ, as the paragraph above says. +name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" +fields = ["family", "given", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing period-marked title word reads as a title" +# 'John Smith Prof.' and thirteen more: a trailing run of period-marked +# LISTED title words is peeled as a title, so the surname behind it +# stops being lost -- title 'Prof.', given 'John', family 'Smith', +# where every baseline read middle 'Smith', family 'Prof.'. A RUN, +# mirroring the leading peel: 'John Smith Prof. Dr.' gives title +# 'Prof. Dr.' and 'Dr. John Smith Prof.' gives 'Dr. Prof.' in input +# order. The family-comma segment takes the same walk, so 'Smith, +# John Prof.' agrees with the bare spelling at last. And the trailing +# title is TRANSPARENT to the post-nominal peel that follows it: +# 'John Smith Jr. Prof.' keeps suffix 'Jr.' rather than promoting the +# generational suffix to the family name, and 'John Prof. MA' reads +# family 'MA' by rules.md#S2's reserve. +# +# Five roles, the union of what the fourteen move; no name moves all +# five, and three move only two -- 'Smith, John Prof.' {title, +# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' +# moving three. `_ambiguities` is not among the five and none of the +# fourteen reports one: the walk leaves a title standing, so H4's +# title half never fires. +# +# TWO of the fourteen end in ' Dr.' and are also reached by +# fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which +# that rule used to explain, and 'John Smith Prof. Dr.'. That rule +# declares {middle, family, suffix} and neither diff fits it now, so +# both fall through to here; its comment records the handover, and +# narrow-first placement is what keeps the pair out of contest. The +# BARE trailing spelling stays with fix(#296): 'Smith Dr' and the 'dr +# ... dr' names wear no abbreviation period and the walk never sees +# them (#316(b), left open). +# +# The last two members arrived with the 2026-09-09 review of the +# docs commit, which added two rules.md examples: 'Mary Jane King.', +# the H5 Accepted clause's collision example (title 'King.', given +# 'Mary', family 'Jane', where every baseline read family 'King.' +# behind middle 'Jane'), and 'Smith Sir.', which is an H1 example. +# 'Smith Sir.' is HERE and not on the fix(#489) run rule above +# because the DIFF decides: it moves {title, family}, the walk +# taking 'Sir.' out of the name, while its given/family reading is +# what every baseline already gave ('Smith' stays the given name on +# both sides). The #489 rule declares {family, given} and does not +# reach a title role at all, so it could not claim the diff even +# placed ahead of this one. +# +# An anchored alternation of the fourteen NAMES rather than a shape. +# What selects them is a SHAPE -- period-marked, listed, trailing -- +# and a member copying TITLES would reach the BARE 'Mary Jane King' +# and 'John Smith Sir', which do not move; the period-marked +# 'Mary Jane King.' is a member precisely because it does, which is +# the line the period draws and the whole of what it draws. A member +# spelled as the shape (a trailing dotted word) would reach 'John +# Smith Xyz.', which is the boundary the leading slot's H2 inference +# is deliberately NOT given here. Those three and 'Smith, Prof.' -- +# the comma shape whose segment holds no name word and which reads +# by a different gate -- are the _MUST_NOT_MATCH probes. The members +# are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins the reach at +# 14 with its digest. +# +# 'Andrew Perkins (Mgr.)' is spelled with \\x28 and \\x29 rather than +# the \\( and \\) the paren-bearing rules above use. Escaped or not, a +# literal parenthesis inside an alternation body hides the WHOLE +# group from test_ledger_guards._alternations -- its member pattern +# excludes '(' and ')' -- and a twelve-member alternation should not +# be invisible to the discovery pass that demands every alternation +# declare what it copies. The two spellings match the same string. +# One set, identical in all four ledgers. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +fields = ["family", "given", "middle", "suffix", "title"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 10435a29..14bf8713 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -379,6 +379,37 @@ issue = "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into s name_regex = "(?<=[^\\s,])(?<!박사)(?<!선생)(?<!교수)(?:박사님|선생님|교수님|박사|씨|님|先生|女士|小姐|教授|様|さん|さま|くん|ちゃん)\\.?(?=$|[ ,])" fields = ["given", "family", "suffix"] +[[change.precedes_narrower]] +issue = "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word" +why = """ +LATENT at this baseline, and the only exemption this file carries. +'田中さん, Dr.' is the one name both regexes reach -- this rule on its +glued さん, the dr rule on the ' Dr.' the string ends with -- and it +produces NO diff against the 2.0.0 wheel at all, so file order decides +nothing here today. It does diff at 1.4.0 and at 2.1.0, and at neither +of those is this pair a contest: the 1.4 ledger carries this regex +under a different issue with different `fields`, and the 2.1 ledger has +no glued-honorific rule to contest with. + +The pair became a contest on 2026-09-08, and not because either rule +was widened. #316 took 'John Smith Dr.' off the dr rule -- a trailing +period-marked title word now routes to `title`, which that rule's +comment records -- and the gate's own OVER-DECLARED check then narrowed +it from {middle, family, suffix} to {family, suffix}, the three names +it still explains moving no middle name. {family, suffix} is a strict +subset of the three roles here where the old list was not, and the +hazard appeared with the subset. + +The declaration says this rule stays first. What it describes is the +COMPOUND: an honorific written against the name is split off its token +and routed to `suffix`, and the peeled name then goes through the +ordinary machinery, which is why `given` and `family` move beside it. +The dr rule describes one vocabulary removal at the END of a string and +knows nothing about the honorific. If '田中さん, Dr.' ever starts +diffing at this baseline, the honorific peel is what its diff is about +and this is the label to read it under. +""" + [[change]] issue = "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix" # '王小明 先生', '김민준 씨', '田中 太郎 様', '田中 殿': #307 ships the @@ -794,18 +825,36 @@ fields = ["title", "given", "middle", "_ambiguities"] [[change]] issue = "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word" -# 'John Smith Dr.' and 'dr Vincent van Gogh dr' with its #100 siblings -# (and, at 2.x only, 'Smith Dr', which 1.4.0 already read as the -# family): 'dr' left SUFFIX_WORDS, where it was v1 -# residue -- 'Dr.' is not a postnominal in any tradition -- so a -# trailing bare 'Dr' is no longer suffix vocabulary and falls to the -# positional read, taking the family name with it. NOT a new class: -# no trailing title-only word routes to title on the no-comma path -# ('John Smith Prof.' reads family 'Prof.' at every baseline), and -# the suffix entry was the only thing making 'dr' the exception. #316 -# is the open question for the class. +# 'dr Vincent van Gogh dr' with its #100 siblings (and, at 2.x only, +# 'Smith Dr', which 1.4.0 already read as the family): 'dr' left +# SUFFIX_WORDS, where it was v1 residue -- 'Dr.' is not a postnominal +# in any tradition -- so a trailing bare 'Dr' is no longer suffix +# vocabulary and falls to the positional read, taking the family name +# with it. +# +# 2026-09-08: 'John Smith Dr.' has LEFT this rule, and the sentence +# that used to close this comment has been retracted with it. It read +# "NOT a new class: no trailing title-only word routes to title on +# the no-comma path ('John Smith Prof.' reads family 'Prof.' at every +# baseline) ... #316 is the open question for the class". #316 is +# answered: a trailing PERIOD-MARKED listed title word now routes to +# title, so 'John Smith Dr.' moves {title, suffix} against this +# baseline and {title, middle, family} against 2.2.0, neither of them +# a subset of the three roles below, and the bundle rule `fix(#316) a +# trailing period-marked title word reads as a title` at the end of +# this file claims it. What stays here is the BARE spelling, which +# #316(b) leaves open: 'Smith Dr' and the 'dr ... dr' names wear no +# abbreviation period and the trailing walk never sees them. This +# rule is narrower than that one and stands first, so the DIFF +# decides which of the two claims a name, not file order. +# +# `middle` left the field list in the same edit: it was there for +# 'John Smith Dr.', whose middle name moved, and the three names the +# rule still explains move `family` and `suffix` alone. The gate said +# so itself -- OVER-DECLARED, "classify() matches by SUBSET, so the +# excess is not inert" (#452). name_regex = "(?i)\\sdr\\.?$" -fields = ["middle", "family", "suffix"] +fields = ["family", "suffix"] [[change]] issue = "fix(#296) a credential-only comma string reads a name and its postnominal" @@ -1717,3 +1766,202 @@ issue = "fix(#462) the facade keeps an initial-shaped conjunction letter" name_regex = "(?:^|[\\s,])(?:[EY]|[EeYy]\\.)(?=[\\s,]|$)" fields = ["_initials"] orders = ["DEFAULT"] + +# --------------------------------------------------------------- +# THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last +# title), #316 (a trailing period-marked title word reads as a +# title), the leading peel's name-word floor, and the `esq` acronym +# drop that shipped beside them. Four rules, one per ARGUMENT rather +# than one per name, over nineteen corpus names. +# +# LAST in the file, and that is narrow-first rather than a +# preference: every rule already here that REACHES one of the +# nineteen declares `fields` that are a strict subset of the bundle +# rule claiming that name, and a wide rule sitting AHEAD of a +# narrower one it shares a corpus name with is an order-decided +# contest the run refuses (#382). Measured 2026-09-08, all four +# ledgers: at 1.4.0 fix(#296)'s trailing `dr` reaches 'John Smith +# Dr.' and 'John Smith Prof. Dr.', the two comma routings reach +# 'Smith, John Prof.', and the two-token `jr` rule reaches 'Dr Jr' +# and 'Sir Jr'; at 2.0.0 and 2.1.0 only fix(#296) reaches any of +# them; at 2.2.0 none does. Not one of those rules ADMITS the diff +# it reaches -- every shape is outside its field list, measured -- +# so nothing below takes a name off a rule that was explaining it, +# and no [[change.precedes_narrower]] block is needed anywhere. +# +# `_initials` appears in none of the four field lists although the +# initials of several of the nineteen do move. The derived view +# enters a diff only where every ROLE and every ambiguity kind +# agrees (compare.py's _RULE_FIELDS, main()'s roles-identical +# guard), and all nineteen move roles -- so no run can produce a +# shape here carrying it, and validate_rules refuses it beside +# another field in any case. +# --------------------------------------------------------------- + +[[change]] +issue = "fix(#489) a title run addresses by its last title" +# 'Her Majesty Queen Elizabeth' and 'Dr. Sir John': the +# given_name_titles lookup was keyed on the WHOLE run, and 'her +# majesty queen' is in no vocabulary and never could be, so the one +# name word behind the run stayed the family. Both sites that ask -- +# rules.md#H1's retag and rules.md#P5's licence -- now test the whole +# run's key OR the last word's, and 'queen' and 'sir' are +# GIVEN_NAME_TITLES entries. The name word behind the run becomes the +# GIVEN name and the family empties, which is H1's Accepted outcome. +# Both roles move together and `fields` declares both. `title` is +# deliberately OUT: the run itself is unchanged, only what it +# addresses as, so a name whose TITLE moved arrives UNEXPLAINED, as +# it should. +# +# 'Reverend Mother Teresa', 'Mr Sir John', 'Xyz. Sir John' and 'Sir +# Sheikh abdul rahman' move the same way and are NOT members: none of +# them is in any corpus file, so a member for one would be a rule +# reaching nothing. +# +# An anchored alternation of the two NAMES rather than a shape. What +# selects them is a SHAPE -- a multi-word title run whose LAST word +# is a given-name title -- and a member copying GIVEN_NAME_TITLES +# would reach 'Sir John Smith' and 'Dr. Smith', which do not move. +# The members are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY in tests/v2/test_ledger_guards.py records; +# _CORPUS_CLAIMS pins the reach at 2 with its digest, and +# _MUST_NOT_MATCH carries three boundaries: a ONE-word run ('Sir John +# Smith'), and two runs whose last word is title vocabulary that is +# not given-name vocabulary ('His Excellency Lord Duncan', 'Her Royal +# Highness Princess Anne' -- the open vocabulary question, #519). One +# set, identical in all four ledgers. +name_regex = "^(?:Dr\\. Sir John|Her Majesty Queen Elizabeth)$" +fields = ["family", "given"] + +[[change]] +issue = "change(suffix-acronym-collisions) esq leaves the acronym set" +# 'John Smith E.S.Q.': 'esq' left SUFFIX_ACRONYMS, where the 2019 bulk +# post-nominal import put it (af5bdab, #93) -- esquire is a +# contraction, not an initialism -- so the multi-dot spelling stops +# being credential vocabulary and falls to the positional read: given +# 'John', middle 'Smith', family 'E.S.Q.', where every release since +# 1.4.0 read family 'Smith', suffix 'E.S.Q.'. +# +# A BEHAVIOR CHANGE rather than a fix, which is what the `change` tag +# says: this is a deliberate parity break at all four baselines, and +# the entry that argues it is decisions.md#suffix-acronym-collisions +# -- there is no issue number to cite, and the release-log bullet +# carries none either. It is not NOT WANTED: that spelling is for a +# reading nobody wants, and this reading is the one the decision +# chose. +# +# Literal, no alternation, and exactly as wide as the diff: 'esq' +# keeps its SUFFIX_WORDS membership, so the acronym entry's only +# unique coverage was this one spelling and this one corpus name. +# _MUST_NOT_MATCH carries the three spellings that did NOT move -- +# 'John Smith Esq' (the word list still reads it), 'Esq. Smith' +# (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone +# post-comma credential). +name_regex = "^John Smith E\\.S\\.Q\\.$" +fields = ["family", "middle", "suffix"] + +[[change]] +issue = "fix(#489) the title peel leaves a name word a suffix cannot be" +# 'Dr King Jr', 'Dr. King MD', 'Dr Jr' and 'Sir Jr': the leading title +# peel's only floor was "leave one piece", so it peeled 'Dr King' +# whole and left the post-nominal to be the name -- title 'Dr King', +# family 'Jr', no suffix at all. leading_titles now also refuses to +# leave a word that is nothing but suffix vocabulary standing as the +# name and gives the run's last word back (rules.md#H3, +# decisions.md#H3), so 'Dr King Jr' reads title 'Dr', family 'King', +# suffix 'Jr' -- what the comma spelling 'King, Dr Jr' always read. +# +# FOUR roles, and no single name moves all four. The two three-word +# names move `title`, `family` and `suffix`; 'Dr Jr' and 'Sir Jr' +# move `given` as well, or instead of `family`, the word given back +# standing alone with the credential peeled off it. At the 2.x +# baselines `_ambiguities` joins the list -- with the title given +# back there is no title left to make the reading H1's, so all four +# report title-or-name through rules.md#H4 -- and it is absent at +# 1.4.0, which has no ambiguity surface to compare (compare.py's +# _RULE_FIELDS). +# +# An anchored alternation of the four NAMES rather than a shape. What +# selects them is a SHAPE -- a title run with nothing but post-nominal +# pieces behind it, whose own last word is not one of those -- and a +# member copying TITLES or SUFFIX_WORDS would reach 'MD DDS', 'Jr. +# Ph. D.' and 'Marquess of Bath', every one of which the floor +# declines and every one of which is a _MUST_NOT_MATCH probe, with +# 'Dr Smith Jr' beside them for the ordinary titled name the floor +# never sees. The members are a list of names and copy no wordlist, +# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins +# the reach at 4 with its digest. One set, identical in all four +# ledgers; only the `fields` differ, as the paragraph above says. +name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" +fields = ["_ambiguities", "family", "given", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing period-marked title word reads as a title" +# 'John Smith Prof.' and thirteen more: a trailing run of period-marked +# LISTED title words is peeled as a title, so the surname behind it +# stops being lost -- title 'Prof.', given 'John', family 'Smith', +# where every baseline read middle 'Smith', family 'Prof.'. A RUN, +# mirroring the leading peel: 'John Smith Prof. Dr.' gives title +# 'Prof. Dr.' and 'Dr. John Smith Prof.' gives 'Dr. Prof.' in input +# order. The family-comma segment takes the same walk, so 'Smith, +# John Prof.' agrees with the bare spelling at last. And the trailing +# title is TRANSPARENT to the post-nominal peel that follows it: +# 'John Smith Jr. Prof.' keeps suffix 'Jr.' rather than promoting the +# generational suffix to the family name, and 'John Prof. MA' reads +# family 'MA' by rules.md#S2's reserve. +# +# Five roles, the union of what the fourteen move; no name moves all +# five, and three move only two -- 'Smith, John Prof.' {title, +# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' +# moving three. `_ambiguities` is not among the five and none of the +# fourteen reports one: the walk leaves a title standing, so H4's +# title half never fires. +# +# TWO of the fourteen end in ' Dr.' and are also reached by +# fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which +# that rule used to explain, and 'John Smith Prof. Dr.'. That rule +# declares {middle, family, suffix} and neither diff fits it now, so +# both fall through to here; its comment records the handover, and +# narrow-first placement is what keeps the pair out of contest. The +# BARE trailing spelling stays with fix(#296): 'Smith Dr' and the 'dr +# ... dr' names wear no abbreviation period and the walk never sees +# them (#316(b), left open). +# +# The last two members arrived with the 2026-09-09 review of the +# docs commit, which added two rules.md examples: 'Mary Jane King.', +# the H5 Accepted clause's collision example (title 'King.', given +# 'Mary', family 'Jane', where every baseline read family 'King.' +# behind middle 'Jane'), and 'Smith Sir.', which is an H1 example. +# 'Smith Sir.' is HERE and not on the fix(#489) run rule above +# because the DIFF decides: it moves {title, family}, the walk +# taking 'Sir.' out of the name, while its given/family reading is +# what every baseline already gave ('Smith' stays the given name on +# both sides). The #489 rule declares {family, given} and does not +# reach a title role at all, so it could not claim the diff even +# placed ahead of this one. +# +# An anchored alternation of the fourteen NAMES rather than a shape. +# What selects them is a SHAPE -- period-marked, listed, trailing -- +# and a member copying TITLES would reach the BARE 'Mary Jane King' +# and 'John Smith Sir', which do not move; the period-marked +# 'Mary Jane King.' is a member precisely because it does, which is +# the line the period draws and the whole of what it draws. A member +# spelled as the shape (a trailing dotted word) would reach 'John +# Smith Xyz.', which is the boundary the leading slot's H2 inference +# is deliberately NOT given here. Those three and 'Smith, Prof.' -- +# the comma shape whose segment holds no name word and which reads +# by a different gate -- are the _MUST_NOT_MATCH probes. The members +# are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins the reach at +# 14 with its digest. +# +# 'Andrew Perkins (Mgr.)' is spelled with \\x28 and \\x29 rather than +# the \\( and \\) the paren-bearing rules above use. Escaped or not, a +# literal parenthesis inside an alternation body hides the WHOLE +# group from test_ledger_guards._alternations -- its member pattern +# excludes '(' and ')' -- and a twelve-member alternation should not +# be invisible to the discovery pass that demands every alternation +# declare what it copies. The two spellings match the same string. +# One set, identical in all four ledgers. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +fields = ["family", "given", "middle", "suffix", "title"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 9c507578..d8157b5f 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -479,18 +479,36 @@ fields = ["title", "given", "middle", "_ambiguities"] [[change]] issue = "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word" -# 'John Smith Dr.' and 'dr Vincent van Gogh dr' with its #100 siblings -# (and, at 2.x only, 'Smith Dr', which 1.4.0 already read as the -# family): 'dr' left SUFFIX_WORDS, where it was v1 -# residue -- 'Dr.' is not a postnominal in any tradition -- so a -# trailing bare 'Dr' is no longer suffix vocabulary and falls to the -# positional read, taking the family name with it. NOT a new class: -# no trailing title-only word routes to title on the no-comma path -# ('John Smith Prof.' reads family 'Prof.' at every baseline), and -# the suffix entry was the only thing making 'dr' the exception. #316 -# is the open question for the class. +# 'dr Vincent van Gogh dr' with its #100 siblings (and, at 2.x only, +# 'Smith Dr', which 1.4.0 already read as the family): 'dr' left +# SUFFIX_WORDS, where it was v1 residue -- 'Dr.' is not a postnominal +# in any tradition -- so a trailing bare 'Dr' is no longer suffix +# vocabulary and falls to the positional read, taking the family name +# with it. +# +# 2026-09-08: 'John Smith Dr.' has LEFT this rule, and the sentence +# that used to close this comment has been retracted with it. It read +# "NOT a new class: no trailing title-only word routes to title on +# the no-comma path ('John Smith Prof.' reads family 'Prof.' at every +# baseline) ... #316 is the open question for the class". #316 is +# answered: a trailing PERIOD-MARKED listed title word now routes to +# title, so 'John Smith Dr.' moves {title, suffix} against this +# baseline and {title, middle, family} against 2.2.0, neither of them +# a subset of the three roles below, and the bundle rule `fix(#316) a +# trailing period-marked title word reads as a title` at the end of +# this file claims it. What stays here is the BARE spelling, which +# #316(b) leaves open: 'Smith Dr' and the 'dr ... dr' names wear no +# abbreviation period and the trailing walk never sees them. This +# rule is narrower than that one and stands first, so the DIFF +# decides which of the two claims a name, not file order. +# +# `middle` left the field list in the same edit: it was there for +# 'John Smith Dr.', whose middle name moved, and the three names the +# rule still explains move `family` and `suffix` alone. The gate said +# so itself -- OVER-DECLARED, "classify() matches by SUBSET, so the +# excess is not inert" (#452). name_regex = "(?i)\\sdr\\.?$" -fields = ["middle", "family", "suffix"] +fields = ["family", "suffix"] [[change]] issue = "fix(#296) a credential-only comma string reads a name and its postnominal" @@ -1667,3 +1685,202 @@ issue = "fix(#462) the facade keeps an initial-shaped conjunction letter" name_regex = "(?:^|[\\s,])(?:[EY]|[EeYy]\\.)(?=[\\s,]|$)" fields = ["_initials"] orders = ["DEFAULT"] + +# --------------------------------------------------------------- +# THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last +# title), #316 (a trailing period-marked title word reads as a +# title), the leading peel's name-word floor, and the `esq` acronym +# drop that shipped beside them. Four rules, one per ARGUMENT rather +# than one per name, over nineteen corpus names. +# +# LAST in the file, and that is narrow-first rather than a +# preference: every rule already here that REACHES one of the +# nineteen declares `fields` that are a strict subset of the bundle +# rule claiming that name, and a wide rule sitting AHEAD of a +# narrower one it shares a corpus name with is an order-decided +# contest the run refuses (#382). Measured 2026-09-08, all four +# ledgers: at 1.4.0 fix(#296)'s trailing `dr` reaches 'John Smith +# Dr.' and 'John Smith Prof. Dr.', the two comma routings reach +# 'Smith, John Prof.', and the two-token `jr` rule reaches 'Dr Jr' +# and 'Sir Jr'; at 2.0.0 and 2.1.0 only fix(#296) reaches any of +# them; at 2.2.0 none does. Not one of those rules ADMITS the diff +# it reaches -- every shape is outside its field list, measured -- +# so nothing below takes a name off a rule that was explaining it, +# and no [[change.precedes_narrower]] block is needed anywhere. +# +# `_initials` appears in none of the four field lists although the +# initials of several of the nineteen do move. The derived view +# enters a diff only where every ROLE and every ambiguity kind +# agrees (compare.py's _RULE_FIELDS, main()'s roles-identical +# guard), and all nineteen move roles -- so no run can produce a +# shape here carrying it, and validate_rules refuses it beside +# another field in any case. +# --------------------------------------------------------------- + +[[change]] +issue = "fix(#489) a title run addresses by its last title" +# 'Her Majesty Queen Elizabeth' and 'Dr. Sir John': the +# given_name_titles lookup was keyed on the WHOLE run, and 'her +# majesty queen' is in no vocabulary and never could be, so the one +# name word behind the run stayed the family. Both sites that ask -- +# rules.md#H1's retag and rules.md#P5's licence -- now test the whole +# run's key OR the last word's, and 'queen' and 'sir' are +# GIVEN_NAME_TITLES entries. The name word behind the run becomes the +# GIVEN name and the family empties, which is H1's Accepted outcome. +# Both roles move together and `fields` declares both. `title` is +# deliberately OUT: the run itself is unchanged, only what it +# addresses as, so a name whose TITLE moved arrives UNEXPLAINED, as +# it should. +# +# 'Reverend Mother Teresa', 'Mr Sir John', 'Xyz. Sir John' and 'Sir +# Sheikh abdul rahman' move the same way and are NOT members: none of +# them is in any corpus file, so a member for one would be a rule +# reaching nothing. +# +# An anchored alternation of the two NAMES rather than a shape. What +# selects them is a SHAPE -- a multi-word title run whose LAST word +# is a given-name title -- and a member copying GIVEN_NAME_TITLES +# would reach 'Sir John Smith' and 'Dr. Smith', which do not move. +# The members are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY in tests/v2/test_ledger_guards.py records; +# _CORPUS_CLAIMS pins the reach at 2 with its digest, and +# _MUST_NOT_MATCH carries three boundaries: a ONE-word run ('Sir John +# Smith'), and two runs whose last word is title vocabulary that is +# not given-name vocabulary ('His Excellency Lord Duncan', 'Her Royal +# Highness Princess Anne' -- the open vocabulary question, #519). One +# set, identical in all four ledgers. +name_regex = "^(?:Dr\\. Sir John|Her Majesty Queen Elizabeth)$" +fields = ["family", "given"] + +[[change]] +issue = "change(suffix-acronym-collisions) esq leaves the acronym set" +# 'John Smith E.S.Q.': 'esq' left SUFFIX_ACRONYMS, where the 2019 bulk +# post-nominal import put it (af5bdab, #93) -- esquire is a +# contraction, not an initialism -- so the multi-dot spelling stops +# being credential vocabulary and falls to the positional read: given +# 'John', middle 'Smith', family 'E.S.Q.', where every release since +# 1.4.0 read family 'Smith', suffix 'E.S.Q.'. +# +# A BEHAVIOR CHANGE rather than a fix, which is what the `change` tag +# says: this is a deliberate parity break at all four baselines, and +# the entry that argues it is decisions.md#suffix-acronym-collisions +# -- there is no issue number to cite, and the release-log bullet +# carries none either. It is not NOT WANTED: that spelling is for a +# reading nobody wants, and this reading is the one the decision +# chose. +# +# Literal, no alternation, and exactly as wide as the diff: 'esq' +# keeps its SUFFIX_WORDS membership, so the acronym entry's only +# unique coverage was this one spelling and this one corpus name. +# _MUST_NOT_MATCH carries the three spellings that did NOT move -- +# 'John Smith Esq' (the word list still reads it), 'Esq. Smith' +# (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone +# post-comma credential). +name_regex = "^John Smith E\\.S\\.Q\\.$" +fields = ["family", "middle", "suffix"] + +[[change]] +issue = "fix(#489) the title peel leaves a name word a suffix cannot be" +# 'Dr King Jr', 'Dr. King MD', 'Dr Jr' and 'Sir Jr': the leading title +# peel's only floor was "leave one piece", so it peeled 'Dr King' +# whole and left the post-nominal to be the name -- title 'Dr King', +# family 'Jr', no suffix at all. leading_titles now also refuses to +# leave a word that is nothing but suffix vocabulary standing as the +# name and gives the run's last word back (rules.md#H3, +# decisions.md#H3), so 'Dr King Jr' reads title 'Dr', family 'King', +# suffix 'Jr' -- what the comma spelling 'King, Dr Jr' always read. +# +# FOUR roles, and no single name moves all four. The two three-word +# names move `title`, `family` and `suffix`; 'Dr Jr' and 'Sir Jr' +# move `given` as well, or instead of `family`, the word given back +# standing alone with the credential peeled off it. At the 2.x +# baselines `_ambiguities` joins the list -- with the title given +# back there is no title left to make the reading H1's, so all four +# report title-or-name through rules.md#H4 -- and it is absent at +# 1.4.0, which has no ambiguity surface to compare (compare.py's +# _RULE_FIELDS). +# +# An anchored alternation of the four NAMES rather than a shape. What +# selects them is a SHAPE -- a title run with nothing but post-nominal +# pieces behind it, whose own last word is not one of those -- and a +# member copying TITLES or SUFFIX_WORDS would reach 'MD DDS', 'Jr. +# Ph. D.' and 'Marquess of Bath', every one of which the floor +# declines and every one of which is a _MUST_NOT_MATCH probe, with +# 'Dr Smith Jr' beside them for the ordinary titled name the floor +# never sees. The members are a list of names and copy no wordlist, +# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins +# the reach at 4 with its digest. One set, identical in all four +# ledgers; only the `fields` differ, as the paragraph above says. +name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" +fields = ["_ambiguities", "family", "given", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing period-marked title word reads as a title" +# 'John Smith Prof.' and thirteen more: a trailing run of period-marked +# LISTED title words is peeled as a title, so the surname behind it +# stops being lost -- title 'Prof.', given 'John', family 'Smith', +# where every baseline read middle 'Smith', family 'Prof.'. A RUN, +# mirroring the leading peel: 'John Smith Prof. Dr.' gives title +# 'Prof. Dr.' and 'Dr. John Smith Prof.' gives 'Dr. Prof.' in input +# order. The family-comma segment takes the same walk, so 'Smith, +# John Prof.' agrees with the bare spelling at last. And the trailing +# title is TRANSPARENT to the post-nominal peel that follows it: +# 'John Smith Jr. Prof.' keeps suffix 'Jr.' rather than promoting the +# generational suffix to the family name, and 'John Prof. MA' reads +# family 'MA' by rules.md#S2's reserve. +# +# Five roles, the union of what the fourteen move; no name moves all +# five, and three move only two -- 'Smith, John Prof.' {title, +# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' +# moving three. `_ambiguities` is not among the five and none of the +# fourteen reports one: the walk leaves a title standing, so H4's +# title half never fires. +# +# TWO of the fourteen end in ' Dr.' and are also reached by +# fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which +# that rule used to explain, and 'John Smith Prof. Dr.'. That rule +# declares {middle, family, suffix} and neither diff fits it now, so +# both fall through to here; its comment records the handover, and +# narrow-first placement is what keeps the pair out of contest. The +# BARE trailing spelling stays with fix(#296): 'Smith Dr' and the 'dr +# ... dr' names wear no abbreviation period and the walk never sees +# them (#316(b), left open). +# +# The last two members arrived with the 2026-09-09 review of the +# docs commit, which added two rules.md examples: 'Mary Jane King.', +# the H5 Accepted clause's collision example (title 'King.', given +# 'Mary', family 'Jane', where every baseline read family 'King.' +# behind middle 'Jane'), and 'Smith Sir.', which is an H1 example. +# 'Smith Sir.' is HERE and not on the fix(#489) run rule above +# because the DIFF decides: it moves {title, family}, the walk +# taking 'Sir.' out of the name, while its given/family reading is +# what every baseline already gave ('Smith' stays the given name on +# both sides). The #489 rule declares {family, given} and does not +# reach a title role at all, so it could not claim the diff even +# placed ahead of this one. +# +# An anchored alternation of the fourteen NAMES rather than a shape. +# What selects them is a SHAPE -- period-marked, listed, trailing -- +# and a member copying TITLES would reach the BARE 'Mary Jane King' +# and 'John Smith Sir', which do not move; the period-marked +# 'Mary Jane King.' is a member precisely because it does, which is +# the line the period draws and the whole of what it draws. A member +# spelled as the shape (a trailing dotted word) would reach 'John +# Smith Xyz.', which is the boundary the leading slot's H2 inference +# is deliberately NOT given here. Those three and 'Smith, Prof.' -- +# the comma shape whose segment holds no name word and which reads +# by a different gate -- are the _MUST_NOT_MATCH probes. The members +# are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins the reach at +# 14 with its digest. +# +# 'Andrew Perkins (Mgr.)' is spelled with \\x28 and \\x29 rather than +# the \\( and \\) the paren-bearing rules above use. Escaped or not, a +# literal parenthesis inside an alternation body hides the WHOLE +# group from test_ledger_guards._alternations -- its member pattern +# excludes '(' and ')' -- and a twelve-member alternation should not +# be invisible to the discovery pass that demands every alternation +# declare what it copies. The two spellings match the same string. +# One set, identical in all four ledgers. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +fields = ["family", "given", "middle", "suffix", "title"] diff --git a/tools/differential/expected_since_2.2.0.toml b/tools/differential/expected_since_2.2.0.toml index a063492d..3a191fa6 100644 --- a/tools/differential/expected_since_2.2.0.toml +++ b/tools/differential/expected_since_2.2.0.toml @@ -326,3 +326,202 @@ issue = "fix(#462) the facade keeps an initial-shaped conjunction letter" name_regex = "(?:^|[\\s,])(?:[EY]|[EeYy]\\.)(?=[\\s,]|$)" fields = ["_initials"] orders = ["DEFAULT"] + +# --------------------------------------------------------------- +# THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last +# title), #316 (a trailing period-marked title word reads as a +# title), the leading peel's name-word floor, and the `esq` acronym +# drop that shipped beside them. Four rules, one per ARGUMENT rather +# than one per name, over nineteen corpus names. +# +# LAST in the file, and that is narrow-first rather than a +# preference: every rule already here that REACHES one of the +# nineteen declares `fields` that are a strict subset of the bundle +# rule claiming that name, and a wide rule sitting AHEAD of a +# narrower one it shares a corpus name with is an order-decided +# contest the run refuses (#382). Measured 2026-09-08, all four +# ledgers: at 1.4.0 fix(#296)'s trailing `dr` reaches 'John Smith +# Dr.' and 'John Smith Prof. Dr.', the two comma routings reach +# 'Smith, John Prof.', and the two-token `jr` rule reaches 'Dr Jr' +# and 'Sir Jr'; at 2.0.0 and 2.1.0 only fix(#296) reaches any of +# them; at 2.2.0 none does. Not one of those rules ADMITS the diff +# it reaches -- every shape is outside its field list, measured -- +# so nothing below takes a name off a rule that was explaining it, +# and no [[change.precedes_narrower]] block is needed anywhere. +# +# `_initials` appears in none of the four field lists although the +# initials of several of the nineteen do move. The derived view +# enters a diff only where every ROLE and every ambiguity kind +# agrees (compare.py's _RULE_FIELDS, main()'s roles-identical +# guard), and all nineteen move roles -- so no run can produce a +# shape here carrying it, and validate_rules refuses it beside +# another field in any case. +# --------------------------------------------------------------- + +[[change]] +issue = "fix(#489) a title run addresses by its last title" +# 'Her Majesty Queen Elizabeth' and 'Dr. Sir John': the +# given_name_titles lookup was keyed on the WHOLE run, and 'her +# majesty queen' is in no vocabulary and never could be, so the one +# name word behind the run stayed the family. Both sites that ask -- +# rules.md#H1's retag and rules.md#P5's licence -- now test the whole +# run's key OR the last word's, and 'queen' and 'sir' are +# GIVEN_NAME_TITLES entries. The name word behind the run becomes the +# GIVEN name and the family empties, which is H1's Accepted outcome. +# Both roles move together and `fields` declares both. `title` is +# deliberately OUT: the run itself is unchanged, only what it +# addresses as, so a name whose TITLE moved arrives UNEXPLAINED, as +# it should. +# +# 'Reverend Mother Teresa', 'Mr Sir John', 'Xyz. Sir John' and 'Sir +# Sheikh abdul rahman' move the same way and are NOT members: none of +# them is in any corpus file, so a member for one would be a rule +# reaching nothing. +# +# An anchored alternation of the two NAMES rather than a shape. What +# selects them is a SHAPE -- a multi-word title run whose LAST word +# is a given-name title -- and a member copying GIVEN_NAME_TITLES +# would reach 'Sir John Smith' and 'Dr. Smith', which do not move. +# The members are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY in tests/v2/test_ledger_guards.py records; +# _CORPUS_CLAIMS pins the reach at 2 with its digest, and +# _MUST_NOT_MATCH carries three boundaries: a ONE-word run ('Sir John +# Smith'), and two runs whose last word is title vocabulary that is +# not given-name vocabulary ('His Excellency Lord Duncan', 'Her Royal +# Highness Princess Anne' -- the open vocabulary question, #519). One +# set, identical in all four ledgers. +name_regex = "^(?:Dr\\. Sir John|Her Majesty Queen Elizabeth)$" +fields = ["family", "given"] + +[[change]] +issue = "change(suffix-acronym-collisions) esq leaves the acronym set" +# 'John Smith E.S.Q.': 'esq' left SUFFIX_ACRONYMS, where the 2019 bulk +# post-nominal import put it (af5bdab, #93) -- esquire is a +# contraction, not an initialism -- so the multi-dot spelling stops +# being credential vocabulary and falls to the positional read: given +# 'John', middle 'Smith', family 'E.S.Q.', where every release since +# 1.4.0 read family 'Smith', suffix 'E.S.Q.'. +# +# A BEHAVIOR CHANGE rather than a fix, which is what the `change` tag +# says: this is a deliberate parity break at all four baselines, and +# the entry that argues it is decisions.md#suffix-acronym-collisions +# -- there is no issue number to cite, and the release-log bullet +# carries none either. It is not NOT WANTED: that spelling is for a +# reading nobody wants, and this reading is the one the decision +# chose. +# +# Literal, no alternation, and exactly as wide as the diff: 'esq' +# keeps its SUFFIX_WORDS membership, so the acronym entry's only +# unique coverage was this one spelling and this one corpus name. +# _MUST_NOT_MATCH carries the three spellings that did NOT move -- +# 'John Smith Esq' (the word list still reads it), 'Esq. Smith' +# (rules.md#H2's leading inference) and 'Smith, Esq.' (the lone +# post-comma credential). +name_regex = "^John Smith E\\.S\\.Q\\.$" +fields = ["family", "middle", "suffix"] + +[[change]] +issue = "fix(#489) the title peel leaves a name word a suffix cannot be" +# 'Dr King Jr', 'Dr. King MD', 'Dr Jr' and 'Sir Jr': the leading title +# peel's only floor was "leave one piece", so it peeled 'Dr King' +# whole and left the post-nominal to be the name -- title 'Dr King', +# family 'Jr', no suffix at all. leading_titles now also refuses to +# leave a word that is nothing but suffix vocabulary standing as the +# name and gives the run's last word back (rules.md#H3, +# decisions.md#H3), so 'Dr King Jr' reads title 'Dr', family 'King', +# suffix 'Jr' -- what the comma spelling 'King, Dr Jr' always read. +# +# FOUR roles, and no single name moves all four. The two three-word +# names move `title`, `family` and `suffix`; 'Dr Jr' and 'Sir Jr' +# move `given` as well, or instead of `family`, the word given back +# standing alone with the credential peeled off it. At the 2.x +# baselines `_ambiguities` joins the list -- with the title given +# back there is no title left to make the reading H1's, so all four +# report title-or-name through rules.md#H4 -- and it is absent at +# 1.4.0, which has no ambiguity surface to compare (compare.py's +# _RULE_FIELDS). +# +# An anchored alternation of the four NAMES rather than a shape. What +# selects them is a SHAPE -- a title run with nothing but post-nominal +# pieces behind it, whose own last word is not one of those -- and a +# member copying TITLES or SUFFIX_WORDS would reach 'MD DDS', 'Jr. +# Ph. D.' and 'Marquess of Bath', every one of which the floor +# declines and every one of which is a _MUST_NOT_MATCH probe, with +# 'Dr Smith Jr' beside them for the ordinary titled name the floor +# never sees. The members are a list of names and copy no wordlist, +# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins +# the reach at 4 with its digest. One set, identical in all four +# ledgers; only the `fields` differ, as the paragraph above says. +name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" +fields = ["_ambiguities", "family", "given", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing period-marked title word reads as a title" +# 'John Smith Prof.' and thirteen more: a trailing run of period-marked +# LISTED title words is peeled as a title, so the surname behind it +# stops being lost -- title 'Prof.', given 'John', family 'Smith', +# where every baseline read middle 'Smith', family 'Prof.'. A RUN, +# mirroring the leading peel: 'John Smith Prof. Dr.' gives title +# 'Prof. Dr.' and 'Dr. John Smith Prof.' gives 'Dr. Prof.' in input +# order. The family-comma segment takes the same walk, so 'Smith, +# John Prof.' agrees with the bare spelling at last. And the trailing +# title is TRANSPARENT to the post-nominal peel that follows it: +# 'John Smith Jr. Prof.' keeps suffix 'Jr.' rather than promoting the +# generational suffix to the family name, and 'John Prof. MA' reads +# family 'MA' by rules.md#S2's reserve. +# +# Five roles, the union of what the fourteen move; no name moves all +# five, and three move only two -- 'Smith, John Prof.' {title, +# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' +# moving three. `_ambiguities` is not among the five and none of the +# fourteen reports one: the walk leaves a title standing, so H4's +# title half never fires. +# +# TWO of the fourteen end in ' Dr.' and are also reached by +# fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which +# that rule used to explain, and 'John Smith Prof. Dr.'. That rule +# declares {middle, family, suffix} and neither diff fits it now, so +# both fall through to here; its comment records the handover, and +# narrow-first placement is what keeps the pair out of contest. The +# BARE trailing spelling stays with fix(#296): 'Smith Dr' and the 'dr +# ... dr' names wear no abbreviation period and the walk never sees +# them (#316(b), left open). +# +# The last two members arrived with the 2026-09-09 review of the +# docs commit, which added two rules.md examples: 'Mary Jane King.', +# the H5 Accepted clause's collision example (title 'King.', given +# 'Mary', family 'Jane', where every baseline read family 'King.' +# behind middle 'Jane'), and 'Smith Sir.', which is an H1 example. +# 'Smith Sir.' is HERE and not on the fix(#489) run rule above +# because the DIFF decides: it moves {title, family}, the walk +# taking 'Sir.' out of the name, while its given/family reading is +# what every baseline already gave ('Smith' stays the given name on +# both sides). The #489 rule declares {family, given} and does not +# reach a title role at all, so it could not claim the diff even +# placed ahead of this one. +# +# An anchored alternation of the fourteen NAMES rather than a shape. +# What selects them is a SHAPE -- period-marked, listed, trailing -- +# and a member copying TITLES would reach the BARE 'Mary Jane King' +# and 'John Smith Sir', which do not move; the period-marked +# 'Mary Jane King.' is a member precisely because it does, which is +# the line the period draws and the whole of what it draws. A member +# spelled as the shape (a trailing dotted word) would reach 'John +# Smith Xyz.', which is the boundary the leading slot's H2 inference +# is deliberately NOT given here. Those three and 'Smith, Prof.' -- +# the comma shape whose segment holds no name word and which reads +# by a different gate -- are the _MUST_NOT_MATCH probes. The members +# are a list of names and copy no wordlist, which is what +# _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins the reach at +# 14 with its digest. +# +# 'Andrew Perkins (Mgr.)' is spelled with \\x28 and \\x29 rather than +# the \\( and \\) the paren-bearing rules above use. Escaped or not, a +# literal parenthesis inside an alternation body hides the WHOLE +# group from test_ledger_guards._alternations -- its member pattern +# excludes '(' and ')' -- and a twelve-member alternation should not +# be invisible to the discovery pass that demands every alternation +# declare what it copies. The two spellings match the same string. +# One set, identical in all four ledgers. +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +fields = ["family", "given", "middle", "suffix", "title"] From e489dc162589d4069e54609370fd83caf7e5573a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 14:56:15 -0700 Subject: [PATCH 07/12] review round: PR #520 findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P5's bound-given reserve counted a trailing period-marked title word as a name word to spare, so `Prof. abdul rahman Prof.` joined the bound pair and read family 'abdul rahman' where `Prof. abdul rahman` reads given 'abdul', family 'rahman'. The reserve now subtracts the H5 run from both views -- assign's second peel runs over the pieces that walk LEFT, so the reserve has to read the same list -- and the same-suffix comparison is read over the shortened lists too, which is what stops `Sir abdul Prof.` joining a title word into the given name. No corpus name moves: the 21-mover list against a0b93f0 is unchanged. Frames are unchanged at 416 parse / 453 facade on py3.11; the reference name never enters the reserve branch, having no bound given word, so its two new calls cost nothing. rules.md#P5 names the H5 walk in the sentence the code cites; H5 gains the P2 and M2 boundaries (`John van der Berg Prof.` keeps family 'van der Berg Prof.', `Mary Smith née Jones Prof.` maiden 'Jones Prof.') and the P5 clause; H5/P2/M2/P5 gain each other's `interacts:`. Tests: the script-order half of "set before the positional read" (`毛 泽东 Dr.`), the conjunction-merged unit that actually pins the one-word gate in `trailing_titles`, `Dr. Do Jr.`'s own test kept out of the no-op-chain parametrization's silence claim, and case rows for the two bound-given readings, `Smith, E.S.Q.`, the two maiden orderings, `Smith, John, Prof.` and `John Smith Prof. and Dr.`. Stale counts and false comments, all re-measured: 12/nineteen -> 14/twenty-one for the title-run bundle in the four ledgers and the guard, the twelve-member alternation is fourteen, the release log's bullet names the accepted `Mary Jane King.` cost, and the ledgers' role paragraph says which baseline has three two-role movers and which has two. `John Prof. MA` does report an ambiguity, so the "none of the fourteen" sentence is corrected to the one that matters (`title-or-name`, still none). The first suffix peel is provisional only where the walk takes something; `trailing_titles` is entered by every parse with a name word to place and not by all 1289 corpus parses (52 return first); its `rest` is the caller's name pieces, and group is now a third caller, which mechanisms.md's census records. Two defensive branches are marked measured-inert over 191,146 generated inputs -- the folded-vs-raw last word and the titled-piece skip -- and the third the review named, `walkable` starting at `n`, is NOT inert: dropping it moves 24 of those inputs, so it is marked load-bearing with the shape that moves. compare.py's RECOMPUTE paragraph: the strict row counts are 38/34/33/8 over 52 names, 50 of them in corpus_issues.jsonl. The every-file pair recorded on 2026-09-05 is retracted rather than bumped -- it exceeded the strict pair, which is impossible -- and replaced with a derivation a reader can run without a baseline wheel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- docs/design/mechanisms.md | 2 +- docs/design/rules.md | 27 +++- docs/release_log.rst | 2 +- nameparser/_lexicon.py | 28 +++- nameparser/_pipeline/_assign.py | 42 ++++-- nameparser/_pipeline/_group.py | 44 +++++-- nameparser/_pipeline/_pieces.py | 19 ++- tests/v2/cases.py | 120 +++++++++++++++++- tests/v2/pipeline/test_assign.py | 20 +++ tests/v2/pipeline/test_pieces.py | 9 ++ tests/v2/test_ledger_guards.py | 30 ++--- tests/v2/test_parser.py | 48 +++++-- tools/differential/compare.py | 40 +++--- tools/differential/corpus_cjk_tolerated.jsonl | 1 + tools/differential/corpus_rules.jsonl | 2 + tools/differential/expected_since_1.4.0.toml | 29 +++-- tools/differential/expected_since_2.0.0.toml | 29 +++-- tools/differential/expected_since_2.1.0.toml | 29 +++-- tools/differential/expected_since_2.2.0.toml | 29 +++-- 19 files changed, 422 insertions(+), 128 deletions(-) diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 2e2094cb..53abef9b 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles, peel_walk and peel_trailing are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` joins that last shape (2026-09-08, the #316/#489 bundle, rules.md#H5): assign alone calls it, at TWO sites — the main walk and the family-comma segment-1 walk — and it is in the leaf rather than inline because each site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. Re-measured 2026-09-08 by call site over `_pipeline/*.py`, the whole census above holds unchanged: is_suffix_piece, leading_titles, peel_walk and peel_trailing shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles, peel_walk and peel_trailing are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` was that last shape for one day (2026-09-08, the #316/#489 bundle, rules.md#H5) and is a shared one since 2026-09-09: assign calls it at two sites — the main walk and the family-comma segment-1 walk — and group's bound-given reserve at a third, because that reserve counts the name words assign will leave and this walk is half of what leaves them (rules.md#P5; counting a trailing title word among them joined 'Prof. abdul rahman Prof.' where 'Prof. abdul rahman' does not). It is in the leaf rather than inline because each assign site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. The reserve's two calls cost the reference name nothing — it never enters that branch, having no bound given word — and the parse and facade frame counts did not move (measured 2026-09-09). Re-measured 2026-09-09 by call site over `_pipeline/*.py`, the rest of the census above holds unchanged: is_suffix_piece, leading_titles, peel_walk, peel_trailing and now trailing_titles shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it diff --git a/docs/design/rules.md b/docs/design/rules.md index 4e7a3814..0e4697cb 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -288,7 +288,18 @@ H5. Rationale: a word abbreviated with a period at the END of a name not a reading a reader would hesitate over — where the doubt is real it is the word left STANDING that carries it, which is H4's report and not this rule's. - history: decisions.md#H5 · interacts: H1, H2, H3, H4, S2, C1 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py + Accepted: the chain reads PIECES, so a join that ran earlier + puts the word out of reach. A particle chain (P2) has already + taken the trailing word into the family name, and a maiden + marker (M2) has already taken it into the maiden name; in + neither is a title word standing in the trailing slot at all. + "John van der Berg Prof." → family="van der Berg Prof." + "Mary Smith née Jones Prof." → maiden="Jones Prof." + Accepted: what the chain leaves is also what counts as a name + word to spare (P5). A trailing title word is not one, so a bound + given-name word behind one joins exactly as it joins with the + title absent. + history: decisions.md#H5 · interacts: H1, H2, H3, H4, M2, P2, P5, S2, C1 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py ## Particles & surname prefixes (P) @@ -421,7 +432,7 @@ P2. Rationale: a particle is written as part of the surname it (#132's ask) has it as the surnames view rather than the family field. "Vincent van Gogh van Beethoven" → surnames="van Gogh van Beethoven" - history: decisions.md#P2 · interacts: P1, P4, M2, S2 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py + history: decisions.md#P2 · interacts: P1, P4, H5, M2, S2 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py P3. Rationale: connective words ("y", "of the") bind name words into one name part; but a single letter in a short name is more @@ -516,9 +527,11 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" particle's attachment (P6) sees the name. What there is to spare is what assign will leave: the join is tried on the pieces as it would - leave them, assign's trailing peel (S2) is read over that, and - the name words it leaves are the words to spare — a trailing - roman numeral, or a bare acronym the peel takes, is no + leave them, assign's trailing peel (S2) is read over that and its + trailing title run (H5) over what that peel leaves, and the name + words the two of them leave are the words to spare — a trailing + roman numeral, or a bare acronym the peel takes, or a trailing + title word the run takes, is no word to spare. The join joins two name words into one and changes no suffix reading: a word the peel reads as a suffix unjoined must read so joined, or the join declines. After a @@ -572,7 +585,7 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" "Sheik abdul salam" family-first → family="abdul salam" "Sheik abdul salam" family-first → given="" "Sheik abdul salam" family-first-given-last → family="abdul salam" - history: decisions.md#P5 · interacts: S2, M2, H1, P2, P4, P6 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py + history: decisions.md#P5 · interacts: S2, M2, H1, H5, P2, P4, P6 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_post_rules.py P6. Rationale: a particle ending the name has nothing to link forward to, so it is not doing a particle's work there. What it @@ -968,7 +981,7 @@ M2. Rationale: a maiden marker announces that what follows it is the is maiden text all the same — the count it needs includes the very words the marker removes, so the reading is left to assign. "John née Jones Smith Ma" → maiden="Jones Smith Ma" - history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2, H1 · implemented: nameparser/_pipeline/_group.py + history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2, H1, H5 · implemented: nameparser/_pipeline/_group.py M3. Rationale: an enclosure says nothing about whether it means maiden, but a recognized marker word inside it does — the clause diff --git a/docs/release_log.rst b/docs/release_log.rst index cbb63b16..e2ad30ae 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -22,7 +22,7 @@ Release Log - **Fix the leading title peel taking a name word and leaving a post-nominal to be the name.** ``HumanName("Dr King Jr")`` gives title ``Dr``, last ``King``, suffix ``Jr``, where every release since 1.4.0 gave title ``Dr King``, last ``Jr`` and no suffix at all; ``Dr. King MD`` moves the same way, and both now read as the comma spelling ``King, Dr Jr`` always has. A title addresses somebody, so the run leaves a name word standing and a post-nominal is not one. A name that is nothing but titles or nothing but post-nominals is untouched, the word given back having to be a name candidate: ``Marquess of Bath``, ``MD DDS`` and ``Jr. Ph. D.`` are unchanged, and so is a title written as one joined unit -- ``Prince of Wales Jr`` keeps title ``Prince of Wales`` rather than losing the title to make a name. Where the run's whole content is the word given back there is no title left, so ``Dr Jr`` gives first ``Dr``, suffix ``Jr`` and reports a title-or-name ambiguity. Four names in the differential corpora read differently for this rule. See the ``H3`` entry of ``docs/design/decisions.md`` - - **Fix a trailing abbreviated title reading as a name word.** ``HumanName("John Smith Prof.")`` gives title ``Prof.``, first ``John``, last ``Smith``, where every release since 1.4.0 gave last ``Prof.`` and lost the surname; ``John Smith Mr.``, ``John Smith Rev.``, ``John Smith Dr.`` and ``Andrew Perkins (Mgr.)`` move the same way. A run chains from the end (``John Smith Prof. Dr.`` gives title ``Prof. Dr.``), a leading title keeps its place (``Dr. John Smith Prof.`` gives title ``Dr. Prof.``), and the comma forms agree with the bare ones now -- ``Smith, John Prof.`` gives title ``Prof.``, first ``John``, last ``Smith`` where it gave middle ``Prof.`` at every release. The trailing title is transparent to the post-nominal reading, so ``John Smith Jr. Prof.`` gives suffix ``Jr.`` rather than promoting the generational suffix to the last name. What does NOT move: an unlisted abbreviation (``John Smith Xyz.`` keeps last ``Xyz.``), a bare title word (``John Smith Sir``, ``Mary Jane King``) and a post-nominal (``John Smith Esq.``). Only a listed title word wearing the abbreviation period is claimed -- the leading slot infers a title from the shape alone, the trailing slot never does. Twelve names in the differential corpora read differently for this rule. See the ``H5`` entry of ``docs/design/decisions.md`` (closes #316) + - **Fix a trailing abbreviated title reading as a name word.** ``HumanName("John Smith Prof.")`` gives title ``Prof.``, first ``John``, last ``Smith``, where every release since 1.4.0 gave last ``Prof.`` and lost the surname; ``John Smith Mr.``, ``John Smith Rev.``, ``John Smith Dr.`` and ``Andrew Perkins (Mgr.)`` move the same way. A run chains from the end (``John Smith Prof. Dr.`` gives title ``Prof. Dr.``), a leading title keeps its place (``Dr. John Smith Prof.`` gives title ``Dr. Prof.``), and the comma forms agree with the bare ones now -- ``Smith, John Prof.`` gives title ``Prof.``, first ``John``, last ``Smith`` where it gave middle ``Prof.`` at every release. The trailing title is transparent to the post-nominal reading, so ``John Smith Jr. Prof.`` gives suffix ``Jr.`` rather than promoting the generational suffix to the last name. What does NOT move: an unlisted abbreviation (``John Smith Xyz.`` keeps last ``Xyz.``), a bare title word (``John Smith Sir``, ``Mary Jane King``) and a post-nominal (``John Smith Esq.``). Only a listed title word wearing the abbreviation period is claimed -- the leading slot infers a title from the shape alone, the trailing slot never does. The reach is the whole title vocabulary, ordinary surnames in it included, so a period written behind one of them takes it out of the name: ``Mary Jane King.`` gives title ``King.``, first ``Mary``, last ``Jane``, where the bare ``Mary Jane King`` keeps last ``King``. That is accepted rather than prevented -- the period is a writing convention and not evidence about the word, and the bare spelling is what the trailing slot is protected from. Fourteen names in the differential corpora read differently for this rule. See the ``H5`` entry of ``docs/design/decisions.md`` (closes #316) - **Remove esq from the default post-nominal acronyms, and assert the two post-nominal sets disjoint.** ``HumanName("John Smith E.S.Q.")`` gives middle ``Smith``, last ``E.S.Q.``, where every release since 1.4.0 gave suffix ``E.S.Q.``. ``Esq``, ``Esq.``, ``ESQ`` and ``esq`` are unchanged, the post-nominal word list carrying every single-token spelling; the acronym entry's only unique coverage was the multi-dot spelling. Esquire is a contraction rather than an initialism, so the initialism set was never its home, and it was the one word in both post-nominal sets -- which is why the sets can now assert they do not overlap, a word in both being matched by two rules that normalize differently. A caller who needs it back adds it: ``Lexicon.default().add(suffix_acronyms={"esq"})``. One name moves in the differential corpora. See the ``suffix-acronym-collisions`` entry of ``docs/design/decisions.md`` diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index c225ad2a..88486d64 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -168,6 +168,22 @@ def _run_addresses_by_given(words: Iterable[str], word of the FOLDED key, not of the raw run, so a run token that folds away cannot empty that arm: the conjunction merge can put a lone '.' in the run ('Sir and . John'), and the fold drops it. + What the drop leaves as the last word can then be the CONJUNCTION + -- 'Sir and . John' keys 'sir and' and reads family 'John', where + 'Sir and Dame John' keys 'sir and dame' and reads given. Harmless + on the shipped vocabulary, which holds no entry ending in a + connective, and a caller who stored one would be asking for it + (measured 2026-09-09). + + Reading the FOLDED key's last word rather than the raw run's is a + defensive branch and a measured-inert one: over 191,146 generated + inputs it is reached 42,413 times and the two never differ, and + swapping it for the raw word changes no parse (2026-09-09) -- not + even on the lone '.' above, whose raw form folds to the empty + string and misses the vocabulary just as 'and' does. Kept as the + honest shape: the key is what the vocabulary is stored as, so the + key is what the lookup reads, and a caller's entry is the thing + that could make the two differ. The whole-run arm is what keeps a caller's multi-word phrase entry working: 'lt col' is stored as one key and matched as one run. Over @@ -176,11 +192,13 @@ def _run_addresses_by_given(words: Iterable[str], one-word run, which is a run the last-word arm reads the same way. H2's unlisted abbreviations ride in the run. One can never match as - the last-word key, being in no vocabulary by definition: 'Xyz. Sir' - keys 'sir' and matches, 'Sir Xyz.' keys 'xyz' and does not. It CAN - sit inside a whole-run key that matches, because given_name_titles - is deliberately not validated against titles: a caller may store - 'sir xyz', and 'Sir Xyz. John' then reads given. + the last-word key, being in no vocabulary by definition -- written + as the inputs that produce those runs, 'Xyz. Sir John' keys 'xyz + sir', matches on 'sir' and reads given 'John', while 'Sir Xyz. + John' keys 'sir xyz', matches on neither arm and reads family + 'John'. It CAN sit inside a whole-run key that matches, because + given_name_titles is deliberately not validated against titles: a + caller who stores 'sir xyz' makes 'Sir Xyz. John' read given. The vocabulary is passed in rather than read off a default: a caller's own Lexicon is the one that has to be consulted, and this diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index c079ef6c..63c9d627 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -16,11 +16,13 @@ (a title needs a following piece, unless the whole name is one title); then positional assignment per name_order with the trailing-suffix rule: the piece from which everything after is a strict suffix is the -last name-position piece, the rest are suffixes. That peel is only -provisional: behind it a trailing run of period-marked title words -chains into the title from the end, leaving one name piece standing, -and the peel then runs ONCE over the pieces with the titled ones -spliced out, so a trailing title is transparent to it. +last name-position piece, the rest are suffixes. Behind that peel a +trailing run of period-marked title words chains into the title from +the end, leaving one name piece standing; where the run TAKES +something the first peel was only provisional and runs again, once, +over the pieces with the titled ones spliced out, so a trailing +title is transparent to it. Where the run takes nothing -- almost +every name -- the first peel is the only one and its answer stands. The v1 single-name+nickname rule lives here (decisions.md#N3): a nonempty nickname beside exactly one piece in total puts that piece in FAMILY. @@ -262,12 +264,14 @@ def _assign_main(seg_idx: int, state: ParseState, # peeled" means depends on name_order. (The roman-numeral fork # needs no such deferral and is reported here.) # - # This first peel is PROVISIONAL: all it settles is where the H5 - # walk below starts. Its roles and its reports are never used -- - # the walk can remove the very word that stopped it, so when the - # walk takes anything the peel is asked again over the pieces as - # they then stand, and that second answer is the only one that - # places a piece or reports a fork. + # This peel is provisional only where the H5 walk below TAKES + # something: the walk can remove the very word that stopped the + # peel, so it is asked again over the pieces as they then stand, + # and that second answer is the only one that places a piece or + # reports a fork. Where the walk takes nothing -- almost every + # name, the walk's own first test being a period match that + # fails -- this peel is the only one, and its roles and its + # reports are the ones the name gets. peeled = peel_trailing(rest, pieces, ptags, tokens) # rules.md#H5: "successive single words that wear the abbreviation # shape and are title vocabulary chain into the title from the end, @@ -564,6 +568,14 @@ def reads_as_a_suffix(m: int, last: int) -> bool: """ if is_suffix_piece(pieces[m], ptags[m], tokens): return True + # DEFENSIVE, and measured inert: the skip fires on 153 + # of 140,227 calls over 191,146 generated inputs, and + # deleting it changes no parse among them (2026-09-09). + # Kept because "as if the titled pieces were absent" is + # the rule this predicate implements, and a caller's + # vocabulary reaches shapes the sweep's word list does + # not -- an inert branch is cheaper than a rule with a + # hole in it. prev = m - 1 while prev in titled_idx: prev -= 1 @@ -627,7 +639,13 @@ def reads_as_a_suffix(m: int, last: int) -> bool: # is why the filter is the walk's own predicate and # not the strict suffix test alone. Piece `n` is # always the given below, whatever that predicate - # would say of it, so it is always a candidate. + # would say of it, so it is always a candidate. That + # `k == n` is LOAD-BEARING, not defensive: it is what + # the walk's floor stands on when the given piece + # itself reads as a suffix, and dropping it leaves + # 'Smith, II Mr. V' a middle 'Mr.' where the title is + # (24 inputs of that shape move, of 191,146 generated, + # measured 2026-09-09). walkable = [k for k in range(n, len(pieces)) if k == n or not reads_as_a_suffix( k, len(pieces) - 1)] diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 1e1bdbce..919b259e 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -46,6 +46,7 @@ from nameparser._pipeline._pieces import ( is_leading_title, is_suffix_piece, is_title_piece, leading_titles, peel_trailing, peel_walk, trailing_start, + trailing_titles, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, @@ -712,11 +713,18 @@ def chain(tail: int) -> None: else: # rules.md#P5: "the join is tried on the pieces as it # would leave them, assign's trailing peel (S2) is read - # over that, and the name words it leaves are the words - # to spare" (history: decisions.md#P5). The view is what + # over that and its trailing title run (H5) over what + # that peel leaves, and the name words the two of them + # leave are the words to spare" + # (history: decisions.md#P5). The view is what # merge() builds -- the same slice assignment, the same - # joined_tags -- and the peel is assign's own, so the - # reserve and the assignment cannot drift. And the join + # joined_tags -- and the peel is assign's own, and so is + # the H5 walk read over what that peel leaves, so the + # reserve and the assignment cannot drift. Both halves + # are needed: the peel alone counted a trailing + # period-marked title word as a name word to spare, and + # 'Prof. abdul rahman Prof.' joined where + # 'Prof. abdul rahman' does not. And the join # changes no suffix reading -- rules.md#P5: "a word the # peel reads as a suffix unjoined must read so joined, # or the join declines" -- compared as the peeled @@ -732,9 +740,21 @@ def chain(tail: int) -> None: drop={"title"})] view_rest = peel_walk(fk, view_tags) after = peel_trailing(view_rest, view, view_tags, tokens) + # rules.md#H5 -- assign's second peel runs over the + # pieces the trailing title walk LEFT, so a + # period-marked title word at the back is not one of the + # name words this counts. Read over the same list assign + # reads it over -- the name pieces the peel left -- on + # both sides, so the two views compare like with like: + # counting the title word made 'Sir abdul Prof.' join it + # into the given name. + before_names = before.names - trailing_titles( + rest[:before.names], pieces, ptags, tokens) + after_names = after.names - trailing_titles( + view_rest[:after.names], view, view_tags, tokens) same_suffixes = ( - [tuple(view[j]) for j in view_rest[after.names:]] - == [tuple(pieces[j]) for j in rest[before.names:]]) + [tuple(view[j]) for j in view_rest[after_names:]] + == [tuple(pieces[j]) for j in rest[before_names:]]) # A given-name title ahead of the bound word asserts # that a given name follows -- the assertion H1 reads # when it keeps "Sir John" a given name -- so behind @@ -745,10 +765,12 @@ def chain(tail: int) -> None: # or that key's LAST word (#489). H2's unlisted # abbreviations ride in the run either way. One is in # no vocabulary by definition, so it never matches as - # the last-word key -- 'Xyz. Sir' keys 'sir' and 'Sir - # Xyz.' keys 'xyz' -- but a caller's phrase entry may - # contain one, and the whole-run arm is what matches - # that. The licence lifts the reserve for two name + # the last-word key -- written as inputs, 'Xyz. Sir + # John' keys 'xyz sir' and matches on 'sir', while + # 'Sir Xyz. John' keys 'sir xyz' and matches on + # neither -- but a caller's phrase entry may contain + # one, and the whole-run arm is what matches that. + # The licence lifts the reserve for two name # WORDS: the piece the join would take must be one word # -- a particle chain is the family name P2 built ('Sir # abdul van der Berg' keeps family 'van der Berg'). @@ -759,7 +781,7 @@ def chain(tail: int) -> None: for i in pieces[k]), given_name_titles)) reserve = BoundJoin.LENIENT if licensed else BoundJoin.STRICT - if same_suffixes and after.names >= reserve: + if same_suffixes and after_names >= reserve: # the pair is a given name whatever tag the word # carried (rules.md#P5); joined_tags says why the # title tag is dropped. Pinned in test_group.py. diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index ba5086d6..325da96c 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -378,7 +378,11 @@ def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken]) -> int: """How many pieces at the END of `rest` are period-marked title - words. `rest` is the NAME pieces the S2 peel left, in piece order. + words. `rest` is the caller's NAME pieces, in piece order: on the + no-comma path what the S2 peel left, after a family comma the + segment's pieces that the segment's own suffix reading does not + claim, and in group's bound-given reserve the peel's leftovers + over the view the join would build. Floor: one name piece stands, so a name is never all title -- and an empty `rest` returns 0, which is what leaves assign's bare-suffix carve-out reached exactly as before. @@ -387,10 +391,15 @@ def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], uses: a joined unit is not the shape this reads, and the tokens of one are not each a title word. - Every parse enters this frame -- assign asks the question here - rather than answering a cheaper version of it inline - (mechanisms.md#ONE-PREDICATE-PER-QUESTION) -- so what it costs an - ordinary name is one frame and one regex match. The shape test + Every parse with a name word to place enters this frame -- assign + asks the question here rather than answering a cheaper version of + it inline (mechanisms.md#ONE-PREDICATE-PER-QUESTION) -- so what it + costs an ordinary name is one frame and one regex match. The + exceptions return before it: a segment that is all title, and a + comma part read wholly as a credential run, have no name piece to + hand this (52 of the 1289 corpus parses, measured 2026-09-09 -- + 'Coach', 'Lord of the Universe', 'Smith, Jr.', 'MD, PHD'). + The shape test runs BEFORE the vocabulary one to keep it at that: _PERIOD_ABBREV is a compiled regex (a C call, no Python frame) where is_title_piece is a call, and almost no name ends in a diff --git a/tests/v2/cases.py b/tests/v2/cases.py index adc4aefe..8f3210b1 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -376,6 +376,22 @@ def _check_cjk_shape_purity(self) -> None: "classification is a slug and not an issue number " "because no issue asked for it; the bundle that " "carried it is #489/#316"), + Case("suffix_acronym_multidot_after_a_family_comma", + "Smith, E.S.Q.", + {"given": "E.S.Q.", "family": "Smith"}, + classification="parity", + notes="the other path the same removal moves, and the one " + "that RESTORES v1: with 'esq' in SUFFIX_ACRONYMS the " + "multi-dot spelling was a suffix piece, so the " + "post-comma segment held no name word and read suffix " + "'E.S.Q.' (2.0.0 through 2.2.0). Out of the set, it is " + "an ordinary name word and the walk's first non-title " + "piece is ALWAYS the given -- which is what 1.4.0 read " + "here, first 'E.S.Q.' / last 'Smith' (measured " + "2026-09-09), so this row is parity where its no-comma " + "sibling above is a parity BREAK. Same entry, opposite " + "directions, because v1 read the two paths " + "differently"), Case("suffix_word_esq_still_reads_as_a_suffix", "John Smith Esq", {"given": "John", "family": "Smith", "suffix": "Esq"}, notes="the other half of the row above, and what the removal " @@ -3414,9 +3430,15 @@ def _check_cjk_shape_purity(self) -> None: "inert-measurement shape. Since #316 the word it " "resets ON is a title here rather than a middle name: " "'I' is what this segment reads as its suffix, so " - "'Dr.' is the trailing piece and the walk takes it, " - "and 'Smith, PSM Dr. I' is 'Smith, PSM I' plus a " - "title. 1.4.0 read suffix 'Dr., I' -- 'dr' was still " + "'Dr.' is the trailing piece and the walk takes it. " + "Transparency does NOT reach this row, and that is the " + "reset itself: 'Smith, PSM I' reads suffix 'PSM I' " + "with no given name at all (measured 2026-09-09), " + "because with no title between them the numeral " + "CONTINUES the credential run. Removing the title " + "removes the reset, so the shorter spelling is a " + "different reading and not this one minus a word. " + "1.4.0 read suffix 'Dr., I' -- 'dr' was still " "postnominal vocabulary before #296's audit, so the " "row's old parity claim had outlived it"), Case("family_comma_run_numeral_after_a_split_credential", @@ -3769,6 +3791,98 @@ def _check_cjk_shape_purity(self) -> None: "John MA'), so in 'Smith, John Prof. MA' a name word " "stands behind 'Prof.' and no title is in trailing " "position at all"), + Case("cjk_trailing_latin_title_keeps_the_script_order", + "毛 泽东 Dr.", + {"title": "Dr.", "given": "泽东", "family": "毛"}, + classification="fix(#316)", + notes="why the walk runs BEFORE the positional read: a Latin " + "title at the back of a wholly-Han name is the one " + "piece that would make the piece set look " + "mixed-script, and a mixed set declines the script " + "order. Taken first, the pieces the script test sees " + "are all Han and the Han order stands -- this is " + "'毛 泽东' plus a title. 1.4.0 read first '毛' / last " + "'泽东' / suffix 'Dr.' and master family 'Dr.' " + "(measured 2026-09-09)", + tolerated=True), + Case("title_word_trailing_behind_a_bound_given_pair", + "Prof. abdul rahman Prof.", + {"title": "Prof. Prof.", "given": "abdul", "family": "rahman"}, + classification="fix(#316)", + notes="P5's reserve counts the name words assign will leave, " + "and a trailing title word is not one of them: this is " + "'Prof. abdul rahman' plus a title, which reads given " + "'abdul' / family 'rahman' because two name words " + "alone do not join. Counting the title word as a word " + "to spare joined the pair and read family 'abdul " + "rahman' (measured on the bundle's third commit). " + "1.4.0 read title 'Prof.' / first 'abdul rahman' / " + "last 'Prof.'"), + Case("title_word_trailing_behind_a_licensed_bound_pair", + "Sir abdul rahman Prof.", + {"title": "Sir Prof.", "family": "abdul rahman"}, + classification="fix(#316)", + notes="the licensed half of the row above, and where the " + "trailing title is NOT transparent: the join fires " + "('sir' asserts a given name follows), and H1 then " + "reads the TITLE ROLE -- both ends of the name -- as " + "one run keyed 'sir prof', which does not address by " + "given name, so the pair becomes the family. 'Sir " + "abdul rahman' alone reads given 'abdul rahman'. The " + "same movement 'Sir John Prof.' shows against 'Sir " + "John', so it is H1's run composition and not P5's " + "reserve; recorded here rather than changed " + "(measured 2026-09-09). 1.4.0 read title 'Sir' / " + "first 'abdul rahman' / last 'Prof.'"), + Case("title_word_trailing_after_a_maiden_take", + "Mary Smith née Jones Prof.", + {"given": "Mary", "family": "Smith", "maiden": "Jones Prof."}, + classification="fix(#274)", + notes="negative control for the trailing walk, and rules.md#" + "H5's M2 boundary: M2's take runs to the name's end " + "and the title is inside what it takes, so no title " + "word is in trailing position at all. Unchanged by " + "#316/#489 -- master reads the same (measured " + "2026-09-09). 1.4.0 had no maiden support and read " + "first 'Mary' / middle 'Smith née Jones' / last " + "'Prof.'"), + Case("title_word_trailing_ahead_of_a_maiden_marker", + "Mary Jones Prof. née Smith", + {"title": "Prof.", "given": "Mary", "family": "Jones", + "maiden": "Smith"}, + classification="fix(#316)", + notes="the mirror: with the marker BEHIND it the title is " + "the last piece the walk sees, so it is taken and " + "'Mary Jones née Smith' is what is left. Master read " + "middle 'Jones' / family 'Prof.'; 1.4.0 read first " + "'Mary' / middle 'Jones Prof. née' / last 'Smith' " + "(measured 2026-09-09)"), + Case("title_word_trailing_in_a_conjunction_unit", + "John Smith Prof. and Dr.", + {"given": "John", "middle": "Smith", + "family": "Prof. and Dr."}, + classification="parity", + notes="negative control for the ONE-WORD-per-piece gate: the " + "conjunction merge made 'Prof. and Dr.' one piece, and " + "the tokens of a joined unit are not each a title " + "word. The row that PINS that gate -- with it deleted " + "the walk takes the unit, because the shape test then " + "runs on the piece's first token and 'Prof.' wears the " + "period ('John de la Prof.' reads 0 either way; both " + "measured 2026-09-09 in test_pieces.py)"), + Case("family_comma_then_a_lone_suffix_word_segment", + "Smith, John, Prof.", + {"given": "John", "family": "Smith", "suffix": "Prof."}, + ambiguities=("comma-structure",), + classification="parity", + notes="the trailing slot is a segment away: a second comma " + "makes the last part its own segment, which the tail " + "consumes as a suffix before any trailing walk reads a " + "piece -- so 'prof' leaving the suffix vocabulary " + "(#296) does not reach this shape and 'Smith, John, " + "Prof.' still reads suffix 'Prof.' where 'Smith, John " + "Prof.' reads title. Unmoved by this bundle and by " + "1.4.0 alike (measured 2026-09-09)"), # -- #271: script-scoped order + segmentation (amendment 2026-07-27) Case("ko_unspaced_default", "김민준", diff --git a/tests/v2/pipeline/test_assign.py b/tests/v2/pipeline/test_assign.py index ce73a9ca..13b03773 100644 --- a/tests/v2/pipeline/test_assign.py +++ b/tests/v2/pipeline/test_assign.py @@ -317,6 +317,26 @@ def test_trailing_title_run_after_a_family_comma() -> None: assert not _by_role(out, Role.MIDDLE) +def test_the_trailing_title_is_taken_before_the_script_order_resolves( +) -> None: + """The other half of "set BEFORE _name_positions". + + The sibling above pins that the walk shortens the piece list in + time for the POSITIONAL read. This pins it for the SCRIPT read, + which is the reason the placement was chosen: a Latin title at + the back of a wholly-Han name is the one piece that would make + the piece set look mixed-script, and a mixed set declines the + script order. Taken first, the pieces the script test sees are + all Han and the Han order stands -- family '毛', given '泽东', + which is not what the default order would have given. + """ + out = _assigned("毛 泽东 Dr.") + assert _by_role(out, Role.TITLE) == "Dr." + assert _by_role(out, Role.FAMILY) == "毛" + assert _by_role(out, Role.GIVEN) == "泽东" + assert out.order != Policy().name_order + + def test_initial_veto_keeps_v_in_middle() -> None: out = _assigned("John V. Smith") assert _by_role(out, Role.MIDDLE) == "V." diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index 4a83b2a3..a4db9ed7 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -214,5 +214,14 @@ def test_the_trailing_run_refuses_a_joined_piece() -> None: 'de la Prof.' is one piece of three tokens, and the tokens of a joined unit are not each a title word -- the particle chain made that unit a name. + + That first row does not PIN the gate, though: with the one-word + test deleted it still reads 0, because the shape test then runs + on the piece's first token and 'de' wears no period (measured + 2026-09-09). The conjunction-merged unit is the row that pins it + -- 'Prof. and Dr.' is one piece whose first token is a + period-marked title word, so without the gate the walk takes it + and the name loses its family (measured 1 under that mutation). """ assert _trailing("John de la Prof.") == 0 + assert _trailing("John Smith Prof. and Dr.") == 0 diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 07ff1dd3..870d7edf 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -2243,9 +2243,9 @@ def _claim(rule: dict) -> _Claim: "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": _Claim(1, ('middle', 'suffix'), "e9f282da0d0f", None), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(108, ('family', 'given', 'middle'), "9a814f70c2dc", None), + _Claim(109, ('family', 'given', 'middle'), "864f9cffa977", None), "fix(#274) maiden markers consumed": - _Claim(32, ('family', 'maiden', 'middle'), "06d199ceb249", None), + _Claim(33, ('family', 'maiden', 'middle'), "6f8bf7136b09", None), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": _Claim(5, ('family', 'given', 'maiden', 'middle'), "bc0e10dd7ec8", None), "fix(#379) a tussenvoegsel after a family comma attaches to the family": @@ -2296,7 +2296,7 @@ def _claim(rule: dict) -> _Claim: # rule at the end of the ledger, and the comment there # records the handover. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(12, ('family', 'suffix'), "c3446b32e8bd", None), + _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": @@ -2452,15 +2452,15 @@ def _claim(rule: dict) -> _Claim: "fix(initials-per-word) a bound-given run initials each word (facade, since 2.0.0)": _Claim(41, ('_initials',), "e99f56c955d5", ('DEFAULT',)), "fix(initials-per-word) a particle chain inside a name part initials each word (facade, since 2.0.0)": - _Claim(108, ('_initials',), "45f0b2c1a7d4", ('DEFAULT',)), + _Claim(109, ('_initials',), "ae9c8f674e0c", ('DEFAULT',)), "fix(initials-per-word) the Ph. D. merge initials each word (facade, since 2.0.0)": _Claim(18, ('_initials',), "f67d8ebddd56", ('DEFAULT',)), # The 2.3 title-run bundle's four rules, last in every # ledger. All four are anchored alternations of NAMES, so the # reach IS the mover list and the four numbers are the four # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 12 for the - # trailing title. Nineteen in all, and every one of them is + # 1 for the esq drop, 4 for the peel floor, 14 for the + # trailing title. Twenty-one in all, and every one of them is # explained by the rule that names it -- these are the rare # rows where reach and explanation coincide, which is what an # anchored name list buys. A widening past those names moves @@ -2577,7 +2577,7 @@ def _claim(rule: dict) -> _Claim: "fix(#379) a tussenvoegsel after a family comma attaches to the family": _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda", None), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(108, ('_ambiguities', 'family', 'given', 'middle'), "9a814f70c2dc", None), + _Claim(109, ('_ambiguities', 'family', 'given', 'middle'), "864f9cffa977", None), # 37 -> 35 with the same 2026-09-05 narrowing as the 1.4 twin, # whose entry carries the reason. Here the one name that # changed hands, '김민준 박사님', goes to the spaced rule @@ -2649,7 +2649,7 @@ def _claim(rule: dict) -> _Claim: # rule at the end of the ledger, and the comment there # records the handover. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(12, ('family', 'suffix'), "c3446b32e8bd", None), + _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": @@ -2721,8 +2721,8 @@ def _claim(rule: dict) -> _Claim: # ledger. All four are anchored alternations of NAMES, so the # reach IS the mover list and the four numbers are the four # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 12 for the - # trailing title. Nineteen in all, and every one of them is + # 1 for the esq drop, 4 for the peel floor, 14 for the + # trailing title. Twenty-one in all, and every one of them is # explained by the rule that names it -- these are the rare # rows where reach and explanation coincide, which is what an # anchored name list buys. A widening past those names moves @@ -2830,8 +2830,8 @@ def _claim(rule: dict) -> _Claim: # ledger. All four are anchored alternations of NAMES, so the # reach IS the mover list and the four numbers are the four # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 12 for the - # trailing title. Nineteen in all, and every one of them is + # 1 for the esq drop, 4 for the peel floor, 14 for the + # trailing title. Twenty-one in all, and every one of them is # explained by the rule that names it -- these are the rare # rows where reach and explanation coincide, which is what an # anchored name list buys. A widening past those names moves @@ -3010,7 +3010,7 @@ def _claim(rule: dict) -> _Claim: # rule at the end of the ledger, and the comment there # records the handover. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(12, ('family', 'suffix'), "c3446b32e8bd", None), + _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": @@ -3074,8 +3074,8 @@ def _claim(rule: dict) -> _Claim: # ledger. All four are anchored alternations of NAMES, so the # reach IS the mover list and the four numbers are the four # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 12 for the - # trailing title. Nineteen in all, and every one of them is + # 1 for the esq drop, 4 for the peel floor, 14 for the + # trailing title. Twenty-one in all, and every one of them is # explained by the rule that names it -- these are the rare # rows where reach and explanation coincide, which is what an # anchored name list buys. A widening past those names moves diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 32d39446..70f7d2d2 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -682,6 +682,19 @@ def test_the_p5_licence_and_h1_read_a_title_run_the_same_way( # agree, run by run. assert (parse(f"{title} John").family == "") == \ (parse(f"{title} abdul rahman").family == "") + # The same invariant with a trailing title behind the pair. P5's + # reserve counts the name words assign's peel AND the H5 walk + # leave, so a period-marked title word at the back is not one of + # them; H1 reads the TITLE role, which by then holds both ends of + # the name. Both sides must still agree about "no family" -- and + # they do run by run, though the two halves reach it differently: + # behind a given-name title the join fires and H1 then hands the + # pair to the family (the run keys 'sir prof'), while behind an + # ordinary one the reserve declines the join and the two words + # split. Either way a family stands, as it does for the one-word + # spelling. + assert (parse(f"{title} John Prof.").family == "") == \ + (parse(f"{title} abdul rahman Prof.").family == "") # The first three reach the chain loop and decline inside it: the piece @@ -695,21 +708,30 @@ def test_the_p5_licence_and_h1_read_a_title_run_the_same_way( # different reasons, one output, and neither may start reporting a fork. @pytest.mark.parametrize("text", [ "Do Van Jr.", "Do Van MD", "St Van Jr.", - "Dr. Van Jr.", "Dr. Van MD", "Dr. Do Jr.", + "Dr. Van Jr.", "Dr. Van MD", ]) def test_no_op_prefix_chain_is_not_a_fork(text: str) -> None: - # Five rows stay fully silent. The sixth is pinned to its exact - # report instead, because since #489 it reports from somewhere else - # entirely: the leading peel's floor gives 'Do' back as the name - # word (the run stood in front of nothing but 'Jr.'), so - # 'Dr. Do Jr.' reads family 'Do', suffix 'Jr.' and H4's title half - # claims the lone name word, which happens to be title vocabulary - # here. That report is about the word left standing, not about a - # chain that never chained -- PARTICLE_OR_GIVEN is still absent. - expected = ((AmbiguityKind.TITLE_OR_NAME,) - if text == "Dr. Do Jr." else ()) - assert tuple(a.kind for a in - _overlap_parser().parse(text).ambiguities) == expected + assert not _overlap_parser().parse(text).ambiguities + + +def test_dr_do_jr_reports_the_word_left_standing() -> None: + """Since #489 this one reports from somewhere else entirely. + + It was the sixth row of the parametrization above until the + 2026-09-09 review, where carrying it made that test assert "no + fork EXCEPT this one" and so stopped saying the thing it exists + to say. The leading peel's floor gives 'Do' back as the name word + (the run stood in front of nothing but 'Jr.'), so 'Dr. Do Jr.' + reads family 'Do', suffix 'Jr.' and H4's title half claims the + lone name word, which happens to be title vocabulary here. That + report is about the word left standing, not about a chain that + never chained -- PARTICLE_OR_GIVEN is still absent, which is what + the kind tuple below pins. + """ + n = _overlap_parser().parse("Dr. Do Jr.") + assert (n.family, n.suffix) == ("Do", "Jr.") + assert tuple(a.kind for a in n.ambiguities) == ( + AmbiguityKind.TITLE_OR_NAME,) def test_a_fork_is_reported_by_exactly_one_stage() -> None: diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 691f640a..01910485 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -1949,11 +1949,11 @@ class _ShapeMismatch(NamedTuple): #: named nowhere under tests/ outside test_ledger_guards.py and #: entered it, so the scan went 52 -> 53 and the roster 50 -> 51. #: Recounted 2026-09-08 with the title-run bundle: one name entered the -#: population, 'John Smith Rev.', the only one of that bundle's nineteen -#: movers no test literal names, and it took a row at each of the four -#: baselines. It is named NOWHERE under tests/, so it counts in both -#: scans -- the every-file figures in the RECOMPUTE paragraph below -#: rise by one each too. That commit moved the ROW counts and nothing +#: population, 'John Smith Rev.', the only one of that bundle's +#: twenty-one movers no test literal names, and it took a row at each +#: of the four baselines. It is named NOWHERE under tests/, so it +#: counts in both scans -- the every-file figures in the RECOMPUTE +#: paragraph below count it too. That commit moved the ROW counts and nothing #: else: it did not re-derive the population clause above, so the #: equality sentence that follows is dated 2026-09-07 and is not #: restated for today. @@ -1962,7 +1962,7 @@ class _ShapeMismatch(NamedTuple): #: five contest rows beyond it having gone to _RECORDED_DIFFS with #501 #: and five more with #498, which left the population by gaining a #: _RECORDED_DIFFS key rather than by ceasing to be watched anywhere. -#: 48 of the 50 sit in corpus_issues.jsonl and 3 in corpus.jsonl, with +#: 50 of the 52 sit in corpus_issues.jsonl and 3 in corpus.jsonl, with #: 'dr Vincent van Gogh dr' in both, so the per-file counts overlap by #: one and are not a partition. Every row is a default-order shape, #: as the roster above's are, so no row here is a declared-order-only @@ -1973,17 +1973,27 @@ class _ShapeMismatch(NamedTuple): #: calls whose order is None and whose rule is not None; apply the #: four clauses above with the literal set from ast.walk over #: tests/**/*.py EXCLUDING test_ledger_guards.py, as the POPULATION -#: clause says -- run over every file it yields 33 / 23 / 22 / 4 rows -#: rather than 37 / 33 / 32 / 7, since _CROSS_RULE_WINNERS' keys and +#: clause says -- run over every file it yields 25 / 23 / 22 / 5 rows +#: rather than 38 / 34 / 33 / 8, since _CROSS_RULE_WINNERS' keys and #: a few guard literals then score as watchers, and #498's fourteen #: keys are exactly that kind of literal -- as are #342's two -#: 2026-09-07 arrivals, both named in _NOT_A_VOCABULARY_COPY, which is -#: why the every-file figures stood still while eight rows landed and -#: one left: the two halves of this -#: sentence moved for different reasons on 2026-09-05, the second -#: because five rows left this dict and the first because those five -#: are watched at 2.x too, where they now score as watched by the -#: guard file -- the tier sets from +#: 2026-09-07 arrivals, both named in _NOT_A_VOCABULARY_COPY. The +#: every-file figures are the strict ones MINUS the roster names that +#: are named as an exact string literal in test_ledger_guards.py and +#: nowhere else under tests/ (13 / 11 / 11 / 3 today), which is a +#: derivation a reader can run in one pass over this dict and that +#: file -- no baseline wheel needed -- and it is how the pair was +#: recomputed on 2026-09-09. That recount RETRACTS the pair recorded +#: on 2026-09-05 (33 / 23 / 22 / 4): the every-file count can never +#: EXCEED the strict one, dropping a name from the population removes +#: rows and never adds them, and 33 stood against a strict 37. The +#: paragraph that carried it argued the every-file figures "stood +#: still" while eight guard-literal rows landed -- which is right +#: about the mechanism and is exactly why the gap between the two +#: pairs widens as #498's and #342's keys arrive, so the two cannot +#: both have stood at 33 and 37. The strict pair is unchanged in +#: method and re-measured here: it is len() over this dict, ledger by +#: ledger -- the tier sets from #: _load_entries over corpus*.jsonl #: through _CORPUS_TIERS, and that ledger's _RECORDED_DIFFS keys. Not #: by replaying the corpus load by hand: the (name, order) dedup, the diff --git a/tools/differential/corpus_cjk_tolerated.jsonl b/tools/differential/corpus_cjk_tolerated.jsonl index f3ef161d..bfa80e04 100644 --- a/tools/differential/corpus_cjk_tolerated.jsonl +++ b/tools/differential/corpus_cjk_tolerated.jsonl @@ -2,6 +2,7 @@ "Dr 김민준씨, Jr." "Dr 김민준씨, V." "威廉·莎士比亚, PhD" +"毛 泽东 Dr." "王先生, V." "田中, 太郎さん" "田中さん II" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b2231511..be9a40b8 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -115,6 +115,7 @@ "John van der Berg" "John van der Berg Ma" "John van der Berg PhD" +"John van der Berg Prof." "John van der Berg Smith" "John van der Berg V" "John van der Berg née Jones" @@ -147,6 +148,7 @@ "Mary Beth Smith" "Mary Jane King" "Mary Jane King." +"Mary Smith née Jones Prof." "Mc Donald" "Mesnil de" "Morse, Det. Insp. Jane" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 887b97f7..19d9d04a 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -2906,11 +2906,11 @@ orders = ["DEFAULT"] # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym # drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over nineteen corpus names. +# than one per name, over twenty-one corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# nineteen declares `fields` that are a strict subset of the bundle +# twenty-one declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -2924,10 +2924,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the nineteen do move. The derived view +# initials of several of the twenty-one do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all nineteen move roles -- so no run can produce a +# guard), and all twenty-one move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -3045,11 +3045,20 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # family 'MA' by rules.md#S2's reserve. # # Five roles, the union of what the fourteen move; no name moves all -# five, and three move only two -- 'Smith, John Prof.' {title, -# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' -# moving three. `_ambiguities` is not among the five and none of the -# fourteen reports one: the walk leaves a title standing, so H4's -# title half never fires. +# five (four is the most, 'John Smith Jr. Prof.'), and two move only +# two at every baseline -- 'Smith, John Prof.' {title, middle} and +# 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. +# A third joins them at 1.4.0, 2.0.0 and 2.1.0, 'John Smith Dr.' +# {title, suffix}, which moves {title, middle, family} at 2.2.0 +# instead: 'dr' was suffix vocabulary until 2.2, so the role it +# vacates differs by baseline (measured 2026-09-09 against all four +# wheels). `_ambiguities` is not among the five, and it is not +# because no name reports one: 'John Prof. MA' reports +# `suffix-or-name`, which is S2's bare-acronym reserve and not this +# rule. None of the fourteen reports `title-or-name` -- the walk +# leaves a title standing, so H4's title half never fires -- and the +# derived view cannot enter a diff here in any case, every one of the +# fourteen moving roles (compare.py's _RULE_FIELDS). # # TWO of the fourteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which @@ -3093,7 +3102,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a twelve-member alternation should not +# excludes '(' and ')' -- and a fourteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 14bf8713..547e46e6 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -1772,11 +1772,11 @@ orders = ["DEFAULT"] # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym # drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over nineteen corpus names. +# than one per name, over twenty-one corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# nineteen declares `fields` that are a strict subset of the bundle +# twenty-one declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -1790,10 +1790,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the nineteen do move. The derived view +# initials of several of the twenty-one do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all nineteen move roles -- so no run can produce a +# guard), and all twenty-one move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -1911,11 +1911,20 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # family 'MA' by rules.md#S2's reserve. # # Five roles, the union of what the fourteen move; no name moves all -# five, and three move only two -- 'Smith, John Prof.' {title, -# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' -# moving three. `_ambiguities` is not among the five and none of the -# fourteen reports one: the walk leaves a title standing, so H4's -# title half never fires. +# five (four is the most, 'John Smith Jr. Prof.'), and two move only +# two at every baseline -- 'Smith, John Prof.' {title, middle} and +# 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. +# A third joins them at 1.4.0, 2.0.0 and 2.1.0, 'John Smith Dr.' +# {title, suffix}, which moves {title, middle, family} at 2.2.0 +# instead: 'dr' was suffix vocabulary until 2.2, so the role it +# vacates differs by baseline (measured 2026-09-09 against all four +# wheels). `_ambiguities` is not among the five, and it is not +# because no name reports one: 'John Prof. MA' reports +# `suffix-or-name`, which is S2's bare-acronym reserve and not this +# rule. None of the fourteen reports `title-or-name` -- the walk +# leaves a title standing, so H4's title half never fires -- and the +# derived view cannot enter a diff here in any case, every one of the +# fourteen moving roles (compare.py's _RULE_FIELDS). # # TWO of the fourteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which @@ -1959,7 +1968,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a twelve-member alternation should not +# excludes '(' and ')' -- and a fourteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index d8157b5f..3cbf5c2a 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -1691,11 +1691,11 @@ orders = ["DEFAULT"] # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym # drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over nineteen corpus names. +# than one per name, over twenty-one corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# nineteen declares `fields` that are a strict subset of the bundle +# twenty-one declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -1709,10 +1709,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the nineteen do move. The derived view +# initials of several of the twenty-one do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all nineteen move roles -- so no run can produce a +# guard), and all twenty-one move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -1830,11 +1830,20 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # family 'MA' by rules.md#S2's reserve. # # Five roles, the union of what the fourteen move; no name moves all -# five, and three move only two -- 'Smith, John Prof.' {title, -# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' -# moving three. `_ambiguities` is not among the five and none of the -# fourteen reports one: the walk leaves a title standing, so H4's -# title half never fires. +# five (four is the most, 'John Smith Jr. Prof.'), and two move only +# two at every baseline -- 'Smith, John Prof.' {title, middle} and +# 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. +# A third joins them at 1.4.0, 2.0.0 and 2.1.0, 'John Smith Dr.' +# {title, suffix}, which moves {title, middle, family} at 2.2.0 +# instead: 'dr' was suffix vocabulary until 2.2, so the role it +# vacates differs by baseline (measured 2026-09-09 against all four +# wheels). `_ambiguities` is not among the five, and it is not +# because no name reports one: 'John Prof. MA' reports +# `suffix-or-name`, which is S2's bare-acronym reserve and not this +# rule. None of the fourteen reports `title-or-name` -- the walk +# leaves a title standing, so H4's title half never fires -- and the +# derived view cannot enter a diff here in any case, every one of the +# fourteen moving roles (compare.py's _RULE_FIELDS). # # TWO of the fourteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which @@ -1878,7 +1887,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a twelve-member alternation should not +# excludes '(' and ')' -- and a fourteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. diff --git a/tools/differential/expected_since_2.2.0.toml b/tools/differential/expected_since_2.2.0.toml index 3a191fa6..a30970e3 100644 --- a/tools/differential/expected_since_2.2.0.toml +++ b/tools/differential/expected_since_2.2.0.toml @@ -332,11 +332,11 @@ orders = ["DEFAULT"] # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym # drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over nineteen corpus names. +# than one per name, over twenty-one corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# nineteen declares `fields` that are a strict subset of the bundle +# twenty-one declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -350,10 +350,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the nineteen do move. The derived view +# initials of several of the twenty-one do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all nineteen move roles -- so no run can produce a +# guard), and all twenty-one move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -471,11 +471,20 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # family 'MA' by rules.md#S2's reserve. # # Five roles, the union of what the fourteen move; no name moves all -# five, and three move only two -- 'Smith, John Prof.' {title, -# middle}, 'Smith Sir.' {title, family}, with 'Mary Jane King.' -# moving three. `_ambiguities` is not among the five and none of the -# fourteen reports one: the walk leaves a title standing, so H4's -# title half never fires. +# five (four is the most, 'John Smith Jr. Prof.'), and two move only +# two at every baseline -- 'Smith, John Prof.' {title, middle} and +# 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. +# A third joins them at 1.4.0, 2.0.0 and 2.1.0, 'John Smith Dr.' +# {title, suffix}, which moves {title, middle, family} at 2.2.0 +# instead: 'dr' was suffix vocabulary until 2.2, so the role it +# vacates differs by baseline (measured 2026-09-09 against all four +# wheels). `_ambiguities` is not among the five, and it is not +# because no name reports one: 'John Prof. MA' reports +# `suffix-or-name`, which is S2's bare-acronym reserve and not this +# rule. None of the fourteen reports `title-or-name` -- the walk +# leaves a title standing, so H4's title half never fires -- and the +# derived view cannot enter a diff here in any case, every one of the +# fourteen moving roles (compare.py's _RULE_FIELDS). # # TWO of the fourteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which @@ -519,7 +528,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a twelve-member alternation should not +# excludes '(' and ')' -- and a fourteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. From 7ac22184d669f1ff78539be0068c6511944b40df Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 15:38:19 -0700 Subject: [PATCH 08/12] review round 2: H1 addresses by the leading run; the tolerated CJK row classified MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit H1 collected every `Role.TITLE` token into ONE run, and with #316(a)'s trailing run shipped a name can carry titles at both ends, so the run behind the name decided the field of the word in front of it. `Sir John Prof.` keyed 'sir prof' -- addressed by neither word -- and read family 'John' where `Sir John` reads given 'John'; `Queen Elizabeth Prof.` lost its given name the same way, and `Dr. Smith Sir.` moved the other direction, the trailing 'sir' making the leading 'dr' address by given name. The rule is #H5's transparency principle applied to the one-word name: `X Prof.` is `X` plus a title, so adding the title cannot change how the words in FRONT of the name are read. The run standing before the one name word addresses; a run standing behind it decides only when none stands before, which is what keeps `Smith Sir.` reading given 'Smith'. Found by this PR's own review of P5: group's licence has always read only the pieces ahead of the bound word (`range(fk)`), so the 2026-08-22 #369 invariant -- H1 and P5 cannot read one run two ways -- had been broken by the composite keying, and `_group.py` now records that beside the licence. The split is at the NAME WORD. Splitting at the first token of another role -- the first shape written -- is defeated by a nickname in front of the titles: `'Smitty' Sir Jones Prof.` found no leading run and read family 'Jones', and `Dr. 'Smitty' Sir John` lost the run it had. H1's rationale decides it, a nickname beside the name word not deciding this reading, so a run written around one is one run. The design review's two-input invariant is what caught it. `Sir abdul rahman Prof.`'s case row flips to given 'abdul rahman': the P5 reserve fix was right AND this one was needed, the join firing for the right reason while the field stayed wrong. Rows added for `Sir John Prof.`, `Queen Elizabeth Prof.` and `Dr. Smith Sir.`; `Smith Sir.` and `Dr. King Sir.` (title 'Dr. King', given 'Sir.', title-or-name) are unmoved. The invariant test's trailing-title half now requires the titled spelling to give the same given and family as the spelling without it, over the nickname spellings too: restoring the whole-titles key fails six of its eleven rows and so does the first split, where comparing the two spellings' "no family" to each other passed under both. rules.md#H1 states the ordering and gains `"Sir John Prof."` and `"Dr. Smith Sir."`; #H5's one-name-word clause is conditioned the same way. Its P5 clause is scoped to the comma-less writing -- after a family comma the reserve reads no peel, so `Berg, abdul Prof.` takes the title into the given name, which is 1.4.0 parity and is recorded with a case row rather than an example, tracked as part of #316. `毛 泽东 Dr.` printed UNCLASSIFIED (radar) at all four gates and now has its own literal rule, `fields` differing per baseline -- it is the one bundle rule that cannot join an alternation, a script-classified member belonging to the honorific pin. The trailing-title alternation goes to sixteen, the bundle to twenty-four corpus names, and the four ledgers, the guard, the release log, decisions.md#H5's population (SEVENTEEN by its own recipe, the CJK name having been missing from it) and compare.py are re-measured. Twenty-four movers against a0b93f0; the only two this commit moves are the two rules.md examples it adds. All four gates read `unexplained: 0; radar unclassified: 0`. Frames unchanged at 416 parse / 453 facade on py3.11 -- the run is built inside H1's guard and the reference name has a family. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- docs/design/decisions.md | 10 +- docs/design/rules.md | 36 ++++- docs/release_log.rst | 2 +- nameparser/_pipeline/_group.py | 13 +- nameparser/_pipeline/_post_rules.py | 43 ++++- tests/v2/cases.py | 86 ++++++++-- tests/v2/test_ledger_guards.py | 157 +++++++++++++------ tests/v2/test_parser.py | 49 +++--- tools/differential/compare.py | 9 +- tools/differential/corpus_rules.jsonl | 2 + tools/differential/expected_since_1.4.0.toml | 72 +++++++-- tools/differential/expected_since_2.0.0.toml | 71 +++++++-- tools/differential/expected_since_2.1.0.toml | 72 +++++++-- tools/differential/expected_since_2.2.0.toml | 73 +++++++-- 14 files changed, 548 insertions(+), 147 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index c6aa64d1..1e2374f8 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -400,6 +400,11 @@ Declined (ambiguity kinds for script-resolved names, 2026-07-27): Frame delta measured at zero on both entry points: each site's read sits inside the branch it serves, and the reference name `Dr. Juan0000 de la Vega III` enters neither — H1's guard wants an unoccupied family, P5's a bound given-name word. TRAILING REACH, added 2026-09-09 in review of the docs commit. H1's site asks which ROLES are unoccupied and never where the title stands, so a run that #H5 chained in BEHIND the one name word decides that word's field exactly as a run in front of it does. Measured on this tree: `Smith Sir.` reads given `Smith` with an empty family and `Smith Queen.` reads given `Smith`, both runs ending in a given-name title, while `Smith Dr.` reads family `Smith`. No code moved for this — the reading has been the shipped one since the trailing walk landed — but the STATEMENT said "a title followed by exactly one name word" and described only the front slot, so what the review changed is the statement, an example line (`"Smith Sir." → given="Smith"`, which puts the name in corpus_rules.jsonl), and the `interacts:` lines, H1 gaining H3 and H5 and H5 gaining H1. The trailing half is where H1 and H5 meet: H5 decides which words leave the name, H1 decides the field of the one left standing. +- 2026-09-09 (the #316/#489 bundle, review round 2) — WHICH run addresses, where a name carries a title run at BOTH ends. The TRAILING REACH paragraph above is right about the trailing slot and was incomplete about the front one: H1 collected every `Role.TITLE` token into ONE run, and with #316(a) shipped that run can span the whole name. `Sir John Prof.` keyed `sir prof`, whose last word `prof` is no given-name title, so it read title `Sir Prof.`, family `John` where `Sir John` reads given `John` — the title written behind the name changing what the title in front of it addressed by. `Queen Elizabeth Prof.` lost its given name the same way, and `Dr. Smith Sir.` moved in the other direction, the trailing `sir` making the leading `dr` address by given name and turning family `Smith` into given `Smith`. + THE RULE: the run standing BEFORE the one name word is the run that addresses; a run standing behind it decides only when none stands before. The split is at the NAME WORD, and the first fix split at the first token of ANOTHER ROLE instead, which a nickname written in front of the titles satisfies: `'Smitty' Sir Jones Prof.` then found no leading run and read family `Jones` where `'Smitty' Sir Jones` reads given, and `Dr. 'Smitty' Sir John` lost the run it had — the same defect one nickname away, caught by the design review's two-input invariant. H1's own rationale is what decides it, saying in as many words that what stands beside the name word does not decide this reading; so a run written around a nickname is ONE run and `Dr. 'Smitty' Sir John` reads given `John` as `Dr. Sir John` does. That is #H5's transparency principle applied to the one-word name — `X Prof.` is `X` plus a title, so adding the title cannot change how the words in front of it are read — and it leaves the trailing slot's own reading untouched, `Smith Sir.` still reading given `Smith` because no run stands in front of it. rules.md#H1's statement carries the ordering and gains two examples, `"Sir John Prof." → given="John"` and `"Dr. Smith Sir." → family="Smith"`; #H5's one-name-word clause is conditioned the same way. + FOUND BY THE PR REVIEW'S P5 FINDING, and that is the part worth keeping. The reserve fix under #P5 was correct on its own terms and left `Sir abdul rahman Prof.` reading family `abdul rahman`: the join fired for the right reason and the field was still wrong. What the review then asked was which run each site reads, and the answer was that they read DIFFERENT runs — group's licence has always read only the pieces ahead of the bound word (`range(fk)`), while H1 read every title token — so the 2026-08-22 #369 invariant under #P5, that H1 and P5 cannot read one run two ways, had been broken by the composite keying without any site's own tests noticing. Both now read the leading run, through the same predicate, and `nameparser/_pipeline/_group.py` records that beside the licence. + MEASURED 2026-09-09, over every distinct name in `tools/differential/corpus*.jsonl`, parsed on the fix's parent (`e489dc1`) and on the fix and diffed across the seven fields and the ambiguity kinds: over the corpus AS THE PARENT HAD IT, not one name moves; over the corpus as this commit leaves it, exactly TWO do and they are the two rules.md examples this entry adds (`Sir John Prof.`, `Dr. Smith Sir.`). The composite shape needs a title run at BOTH ends and no corpus name had one before those two, which is why a defect this plain sat behind four green gates. `Dr. King Sir.` is untouched and stays #H4's: its leading run is `dr king`, whose last word `king` IS a given-name title, so the word the peel left standing reads given `Sir.` with `title-or-name` reported. The end-to-end invariant is `tests/v2/test_parser.py::test_the_p5_licence_and_h1_read_a_title_run_the_same_way`, whose trailing-title half now requires `f"{title} abdul rahman Prof."` and `f"{title} John Prof."` to give the same given and family as the spellings without the trailing title; restoring the whole-`titles` key fails six of its eleven rows, which is the mutation that proves it discriminates. It carries the nickname spellings as well, and those are the only rows that separate the two splits above. Frame delta zero on both entry points: the run is built inside H1's guard, after the role counts have short-circuited, and the reference name `Dr. Juan0000 de la Vega III` has a family and never enters it. + ### H2 — the leading-abbreviation title - 2026-06-30 (leading-period-title design; v2 core, PR #288) — the shape test is v1 parity (period_abbreviation): two-plus letters then a period, bare initials exempt. Its site is the head of the part CARRYING THE GIVEN NAME — the whole name, or the post-comma part under a family comma — not "the head of the name"; that scope correction is PR #315 (2026-08-01, docs-only), verified against 1.4.0 from PyPI, so the parity claim is real and the narrower description never was. The extraction litmus (2026-08-15): the spec drafted this rule as @@ -440,11 +445,12 @@ Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparse - **The predicate refuses `is_leading_title` and that refusal is the rule.** The walk uses `is_title_piece`, the vocabulary read, shared with the leading run so the two cannot disagree about what a title WORD is while disagreeing, deliberately, about what a title SHAPE is. `is_leading_title` carries H2's unlisted-abbreviation inference; with it, `John Smith Xyz.` would lose its family name to a title. That is the plan's mutation check for this commit. - **No fork is reported**, which is #316's open question 4. Under the input-is-a-name premise a period-marked title word is not a reading a reader would hesitate over; where the doubt is real it is the word left STANDING that carries it, and #H4 already reports that. No `AmbiguityKind` is added by this bundle and `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` is unchanged. - **A1, the peel order, decided by Derek: the first peel is PROVISIONAL and a trailing title is TRANSPARENT to the suffix reading.** `X Prof. Y` reads exactly as `X Y` reads, plus the title. SCOPED 2026-09-09, in review of the docs commit: that is a claim about inputs where a name word still stands on both sides of the chain — two or more name words. Where the chain leaves ONE name word there is no second reading for it to be transparent to, and #H1 decides the field instead: `Smith Prof.` reads family `Smith` where `Smith` alone reads given `Smith` and reports `given-or-family`, and `Smith Sir.` reads given `Smith` with an empty family, `sir` being a given-name title. All three measured on this tree; rules.md#H5's statement carries the scope and `_assign.py`'s excerpt of it was updated in the same edit. The problem is order: with ONE peel, `Prof.` standing behind `Jr.` stopped the suffix peel before `Jr.`, and the title walk then removed the very word that had been blocking it — `John Smith Jr. Prof.` read family `Jr.`, a generational suffix promoted to the family name, which is worse output than the reading it replaced. So the pieces the walk takes are spliced out and ONE peel runs over what stands, in original order, and that second answer alone places a piece or reports a fork. Measured: `John Smith Jr. Prof.` reads suffix `Jr.`; `John Prof. MA` reads the family `MA` that `John MA` reads, S2's reserve keeping a bare ambiguous acronym the family of a two-word name where a second peel laid over a first read family `John`, suffix `MA`; and `John Smith V Prof. VI` reads what `John Smith V VI` reads — middle `Smith V`, family `VI`, nothing reported — where two peels each reporting their own last piece reported twice. The choice is CORPUS-NEUTRAL: both variants move the same eight names, no more and no fewer, so it is a question about output quality alone. -- **A2, the family-comma segment-1 path, decided: it gets the walk.** The spec's condition was "if the comma segment gate already routes those, say so and leave it". Measured, it does not: `Smith, John Prof.` read middle `Prof.` at 1.4.0, 2.0.0, 2.1.0, 2.2.0 and at this branch's parent — all five measured — while `Smith, John Prof. Dr.` read middle `Prof.`, suffix `Dr.` through 2.1.0 and middle `Prof. Dr.` from 2.2.0, `dr` having left the suffix vocabulary in #296. The eleven comma rows the spec calls "already routed" are the `Smith, Prof.` shape, where segment 1 holds NO name word and `segment_suffix_reading` reads it piece by piece — a different gate and a different mechanism. So the segment-1 walk was genuinely missing; with it, `Smith, John Prof.` reads title `Prof.`, given `John`, family `Smith`. It moved NO corpus name at the fix; `Smith, John Prof.` is a rules.md#H5 example, so it ENTERS the corpus in the docs commit that follows and is one of the fourteen in the population bullet below. Case rows pin it either way. The segment's walk follows the same transparency principle with its own lenient loop, which is why the candidates are the pieces that walk would not read as a suffix rather than the strict suffix test alone (`Smith, John Prof. Jr.` must reach past the post-nominal, `Smith, John Prof. V` past the numeral the lenient tail test claims, #144). +- **A2, the family-comma segment-1 path, decided: it gets the walk.** The spec's condition was "if the comma segment gate already routes those, say so and leave it". Measured, it does not: `Smith, John Prof.` read middle `Prof.` at 1.4.0, 2.0.0, 2.1.0, 2.2.0 and at this branch's parent — all five measured — while `Smith, John Prof. Dr.` read middle `Prof.`, suffix `Dr.` through 2.1.0 and middle `Prof. Dr.` from 2.2.0, `dr` having left the suffix vocabulary in #296. The eleven comma rows the spec calls "already routed" are the `Smith, Prof.` shape, where segment 1 holds NO name word and `segment_suffix_reading` reads it piece by piece — a different gate and a different mechanism. So the segment-1 walk was genuinely missing; with it, `Smith, John Prof.` reads title `Prof.`, given `John`, family `Smith`. It moved NO corpus name at the fix; `Smith, John Prof.` is a rules.md#H5 example, so it ENTERS the corpus in the docs commit that follows and is one of the seventeen in the population bullet below. Case rows pin it either way. The segment's walk follows the same transparency principle with its own lenient loop, which is why the candidates are the pieces that walk would not read as a suffix rather than the strict suffix test alone (`Smith, John Prof. Jr.` must reach past the post-nominal, `Smith, John Prof. V` past the numeral the lenient tail test claims, #144). - **What A2 does NOT reach, and why that is right.** `Smith, John Prof. MA` still reads middle `Prof. MA`: after a family comma a bare ambiguous acronym is a MIDDLE name and has been since 2.0 (the `Smith, Ed` cost S2 already accepted), so `MA` stands at the end of the segment, is not a period-marked title word, and stops the walk before it starts. The walk reads from the end; it does not hunt. -- **Measured population: FIVE corpus names at the fix, FOURTEEN after the docs commit that follows it.** The five are the four planted no-comma rows in corpus_issues.jsonl (`John Smith Dr.`, `John Smith Mr.`, `John Smith Prof.`, `John Smith Rev.`) plus `Andrew Perkins (Mgr.)`, which the drafting did not expect. The other nine are rules.md examples entering corpus_rules.jsonl in the docs commit, eight of them this rule's own — `Smith Prof.`, `Dr. John Smith Prof.`, `John Smith Prof. Dr.`, `John Smith Prof. Jr.`, `John Smith Jr. Prof.`, `John Prof. MA`, `Smith, John Prof.` and `Mary Jane King.`, the last of those added by the 2026-09-09 review of that commit — plus `Smith Sir.`, which is #H1's example and reaches this walk because the run it puts behind the name word is what the walk takes. So the growth is documentation rather than reach, and both numbers come off the same recipe run on the two trees. Re-measured on the docs tree with `_pieces.trailing_titles` stubbed to return 0, which disables the walk at both sites. That fifth is a genuine hit rather than a misfire: rules.md#S1 drops the brackets and reads the content exactly as if written bare, so `(Mgr.)` is a trailing period-marked title word and reads as one; its own test stays green. Recompute by parsing every name in `tools/differential/corpus*.jsonl` on the tree and on the branch's third commit's parent and diffing the seven name fields plus `ambiguities`. Baselines differ per name — `John Smith Dr.` read suffix `Dr.` at 1.4.0, 2.0.0 and 2.1.0 (`dr` left the suffix vocabulary in 2.2, #296) and family `Dr.` at 2.2.0 — so the ledger entries are per-baseline. +- **Measured population: FIVE corpus names at the fix, SEVENTEEN after the docs commit that follows it and the two review rounds on it.** The five are the four planted no-comma rows in corpus_issues.jsonl (`John Smith Dr.`, `John Smith Mr.`, `John Smith Prof.`, `John Smith Rev.`) plus `Andrew Perkins (Mgr.)`, which the drafting did not expect. Eleven of the other twelve are rules.md examples entering corpus_rules.jsonl, eight of them this rule's own — `Smith Prof.`, `Dr. John Smith Prof.`, `John Smith Prof. Dr.`, `John Smith Prof. Jr.`, `John Smith Jr. Prof.`, `John Prof. MA`, `Smith, John Prof.` and `Mary Jane King.`, the last of those added by the first 2026-09-09 review of that commit — plus three of #H1's, which reach this walk because the run each puts behind the name word is what the walk takes: `Smith Sir.`, and `Sir John Prof.` and `Dr. Smith Sir.` from the second review round the same day. The twelfth is `毛 泽东 Dr.`, which entered corpus_cjk_tolerated.jsonl in the FIRST review round and is the one member of the population in a native script: the walk runs before the positional read, so the pieces the script test sees are all Han and the family-first order stands. It is a corpus name the walk moves like any other, and the reason it took its own ledger rule rather than a seventeenth alternative is a guard about alternations, not a difference in the argument (the ledger comment beside it says which). So the growth is documentation rather than reach, and both numbers come off the same recipe run on the two trees. Re-measured on the docs tree with `_pieces.trailing_titles` stubbed to return 0, which disables the walk at both sites. That fifth is a genuine hit rather than a misfire: rules.md#S1 drops the brackets and reads the content exactly as if written bare, so `(Mgr.)` is a trailing period-marked title word and reads as one; its own test stays green. Recompute by parsing every name in `tools/differential/corpus*.jsonl` on the tree and on the branch's third commit's parent and diffing the seven name fields plus `ambiguities`. Baselines differ per name — `John Smith Dr.` read suffix `Dr.` at 1.4.0, 2.0.0 and 2.1.0 (`dr` left the suffix vocabulary in 2.2, #296) and family `Dr.` at 2.2.0 — so the ledger entries are per-baseline. - **The inline frame-free gates at the two walk sites were REMOVED in review**, and this is the ONE-PREDICATE-PER-QUESTION half of the entry. Each site had a cheap inline test written to match the walk's own first condition; that is a second implementation of the question, and the measurement that justified it did not survive re-running. The band test runs early in a session where the facade sits at 453, so the "one frame of headroom" claim did not reproduce. With the gates gone the walk costs +1 frame on each entry point, inside the plan's target of two and inside `test_facade_cost_stays_within_its_band`. The walk's own cheapness is where the saving lives instead: the abbreviation-shape test is a compiled regex (a C call, no Python frame) and runs BEFORE the vocabulary call, and almost no name ends in a period-marked word, so the ordinary parse pays one match and stops. - **A pre-existing detail mismatch, recorded and NOT fixed.** #H4's join shape reports `title-or-name` with a `detail` that says the unit was "read as a given name by convention", while under the default order H1 retags the unit to the family — so `Dr. John of Prince` reports that text with the unit in `family`. This rule adds a second input with the same mismatch, `John of Prince Prof.`, and fixes neither: the wording predates this bundle, the fork the kind reports is title-versus-name which no field answers either way, and #H4 already records why the detail names no field for the peel shape. +- 2026-09-09 (review round 2) — **A1's transparency holds for the ONE-name-word case too, and the scope written a day earlier was the defect rather than the boundary.** A1 was scoped that morning to inputs where a name word still stands on both sides of the chain, handing the one-word case to #H1 unconditionally: "the title behind that word decides its field as a title in front of it would". #H1 was then reading BOTH ends of the name as one title run, so `Sir John Prof.` read family `John` where `Sir John` reads given `John` — the trailing title changing what the leading one addressed by, which is exactly the non-transparency A1 denies. The clause is now conditioned on there being no run in FRONT of the word: where one stands it addresses and the chained trailing title only joins the title field, so `X Prof.` reads as `X` plus the title for one name word as for two. Where no run stands in front the trailing one still decides, unchanged — `Smith Prof.` reads family `Smith`, `Smith Sir.` given `Smith`. Nothing in the walk moved; the fix is entirely in which run H1 keys, and #H1's 2026-09-09 entry carries it, the measurement and the mutation. The #P5 clause below reads the same way afterwards and gains its second half: a bound given-name word behind a trailing title joins as it joins with the title absent AND lands in the same field. Said of the COMMA-LESS writing, and the review that found the H1 defect measured why the qualifier is needed: after a family comma the reserve reads no peel at all, so the join fires over the trailing title word and takes it into the given name — `Berg, abdul Prof.` reads given `abdul Prof.` where `Berg, John Prof.` reads title `Prof.`. That is PARITY (1.4.0 reads it the same, measured) and a gap in the segment path rather than a boundary of the rule, so it is recorded rather than fixed here: the clause carries the scope, and the case row `title_word_trailing_behind_a_bound_pair_after_a_comma` pins it without making it normative. Tracked as part of #316. ### W1 — unspaced CJK division diff --git a/docs/design/rules.md b/docs/design/rules.md index 0e4697cb..bece775d 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -41,19 +41,29 @@ H1. Rationale: a title normally addresses by surname, so a title a suffix, a nickname, a maiden name — does not make the name any longer, so it does not decide this reading. And a run written BEHIND the one name word is the same form of address written on - the other side of it, so it decides the same reading. + the other side of it, so where no title run stands in front of + that word it decides the same reading. Where one does stand in + front, that run addresses: a title written behind the name is + the name plus that title (H5), and adding it cannot change how + the words in front of the name are read. Which side a title is + on is decided by the NAME WORD and by nothing else standing + beside it, so a run written around a nickname is one run: the + name Dr. 'Smitty' Sir John reads as Dr. Sir John does. A title followed by exactly one name word makes that word the family name, whatever suffix, nickname or maiden name stands beside it, unless the title is a given-name title, which keeps it the given name; a run of several titles addresses as its last - title does, and a title run standing BEHIND the one name word - decides that word's field the same way. + title does, and where a run stands BEFORE the one name word it + is the run that addresses, a run standing behind it deciding + that word's field only when none stands before. "Mr. Johnson" → family="Johnson" "Mrs. Garcia" → family="Garcia" "Dr. Smith née Jones" → family="Smith" "Her Majesty Queen Elizabeth" → given="Elizabeth" "Dr. Sir John" → given="John" "Smith Sir." → given="Smith" + "Sir John Prof." → given="John" + "Dr. Smith Sir." → family="Smith" "His Excellency Lord Duncan" → family="Duncan" "Sir John" → given="John" · boundary Accepted: a given-name title plus one name word leaves the @@ -246,9 +256,12 @@ H5. Rationale: a word abbreviated with a period at the END of a name TRANSPARENT to the suffix reading: where two or more name words stand, what stands once the chain is taken reads exactly as it would read written without the title, plus the title. Where the - chain leaves ONE name word, there is no second reading for it to - be transparent to, and the title behind that word decides its - field as a title in front of it would (H1). + chain leaves ONE name word and no title run stands in front of + it, there is no second reading for it to be transparent to, and + the title behind that word decides its field as a title in front + of it would (H1). Where a run DOES stand in front, that run + addresses and the chained title only joins the title field, so + the transparency holds for the one-word name too. "John Smith Prof." → title="Prof." "John Smith Prof." → family="Smith" "John Smith Prof. Dr." → title="Prof. Dr." @@ -298,7 +311,16 @@ H5. Rationale: a word abbreviated with a period at the END of a name Accepted: what the chain leaves is also what counts as a name word to spare (P5). A trailing title word is not one, so a bound given-name word behind one joins exactly as it joins with the - title absent. + title absent — and lands in the same field, the run in FRONT of + the joined pair being the run H1 asks about. Said of the + comma-less writing, which is where the chain runs before the + reserve is read. After a family comma the reserve reads no peel + at all (P5), so the join there takes the trailing title word into + the given name and the chain never sees it, where the same + segment writing with an ordinary given name reads the title. A + gap in the segment path rather than a boundary of this rule, and + tracked as part of #316. Pinned by a case row rather than by an + example here, so recording the gap does not make it normative. history: decisions.md#H5 · interacts: H1, H2, H3, H4, M2, P2, P5, S2, C1 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_pieces.py ## Particles & surname prefixes (P) diff --git a/docs/release_log.rst b/docs/release_log.rst index e2ad30ae..718a8f75 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -22,7 +22,7 @@ Release Log - **Fix the leading title peel taking a name word and leaving a post-nominal to be the name.** ``HumanName("Dr King Jr")`` gives title ``Dr``, last ``King``, suffix ``Jr``, where every release since 1.4.0 gave title ``Dr King``, last ``Jr`` and no suffix at all; ``Dr. King MD`` moves the same way, and both now read as the comma spelling ``King, Dr Jr`` always has. A title addresses somebody, so the run leaves a name word standing and a post-nominal is not one. A name that is nothing but titles or nothing but post-nominals is untouched, the word given back having to be a name candidate: ``Marquess of Bath``, ``MD DDS`` and ``Jr. Ph. D.`` are unchanged, and so is a title written as one joined unit -- ``Prince of Wales Jr`` keeps title ``Prince of Wales`` rather than losing the title to make a name. Where the run's whole content is the word given back there is no title left, so ``Dr Jr`` gives first ``Dr``, suffix ``Jr`` and reports a title-or-name ambiguity. Four names in the differential corpora read differently for this rule. See the ``H3`` entry of ``docs/design/decisions.md`` - - **Fix a trailing abbreviated title reading as a name word.** ``HumanName("John Smith Prof.")`` gives title ``Prof.``, first ``John``, last ``Smith``, where every release since 1.4.0 gave last ``Prof.`` and lost the surname; ``John Smith Mr.``, ``John Smith Rev.``, ``John Smith Dr.`` and ``Andrew Perkins (Mgr.)`` move the same way. A run chains from the end (``John Smith Prof. Dr.`` gives title ``Prof. Dr.``), a leading title keeps its place (``Dr. John Smith Prof.`` gives title ``Dr. Prof.``), and the comma forms agree with the bare ones now -- ``Smith, John Prof.`` gives title ``Prof.``, first ``John``, last ``Smith`` where it gave middle ``Prof.`` at every release. The trailing title is transparent to the post-nominal reading, so ``John Smith Jr. Prof.`` gives suffix ``Jr.`` rather than promoting the generational suffix to the last name. What does NOT move: an unlisted abbreviation (``John Smith Xyz.`` keeps last ``Xyz.``), a bare title word (``John Smith Sir``, ``Mary Jane King``) and a post-nominal (``John Smith Esq.``). Only a listed title word wearing the abbreviation period is claimed -- the leading slot infers a title from the shape alone, the trailing slot never does. The reach is the whole title vocabulary, ordinary surnames in it included, so a period written behind one of them takes it out of the name: ``Mary Jane King.`` gives title ``King.``, first ``Mary``, last ``Jane``, where the bare ``Mary Jane King`` keeps last ``King``. That is accepted rather than prevented -- the period is a writing convention and not evidence about the word, and the bare spelling is what the trailing slot is protected from. Fourteen names in the differential corpora read differently for this rule. See the ``H5`` entry of ``docs/design/decisions.md`` (closes #316) + - **Fix a trailing abbreviated title reading as a name word.** ``HumanName("John Smith Prof.")`` gives title ``Prof.``, first ``John``, last ``Smith``, where every release since 1.4.0 gave last ``Prof.`` and lost the surname; ``John Smith Mr.``, ``John Smith Rev.``, ``John Smith Dr.`` and ``Andrew Perkins (Mgr.)`` move the same way. A run chains from the end (``John Smith Prof. Dr.`` gives title ``Prof. Dr.``), a leading title keeps its place (``Dr. John Smith Prof.`` gives title ``Dr. Prof.``), and the comma forms agree with the bare ones now -- ``Smith, John Prof.`` gives title ``Prof.``, first ``John``, last ``Smith`` where it gave middle ``Prof.`` at every release. The trailing title is transparent to the post-nominal reading, so ``John Smith Jr. Prof.`` gives suffix ``Jr.`` rather than promoting the generational suffix to the last name. What does NOT move: an unlisted abbreviation (``John Smith Xyz.`` keeps last ``Xyz.``), a bare title word (``John Smith Sir``, ``Mary Jane King``) and a post-nominal (``John Smith Esq.``). Only a listed title word wearing the abbreviation period is claimed -- the leading slot infers a title from the shape alone, the trailing slot never does. The reach is the whole title vocabulary, ordinary surnames in it included, so a period written behind one of them takes it out of the name: ``Mary Jane King.`` gives title ``King.``, first ``Mary``, last ``Jane``, where the bare ``Mary Jane King`` keeps last ``King``. That is accepted rather than prevented -- the period is a writing convention and not evidence about the word, and the bare spelling is what the trailing slot is protected from. A title run in FRONT of the name still does the addressing, so the trailing title is transparent to that reading too: ``Sir John Prof.`` gives title ``Sir Prof.``, first ``John`` -- ``Sir John`` plus a title -- and ``Dr. Smith Sir.`` gives title ``Dr. Sir.``, last ``Smith``. Where NO title stands in front, the trailing one decides the field: ``Smith Sir.`` gives first ``Smith`` and ``Smith Prof.`` gives last ``Smith``. Sixteen names in the differential corpora read differently for this rule, and a seventeenth for the same argument in a native script: ``毛 泽东 Dr.`` gives title ``Dr.``, first ``泽东``, last ``毛``, where 2.2.0 gave last ``Dr.`` and lost the family-first order. See the ``H5`` entry of ``docs/design/decisions.md`` (closes #316) - **Remove esq from the default post-nominal acronyms, and assert the two post-nominal sets disjoint.** ``HumanName("John Smith E.S.Q.")`` gives middle ``Smith``, last ``E.S.Q.``, where every release since 1.4.0 gave suffix ``E.S.Q.``. ``Esq``, ``Esq.``, ``ESQ`` and ``esq`` are unchanged, the post-nominal word list carrying every single-token spelling; the acronym entry's only unique coverage was the multi-dot spelling. Esquire is a contraction rather than an initialism, so the initialism set was never its home, and it was the one word in both post-nominal sets -- which is why the sets can now assert they do not overlap, a word in both being matched by two rules that normalize differently. A caller who needs it back adds it: ``Lexicon.default().add(suffix_acronyms={"esq"})``. One name moves in the differential corpora. See the ``suffix-acronym-collisions`` entry of ``docs/design/decisions.md`` diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 919b259e..f0547ea5 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -762,7 +762,18 @@ def chain(tail: int) -> None: # title run through the ONE predicate post_rules asks # for H1, so the two rules cannot disagree about what # one run asserts; what it reads is the whole run's key - # or that key's LAST word (#489). H2's unlisted + # or that key's LAST word (#489). And it is the same + # RUN on both sides: `range(fk)` is the pieces AHEAD of + # the bound word, and H1 asks its own question of the + # leading run too (_post_rules._addressing_run). They + # did disagree for one commit -- H1 keyed every TITLE + # token, so the trailing title in 'Sir abdul rahman + # Prof.' joined this run and flipped the join's own + # premise, handing the licensed pair to the family. + # Reading the leading run at both sites is what makes + # that unreachable rather than merely unlikely: a + # trailing title is behind the word, and neither site + # can see it. H2's unlisted # abbreviations ride in the run either way. One is in # no vocabulary by definition, so it never matches as # the last-word key -- written as inputs, 'Xyz. Sir diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 810369ff..1d23a7f1 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -344,6 +344,36 @@ def _is_lone_never_given_particle(site: tuple[int, ...], and "vocab:particle-ambiguous" not in tokens[site[0]].tags) +def _addressing_run(titles: list[int], name_word: int) -> list[int]: + """The title run H1 asks about: the LEADING one where one stands, + else the whole (trailing) run -- rules.md#H1 -- the run standing + BEFORE the one name word being the run that addresses, and a run + standing behind it deciding that word's field only when none + stands before. + + Keeping the two ends apart is what makes a trailing title + TRANSPARENT (rules.md#H5): `Sir John Prof.` is `Sir John` plus a + title, and reading both ends as one run keyed 'sir prof' made + adding the title flip the name word's field (#489, #316). + + The split is at the NAME WORD, not at the first token of another + role, and the difference is a nickname or a maiden name written + among the titles. H1's own rationale says what stands beside the + name word "does not make the name any longer, so it does not + decide this reading", and that has to hold for WHICH run + addresses as well as for how many words the name has: `Dr. + 'Smitty' Sir John` is one run written around a nickname and reads + given 'John' as `Dr. Sir John` does, while `'Smitty' Dr. Jones + Sir.` keeps family 'Jones' as `Dr. Jones Sir.` does. Splitting on + the first non-title token got both wrong (measured 2026-09-09). + + Called only from inside H1's guard, after the role counts have + short-circuited, so a name with a family never builds this list; + `name_word` is the first GIVEN, which that guard has already + proved is the only name word there is.""" + return [i for i in titles if i < name_word] or titles + + def post_rules(state: ParseState) -> ParseState: tokens = list(state.tokens) ambiguities = list(state.ambiguities) @@ -362,14 +392,21 @@ def post_rules(state: ParseState) -> ParseState: # not count units -- decisions.md#H1) (v1 handle_firstnames) # # rules.md#H1: "a run of several titles addresses as its last - # title does" -- #489, so 'Her Majesty Queen Elizabeth' reads given - # 'Elizabeth': the run is not a given-name title but 'queen' is. + # title does, and where a run stands BEFORE the one name word it + # is the run that addresses, a run standing behind it deciding + # that word's field only when none stands before" -- #489, so 'Her + # Majesty Queen Elizabeth' reads given 'Elizabeth': the run is not + # a given-name title but 'queen' is. WHICH run is _addressing_run's + # question; every title token is in the TITLE role by now, both + # ends of 'Sir John Prof.' among them, and keying the two ends as + # one run made the trailing title change the leading one's reading. # The predicate lives beside _title_key because the P5 licence in # group asks the same question of the same run, and a run read two # ways is a rule contradicting itself (decisions.md#P5, #369). if (titles and givens and not middles and not families and not _run_addresses_by_given( - (tokens[i].text for i in titles), + (tokens[i].text + for i in _addressing_run(titles, givens[0])), state.lexicon.given_name_titles)): for i in givens: _retag(tokens, i, Role.FAMILY) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 8f3210b1..78cb6f2e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -3820,20 +3820,80 @@ def _check_cjk_shape_purity(self) -> None: "last 'Prof.'"), Case("title_word_trailing_behind_a_licensed_bound_pair", "Sir abdul rahman Prof.", - {"title": "Sir Prof.", "family": "abdul rahman"}, + {"title": "Sir Prof.", "given": "abdul rahman"}, classification="fix(#316)", - notes="the licensed half of the row above, and where the " - "trailing title is NOT transparent: the join fires " - "('sir' asserts a given name follows), and H1 then " - "reads the TITLE ROLE -- both ends of the name -- as " - "one run keyed 'sir prof', which does not address by " - "given name, so the pair becomes the family. 'Sir " - "abdul rahman' alone reads given 'abdul rahman'. The " - "same movement 'Sir John Prof.' shows against 'Sir " - "John', so it is H1's run composition and not P5's " - "reserve; recorded here rather than changed " - "(measured 2026-09-09). 1.4.0 read title 'Sir' / " - "first 'abdul rahman' / last 'Prof.'"), + notes="the licensed half of the row above, and transparent " + "at both ends: the join fires ('sir' asserts a given " + "name follows) and the joined pair stays the GIVEN " + "name, which is what 'Sir abdul rahman' reads without " + "the trailing title. Two fixes were needed and this " + "row took both. P5's reserve had counted the trailing " + "title word as a name word to spare, which made the " + "join fire for the wrong reason; with that corrected " + "the join is right and the field was still wrong, H1 " + "reading the TITLE ROLE -- both ends of the name by " + "then -- as one run keyed 'sir prof', which addresses " + "by neither word, so the pair became the family. H1 " + "now asks the LEADING run, and 'Sir John Prof.' " + "against 'Sir John' is the same movement in one word " + "(rules.md#H1, both are examples there). 1.4.0 read " + "title 'Sir' / first 'abdul rahman' / last 'Prof.' " + "(measured 2026-09-09)"), + Case("title_word_trailing_behind_a_bound_pair_after_a_comma", + "Berg, abdul Prof.", + {"given": "abdul Prof.", "family": "Berg"}, + notes="the recorded GAP in rules.md#H5's P5 clause, pinned " + "here rather than by an example there so that " + "recording it does not make it normative. After a " + "family comma the reserve reads no peel (rules.md#P5), " + "so the bound join fires over the trailing title word " + "and takes it into the given name; the trailing walk " + "never sees it. The comma-less spelling is " + "transparent -- 'Sir abdul rahman Prof.' above -- and " + "so is this segment with an ordinary given name, " + "'Smith, John Prof.' reading title 'Prof.'. PARITY, " + "which is why it is not a fix row: 1.4.0 read first " + "'abdul Prof.', last 'Berg' too (measured 2026-09-09). " + "Tracked as part of #316"), + Case("title_run_leading_addresses_over_a_trailing_title", + "Sir John Prof.", + {"title": "Sir Prof.", "given": "John"}, + classification="fix(#316)", + notes="rules.md#H1's clause in one word: the run BEFORE the " + "one name word addresses, so this is 'Sir John' plus a " + "title and the given name survives the title being " + "added. Keying every TITLE token as one run gave 'sir " + "prof', which addresses by neither, and read family " + "'John'. 1.4.0 read title 'Sir' / first 'John' / last " + "'Prof.' (measured 2026-09-09)"), + Case("title_run_leading_given_name_title_over_a_trailing_title", + "Queen Elizabeth Prof.", + {"title": "Queen Prof.", "given": "Elizabeth"}, + classification="fix(#316)", + notes="the same clause where the leading run is a ONE-word " + "given-name title that is also an ordinary surname. " + "'Queen Elizabeth' reads given 'Elizabeth' and adding " + "the trailing title does not move it; the composite " + "key 'queen prof' read family 'Elizabeth'. Not a " + "rules.md example -- 'Sir John Prof.' carries the " + "clause there -- and kept because `queen` is the " + "vocabulary entry #489's run arm turns on. 1.4.0 read " + "title 'Queen' / first 'Elizabeth' / last 'Prof.' " + "(measured 2026-09-09)"), + Case("title_run_leading_addresses_over_a_trailing_given_name_title", + "Dr. Smith Sir.", + {"title": "Dr. Sir.", "family": "Smith"}, + classification="fix(#316)", + notes="the mirror, and the reading the clause's ordering " + "decides: a trailing given-name title does NOT lift " + "the leading run's reading, so this is 'Dr. Smith' " + "plus a title and the family stands. The composite key " + "'dr sir' matched on 'sir' and read given 'Smith'. " + "Where NO run stands in front the trailing one does " + "decide -- 'Smith Sir.' reads given 'Smith', a " + "rules.md#H1 example unchanged by this. 1.4.0 read " + "title 'Dr.' / first 'Smith' / last 'Sir.' (measured " + "2026-09-09)"), Case("title_word_trailing_after_a_maiden_take", "Mary Smith née Jones Prof.", {"given": "Mary", "family": "Smith", "maiden": "Jones Prof."}, diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 870d7edf..438622b0 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -990,6 +990,16 @@ def test_case_shape_ids_exist_in_the_inventory() -> None: "fix(#316) a trailing period-marked title word reads as a title": ("John Smith Xyz.", "John Smith Sir", "Mary Jane King", "John Smith Esq.", "Smith, Prof."), + # The CJK member of the same argument is its own literal rule -- + # an alternation holding a script-classified member belongs to the + # honorific pin -- so a reach of 1 is one _CORPUS_CLAIMS cannot + # police on its own, and these are the wall. All three are + # tolerated CJK corpus names the rule must not take: a Latin + # post-nominal behind a comma, the comma spelling of this very + # shape (routed by the segment gate), and a spaced Latin + # post-nominal that is not title vocabulary. + "fix(#316) a trailing Latin title on a native-script name is a title": + ("王先生, V.", "田中さん, Dr.", "田中さん II"), # The esq boundary is every spelling SUFFIX_WORDS still carries, # in each of the three positions the corpora write it in. "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -1785,17 +1795,22 @@ class _LatinCopy(NamedTuple): # first is the period-marked spelling of the very name the # wordlist test above uses, which is the point of it -- the bare # word is protected and the abbreviated one is not, and the member - # says so by carrying the period. 'Andrew Perkins \x28Mgr\.\x29' + # says so by carrying the period. 'Sir John Prof\.' and + # 'Dr\. Smith Sir\.' joined in the SECOND review round the same + # day, as the rules.md examples of H1's leading-run clause: both + # move `title` -- the walk takes the trailing word out of the name + # -- which is what puts them here rather than on the #489 run + # rule, whose `fields` reach no title role at all. 'Andrew Perkins \x28Mgr\.\x29' # carries the ledger's \x28/\x29 spelling of the parentheses, # without which this whole alternation would be invisible to # _alternations above. One set, identical in all four ledgers. frozenset({r"Andrew Perkins \x28Mgr\.\x29", r"Dr\. John Smith Prof\.", - r"John Prof\. MA", r"John Smith Dr\.", + r"Dr\. Smith Sir\.", r"John Prof\. MA", r"John Smith Dr\.", r"John Smith Jr\. Prof\.", r"John Smith Mr\.", r"John Smith Prof\.", r"John Smith Prof\. Dr\.", r"John Smith Prof\. Jr\.", r"John Smith Rev\.", - r"Mary Jane King\.", r"Smith Prof\.", r"Smith Sir\.", - r"Smith, John Prof\."}), + r"Mary Jane King\.", r"Sir John Prof\.", r"Smith Prof\.", + r"Smith Sir\.", r"Smith, John Prof\."}), }) def _unjustified_reach(name_regex: str, members: set[str]) -> list[str]: @@ -2455,16 +2470,17 @@ def _claim(rule: dict) -> _Claim: _Claim(109, ('_initials',), "ae9c8f674e0c", ('DEFAULT',)), "fix(initials-per-word) the Ph. D. merge initials each word (facade, since 2.0.0)": _Claim(18, ('_initials',), "f67d8ebddd56", ('DEFAULT',)), - # The 2.3 title-run bundle's four rules, last in every - # ledger. All four are anchored alternations of NAMES, so the - # reach IS the mover list and the four numbers are the four - # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 14 for the - # trailing title. Twenty-one in all, and every one of them is - # explained by the rule that names it -- these are the rare - # rows where reach and explanation coincide, which is what an - # anchored name list buys. A widening past those names moves - # the digest here before it can reach the gate. + # The 2.3 title-run bundle's five rules, last in every + # ledger. All five are anchored on NAMES, so the reach IS the + # mover list: 2 names for the run keying, 1 for the esq drop, + # 4 for the peel floor, 16 for the trailing title and 1 for + # the trailing title on a native-script name, which is the + # same argument on the one name a script-classified + # alternation may not hold. Twenty-four in all, and every one + # of them is explained by the rule that names it -- these are + # the rare rows where reach and explanation coincide, which is + # what an anchored name list buys. A widening past those names + # moves the digest here before it can reach the gate. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -2477,8 +2493,17 @@ def _claim(rule: dict) -> _Claim: "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(4, ('family', 'given', 'suffix', 'title'), "ac7318881b28", None), "fix(#316) a trailing period-marked title word reads as a title": - _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), - "4130dc8bbf40", None), + _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), + "562e0e82a22b", None), + # The CJK member of the same argument, its own literal rule + # (an alternation holding a script-classified member belongs + # to the honorific pin above). ONE corpus name, and the roles + # are the one thing that differs by ledger here: this baseline + # read the Latin word as a post-nominal and the Han words + # given-first, so all four move. + "fix(#316) a trailing Latin title on a native-script name is a title": + _Claim(1, ('family', 'given', 'suffix', 'title'), + "567f09dc9b45", None), }, "expected_since_2.0.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -2717,16 +2742,17 @@ def _claim(rule: dict) -> _Claim: # regex is the same string in each. "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), - # The 2.3 title-run bundle's four rules, last in every - # ledger. All four are anchored alternations of NAMES, so the - # reach IS the mover list and the four numbers are the four - # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 14 for the - # trailing title. Twenty-one in all, and every one of them is - # explained by the rule that names it -- these are the rare - # rows where reach and explanation coincide, which is what an - # anchored name list buys. A widening past those names moves - # the digest here before it can reach the gate. + # The 2.3 title-run bundle's five rules, last in every + # ledger. All five are anchored on NAMES, so the reach IS the + # mover list: 2 names for the run keying, 1 for the esq drop, + # 4 for the peel floor, 16 for the trailing title and 1 for + # the trailing title on a native-script name, which is the + # same argument on the one name a script-classified + # alternation may not hold. Twenty-four in all, and every one + # of them is explained by the rule that names it -- these are + # the rare rows where reach and explanation coincide, which is + # what an anchored name list buys. A widening past those names + # moves the digest here before it can reach the gate. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -2739,8 +2765,15 @@ def _claim(rule: dict) -> _Claim: "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), "fix(#316) a trailing period-marked title word reads as a title": - _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), - "4130dc8bbf40", None), + _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), + "562e0e82a22b", None), + # The CJK member of the same argument, its own literal rule + # (an alternation holding a script-classified member belongs + # to the honorific pin above). ONE corpus name; this baseline + # reads as 1.4.0 does, so the roles are the same four. + "fix(#316) a trailing Latin title on a native-script name is a title": + _Claim(1, ('family', 'given', 'suffix', 'title'), + "567f09dc9b45", None), }, # The 2.3 cycle's first rule, and a facade-only render fix: every # role is identical, so `_initials` alone. Reach and digest as in @@ -2826,16 +2859,17 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('suffix',), "1b67339cf744", None), "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), - # The 2.3 title-run bundle's four rules, last in every - # ledger. All four are anchored alternations of NAMES, so the - # reach IS the mover list and the four numbers are the four - # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 14 for the - # trailing title. Twenty-one in all, and every one of them is - # explained by the rule that names it -- these are the rare - # rows where reach and explanation coincide, which is what an - # anchored name list buys. A widening past those names moves - # the digest here before it can reach the gate. + # The 2.3 title-run bundle's five rules, last in every + # ledger. All five are anchored on NAMES, so the reach IS the + # mover list: 2 names for the run keying, 1 for the esq drop, + # 4 for the peel floor, 16 for the trailing title and 1 for + # the trailing title on a native-script name, which is the + # same argument on the one name a script-classified + # alternation may not hold. Twenty-four in all, and every one + # of them is explained by the rule that names it -- these are + # the rare rows where reach and explanation coincide, which is + # what an anchored name list buys. A widening past those names + # moves the digest here before it can reach the gate. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -2848,8 +2882,17 @@ def _claim(rule: dict) -> _Claim: "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), "fix(#316) a trailing period-marked title word reads as a title": - _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), - "4130dc8bbf40", None), + _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), + "562e0e82a22b", None), + # The CJK member of the same argument, its own literal rule + # (an alternation holding a script-classified member belongs + # to the honorific pin above). ONE corpus name; 2.2.0 took + # `dr` out of the post-nominal vocabulary, so the Latin word + # was a name word there and `suffix` is empty on both sides + # while `middle` moves instead. + "fix(#316) a trailing Latin title on a native-script name is a title": + _Claim(1, ('family', 'given', 'middle', 'title'), + "567f09dc9b45", None), }, "expected_since_2.1.0.toml": { # #436/#437's Latin alternation, first in every ledger. @@ -3070,16 +3113,17 @@ def _claim(rule: dict) -> _Claim: # in every 2.x wheel, so the baseline makes no difference. "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), - # The 2.3 title-run bundle's four rules, last in every - # ledger. All four are anchored alternations of NAMES, so the - # reach IS the mover list and the four numbers are the four - # release-log bullets one for one: 2 names for the run keying, - # 1 for the esq drop, 4 for the peel floor, 14 for the - # trailing title. Twenty-one in all, and every one of them is - # explained by the rule that names it -- these are the rare - # rows where reach and explanation coincide, which is what an - # anchored name list buys. A widening past those names moves - # the digest here before it can reach the gate. + # The 2.3 title-run bundle's five rules, last in every + # ledger. All five are anchored on NAMES, so the reach IS the + # mover list: 2 names for the run keying, 1 for the esq drop, + # 4 for the peel floor, 16 for the trailing title and 1 for + # the trailing title on a native-script name, which is the + # same argument on the one name a script-classified + # alternation may not hold. Twenty-four in all, and every one + # of them is explained by the rule that names it -- these are + # the rare rows where reach and explanation coincide, which is + # what an anchored name list buys. A widening past those names + # moves the digest here before it can reach the gate. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -3092,8 +3136,17 @@ def _claim(rule: dict) -> _Claim: "fix(#489) the title peel leaves a name word a suffix cannot be": _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), "fix(#316) a trailing period-marked title word reads as a title": - _Claim(14, ('family', 'given', 'middle', 'suffix', 'title'), - "4130dc8bbf40", None), + _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), + "562e0e82a22b", None), + # The CJK member of the same argument, its own literal rule + # (an alternation holding a script-classified member belongs + # to the honorific pin above). ONE corpus name, and the FEWEST + # roles of any ledger: 2.1.0 shipped the Han family-first + # order, so only the Latin word moves, out of `suffix` and + # into `title`. + "fix(#316) a trailing Latin title on a native-script name is a title": + _Claim(1, ('suffix', 'title'), + "567f09dc9b45", None), }, } diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 70f7d2d2..9d3ab647 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -675,26 +675,39 @@ def test_the_p5_licence_and_h1_read_a_title_run_the_same_way( title: str) -> None: # The licence's one invariant, as a contract: P5 lifts the reserve # behind a title run exactly when H1 keeps the one word after that - # run a given name. Both ask _run_addresses_by_given -- the whole - # run's key, or the run's last word's (#489); if either side's read - # drifted, a run P5 licensed that H1 then read as title-plus-family - # would hand the joined pair to the family. So "no family" must - # agree, run by run. + # run a given name. Both ask _run_addresses_by_given, of the + # LEADING run, for the whole run's key or the run's last word's + # (#489); if either side's read drifted, a run P5 licensed that H1 + # then read as title-plus-family would hand the joined pair to the + # family. So "no family" must agree, run by run. assert (parse(f"{title} John").family == "") == \ (parse(f"{title} abdul rahman").family == "") - # The same invariant with a trailing title behind the pair. P5's - # reserve counts the name words assign's peel AND the H5 walk - # leave, so a period-marked title word at the back is not one of - # them; H1 reads the TITLE role, which by then holds both ends of - # the name. Both sides must still agree about "no family" -- and - # they do run by run, though the two halves reach it differently: - # behind a given-name title the join fires and H1 then hands the - # pair to the family (the run keys 'sir prof'), while behind an - # ordinary one the reserve declines the join and the two words - # split. Either way a family stands, as it does for the one-word - # spelling. - assert (parse(f"{title} John Prof.").family == "") == \ - (parse(f"{title} abdul rahman Prof.").family == "") + # The same invariant with a trailing title behind the pair, and + # stated the way rules.md#H5's transparency clause states it + # rather than as agreement between the two halves: `X Prof.` is + # `X` plus a title, so adding the title moves NEITHER field. That + # is what discriminates. Comparing the two spellings' "no family" + # to each other passes under the composite keying this replaced, + # where H1 read the TITLE role -- both ends of the name by then -- + # as one run: `sir prof` addresses by neither word, so both + # spellings handed their name words to the family together and + # agreed while both were wrong. Pinned by mutation: restore the + # whole-`titles` key and every given-name-title row below fails + # here (2026-09-09). + # + # The NICKNAME spellings are here because the first fix for this + # split the run at the first token of another role, which a + # nickname written in front of the titles satisfies -- so + # "'Smitty' Sir Jones Prof." found no leading run and read family + # 'Jones' where "'Smitty' Sir Jones" reads given. What decides is + # the name WORD, H1's rationale saying in as many words that a + # nickname beside it does not decide the reading, and only these + # rows can tell the two splits apart. + for suffixless in (f"{title} John", f"{title} abdul rahman", + f"'Smitty' {title} John", + f"'Smitty' {title} abdul rahman"): + plain, titled = parse(suffixless), parse(f"{suffixless} Prof.") + assert (titled.given, titled.family) == (plain.given, plain.family) # The first three reach the chain loop and decline inside it: the piece diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 01910485..276b0f3c 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -1950,8 +1950,13 @@ class _ShapeMismatch(NamedTuple): #: entered it, so the scan went 52 -> 53 and the roster 50 -> 51. #: Recounted 2026-09-08 with the title-run bundle: one name entered the #: population, 'John Smith Rev.', the only one of that bundle's -#: twenty-one movers no test literal names, and it took a row at each -#: of the four baselines. It is named NOWHERE under tests/, so it +#: movers no test literal names, and it took a row at each +#: of the four baselines. Re-checked 2026-09-09 in that bundle's second +#: review round, when its mover list went from twenty-one to +#: twenty-four: none of the three arrivals joins the population, all +#: three being named as case-row literals in tests/v2/cases.py +#: ('Sir John Prof.', 'Dr. Smith Sir.', '毛 泽东 Dr.'), so the row +#: counts below are unmoved. It is named NOWHERE under tests/, so it #: counts in both scans -- the every-file figures in the RECOMPUTE #: paragraph below count it too. That commit moved the ROW counts and nothing #: else: it did not re-derive the population clause above, so the diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index be9a40b8..0ad6c230 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -41,6 +41,7 @@ "Dr. King MD" "Dr. Sir John" "Dr. Smith" +"Dr. Smith Sir." "Dr. Smith née Jones" "Dr. Smith, John" "Dr. abdul salam" @@ -180,6 +181,7 @@ "Shirley Maclaine" "Sidorov Ivan Petrovich Jr." "Sir John" +"Sir John Prof." "Sir Jr" "Sir Ph. D. Van Johnson" "Sir abdul van der Berg" diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 19d9d04a..055e664e 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -2905,12 +2905,15 @@ orders = ["DEFAULT"] # THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym -# drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over twenty-one corpus names. +# drop that shipped beside them. Five rules, four of them one per +# ARGUMENT rather +# than one per name, plus a fifth that is #316's argument again on +# the one name a script-classified alternation cannot hold. Over +# twenty-four corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# twenty-one declares `fields` that are a strict subset of the bundle +# twenty-four declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -2924,10 +2927,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the twenty-one do move. The derived view +# initials of several of the twenty-four do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all twenty-one move roles -- so no run can produce a +# guard), and all twenty-four move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -3044,7 +3047,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # -# Five roles, the union of what the fourteen move; no name moves all +# Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and # 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. @@ -3055,12 +3058,12 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # wheels). `_ambiguities` is not among the five, and it is not # because no name reports one: 'John Prof. MA' reports # `suffix-or-name`, which is S2's bare-acronym reserve and not this -# rule. None of the fourteen reports `title-or-name` -- the walk +# rule. None of the sixteen reports `title-or-name` -- the walk # leaves a title standing, so H4's title half never fires -- and the # derived view cannot enter a diff here in any case, every one of the -# fourteen moving roles (compare.py's _RULE_FIELDS). +# sixteen moving roles (compare.py's _RULE_FIELDS). # -# TWO of the fourteen end in ' Dr.' and are also reached by +# TWO of the sixteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which # that rule used to explain, and 'John Smith Prof. Dr.'. That rule # declares {middle, family, suffix} and neither diff fits it now, so @@ -3083,7 +3086,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # reach a title role at all, so it could not claim the diff even # placed ahead of this one. # -# An anchored alternation of the fourteen NAMES rather than a shape. +# An anchored alternation of the sixteen NAMES rather than a shape. # What selects them is a SHAPE -- period-marked, listed, trailing -- # and a member copying TITLES would reach the BARE 'Mary Jane King' # and 'John Smith Sir', which do not move; the period-marked @@ -3102,9 +3105,54 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a fourteen-member alternation should not +# excludes '(' and ')' -- and a sixteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing Latin title on a native-script name is a title" +# '毛 泽东 Dr.': the rule above's walk, on the one corpus name it +# reaches that is not Latin-only. The walk runs BEFORE the positional +# read, so the pieces the script test sees are all Han and the +# family-first order stands -- this is '毛 泽东' plus a title, and +# it reads title 'Dr.', given '泽东', family '毛'. A Latin title at +# the back of a wholly-Han name is the one piece that would make the +# piece set look mixed-script, which is what the ordering of the two +# buys. +# +# ITS OWN rule rather than a seventeenth alternative above, and not by +# preference: an alternation holding a script-classified member is +# claimed by the honorific pin in tests/v2/test_ledger_guards.py, +# which requires such an alternation's members to BE the config's CJK +# honorific entries. So this stands alone, literal-anchored, reaching +# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with +# _MUST_NOT_MATCH probes for the boundary -- the same shape the four +# #436/#437 CJK rules above take, and for the same reason. +# +# Classified 2026-09-09, in the review round that fixed which title +# run H1 addresses by; the reading itself is untouched by that fix. +# The name is RADAR tier (corpus_cjk_tolerated.jsonl, tolerated since +# the 2026-09-01 CJK demotion) and it had been printing UNCLASSIFIED +# at all four gates since the trailing walk landed. Radar never +# blocks, so nothing failed -- and this bundle's convention is that +# every gate line reads `radar unclassified: 0`, which is the whole +# reason the row is here rather than left to print. +# +# FOUR roles here, and NOT the same four at every baseline, which is +# why this is the one bundle rule whose `fields` differ per ledger. +# This baseline read first '毛', last '泽东', suffix 'Dr.' -- the +# Latin word taken as a post-nominal and the two Han words read +# given-first -- so `title`, `given`, `family` and `suffix` all move. +# +# Literal and anchored, so the reach cannot widen into the other +# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# them as probes: '王先生, V.' (a Latin post-nominal behind a +# comma, which the trailing walk never sees), '田中さん, Dr.' (the +# comma spelling of this very shape, routed by the segment gate) and +# '田中さん II' (a spaced Latin post-nominal that is not title +# vocabulary). +name_regex = "^毛 泽东 Dr\\.$" +fields = ["family", "given", "suffix", "title"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 547e46e6..93b7f23b 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -1771,12 +1771,15 @@ orders = ["DEFAULT"] # THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym -# drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over twenty-one corpus names. +# drop that shipped beside them. Five rules, four of them one per +# ARGUMENT rather +# than one per name, plus a fifth that is #316's argument again on +# the one name a script-classified alternation cannot hold. Over +# twenty-four corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# twenty-one declares `fields` that are a strict subset of the bundle +# twenty-four declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -1790,10 +1793,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the twenty-one do move. The derived view +# initials of several of the twenty-four do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all twenty-one move roles -- so no run can produce a +# guard), and all twenty-four move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -1910,7 +1913,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # -# Five roles, the union of what the fourteen move; no name moves all +# Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and # 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. @@ -1921,12 +1924,12 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # wheels). `_ambiguities` is not among the five, and it is not # because no name reports one: 'John Prof. MA' reports # `suffix-or-name`, which is S2's bare-acronym reserve and not this -# rule. None of the fourteen reports `title-or-name` -- the walk +# rule. None of the sixteen reports `title-or-name` -- the walk # leaves a title standing, so H4's title half never fires -- and the # derived view cannot enter a diff here in any case, every one of the -# fourteen moving roles (compare.py's _RULE_FIELDS). +# sixteen moving roles (compare.py's _RULE_FIELDS). # -# TWO of the fourteen end in ' Dr.' and are also reached by +# TWO of the sixteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which # that rule used to explain, and 'John Smith Prof. Dr.'. That rule # declares {middle, family, suffix} and neither diff fits it now, so @@ -1949,7 +1952,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # reach a title role at all, so it could not claim the diff even # placed ahead of this one. # -# An anchored alternation of the fourteen NAMES rather than a shape. +# An anchored alternation of the sixteen NAMES rather than a shape. # What selects them is a SHAPE -- period-marked, listed, trailing -- # and a member copying TITLES would reach the BARE 'Mary Jane King' # and 'John Smith Sir', which do not move; the period-marked @@ -1968,9 +1971,53 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a fourteen-member alternation should not +# excludes '(' and ')' -- and a sixteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing Latin title on a native-script name is a title" +# '毛 泽东 Dr.': the rule above's walk, on the one corpus name it +# reaches that is not Latin-only. The walk runs BEFORE the positional +# read, so the pieces the script test sees are all Han and the +# family-first order stands -- this is '毛 泽东' plus a title, and +# it reads title 'Dr.', given '泽东', family '毛'. A Latin title at +# the back of a wholly-Han name is the one piece that would make the +# piece set look mixed-script, which is what the ordering of the two +# buys. +# +# ITS OWN rule rather than a seventeenth alternative above, and not by +# preference: an alternation holding a script-classified member is +# claimed by the honorific pin in tests/v2/test_ledger_guards.py, +# which requires such an alternation's members to BE the config's CJK +# honorific entries. So this stands alone, literal-anchored, reaching +# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with +# _MUST_NOT_MATCH probes for the boundary -- the same shape the four +# #436/#437 CJK rules above take, and for the same reason. +# +# Classified 2026-09-09, in the review round that fixed which title +# run H1 addresses by; the reading itself is untouched by that fix. +# The name is RADAR tier (corpus_cjk_tolerated.jsonl, tolerated since +# the 2026-09-01 CJK demotion) and it had been printing UNCLASSIFIED +# at all four gates since the trailing walk landed. Radar never +# blocks, so nothing failed -- and this bundle's convention is that +# every gate line reads `radar unclassified: 0`, which is the whole +# reason the row is here rather than left to print. +# +# FOUR roles here, and NOT the same four at every baseline, which is +# why this is the one bundle rule whose `fields` differ per ledger. +# This baseline reads as 1.4.0 does -- first '毛', last '泽东', +# suffix 'Dr.' -- so `title`, `given`, `family` and `suffix` all move. +# +# Literal and anchored, so the reach cannot widen into the other +# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# them as probes: '王先生, V.' (a Latin post-nominal behind a +# comma, which the trailing walk never sees), '田中さん, Dr.' (the +# comma spelling of this very shape, routed by the segment gate) and +# '田中さん II' (a spaced Latin post-nominal that is not title +# vocabulary). +name_regex = "^毛 泽东 Dr\\.$" +fields = ["family", "given", "suffix", "title"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 3cbf5c2a..acec949e 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -1690,12 +1690,15 @@ orders = ["DEFAULT"] # THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym -# drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over twenty-one corpus names. +# drop that shipped beside them. Five rules, four of them one per +# ARGUMENT rather +# than one per name, plus a fifth that is #316's argument again on +# the one name a script-classified alternation cannot hold. Over +# twenty-four corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# twenty-one declares `fields` that are a strict subset of the bundle +# twenty-four declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -1709,10 +1712,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the twenty-one do move. The derived view +# initials of several of the twenty-four do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all twenty-one move roles -- so no run can produce a +# guard), and all twenty-four move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -1829,7 +1832,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # -# Five roles, the union of what the fourteen move; no name moves all +# Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and # 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. @@ -1840,12 +1843,12 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # wheels). `_ambiguities` is not among the five, and it is not # because no name reports one: 'John Prof. MA' reports # `suffix-or-name`, which is S2's bare-acronym reserve and not this -# rule. None of the fourteen reports `title-or-name` -- the walk +# rule. None of the sixteen reports `title-or-name` -- the walk # leaves a title standing, so H4's title half never fires -- and the # derived view cannot enter a diff here in any case, every one of the -# fourteen moving roles (compare.py's _RULE_FIELDS). +# sixteen moving roles (compare.py's _RULE_FIELDS). # -# TWO of the fourteen end in ' Dr.' and are also reached by +# TWO of the sixteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which # that rule used to explain, and 'John Smith Prof. Dr.'. That rule # declares {middle, family, suffix} and neither diff fits it now, so @@ -1868,7 +1871,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # reach a title role at all, so it could not claim the diff even # placed ahead of this one. # -# An anchored alternation of the fourteen NAMES rather than a shape. +# An anchored alternation of the sixteen NAMES rather than a shape. # What selects them is a SHAPE -- period-marked, listed, trailing -- # and a member copying TITLES would reach the BARE 'Mary Jane King' # and 'John Smith Sir', which do not move; the period-marked @@ -1887,9 +1890,54 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a fourteen-member alternation should not +# excludes '(' and ')' -- and a sixteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing Latin title on a native-script name is a title" +# '毛 泽东 Dr.': the rule above's walk, on the one corpus name it +# reaches that is not Latin-only. The walk runs BEFORE the positional +# read, so the pieces the script test sees are all Han and the +# family-first order stands -- this is '毛 泽东' plus a title, and +# it reads title 'Dr.', given '泽东', family '毛'. A Latin title at +# the back of a wholly-Han name is the one piece that would make the +# piece set look mixed-script, which is what the ordering of the two +# buys. +# +# ITS OWN rule rather than a seventeenth alternative above, and not by +# preference: an alternation holding a script-classified member is +# claimed by the honorific pin in tests/v2/test_ledger_guards.py, +# which requires such an alternation's members to BE the config's CJK +# honorific entries. So this stands alone, literal-anchored, reaching +# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with +# _MUST_NOT_MATCH probes for the boundary -- the same shape the four +# #436/#437 CJK rules above take, and for the same reason. +# +# Classified 2026-09-09, in the review round that fixed which title +# run H1 addresses by; the reading itself is untouched by that fix. +# The name is RADAR tier (corpus_cjk_tolerated.jsonl, tolerated since +# the 2026-09-01 CJK demotion) and it had been printing UNCLASSIFIED +# at all four gates since the trailing walk landed. Radar never +# blocks, so nothing failed -- and this bundle's convention is that +# every gate line reads `radar unclassified: 0`, which is the whole +# reason the row is here rather than left to print. +# +# TWO roles here, the fewest at any baseline, and NOT the same set as +# elsewhere -- this is the one bundle rule whose `fields` differ per +# ledger. 2.1.0 shipped the Han family-first order, so it already read +# first '泽东', last '毛'; only the Latin word moves, out of +# `suffix` and into `title`. +# +# Literal and anchored, so the reach cannot widen into the other +# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# them as probes: '王先生, V.' (a Latin post-nominal behind a +# comma, which the trailing walk never sees), '田中さん, Dr.' (the +# comma spelling of this very shape, routed by the segment gate) and +# '田中さん II' (a spaced Latin post-nominal that is not title +# vocabulary). +name_regex = "^毛 泽东 Dr\\.$" +fields = ["suffix", "title"] diff --git a/tools/differential/expected_since_2.2.0.toml b/tools/differential/expected_since_2.2.0.toml index a30970e3..5f4d77a4 100644 --- a/tools/differential/expected_since_2.2.0.toml +++ b/tools/differential/expected_since_2.2.0.toml @@ -331,12 +331,15 @@ orders = ["DEFAULT"] # THE 2.3 TITLE-RUN BUNDLE: #489 (a title run addresses by its last # title), #316 (a trailing period-marked title word reads as a # title), the leading peel's name-word floor, and the `esq` acronym -# drop that shipped beside them. Four rules, one per ARGUMENT rather -# than one per name, over twenty-one corpus names. +# drop that shipped beside them. Five rules, four of them one per +# ARGUMENT rather +# than one per name, plus a fifth that is #316's argument again on +# the one name a script-classified alternation cannot hold. Over +# twenty-four corpus names. # # LAST in the file, and that is narrow-first rather than a # preference: every rule already here that REACHES one of the -# twenty-one declares `fields` that are a strict subset of the bundle +# twenty-four declares `fields` that are a strict subset of the bundle # rule claiming that name, and a wide rule sitting AHEAD of a # narrower one it shares a corpus name with is an order-decided # contest the run refuses (#382). Measured 2026-09-08, all four @@ -350,10 +353,10 @@ orders = ["DEFAULT"] # and no [[change.precedes_narrower]] block is needed anywhere. # # `_initials` appears in none of the four field lists although the -# initials of several of the twenty-one do move. The derived view +# initials of several of the twenty-four do move. The derived view # enters a diff only where every ROLE and every ambiguity kind # agrees (compare.py's _RULE_FIELDS, main()'s roles-identical -# guard), and all twenty-one move roles -- so no run can produce a +# guard), and all twenty-four move roles -- so no run can produce a # shape here carrying it, and validate_rules refuses it beside # another field in any case. # --------------------------------------------------------------- @@ -470,7 +473,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # generational suffix to the family name, and 'John Prof. MA' reads # family 'MA' by rules.md#S2's reserve. # -# Five roles, the union of what the fourteen move; no name moves all +# Five roles, the union of what the sixteen move; no name moves all # five (four is the most, 'John Smith Jr. Prof.'), and two move only # two at every baseline -- 'Smith, John Prof.' {title, middle} and # 'Smith Sir.' {title, family} -- with 'Mary Jane King.' moving three. @@ -481,12 +484,12 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # wheels). `_ambiguities` is not among the five, and it is not # because no name reports one: 'John Prof. MA' reports # `suffix-or-name`, which is S2's bare-acronym reserve and not this -# rule. None of the fourteen reports `title-or-name` -- the walk +# rule. None of the sixteen reports `title-or-name` -- the walk # leaves a title standing, so H4's title half never fires -- and the # derived view cannot enter a diff here in any case, every one of the -# fourteen moving roles (compare.py's _RULE_FIELDS). +# sixteen moving roles (compare.py's _RULE_FIELDS). # -# TWO of the fourteen end in ' Dr.' and are also reached by +# TWO of the sixteen end in ' Dr.' and are also reached by # fix(#296)'s trailing-`dr` rule above -- 'John Smith Dr.', which # that rule used to explain, and 'John Smith Prof. Dr.'. That rule # declares {middle, family, suffix} and neither diff fits it now, so @@ -509,7 +512,7 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # reach a title role at all, so it could not claim the diff even # placed ahead of this one. # -# An anchored alternation of the fourteen NAMES rather than a shape. +# An anchored alternation of the sixteen NAMES rather than a shape. # What selects them is a SHAPE -- period-marked, listed, trailing -- # and a member copying TITLES would reach the BARE 'Mary Jane King' # and 'John Smith Sir', which do not move; the period-marked @@ -528,9 +531,55 @@ issue = "fix(#316) a trailing period-marked title word reads as a title" # the \\( and \\) the paren-bearing rules above use. Escaped or not, a # literal parenthesis inside an alternation body hides the WHOLE # group from test_ledger_guards._alternations -- its member pattern -# excludes '(' and ')' -- and a fourteen-member alternation should not +# excludes '(' and ')' -- and a sixteen-member alternation should not # be invisible to the discovery pass that demands every alternation # declare what it copies. The two spellings match the same string. # One set, identical in all four ledgers. -name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" +name_regex = "^(?:Andrew Perkins \\x28Mgr\\.\\x29|Dr\\. John Smith Prof\\.|Dr\\. Smith Sir\\.|John Prof\\. MA|John Smith Dr\\.|John Smith Jr\\. Prof\\.|John Smith Mr\\.|John Smith Prof\\.|John Smith Prof\\. Dr\\.|John Smith Prof\\. Jr\\.|John Smith Rev\\.|Mary Jane King\\.|Sir John Prof\\.|Smith Prof\\.|Smith Sir\\.|Smith, John Prof\\.)$" fields = ["family", "given", "middle", "suffix", "title"] + +[[change]] +issue = "fix(#316) a trailing Latin title on a native-script name is a title" +# '毛 泽东 Dr.': the rule above's walk, on the one corpus name it +# reaches that is not Latin-only. The walk runs BEFORE the positional +# read, so the pieces the script test sees are all Han and the +# family-first order stands -- this is '毛 泽东' plus a title, and +# it reads title 'Dr.', given '泽东', family '毛'. A Latin title at +# the back of a wholly-Han name is the one piece that would make the +# piece set look mixed-script, which is what the ordering of the two +# buys. +# +# ITS OWN rule rather than a seventeenth alternative above, and not by +# preference: an alternation holding a script-classified member is +# claimed by the honorific pin in tests/v2/test_ledger_guards.py, +# which requires such an alternation's members to BE the config's CJK +# honorific entries. So this stands alone, literal-anchored, reaching +# exactly one corpus name (_CORPUS_CLAIMS carries the 1) with +# _MUST_NOT_MATCH probes for the boundary -- the same shape the four +# #436/#437 CJK rules above take, and for the same reason. +# +# Classified 2026-09-09, in the review round that fixed which title +# run H1 addresses by; the reading itself is untouched by that fix. +# The name is RADAR tier (corpus_cjk_tolerated.jsonl, tolerated since +# the 2026-09-01 CJK demotion) and it had been printing UNCLASSIFIED +# at all four gates since the trailing walk landed. Radar never +# blocks, so nothing failed -- and this bundle's convention is that +# every gate line reads `radar unclassified: 0`, which is the whole +# reason the row is here rather than left to print. +# +# FOUR roles here, and NOT the same four at every baseline, which is +# why this is the one bundle rule whose `fields` differ per ledger. +# 2.2.0 took `dr` out of the post-nominal vocabulary, so 'Dr.' became +# a name word: first '毛', middle '泽东', last 'Dr.', the mixed-script +# piece set declining the Han order it had had since 2.1.0. `title`, +# `given`, `middle` and `family` move; `suffix` is empty on both sides. +# +# Literal and anchored, so the reach cannot widen into the other +# twenty-nine tolerated CJK names. _MUST_NOT_MATCH carries three of +# them as probes: '王先生, V.' (a Latin post-nominal behind a +# comma, which the trailing walk never sees), '田中さん, Dr.' (the +# comma spelling of this very shape, routed by the segment gate) and +# '田中さん II' (a spaced Latin post-nominal that is not title +# vocabulary). +name_regex = "^毛 泽东 Dr\\.$" +fields = ["family", "given", "middle", "title"] From 96a511b694bc8a6f6d9efe8c8da8550db309e0f3 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 20:28:02 -0700 Subject: [PATCH 09/12] refactor: one tail reading for assign and the P5 reserve; /simplify round on PR #520 The trailing peel (S2) and the trailing title chain (H5) were derived twice: assign ran peel -> chain -> splice -> re-peel inline, and group's bound-given reserve modelled that as the peel's count minus the chain's take on each of the two views the join compares. The two disagreed at S2's bare-ambiguous reserve -- `abdul rahman MA` declined the join while `abdul rahman MA Prof.` took it, contradicting H5's stated clause that a bound given-name word behind a trailing title joins as it joins with the title absent. And assign's re-peel ran once, so `John Prof. MA Prof.` un-peeled the acronym and re-exposed the first title, reading family `Prof.` where `John Prof. MA` reads family `MA`. _pieces.tail_reading is now the one function: peel, chain, splice, peel again until the chain takes nothing, returning the walk, the chained pieces and the final peel. Assign calls it in place of its inline sequence; the reserve calls it on both views and compares the suffix lists, both subtractions deleted, with the join's refusal to take a chained piece said as the rule it is rather than left to the arithmetic. trailing_titles returns the count it LEAVES STANDING, the way peel_trailing counts, so the two compose without arithmetic at all. The two readings above are the only behaviour that moves: all seven fields, the ambiguity kinds and the recorded order over every corpus name and every cases.py literal are identical, 2674 names on both trees, none moved. Both take case rows. The helper costs one frame on the common path, paid by _effective_order taking the name-piece indices instead of a list built for it; the reference name stays at 416 parse / 453 facade on 3.11. Also in this round: leading_titles' floor as a for/else; the family-comma walk's `titled` as a parameter rather than a closure over a rebound local, one "previous kept piece" helper for both `prev` and the name's end, and the placement pass reading the first pass's own memo where the chain took nothing (-7 frames on `Smith, John Quincy`); _lexicon's fold factored so "the last word of the FOLDED key" is structural; suffixes.py's unreachable third assert folded into the disjointness one; duplicated prose in _group, _post_rules, compare.py and test_ledger_guards reduced to one copy; and the `Sir Jr` example and case row dropped, `Dr Jr` reading by the same branches -- which moves two ledger rosters (7 -> 6, 4 -> 3). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- AGENTS.md | 2 +- docs/design/decisions.md | 11 +- docs/design/rules.md | 16 +- docs/release_log.rst | 2 +- nameparser/_lexicon.py | 67 ++++---- nameparser/_pipeline/_assign.py | 220 ++++++++++++++------------ nameparser/_pipeline/_group.py | 81 +++++----- nameparser/_pipeline/_pieces.py | 106 ++++++++++--- nameparser/_pipeline/_post_rules.py | 16 +- nameparser/config/suffixes.py | 22 +-- tests/v2/cases.py | 72 ++++++--- tests/v2/pipeline/test_pieces.py | 34 ++-- tests/v2/test_ledger_guards.py | 82 +++------- tests/v2/test_lexicon.py | 9 +- tools/differential/compare.py | 45 ++---- tools/differential/corpus_rules.jsonl | 1 - 16 files changed, 426 insertions(+), 360 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a811e883..24c297de 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -370,7 +370,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **`Lexicon.given_name_titles` is deliberately unvalidated against `titles` — do not add a check** — the lookup key is the space-joined run of `Role.TITLE` tokens, built by the parse — matched whole OR by that key's LAST word since #489, a run addressing the way its final title does — and a conjunction inside a run is itself tagged `Role.TITLE`, so `"sir and dame"` is a matchable key whose middle word lives in `conjunctions`. A whole-entry check rejected multi-word entries; a per-word check rejected that one. No static relation over the vocabulary sets decides reachability. An unreachable entry is inert — nothing consults it and nothing misparses — so the condition being "guarded" costs the user nothing while each guard cost a working configuration. If a diagnostic is wanted, it must be non-blocking — the multi-word UserWarning in `_normset` is the shipped example; a raise remains wrong. Note the SHIPPED data cannot carry a multi-word given-name title without a spurious warning: `GIVEN_NAME_TITLES ⊆ TITLES` puts the entry in `titles` too, which is per-word warned; user-supplied v2 Lexicons are unaffected (`add(given_name_titles=...)` alone is silent). -**`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_title_key` joins per-word `_normalize` and DROPS words that fold away; keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". +**`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_fold_words` runs `_normalize` per word and DROPS the words that fold away (`_title_key` is that list space-joined, and `_run_addresses_by_given` reads the list itself, so its last-word arm is the last word of the FOLDED key by construction); keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version". **Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over ten shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The ten shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — the other nine are pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured); measure before pruning one. A stage gated on an opt-in `Policy` field needs a `_POLICY_SHAPES` entry instead, since bare `parse()` never enters it — and that table's rows carry a **reachability probe** run before the measurement, because a precedence change can quietly stop the shape reaching the stage and leave a green test measuring a no-op (`_POLICY_SHAPES` is also asserted non-empty: an empty `parametrize` is a skip, not a failure, so deleting its last row would retire the guard silently). diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 1e2374f8..67f71c44 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -416,10 +416,10 @@ Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparse ### H3 — the title run's floor - 2026-09-08 (the #316/#489 bundle) — the leading title run leaves a name word standing, and a post-nominal is not one. Derek's question is what opened it, verbatim: "Why does `Dr King Jr` need to parse `king` differently than `Dr King`? Jr is a recognized suffix, so it could not count as a following name." The mechanism behind the old reading is an ORDER: the leading peel runs before the trailing suffix peel and its only floor was "a title needs a following piece", so it took `Dr King` whole and left `Jr` to be the name — title `Dr King`, family `Jr`, no suffix at all. The floor added here is a second one, asked of the run's last word: where everything behind the run is a suffix piece and that word is not itself suffix vocabulary, the run gives it back. `Dr King Jr` now reads title `Dr`, family `King`, suffix `Jr` — what v1 wanted (#v1-xfail-triage), what rules.md#S2's descriptive note predicted, and what the comma spelling `King, Dr Jr` has always given. -- Measured over the differential corpus glob as it stood at the fix — 1123 distinct names, 1263 rows, before this bundle's own rules.md examples entered corpus_rules.jsonl and made it 1136 and 1284 — tree against the same tree with the floor removed: exactly one name changes, `Dr King Jr`. FOUR after the docs commit, which puts `Dr. King MD`, `Dr Jr` and `Sir Jr` in corpus_rules.jsonl as examples of the floor and its edge — re-measured there, and `Prince of Wales Jr` enters with them and does NOT move, which is the one-word gate witnessed in the corpus. The case row `Dr. King MD` moves the same way. Recompute by parsing every name in `tools/differential/corpus*.jsonl` twice, once on the tree and once with `_pieces.leading_titles` monkeypatched back to the bare "a title needs a following piece" loop, and diffing the seven name fields plus `ambiguities`; 1.4.0 gives the degenerate reading for both names, so the change diffs at every baseline. +- Measured over the differential corpus glob as it stood at the fix — 1123 distinct names, 1263 rows, before this bundle's own rules.md examples entered corpus_rules.jsonl and made it 1136 and 1284 — tree against the same tree with the floor removed: exactly one name changes, `Dr King Jr`. FOUR after the docs commit, which puts `Dr. King MD`, `Dr Jr` and `Sir Jr` in corpus_rules.jsonl as examples of the floor and its edge (THREE from 2026-09-09, the /simplify round having taken `Sir Jr` out of rules.md — see #H5 — where `Dr Jr` pins the same reading) — re-measured there, and `Prince of Wales Jr` enters with them and does NOT move, which is the one-word gate witnessed in the corpus. The case row `Dr. King MD` moves the same way. Recompute by parsing every name in `tools/differential/corpus*.jsonl` twice, once on the tree and once with `_pieces.leading_titles` monkeypatched back to the bare "a title needs a following piece" loop, and diffing the seven name fields plus `ambiguities`; 1.4.0 gives the degenerate reading for both names, so the change diffs at every baseline. - The word given back must be a NAME CANDIDATE, and that is what leaves the degenerate inputs alone. All-suffix inputs are untouched because the run's last word is itself suffix vocabulary — `MD DDS` keeps title `MD`, family `DDS`, and `Jr. Ph. D.` keeps title `Jr.`, suffix `Ph. D.` — and all-title inputs are untouched because there is no rest for the floor to read: `Dr.`, `Marquess of Bath` and `Coach` are unchanged. - ONE WORD is the gate, and it is a Task 1 review outcome rather than the drafting's shape. A joined title unit given back would lose the title entirely, which is worse than the reading it replaces: `Prince of Wales Jr` keeps title `Prince of Wales`, family `Jr` rather than becoming given `Prince of Wales` with no title at all. `Lord Chancellor Jr`, whose run is two separate words, does move — title `Lord`, family `Chancellor`, suffix `Jr`. -- The edge accepted: where the run's whole content is the word given back, no title is left to make the reading H1's, so the word stands as the NAME rather than as the family. `Dr Jr` reads given `Dr`, suffix `Jr` and `Sir Jr` given `Sir`, suffix `Jr`, both reporting `title-or-name` by H4's lone-title-word convention. That is the residue rules.md#S2's note is now scoped to. +- The edge accepted: where the run's whole content is the word given back, no title is left to make the reading H1's, so the word stands as the NAME rather than as the family. `Dr Jr` reads given `Dr`, suffix `Jr` and `Sir Jr` given `Sir`, suffix `Jr`, both reporting `title-or-name` by H4's lone-title-word convention (2026-09-09: `Sir Jr` is no longer written down anywhere but this sentence — once the floor empties the run no branch reads `vocab:given-title`, so the two readings are one and `Dr Jr` carries it). That is the residue rules.md#S2's note is now scoped to. - Two residuals the floor does not reach, recorded because they look like misses and are not. A run whose LAST word is itself suffix vocabulary is not given back, so `Dr King MD PhD` still reads title `Dr King MD`, family `PhD` — the floor asks about one word, not about the run's contents. And the floor asks `is_suffix_piece`, which vetoes a bare initial-shaped numeral, so `Dr King V` keeps the whole run as the title and reads given `V` (`king` being a given-name title and the run's last word, which is #H1's 2026-09-08 amendment), where `Dr Smith V` reads family `Smith`, suffix `V`. That numeral fork is #401/#421's territory and is deliberately outside this floor. - The floor lands in `leading_titles` rather than in assign because that predicate is the ONE answer to where the leading run ends (mechanisms.md#ONE-PREDICATE-PER-QUESTION): assign sets the roles from it and group's chain guard reads the same count, so a floor in assign would have given the two sites different runs. One cost measured and one shape forced by it: the branch's entry gate is two inline tag reads rather than `is_suffix_piece`, because `leading_titles` runs four times per parse and asking the authoritative predicate first cost 8 calls per parse of the reference name against a band with room for two (#parse-cost). The tag reads are the cheapest NECESSARY condition for the next piece to be a suffix piece at all, so an ordinary titled name leaves the branch without entering a frame; `is_suffix_piece` stays the predicate that ANSWERS. Frame delta measured at zero on both entry points. @@ -451,6 +451,13 @@ Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparse - **The inline frame-free gates at the two walk sites were REMOVED in review**, and this is the ONE-PREDICATE-PER-QUESTION half of the entry. Each site had a cheap inline test written to match the walk's own first condition; that is a second implementation of the question, and the measurement that justified it did not survive re-running. The band test runs early in a session where the facade sits at 453, so the "one frame of headroom" claim did not reproduce. With the gates gone the walk costs +1 frame on each entry point, inside the plan's target of two and inside `test_facade_cost_stays_within_its_band`. The walk's own cheapness is where the saving lives instead: the abbreviation-shape test is a compiled regex (a C call, no Python frame) and runs BEFORE the vocabulary call, and almost no name ends in a period-marked word, so the ordinary parse pays one match and stops. - **A pre-existing detail mismatch, recorded and NOT fixed.** #H4's join shape reports `title-or-name` with a `detail` that says the unit was "read as a given name by convention", while under the default order H1 retags the unit to the family — so `Dr. John of Prince` reports that text with the unit in `family`. This rule adds a second input with the same mismatch, `John of Prince Prof.`, and fixes neither: the wording predates this bundle, the fork the kind reports is title-versus-name which no field answers either way, and #H4 already records why the detail names no field for the peel shape. - 2026-09-09 (review round 2) — **A1's transparency holds for the ONE-name-word case too, and the scope written a day earlier was the defect rather than the boundary.** A1 was scoped that morning to inputs where a name word still stands on both sides of the chain, handing the one-word case to #H1 unconditionally: "the title behind that word decides its field as a title in front of it would". #H1 was then reading BOTH ends of the name as one title run, so `Sir John Prof.` read family `John` where `Sir John` reads given `John` — the trailing title changing what the leading one addressed by, which is exactly the non-transparency A1 denies. The clause is now conditioned on there being no run in FRONT of the word: where one stands it addresses and the chained trailing title only joins the title field, so `X Prof.` reads as `X` plus the title for one name word as for two. Where no run stands in front the trailing one still decides, unchanged — `Smith Prof.` reads family `Smith`, `Smith Sir.` given `Smith`. Nothing in the walk moved; the fix is entirely in which run H1 keys, and #H1's 2026-09-09 entry carries it, the measurement and the mutation. The #P5 clause below reads the same way afterwards and gains its second half: a bound given-name word behind a trailing title joins as it joins with the title absent AND lands in the same field. Said of the COMMA-LESS writing, and the review that found the H1 defect measured why the qualifier is needed: after a family comma the reserve reads no peel at all, so the join fires over the trailing title word and takes it into the given name — `Berg, abdul Prof.` reads given `abdul Prof.` where `Berg, John Prof.` reads title `Prof.`. That is PARITY (1.4.0 reads it the same, measured) and a gap in the segment path rather than a boundary of the rule, so it is recorded rather than fixed here: the clause carries the scope, and the case row `title_word_trailing_behind_a_bound_pair_after_a_comma` pins it without making it normative. Tracked as part of #316. +- 2026-09-09 (the /simplify round on the bundle's PR) — **the trailing peel and the title chain are ONE reading run to a FIXED POINT, and A1's "ONE peel runs over what stands" was a truncation of it.** Two contrived readings contradicted clauses this section states, both found by an altitude review reading the code against the rule rather than by a test. (1) THE SUBTRACTION MODEL. #P5's reserve modelled assign's reading as the peel's count minus the chain's take, on each of the two views the join compares. That arithmetic equals assign's answer only while the chain's take does not change what the peel would do, and at #S2's bare-ambiguous reserve it does: `abdul rahman MA` reads given `abdul`, family `rahman`, suffix `MA` — the join declining because peeling the acronym unjoined and not joined is a suffix reading the join would change — while `abdul rahman MA Prof.` read given `abdul rahman`, family `MA`, the subtraction having counted `MA` a name word to spare. The Accepted clause below, "a bound given-name word behind one joins exactly as it joins with the title absent", was false as stated. (2) THE TRUNCATED FIXED POINT. The re-peel ran ONCE, so a second title was read half way: `John Prof. MA` reads title `Prof.`, given `John`, family `MA`, and `John Prof. MA Prof.` read title `Prof.`, given `John`, family `Prof.`, suffix `MA` — the re-peel un-peeled the acronym and re-exposed the FIRST title, which nothing then took. Both measured on the branch tip, 2026-09-09. +- **The fix is one function, `_pieces.tail_reading`:** peel, chain, splice the chained pieces out, peel again over what is left, until the chain takes nothing. Assign calls it in place of its inline sequence and all its reports still come from the final peel; the reserve calls it on both views and compares the suffix lists it returns, and both subtractions are deleted. `abdul rahman MA Prof.` now reads given `abdul`, family `rahman`, suffix `MA` under title `Prof.`, and `John Prof. MA Prof.` reads title `Prof. Prof.`, given `John`, family `MA`. One further clause had to be said in the code rather than left to the arithmetic: the join must not take a piece the chain takes (#P5 — "a trailing title word the run takes is no word to spare"), which the subtraction had enforced by accident, by leaving that piece in the tail it compared. Without it `Sir abdul Prof.` joins the title into the given name; with it the reading is title `Sir Prof.`, given `abdul`, as before. `trailing_titles` returns the count it LEAVES STANDING in the same edit, the way `peel_trailing` counts, so the two compose at the call site without arithmetic at all. +- **Cost, and what paid it.** The shared helper is one frame on the common path, where the inline sequence was none. It is paid by `_effective_order` taking the segment's pieces and the name-piece indices — the shape every piece-layer predicate takes — instead of a list built for it, which on 3.11 is a comprehension frame on every parse; the reference name stays at 416 parse / 453 facade. The helper returns a bare 3-tuple rather than a NamedTuple for the same budget: a NamedTuple's `__new__` is itself a frame, measured at +1 here. +- **How big the class is, measured as the rule states it.** The two readings named above are the shapes that made the defects visible; the class they belong to is every input carrying TWO trailing period-marked title words, and it is 88 shapes wide over a generated sweep. Recipe: take the alphabet `John Smith abdul rahman Prof. Dr. Sir MA Jr. V Xyz. King de née`, form every 3- and 4-word arrangement ending in `Prof.` or `Dr.` whose spelling WITHOUT that last word still fills two of given/middle/family (3,210 of them — the two-or-more-name-words scope the A1 clause is stated for), and compare the seven fields other than `title` against that shorter spelling. rules.md#H5 says they must be equal. On the branch tip 786 differ; on this commit 698 do, none of them new, and all 698 are the P2/M2 residue the rule already accepts as a boundary — a particle chain or a maiden marker took the trailing word before the H5 chain could see it (`John de Prof.`, `Mary née Prof.`). The 88 the fix closes are all the two-title shape. Over the same sweep's 82,712 inputs (both comma-less and family-comma spellings, 2 to 4 words) exactly 156 parses move, and every one ends in that shape. + +- **Nothing else moved, and it was measured rather than argued.** Snapshot all seven fields, the ambiguity kinds and the recorded `order` for every name in `tools/differential/corpus*.jsonl` plus every string literal in `tests/v2/cases.py` (`ast.walk` over that file, which sweeps up the notes and the case ids too — harmless, they parse like anything else), on the branch tip and on the fix, and diff: 2682 strings before and 2681 after, 2674 of them on both trees, and NONE of those 2674 reads differently. The seven that differ are the case rows this round added and dropped, not parses that moved. The two readings above are inputs no corpus holds — as is every other member of the class the bullet below measures — and each takes a case row (`title_word_trailing_run_is_read_to_a_fixed_point`, `title_word_trailing_behind_a_bound_pair_at_the_peel_reserve`). The same round dropped the `Sir Jr` example from rules.md#S2 and its case row: it reads by the same branches as `Dr Jr`, the same roles and the same reported kind, the run being empty by then and no branch reading `vocab:given-title` — so the pair pinned one reading twice. Its leaving corpus_rules.jsonl moves two ledger rosters: the jr suffix-routing rule 7 → 6 corpus names in the 1.4.0 ledger, and the #489 peel-floor rule 4 → 3 in all four. + ### W1 — unspaced CJK division diff --git a/docs/design/rules.md b/docs/design/rules.md index bece775d..ba455a15 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -549,9 +549,11 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" particle's attachment (P6) sees the name. What there is to spare is what assign will leave: the join is tried on the pieces as it would - leave them, assign's trailing peel (S2) is read over that and its - trailing title run (H5) over what that peel leaves, and the name - words the two of them leave are the words to spare — a trailing + leave them, and the same reading assign runs over them — its + trailing peel (S2) and its trailing title run (H5), each read + over what the other leaves until neither takes anything more — + is read over that view, the name words it leaves being the words + to spare. A trailing roman numeral, or a bare acronym the peel takes, or a trailing title word the run takes, is no word to spare. The join joins two name words into one and @@ -836,12 +838,14 @@ S2. Rationale: generational suffixes and credentials are recognized and is small: where the run's whole content is the word given back, no title is left to name anybody, so the word stands as the name rather than as the family — `Dr Jr` reads given `Dr`, - suffix `Jr` and `Sir Jr` given `Sir`, suffix `Jr`, both - reporting `title-or-name` (H4). The vocabulary half is decided + suffix `Jr`, reporting `title-or-name` (H4), and `Sir Jr`, the + spelling this note first named, reads by the same branches — + given `Sir`, suffix `Jr`, the same kind reported — the run + being empty by then and no branch reading `vocab:given-title`. + The vocabulary half is decided and unchanged (decisions.md#v1-xfail-triage: `king` stays a title, for the addressing forms). "Dr Jr" → suffix="Jr" - "Sir Jr" → suffix="Jr" interacts: H1, H2, H3, H5, C1 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py S3. Rationale: credentials are often written run together with diff --git a/docs/release_log.rst b/docs/release_log.rst index 718a8f75..67b2b31c 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -20,7 +20,7 @@ Release Log - **Fix a title run addressing by its first title rather than its last.** ``HumanName("Her Majesty Queen Elizabeth")`` gives first ``Elizabeth`` with an empty last name, where every release since 1.4.0 gave last ``Elizabeth``. Several titles written together are one form of address and the one that does the addressing is the last, so the run is now matched whole or by its last word: ``Reverend Mother Teresa``, ``Dr. Sir John`` and ``Mr Sir John`` move the same way, and ``Sir Sheikh abdul rahman`` gives first ``abdul rahman``. What does NOT move is a run whose last word addresses by surname: ``His Excellency Lord Duncan`` still gives last ``Duncan`` and ``Her Royal Highness Princess Anne`` last ``Anne``, ``lord`` and ``princess`` not being given-name titles -- a vocabulary question with its own argument, filed separately (#519). A caller's multi-word entry still matches as a phrase. Two names in the differential corpora read differently for this rule. See the ``H1`` entry of ``docs/design/decisions.md`` (closes #489) - - **Fix the leading title peel taking a name word and leaving a post-nominal to be the name.** ``HumanName("Dr King Jr")`` gives title ``Dr``, last ``King``, suffix ``Jr``, where every release since 1.4.0 gave title ``Dr King``, last ``Jr`` and no suffix at all; ``Dr. King MD`` moves the same way, and both now read as the comma spelling ``King, Dr Jr`` always has. A title addresses somebody, so the run leaves a name word standing and a post-nominal is not one. A name that is nothing but titles or nothing but post-nominals is untouched, the word given back having to be a name candidate: ``Marquess of Bath``, ``MD DDS`` and ``Jr. Ph. D.`` are unchanged, and so is a title written as one joined unit -- ``Prince of Wales Jr`` keeps title ``Prince of Wales`` rather than losing the title to make a name. Where the run's whole content is the word given back there is no title left, so ``Dr Jr`` gives first ``Dr``, suffix ``Jr`` and reports a title-or-name ambiguity. Four names in the differential corpora read differently for this rule. See the ``H3`` entry of ``docs/design/decisions.md`` + - **Fix the leading title peel taking a name word and leaving a post-nominal to be the name.** ``HumanName("Dr King Jr")`` gives title ``Dr``, last ``King``, suffix ``Jr``, where every release since 1.4.0 gave title ``Dr King``, last ``Jr`` and no suffix at all; ``Dr. King MD`` moves the same way, and both now read as the comma spelling ``King, Dr Jr`` always has. A title addresses somebody, so the run leaves a name word standing and a post-nominal is not one. A name that is nothing but titles or nothing but post-nominals is untouched, the word given back having to be a name candidate: ``Marquess of Bath``, ``MD DDS`` and ``Jr. Ph. D.`` are unchanged, and so is a title written as one joined unit -- ``Prince of Wales Jr`` keeps title ``Prince of Wales`` rather than losing the title to make a name. Where the run's whole content is the word given back there is no title left, so ``Dr Jr`` gives first ``Dr``, suffix ``Jr`` and reports a title-or-name ambiguity. Three names in the differential corpora read differently for this rule. See the ``H3`` entry of ``docs/design/decisions.md`` - **Fix a trailing abbreviated title reading as a name word.** ``HumanName("John Smith Prof.")`` gives title ``Prof.``, first ``John``, last ``Smith``, where every release since 1.4.0 gave last ``Prof.`` and lost the surname; ``John Smith Mr.``, ``John Smith Rev.``, ``John Smith Dr.`` and ``Andrew Perkins (Mgr.)`` move the same way. A run chains from the end (``John Smith Prof. Dr.`` gives title ``Prof. Dr.``), a leading title keeps its place (``Dr. John Smith Prof.`` gives title ``Dr. Prof.``), and the comma forms agree with the bare ones now -- ``Smith, John Prof.`` gives title ``Prof.``, first ``John``, last ``Smith`` where it gave middle ``Prof.`` at every release. The trailing title is transparent to the post-nominal reading, so ``John Smith Jr. Prof.`` gives suffix ``Jr.`` rather than promoting the generational suffix to the last name. What does NOT move: an unlisted abbreviation (``John Smith Xyz.`` keeps last ``Xyz.``), a bare title word (``John Smith Sir``, ``Mary Jane King``) and a post-nominal (``John Smith Esq.``). Only a listed title word wearing the abbreviation period is claimed -- the leading slot infers a title from the shape alone, the trailing slot never does. The reach is the whole title vocabulary, ordinary surnames in it included, so a period written behind one of them takes it out of the name: ``Mary Jane King.`` gives title ``King.``, first ``Mary``, last ``Jane``, where the bare ``Mary Jane King`` keeps last ``King``. That is accepted rather than prevented -- the period is a writing convention and not evidence about the word, and the bare spelling is what the trailing slot is protected from. A title run in FRONT of the name still does the addressing, so the trailing title is transparent to that reading too: ``Sir John Prof.`` gives title ``Sir Prof.``, first ``John`` -- ``Sir John`` plus a title -- and ``Dr. Smith Sir.`` gives title ``Dr. Sir.``, last ``Smith``. Where NO title stands in front, the trailing one decides the field: ``Smith Sir.`` gives first ``Smith`` and ``Smith Prof.`` gives last ``Smith``. Sixteen names in the differential corpora read differently for this rule, and a seventeenth for the same argument in a native script: ``毛 泽东 Dr.`` gives title ``Dr.``, first ``泽东``, last ``毛``, where 2.2.0 gave last ``Dr.`` and lost the family-first order. See the ``H5`` entry of ``docs/design/decisions.md`` (closes #316) diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 88486d64..ff654dd1 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -127,27 +127,45 @@ def _normalize(word: str) -> str: word = stripped -def _title_key(words: Iterable[str]) -> str: - """The given_name_titles lookup key for a run of title words. +def _fold_words(words: Iterable[str]) -> list[str]: + """The words of a title run, folded for storage and lookup. A multi-word title is matched as one key ('lt col'), so the fold has - to run per word and rejoin -- _normalize on the whole phrase would - leave interior periods. Defined once because it is built at match - time (_run_addresses_by_given, which is how post_rules' H1 and - group's P5 licence both reach it) and at translation time - (_config_shim's first_name_titles), and a divergence between them - fails silently: the entry simply stops matching. - - Words that fold away are DROPPED, not joined as empty. Keeping the - gap makes the fold non-idempotent -- 'lt .' would store 'lt ', which + to run per word -- _normalize on the whole phrase would leave + interior periods. + + Words that fold away are DROPPED, not kept as empty. Keeping the + gap makes the key non-idempotent -- 'lt .' would store 'lt ', which match time can never build: a run CAN carry a lone '.' (the conjunction merge puts one there), but the fold drops the empty word, so the key is 'lt' and the stored entry is inert. Storage re-runs this fold on unpickle and on every dataclasses.replace, so a value that changes under a second pass is one Lexicon later rejects as "not written by this version". _normalize converges for the same - reason; so must anything built on top of it.""" - return " ".join(filter(None, (_normalize(w) for w in words))) + reason; so must anything built on top of it. + + A LIST, so that _run_addresses_by_given's last-word arm can be the + last word of this fold rather than a re-split of the joined key -- + "the last word of the FOLDED key" is then structural, and the two + arms read one fold between them. + + map/filter rather than a comprehension: both are C calls where a + comprehension is a Python frame on 3.11, and this runs on the parse + path for every name carrying a title run (decisions.md#parse-cost). + """ + return list(filter(None, map(_normalize, words))) + + +def _title_key(words: Iterable[str]) -> str: + """The given_name_titles lookup key for a run of title words: the + folded words, space-joined. + + Defined once because it is built at match time + (_run_addresses_by_given, which is how post_rules' H1 and group's + P5 licence both reach it) and at translation time (_config_shim's + first_name_titles), and a divergence between them fails silently: + the entry simply stops matching.""" + return " ".join(_fold_words(words)) def _run_addresses_by_given(words: Iterable[str], @@ -164,8 +182,9 @@ def _run_addresses_by_given(words: Iterable[str], so 'Her Majesty Queen' addresses by given name because 'queen' does. The two arms are asked of one key and neither is the other's fallback -- the `or` short-circuits but decides nothing, since for - a one-word run the key IS its last word. The last word is the last - word of the FOLDED key, not of the raw run, so a run token that + a one-word run the key IS its last word. Both arms read one + _fold_words list, so the last word is the last word of the FOLDED + key by construction rather than by agreement, and a run token that folds away cannot empty that arm: the conjunction merge can put a lone '.' in the run ('Sir and . John'), and the fold drops it. What the drop leaves as the last word can then be the CONJUNCTION @@ -175,16 +194,6 @@ def _run_addresses_by_given(words: Iterable[str], connective, and a caller who stored one would be asking for it (measured 2026-09-09). - Reading the FOLDED key's last word rather than the raw run's is a - defensive branch and a measured-inert one: over 191,146 generated - inputs it is reached 42,413 times and the two never differ, and - swapping it for the raw word changes no parse (2026-09-09) -- not - even on the lone '.' above, whose raw form folds to the empty - string and misses the vocabulary just as 'and' does. Kept as the - honest shape: the key is what the vocabulary is stored as, so the - key is what the lookup reads, and a caller's entry is the thing - that could make the two differ. - The whole-run arm is what keeps a caller's multi-word phrase entry working: 'lt col' is stored as one key and matched as one run. Over the SHIPPED vocabulary it is dead -- every shipped entry is a single @@ -203,8 +212,12 @@ def _run_addresses_by_given(words: Iterable[str], The vocabulary is passed in rather than read off a default: a caller's own Lexicon is the one that has to be consulted, and this module is where Lexicon is defined.""" - key = _title_key(words) - return key in vocabulary or key.rpartition(" ")[2] in vocabulary + folded = _fold_words(words) + if not folded: + # every word folded away, so there is no key and no last word: + # the joined spelling built "" here, which matched nothing + return False + return " ".join(folded) in vocabulary or folded[-1] in vocabulary def _reject_buffer(value: object, label: str, plural: str) -> None: diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 63c9d627..7ea9929a 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -19,10 +19,12 @@ last name-position piece, the rest are suffixes. Behind that peel a trailing run of period-marked title words chains into the title from the end, leaving one name piece standing; where the run TAKES -something the first peel was only provisional and runs again, once, -over the pieces with the titled ones spliced out, so a trailing -title is transparent to it. Where the run takes nothing -- almost -every name -- the first peel is the only one and its answer stands. +something the peel was only provisional and runs again over the pieces +with the titled ones spliced out, the two alternating until the run +takes nothing, so a trailing title is transparent to the suffix +reading however many titles are written (_pieces.tail_reading). Where +the run takes nothing -- almost every name -- the first peel is the +only one and its answer stands. The v1 single-name+nickname rule lives here (decisions.md#N3): a nonempty nickname beside exactly one piece in total puts that piece in FAMILY. @@ -63,8 +65,8 @@ effective_script, is_suffix_lenient, resolve_script_set, ) from nameparser._pipeline._pieces import ( - is_suffix_piece, leading_titles, peel_trailing, peel_walk, - segment_suffix_reading, trailing_titles, + is_suffix_piece, leading_titles, peel_walk, segment_suffix_reading, + tail_reading, trailing_titles, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, _NEVER_FLIPPED, @@ -125,7 +127,8 @@ class EffectiveOrder(NamedTuple): # order the caller declared; a wholly-katakana name keeps the declared # order" (history: decisions.md#W4) def _effective_order(policy: Policy, - pieces: list[tuple[int, ...]], + pieces: Sequence[tuple[int, ...]], + name_pieces: Sequence[int], tokens: list[WorkToken], *, dot_divided: bool) -> EffectiveOrder: """script_orders resolution (#271): when every name piece is @@ -153,6 +156,12 @@ def _effective_order(policy: Policy, resolves the SCRIPT for a single token. This function calls that one per token below. + Takes the segment's pieces and WHICH of them the name kept, the + shape every piece-layer predicate takes: a caller holding those + indices had to build a second list of the same pieces to hand + over otherwise, which on 3.11 is a comprehension frame on every + parse (decisions.md#parse-cost). + Returns an EffectiveOrder: the order triple, and `by_script` set only on the one path where an entry answered. Every fallback below is a script rule DECLINING, and reports it as such. @@ -169,8 +178,8 @@ def _effective_order(policy: Policy, # piece and a Hiragana piece only license together, never one at a # time), so resolution is deferred to resolve_script_set below. found: set[Script] = set() - for piece in pieces: - for i in piece: + for piece_idx in name_pieces: + for i in pieces[piece_idx]: script = effective_script(tokens[i].text) if script is None: # Latin, mixed, or a script with no entry: never a key @@ -235,10 +244,14 @@ def _assign_main(seg_idx: int, state: ParseState, # group-flagged suffix pieces (the ph-d merge) are suffixes at ANY # position -- v1's fix_phd extracted the credential from the string # before parsing, so position never mattered (PR review I3). - # Walked rather than collected first: a comprehension is a frame of - # its own on 3.11 and the list was never read again, which is one - # frame back against the one the H5 walk below costs - # (decisions.md#parse-cost). + # Walked rather than collected first: the list was never read + # again, and on 3.11 a comprehension is a frame of its own, which + # is one frame back against the one the H5 reading below costs. A + # 3.11 FACT and not a portable one: PEP 709 inlines comprehensions + # from 3.12, where this tree's reference parse costs 395 written + # either way (measured 2026-09-09, against 416 and 417 on 3.11). + # 3.11 is the interpreter the band is quoted for, so 3.11 is what + # the shape is chosen on (decisions.md#parse-cost). for k in range(n, len(pieces)): if "suffix" in ptags[k]: _set_roles(tokens, pieces[k], Role.SUFFIX) @@ -254,52 +267,31 @@ def _assign_main(seg_idx: int, state: ParseState, if len(pieces) == 1 and len(rest) == 1 and has_nickname: _set_roles(tokens, pieces[rest[0]], Role.FAMILY) return None - # peel the trailing suffix run: k = first index in rest from which - # every piece is a suffix. The walk is _pieces.peel_trailing since - # #425 -- one walk, shared with the bound-given reserve, and - # documented there. Every bare ambiguous acronym it had to resolve - # is one coin-flip each, in either direction, so the report - # collects rather than overwrites. Deferred to after assignment - # because the wording reads the role back, and which role "not - # peeled" means depends on name_order. (The roman-numeral fork - # needs no such deferral and is reported here.) + # rules.md#S2's trailing peel and rules.md#H5's title chain, read + # together to their fixed point by _pieces.tail_reading -- one + # function since the /simplify round, shared with the bound-given + # reserve (P5), which must count the name words this leaves. # - # This peel is provisional only where the H5 walk below TAKES - # something: the walk can remove the very word that stopped the - # peel, so it is asked again over the pieces as they then stand, - # and that second answer is the only one that places a piece or - # reports a fork. Where the walk takes nothing -- almost every - # name, the walk's own first test being a period match that - # fails -- this peel is the only one, and its roles and its - # reports are the ones the name gets. - peeled = peel_trailing(rest, pieces, ptags, tokens) # rules.md#H5: "successive single words that wear the abbreviation # shape and are title vocabulary chain into the title from the end, - # leaving one name word standing" - # -- read over what the suffix - # peel left and set BEFORE _name_positions, so the shortened list is - # what the positional read and the script test both see (a trailing - # Latin title must not make a wholly-CJK name look mixed-script, the - # same reason the leading peel runs first). Placed ahead of the - # bare-suffix carve-out because with no name piece left there is - # nothing for the walk to read: its floor returns 0 on an empty - # list, so the branch below is reached exactly as before. - titled_tail = trailing_titles(rest[:peeled.names], pieces, ptags, - tokens) - if titled_tail: - cut = peeled.names - titled_tail - for piece_idx in rest[cut:peeled.names]: - _set_roles(tokens, pieces[piece_idx], Role.TITLE) - # The title is TRANSPARENT to the suffix peel: the pieces the - # walk took are spliced out and the peel runs once over what - # is left -- the name pieces the walk kept, then the pieces - # the provisional peel had taken, in original order -- so - # 'X Prof. Y' reads exactly as 'X Y' reads, plus the title. - # 'John Smith Jr. Prof.' reads suffix 'Jr.' where a single - # pass promoted a generational suffix to the family name, and - # 'John Prof. MA' reads the family 'MA' that 'John MA' reads. - rest = rest[:cut] + rest[peeled.names:] - peeled = peel_trailing(rest, pieces, ptags, tokens) + # leaving one name word standing" -- the titles are set BEFORE + # _name_positions, so the shortened list is what the positional + # read and the script test both see (a trailing Latin title must + # not make a wholly-CJK name look mixed-script, the same reason the + # leading peel runs first). Read ahead of the bare-suffix carve-out + # because with no name piece left there is nothing for the chain to + # read: its floor keeps 0 pieces of an empty list, so the branch + # below is reached exactly as before. + # + # Every bare ambiguous acronym the FINAL peel had to resolve is one + # coin-flip each, in either direction, so the report collects + # rather than overwrites. Deferred to after assignment because the + # wording reads the role back, and which role "not peeled" means + # depends on name_order. (The roman-numeral fork needs no such + # deferral and is reported here.) + rest, titled_tail, peeled = tail_reading(rest, pieces, ptags, tokens) + for piece_idx in titled_tail: + _set_roles(tokens, pieces[piece_idx], Role.TITLE) if peeled.numeral is not None: # a trailing single letter is a name part unless it happens # to be a roman numeral -- and V/X/I are ordinary middle @@ -314,11 +306,11 @@ def _assign_main(seg_idx: int, state: ParseState, if peeled.names == 0: # everything suffix-shaped after titles: first one is the name name_pieces, suffix_pieces = suffix_pieces[:1], suffix_pieces[1:] - # AFTER both peels, and load-bearing: the script test sees the NAME - # pieces only, so a Latin title or suffix ('Dr. 毛 泽东', '毛 泽东, - # PhD') cannot make a wholly-CJK name look mixed-script. - resolved = _effective_order(state.policy, - [pieces[i] for i in name_pieces], tokens, + # AFTER the whole tail reading, and load-bearing: the script test + # sees the NAME pieces only, so a Latin title or suffix ('Dr. 毛 + # 泽东', '毛 泽东, PhD') cannot make a wholly-CJK name look + # mixed-script. + resolved = _effective_order(state.policy, pieces, name_pieces, tokens, dot_divided=bool(state.interpunct_offsets)) order = resolved.order roles = _name_positions(order, len(name_pieces)) @@ -546,45 +538,66 @@ def assign(state: ParseState) -> ParseState: if len(state.segments) > 1: pieces = state.pieces[1] ptags = state.piece_tags[1] + # Both are the walk's, and both are empty on the gate's + # path below, which reads the whole segment as a + # credential run and leaves no piece for the walk to + # place. titled_idx: tuple[int, ...] = () + walkable: list[int] = [] - def reads_as_a_suffix(m: int, last: int) -> bool: + def previous_kept(m: int, titled: tuple[int, ...]) -> int: + """The piece before `m` that the H5 chain did NOT + take. Both readings this segment needs are that one: + the piece the lenient tail test measures against, and + -- asked of one past the end -- where the segment's + name ENDS, since a title the chain took is not where a + name ends (#144). One walk for both, so 'as if the + titled pieces were absent' cannot come to mean two + things. + + DEFENSIVE, and measured inert: over 191,146 generated + inputs the skip fired on 153 of the 140,227 walks the + lenient test's `prev` asked for, and deleting it + changed no parse among them (2026-09-09, on the shape + this replaced, where the name's END walked past the + same pieces in a copy of this loop). Kept because "as + if the titled pieces were absent" is the rule the + predicate below implements, and a caller's vocabulary + reaches shapes the sweep's word list does not -- an + inert branch is cheaper than a rule with a hole in it. + """ + m -= 1 + while m in titled: + m -= 1 + return m + + def reads_as_a_suffix(m: int, titled: tuple[int, ...]) -> bool: """Does this segment's walk read piece `m` as a suffix? Asked twice, and by one predicate rather than by two conditions written to match (mechanisms.md#ONE-PREDICATE-PER-QUESTION): once to - find the pieces the H5 title walk must not reach past, - and once by the walk order below, which is the site - that places them. `last` is where this segment's name - ends -- provisionally the segment's last piece, and - after the title walk the last piece the walk left, - since a title it took is not where a name ends. - - `titled_idx` is read at CALL time and is empty on the - first pass, which is what makes the second reading the - one 'as if the titled pieces were absent': the lenient - test's preceding piece skips them too. + find the pieces the H5 title chain must not reach + past, and once by the walk order below, which is the + site that places them. + + `titled` is a PARAMETER because the two passes hand it + different values -- () on the first, the chain's own + pieces on the second, which is what makes that second + reading the one 'as if the titled pieces were absent'. + A closure over the caller's local said the same thing, + but only by WHEN it was rebound. """ if is_suffix_piece(pieces[m], ptags[m], tokens): return True - # DEFENSIVE, and measured inert: the skip fires on 153 - # of 140,227 calls over 191,146 generated inputs, and - # deleting it changes no parse among them (2026-09-09). - # Kept because "as if the titled pieces were absent" is - # the rule this predicate implements, and a caller's - # vocabulary reaches shapes the sweep's word list does - # not -- an inert branch is cheaper than a rule with a - # hole in it. - prev = m - 1 - while prev in titled_idx: - prev -= 1 + prev = previous_kept(m, titled) # trailing piece of a two-part name is unambiguously # positioned: v1 accepts the lenient test there # ('Smith, John V' -> suffix='V', #144); with a third # comma part the trailing token is more likely a middle # initial, so strict only - return (m == last and len(state.segments) == 2 + return (m == previous_kept(len(pieces), titled) + and len(state.segments) == 2 and len(pieces[m]) == 1 and _reads_as_a_trailing_suffix( pieces[m], pieces[prev], ptags[prev], @@ -647,13 +660,11 @@ def reads_as_a_suffix(m: int, last: int) -> bool: # (24 inputs of that shape move, of 191,146 generated, # measured 2026-09-09). walkable = [k for k in range(n, len(pieces)) - if k == n or not reads_as_a_suffix( - k, len(pieces) - 1)] - taken = trailing_titles(walkable, pieces, ptags, tokens) - if taken: - titled_idx = tuple(walkable[len(walkable) - taken:]) - for k in titled_idx: - _set_roles(tokens, pieces[k], Role.TITLE) + if k == n or not reads_as_a_suffix(k, ())] + kept = trailing_titles(walkable, pieces, ptags, tokens) + titled_idx = tuple(walkable[kept:]) + for k in titled_idx: + _set_roles(tokens, pieces[k], Role.TITLE) # v1 walk order: the first non-title piece is ALWAYS the # given, before any suffix check -- 'Hardman, RN - CRNA' # keeps first='RN'. The one deliberate 2.0 deviation, @@ -665,20 +676,23 @@ def reads_as_a_suffix(m: int, last: int) -> bool: # so the walk here never meets the case. if n < len(pieces): _set_roles(tokens, pieces[n], Role.GIVEN) - # the walk's floor leaves a name piece standing, so the - # given above is never one of the pieces it took; what the - # walk DOES move is where this segment's name ends, and the - # lenient tail test below turns on that (#144) - last_kept = len(pieces) - 1 - while last_kept in titled_idx: - last_kept -= 1 + # The chain's floor leaves a name piece standing, so the + # given above is never one of the pieces it took. What the + # chain DOES move is where this segment's name ends, which + # the lenient tail test turns on (#144) -- so where it took + # something the question is re-asked with the pieces it + # left. Where it took nothing, `walkable` already IS this + # predicate's answer for every piece: the first pass's own + # memo, not a second spelling of the question, and it + # cannot have gone stale because nothing the chain does + # moved the end of the name. for m in range(n + 1, len(pieces)): if m in titled_idx: continue - if reads_as_a_suffix(m, last_kept): - _set_roles(tokens, pieces[m], Role.SUFFIX) - else: - _set_roles(tokens, pieces[m], Role.MIDDLE) + suffix_here = (reads_as_a_suffix(m, titled_idx) + if titled_idx else m not in walkable) + _set_roles(tokens, pieces[m], + Role.SUFFIX if suffix_here else Role.MIDDLE) if reading is not None and sum( 1 for k, piece in enumerate(fam_pieces) if not is_suffix_piece(piece, fam_tags[k], tokens)) > 1: diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index f0547ea5..c9817fee 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -45,8 +45,7 @@ from nameparser._lexicon import _run_addresses_by_given from nameparser._pipeline._pieces import ( is_leading_title, is_suffix_piece, is_title_piece, - leading_titles, peel_trailing, peel_walk, trailing_start, - trailing_titles, + leading_titles, peel_walk, tail_reading, trailing_start, ) from nameparser._pipeline._state import ( ParseState, PendingAmbiguity, Structure, WorkToken, @@ -712,19 +711,22 @@ def chain(tail: int) -> None: merge(fk, fk + 2, drop={"title"}) else: # rules.md#P5: "the join is tried on the pieces as it - # would leave them, assign's trailing peel (S2) is read - # over that and its trailing title run (H5) over what - # that peel leaves, and the name words the two of them - # leave are the words to spare" + # would leave them, and the same reading assign runs + # over them — its trailing peel (S2) and its trailing + # title run (H5), each read over what the other leaves + # until neither takes anything more — is read over that + # view, the name words it leaves being the words to + # spare" # (history: decisions.md#P5). The view is what # merge() builds -- the same slice assignment, the same - # joined_tags -- and the peel is assign's own, and so is - # the H5 walk read over what that peel leaves, so the - # reserve and the assignment cannot drift. Both halves - # are needed: the peel alone counted a trailing - # period-marked title word as a name word to spare, and - # 'Prof. abdul rahman Prof.' joined where - # 'Prof. abdul rahman' does not. And the join + # joined_tags -- and the reading is assign's own, the + # ONE function that runs the peel and the H5 chain to + # their fixed point (_pieces.tail_reading), so the + # reserve and the assignment cannot drift. Modelling it + # here as a subtraction instead is what let them: 'abdul + # rahman MA' declined the join and 'abdul rahman MA + # Prof.' took it, where H5 says the title changes + # nothing (decisions.md#H5, 2026-09-09). And the join # changes no suffix reading -- rules.md#P5: "a word the # peel reads as a suffix unjoined must read so joined, # or the join declines" -- compared as the peeled @@ -732,29 +734,32 @@ def chain(tail: int) -> None: # nothing joined, 'abdul Smith Ma' peels the acronym # unjoined and keeps it joined. Shapes pinned in # test_group.py. - rest = peel_walk(fk, ptags) - before = peel_trailing(rest, pieces, ptags, tokens) + rest, chain_took, before = tail_reading( + peel_walk(fk, ptags), pieces, ptags, tokens) view, view_tags = list(pieces), list(ptags) view[fk:fk + 2] = [pieces[fk] + pieces[fk + 1]] view_tags[fk:fk + 2] = [joined_tags(fk, fk + 2, drop={"title"})] - view_rest = peel_walk(fk, view_tags) - after = peel_trailing(view_rest, view, view_tags, tokens) - # rules.md#H5 -- assign's second peel runs over the - # pieces the trailing title walk LEFT, so a - # period-marked title word at the back is not one of the - # name words this counts. Read over the same list assign - # reads it over -- the name pieces the peel left -- on - # both sides, so the two views compare like with like: - # counting the title word made 'Sir abdul Prof.' join it - # into the given name. - before_names = before.names - trailing_titles( - rest[:before.names], pieces, ptags, tokens) - after_names = after.names - trailing_titles( - view_rest[:after.names], view, view_tags, tokens) + view_rest, _, after = tail_reading( + peel_walk(fk, view_tags), view, view_tags, tokens) same_suffixes = ( - [tuple(view[j]) for j in view_rest[after_names:]] - == [tuple(pieces[j]) for j in rest[before_names:]]) + [tuple(view[j]) for j in view_rest[after.names:]] + == [tuple(pieces[j]) for j in rest[before.names:]]) + # rules.md#P5: "a trailing roman numeral, or a bare + # acronym the peel takes, or a trailing title word the + # run takes, is no word to spare" -- and the join joins + # two NAME words, so a piece the chain takes is no more + # joinable than a marker or a suffix piece is: 'Sir + # abdul Prof.' reads title 'Sir Prof.', given 'abdul'. + # Read off the UNJOINED view, the one that still has + # the title as a piece of its own -- the join would + # swallow it, and a swallowed title is a title the + # joined view can no longer see. The suffix comparison + # alone said this while the counts were subtractions, + # by leaving the chained piece in the tail it compared; + # under the shared reading the chain takes it out of + # both views, so the rule is asked as the rule. + chained = fk + 1 in chain_took # A given-name title ahead of the bound word asserts # that a given name follows -- the assertion H1 reads # when it keeps "Sir John" a given name -- so behind @@ -773,14 +778,9 @@ def chain(tail: int) -> None: # Reading the leading run at both sites is what makes # that unreachable rather than merely unlikely: a # trailing title is behind the word, and neither site - # can see it. H2's unlisted - # abbreviations ride in the run either way. One is in - # no vocabulary by definition, so it never matches as - # the last-word key -- written as inputs, 'Xyz. Sir - # John' keys 'xyz sir' and matches on 'sir', while - # 'Sir Xyz. John' keys 'sir xyz' and matches on - # neither -- but a caller's phrase entry may contain - # one, and the whole-run arm is what matches that. + # can see it. What H2's unlisted abbreviations do + # inside such a run is the predicate's own business, + # and its docstring is where they are worked through. # The licence lifts the reserve for two name # WORDS: the piece the join would take must be one word # -- a particle chain is the family name P2 built ('Sir @@ -792,7 +792,8 @@ def chain(tail: int) -> None: for i in pieces[k]), given_name_titles)) reserve = BoundJoin.LENIENT if licensed else BoundJoin.STRICT - if same_suffixes and after_names >= reserve: + if (not chained and same_suffixes + and after.names >= reserve): # the pair is a given name whatever tag the word # carried (rules.md#P5); joined_tags says why the # title tag is dropped. Pinned in test_group.py. diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 325da96c..384a1632 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -29,7 +29,10 @@ peel_walk, peel_trailing and trailing_start together -- though only the first two cross a stage boundary. trailing_titles joins them because it reads what that peel left: the two answer one question -between them, where the tail of a name stops being the name. +between them, where the tail of a name stops being the name -- and +tail_reading is that one question, running them against each other to +their fixed point for the two stages that must not disagree about the +answer. Layering: imports _state and _vocab only; _group and _assign import it, and neither of the two it imports imports it back. @@ -90,10 +93,10 @@ def leading_titles(pieces: Sequence[Sequence[int]], segment is one title (v1 parity). And the run gives back its last piece when that piece is a name candidate: where everything behind the run is suffix pieces, the run gives back its last piece, when - that piece is one word and is not itself suffix vocabulary -- the - one-word half is what leaves 'Prince of Wales Jr' alone and the - vocabulary half what leaves 'MD DDS' and 'Jr. Ph. D.' alone - (rules.md#H3, decisions.md#H3). + that piece is one word and is not itself suffix vocabulary + (rules.md#H3, decisions.md#H3 -- the block at the floor below + carries the examples of each half, and the ordering its two inline + tag reads were measured on). One definition, read by assign (which sets the roles) and by the chain's trailing-run walk; the leading-particle scan shares the predicate, is_leading_title, but stops at a title-and-particle @@ -139,11 +142,11 @@ def leading_titles(pieces: Sequence[Sequence[int]], and len(pieces[n - 1]) == 1 and not is_suffix_piece(pieces[n - 1], ptags[n - 1], tokens)): - k = n - while k < len(pieces) and is_suffix_piece(pieces[k], ptags[k], - tokens): - k += 1 - if k == len(pieces): + for k in range(n, len(pieces)): + if not is_suffix_piece(pieces[k], ptags[k], tokens): + break + else: + # nothing behind the run but suffix pieces n -= 1 return n @@ -377,12 +380,15 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken]) -> int: - """How many pieces at the END of `rest` are period-marked title - words. `rest` is the caller's NAME pieces, in piece order: on the - no-comma path what the S2 peel left, after a family comma the - segment's pieces that the segment's own suffix reading does not - claim, and in group's bound-given reserve the peel's leftovers - over the view the join would build. + """How many pieces of `rest` the trailing title chain LEAVES + standing: `rest[:kept]` are the name pieces and `rest[kept:]` the + period-marked title words the chain took, in piece order. Counted + the way `peel_trailing` counts, so the two answers compose without + arithmetic at the call site. `rest` is the caller's NAME pieces: + on the no-comma path what the S2 peel left, after a family comma + the segment's pieces that the segment's own suffix reading does + not claim, and in `tail_reading` the leftovers of whichever peel + is current. Floor: one name piece stands, so a name is never all title -- and an empty `rest` returns 0, which is what leaves assign's bare-suffix carve-out reached exactly as before. @@ -406,14 +412,72 @@ def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], period-marked word, so the ordinary parse pays the one match and stops (decisions.md#parse-cost). """ - n = 0 - while len(rest) - n > 1: - idx = rest[len(rest) - n - 1] + k = len(rest) + while k > 1: + idx = rest[k - 1] piece = pieces[idx] if (len(piece) == 1 and _PERIOD_ABBREV.match(tokens[piece[0]].text) and is_title_piece(piece, ptags[idx], tokens)): - n += 1 + k -= 1 continue break - return n + return k + + +# rules.md#H5: "the title is TRANSPARENT to the suffix reading: where +# two or more name words stand, what stands once the chain is taken +# reads exactly as it would read written without the title, plus the +# title" +def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]], + ptags: Sequence[Set[str]], + tokens: Sequence[WorkToken], + ) -> tuple[list[int], tuple[int, ...], Peel]: + """The S2 peel and the H5 chain read together to a FIXED POINT: + peel, chain, splice the chained pieces out, peel again over what + is left -- the name pieces the chain kept, then the pieces the + peel had taken, in original order -- until the chain takes + nothing. Where it takes nothing on the first pass, which is almost + every name, that first peel is the answer and the loop costs one + comparison. + + Returns the walk with the chained pieces spliced out, the pieces + the chain took, and the FINAL peel -- whose numeral fork and + ambiguous picks are the ones assign reports. The walk is + partitioned by that peel exactly as a caller partitions its own: + `rest[:peel.names]` the name pieces, `rest[peel.names:]` the + suffixes. A bare tuple rather than a named one because a + NamedTuple's __new__ is a frame of its own on every parse + (decisions.md#parse-cost), and `_group_segment` returns its three + the same way. + + Transparency is what the fixed point buys: 'X Prof. Y' reads + exactly as 'X Y' reads plus the title, however many titles are + written and wherever the peel then stops. Iterating ONCE reads a + second title only half way -- 'John Prof. MA Prof.' un-peeled the + acronym and re-exposed the first title, reading family 'Prof.' + with suffix 'MA' where 'John Prof. MA' reads family 'MA'. + + One function for two readers -- assign's placement and group's + bound-given reserve (P5), which must count the name words assign + will leave. Deriving that agreement twice is what left the two + disagreeing at S2's bare-ambiguous reserve: 'abdul rahman MA' + declined the join and 'abdul rahman MA Prof.' took it + (mechanisms.md#ONE-PREDICATE-PER-QUESTION). + + `rest` is a peel_walk list and is not mutated -- the splice + rebinds this local -- so a caller's own reference still names the + walk it built. Both callers read the one returned here instead, + which is the one the final peel partitions. + """ + titled: list[int] = [] + while True: + peeled = peel_trailing(rest, pieces, ptags, tokens) + kept = trailing_titles(rest[:peeled.names], pieces, ptags, + tokens) + if kept == peeled.names: + return rest, tuple(titled), peeled + # the chain's pieces reach this list back to front, so each + # run goes in FRONT of what the pass before it took + titled[:0] = rest[kept:peeled.names] + rest = rest[:kept] + rest[peeled.names:] diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 1d23a7f1..7c9dc1ec 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -351,7 +351,9 @@ def _addressing_run(titles: list[int], name_word: int) -> list[int]: standing behind it deciding that word's field only when none stands before. - Keeping the two ends apart is what makes a trailing title + Every title token is in the TITLE role by the time this runs, both + ends of `Sir John Prof.` among them, so `titles` is not a run -- + keeping the two ends apart is what makes a trailing title TRANSPARENT (rules.md#H5): `Sir John Prof.` is `Sir John` plus a title, and reading both ends as one run keyed 'sir prof' made adding the title flip the name word's field (#489, #316). @@ -394,15 +396,9 @@ def post_rules(state: ParseState) -> ParseState: # rules.md#H1: "a run of several titles addresses as its last # title does, and where a run stands BEFORE the one name word it # is the run that addresses, a run standing behind it deciding - # that word's field only when none stands before" -- #489, so 'Her - # Majesty Queen Elizabeth' reads given 'Elizabeth': the run is not - # a given-name title but 'queen' is. WHICH run is _addressing_run's - # question; every title token is in the TITLE role by now, both - # ends of 'Sir John Prof.' among them, and keying the two ends as - # one run made the trailing title change the leading one's reading. - # The predicate lives beside _title_key because the P5 licence in - # group asks the same question of the same run, and a run read two - # ways is a rule contradicting itself (decisions.md#P5, #369). + # that word's field only when none stands before" -- #489. WHICH + # run that is, and why the two ends of a name are not one, are + # _addressing_run's; its docstring carries the history. if (titles and givens and not middles and not families and not _run_addresses_by_given( (tokens[i].text diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index 791a487c..6ce0be9d 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -931,20 +931,20 @@ # a bulk import from quietly re-creating one. It guards the SHIPPED sets # only: Lexicon has no matching invariant, so a caller who wants the # overlap in their own vocabulary may still have it. +# +# It carries the AMBIGUOUS set's stake too, which is the sharper one: +# suffix_as_written ORs the word branch and the acronym branch, so an +# ambiguous acronym that were also a suffix word would be claimed +# through the word membership and bypass the period gate the ambiguous +# set exists to impose. The ambiguous set is a subset of the acronyms +# (the assert above), so this one covers it -- a separate assert of +# `SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_WORDS` cannot fail while both of +# these hold. assert not (SUFFIX_ACRONYMS & SUFFIX_WORDS), \ "a post-nominal belongs to one set or the other, never both (the " \ - "two normalize differently): " \ + "two normalize differently, and the word branch would bypass an " \ + "ambiguous acronym's period gate): " \ f"{sorted(SUFFIX_ACRONYMS & SUFFIX_WORDS)}" -# The narrower claim, kept for the message it prints: an ambiguous -# acronym must not also be a plain suffix word, because suffix_as_written -# ORs the two branches, so the word membership would bypass the period -# gate the ambiguous set exists to impose. Implied by the disjointness -# above for as long as the ambiguous set stays a subset of the acronyms, -# which is what the first assert holds. -assert not (SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_WORDS), \ - "an ambiguous acronym must not also be a suffix word (the word " \ - "branch bypasses its period gate): " \ - f"{sorted(SUFFIX_ACRONYMS_AMBIGUOUS & SUFFIX_WORDS)}" # The peel splits its tail off as a TOKEN and suffix classification is # what claims it downstream, so a tail that is not also a suffix word # would split the name and then leave the piece sitting in it. The diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 78cb6f2e..67bc1013 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -2385,7 +2385,14 @@ def _check_cjk_shape_purity(self) -> None: notes="the accepted edge: with `Dr` given back and `Jr` " "peeled as the suffix, no title is left to make the " "reading H1's, so the lone name word is H4's -- " - "`dr` is title vocabulary and the word standing is it"), + "`dr` is title vocabulary and the word standing is it. " + "This shape is what rules.md#S2's descriptive note " + "named, in its `Sir Jr` spelling, as the reading it " + "predicted and did not get; once the floor empties the " + "run no branch reads `vocab:given-title` at all, so " + "`Sir Jr` is this row in every part -- same roles, same " + "kind, the detail naming a different word -- and it is " + "not pinned twice"), Case("title_run_floor_gives_back_the_last_of_a_run", "Lord Chancellor Jr", {"title": "Lord", "family": "Chancellor", "suffix": "Jr"}, @@ -2420,17 +2427,6 @@ def _check_cjk_shape_purity(self) -> None: classification="parity", notes="negative control: nothing stands behind the run, so " "there is no all-suffix rest for the floor to see"), - Case("title_run_floor_gives_back_a_given_name_title_run", "Sir Jr", - {"given": "Sir", "suffix": "Jr"}, - ambiguities=("title-or-name",), classification="fix(#489)", - notes="`Sir Jr` is the input rules.md#S2's descriptive note " - "names -- it read given `Jr` and now reads given `Sir`, " - "suffix `Jr`. The row is here for that provenance and " - "not for a second mechanism: the reading is `Dr Jr`'s " - "in every part -- same roles, same kind, the detail " - "being the word left standing -- because once the floor " - "empties the run no branch reads `vocab:given-title` " - "at all"), Case("title_run_addresses_by_its_last_title", "Her Majesty Queen Elizabeth", {"title": "Her Majesty Queen", "given": "Elizabeth"}, @@ -3652,6 +3648,23 @@ def _check_cjk_shape_purity(self) -> None: "the reports come from the single peel over the " "spliced pieces. Collecting both peels' picks instead " "reported the same coin flip twice"), + Case("title_word_trailing_run_is_read_to_a_fixed_point", + "John Prof. MA Prof.", + {"title": "Prof. Prof.", "given": "John", "family": "MA"}, + ambiguities=("suffix-or-name",), classification="fix(#316)", + notes="transparency for a SECOND title, which peel and chain " + "reading to a FIXED POINT buys and one re-peel did " + "not: the chain takes the last 'Prof.', the peel over " + "what is left un-peels 'MA' and re-exposes the first " + "'Prof.', and only asking the chain again takes it. " + "Iterating once read title 'Prof.', given 'John', " + "family 'Prof.', suffix 'MA' -- which is 'John Prof. " + "MA' plus a title in no reading at all. Found by the " + "/simplify round on the bundle, 2026-09-09; 1.4.0 read " + "first 'John' / middle 'Prof. MA' / last 'Prof.' " + "(measured 2026-09-09), so the row is #316's parity " + "break either way and the round decided only which " + "reading it is"), Case("title_word_trailing_keeps_the_bare_acronym_reserve", "John Prof. MA", {"title": "Prof.", "given": "John", "family": "MA"}, @@ -3818,6 +3831,27 @@ def _check_cjk_shape_purity(self) -> None: "rahman' (measured on the bundle's third commit). " "1.4.0 read title 'Prof.' / first 'abdul rahman' / " "last 'Prof.'"), + Case("title_word_trailing_behind_a_bound_pair_at_the_peel_reserve", + "abdul rahman MA Prof.", + {"title": "Prof.", "given": "abdul", "family": "rahman", + "suffix": "MA"}, ambiguities=("suffix-or-name",), + classification="fix(#316)", + notes="the row above's shape at S2's bare-ambiguous reserve, " + "which is the one place the reserve's own model of " + "assign's reading and assign's reading disagreed. This " + "is 'abdul rahman MA' plus a title, and that reads " + "given 'abdul' / family 'rahman' / suffix 'MA': the " + "join declines because peeling the acronym unjoined " + "and not joined is a suffix reading the join would " + "change. Modelling the peel and the chain as a " + "subtraction, the reserve counted 'MA' a name word to " + "spare here and not without the title, so the join " + "fired and read given 'abdul rahman', family 'MA'. One " + "shared reading of the tail is what makes the two " + "agree -- found by the /simplify round on the bundle, " + "2026-09-09 (decisions.md#H5). 1.4.0 read first 'abdul " + "rahman' / middle 'MA' / last 'Prof.' (measured " + "2026-09-09)"), Case("title_word_trailing_behind_a_licensed_bound_pair", "Sir abdul rahman Prof.", {"title": "Sir Prof.", "given": "abdul rahman"}, @@ -3866,20 +3900,6 @@ def _check_cjk_shape_purity(self) -> None: "prof', which addresses by neither, and read family " "'John'. 1.4.0 read title 'Sir' / first 'John' / last " "'Prof.' (measured 2026-09-09)"), - Case("title_run_leading_given_name_title_over_a_trailing_title", - "Queen Elizabeth Prof.", - {"title": "Queen Prof.", "given": "Elizabeth"}, - classification="fix(#316)", - notes="the same clause where the leading run is a ONE-word " - "given-name title that is also an ordinary surname. " - "'Queen Elizabeth' reads given 'Elizabeth' and adding " - "the trailing title does not move it; the composite " - "key 'queen prof' read family 'Elizabeth'. Not a " - "rules.md example -- 'Sir John Prof.' carries the " - "clause there -- and kept because `queen` is the " - "vocabulary entry #489's run arm turns on. 1.4.0 read " - "title 'Queen' / first 'Elizabeth' / last 'Prof.' " - "(measured 2026-09-09)"), Case("title_run_leading_addresses_over_a_trailing_given_name_title", "Dr. Smith Sir.", {"title": "Dr. Sir.", "family": "Smith"}, diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index a4db9ed7..462108c6 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -157,7 +157,11 @@ def test_the_leading_run_keeps_a_joined_unit_it_cannot_give_back() -> None: def _trailing(text: str) -> int: """trailing_titles over the rest assign hands it: the name pieces - the S2 peel left, after the leading run is counted off.""" + the S2 peel left, after the leading run is counted off. + + The count is what the chain LEAVES STANDING, the way peel_trailing + counts -- so a walk that takes nothing returns the length of the + rest it was handed, and each title taken is one off that.""" state = _through_group(text) pieces, ptags = state.pieces[0], state.piece_tags[0] tokens = list(state.tokens) @@ -173,9 +177,9 @@ def test_the_trailing_run_chains_period_marked_title_words() -> None: reading below, which is the reason the walk chains rather than taking the last piece and stopping. """ - assert _trailing("John Smith Prof. Dr.") == 2 - assert _trailing("John Smith Prof.") == 1 - assert _trailing("Dr. John Smith Prof.") == 1 # leading run too + assert _trailing("John Smith Prof. Dr.") == 2 # of four: both taken + assert _trailing("John Smith Prof.") == 2 # of three + assert _trailing("Dr. John Smith Prof.") == 2 # leading run too def test_the_trailing_run_leaves_one_name_piece_standing() -> None: @@ -187,10 +191,10 @@ def test_the_trailing_run_leaves_one_name_piece_standing() -> None: where the same floor returns 0 and leaves assign's bare-suffix carve-out reached exactly as before. """ - assert _trailing("Smith Prof.") == 1 - assert _trailing("Dr. Prof.") == 0 # the floor, on a rest the + assert _trailing("Smith Prof.") == 1 # of two: the title taken + assert _trailing("Dr. Prof.") == 1 # the floor, on a rest the # walk would otherwise take - assert _trailing("Smith") == 0 # one-piece rest + assert _trailing("Smith") == 1 # one-piece rest assert _trailing("Prof.") == 0 # empty rest @@ -203,9 +207,10 @@ def test_the_trailing_run_reads_vocabulary_and_not_shape() -> None: is claimed there only when the vocabulary claims it, and a bare title word is claimed not at all. """ - assert _trailing("John Smith Xyz.") == 0 # unlisted abbreviation - assert _trailing("John Smith Sir") == 0 # no period - assert _trailing("John Smith Esq.") == 0 # the peel took it first + # nothing claimed: the walk leaves every piece it was handed + assert _trailing("John Smith Xyz.") == 3 # unlisted abbreviation + assert _trailing("John Smith Sir") == 3 # no period + assert _trailing("John Smith Esq.") == 2 # the peel took it first def test_the_trailing_run_refuses_a_joined_piece() -> None: @@ -216,12 +221,13 @@ def test_the_trailing_run_refuses_a_joined_piece() -> None: that unit a name. That first row does not PIN the gate, though: with the one-word - test deleted it still reads 0, because the shape test then runs + test deleted it stands unchanged, because the shape test then runs on the piece's first token and 'de' wears no period (measured 2026-09-09). The conjunction-merged unit is the row that pins it -- 'Prof. and Dr.' is one piece whose first token is a period-marked title word, so without the gate the walk takes it - and the name loses its family (measured 1 under that mutation). + and the name loses its family (measured one piece fewer standing + under that mutation). """ - assert _trailing("John de la Prof.") == 0 - assert _trailing("John Smith Prof. and Dr.") == 0 + assert _trailing("John de la Prof.") == 2 + assert _trailing("John Smith Prof. and Dr.") == 3 diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 438622b0..3cf92310 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -2431,8 +2431,11 @@ def _claim(rule: dict) -> _Claim: # `title` and `given` too -- so both fall through to the # peel-floor rule at the end of the ledger, and the surplus # is the one the paragraph above says these four carry. + # 7 -> 6 on 2026-09-09, in the /simplify round: `Sir Jr` left + # rules.md and so left the corpus; `Dr Jr` reads the same way + # and pins the shape alone. "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`": - _Claim(7, ('family', 'suffix'), "4cd8e7fbd20d", None), + _Claim(6, ('family', 'suffix'), "dd3fc23d90a1", None), "fix(suffix-routing) a two-token name ending in a credential acronym keeps it in `suffix`": _Claim(2, ('family', 'suffix'), "ed72c9672214", None), "fix(suffix-routing) the dotted M.A. spelling reads as a credential (ma-do)": @@ -2473,14 +2476,17 @@ def _claim(rule: dict) -> _Claim: # The 2.3 title-run bundle's five rules, last in every # ledger. All five are anchored on NAMES, so the reach IS the # mover list: 2 names for the run keying, 1 for the esq drop, - # 4 for the peel floor, 16 for the trailing title and 1 for + # 3 for the peel floor, 16 for the trailing title and 1 for # the trailing title on a native-script name, which is the # same argument on the one name a script-classified - # alternation may not hold. Twenty-four in all, and every one + # alternation may not hold. Twenty-three in all, and every one # of them is explained by the rule that names it -- these are # the rare rows where reach and explanation coincide, which is # what an anchored name list buys. A widening past those names # moves the digest here before it can reach the gate. + # 24 -> 23 on 2026-09-09, in the /simplify round: `Sir Jr` + # left rules.md, where it pinned nothing `Dr Jr` does not, + # and so left the rules corpus and the peel floor's reach. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -2491,7 +2497,7 @@ def _claim(rule: dict) -> _Claim: # digest are identical in all four ledgers and only the roles # move. "fix(#489) the title peel leaves a name word a suffix cannot be": - _Claim(4, ('family', 'given', 'suffix', 'title'), "ac7318881b28", None), + _Claim(3, ('family', 'given', 'suffix', 'title'), "07b02286cd81", None), "fix(#316) a trailing period-marked title word reads as a title": _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), "562e0e82a22b", None), @@ -2663,16 +2669,9 @@ def _claim(rule: dict) -> _Claim: _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), - # `middle` left the ROLES in the same edit, at the gate's - # own OVER-DECLARED insistence: with 'John Smith Dr.' gone, - # no name the rule still explains moves a middle name. - # 11 -> 12 on 2026-09-08: the 2.3 title-run bundle put - # 'John Smith Prof. Dr.' in the rules corpus, and the - # trailing-`dr` regex reaches any name ending in " Dr.". - # Reach, not explanation -- the rule explains NEITHER of - # the two 'Dr.' names now, both having moved to the #316 - # rule at the end of the ledger, and the comment there - # records the handover. + # `middle` left the ROLES in the same edit, and the reach + # grew with the rules corpus; the 1.4.0 roster above carries + # both, and says the same at the other two baselines. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": @@ -2743,16 +2742,8 @@ def _claim(rule: dict) -> _Claim: "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), # The 2.3 title-run bundle's five rules, last in every - # ledger. All five are anchored on NAMES, so the reach IS the - # mover list: 2 names for the run keying, 1 for the esq drop, - # 4 for the peel floor, 16 for the trailing title and 1 for - # the trailing title on a native-script name, which is the - # same argument on the one name a script-classified - # alternation may not hold. Twenty-four in all, and every one - # of them is explained by the rule that names it -- these are - # the rare rows where reach and explanation coincide, which is - # what an anchored name list buys. A widening past those names - # moves the digest here before it can reach the gate. + # ledger, and the same reach at all four: the 1.4.0 roster + # above carries the argument. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -2763,7 +2754,7 @@ def _claim(rule: dict) -> _Claim: # digest are identical in all four ledgers and only the roles # move. "fix(#489) the title peel leaves a name word a suffix cannot be": - _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), + _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "07b02286cd81", None), "fix(#316) a trailing period-marked title word reads as a title": _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), "562e0e82a22b", None), @@ -2860,16 +2851,8 @@ def _claim(rule: dict) -> _Claim: "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), # The 2.3 title-run bundle's five rules, last in every - # ledger. All five are anchored on NAMES, so the reach IS the - # mover list: 2 names for the run keying, 1 for the esq drop, - # 4 for the peel floor, 16 for the trailing title and 1 for - # the trailing title on a native-script name, which is the - # same argument on the one name a script-classified - # alternation may not hold. Twenty-four in all, and every one - # of them is explained by the rule that names it -- these are - # the rare rows where reach and explanation coincide, which is - # what an anchored name list buys. A widening past those names - # moves the digest here before it can reach the gate. + # ledger, and the same reach at all four: the 1.4.0 roster + # above carries the argument. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -2880,7 +2863,7 @@ def _claim(rule: dict) -> _Claim: # digest are identical in all four ledgers and only the roles # move. "fix(#489) the title peel leaves a name word a suffix cannot be": - _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), + _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "07b02286cd81", None), "fix(#316) a trailing period-marked title word reads as a title": _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), "562e0e82a22b", None), @@ -3042,16 +3025,9 @@ def _claim(rule: dict) -> _Claim: _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), - # `middle` left the ROLES in the same edit, at the gate's - # own OVER-DECLARED insistence: with 'John Smith Dr.' gone, - # no name the rule still explains moves a middle name. - # 11 -> 12 on 2026-09-08: the 2.3 title-run bundle put - # 'John Smith Prof. Dr.' in the rules corpus, and the - # trailing-`dr` regex reaches any name ending in " Dr.". - # Reach, not explanation -- the rule explains NEITHER of - # the two 'Dr.' names now, both having moved to the #316 - # rule at the end of the ledger, and the comment there - # records the handover. + # `middle` left the ROLES in the same edit, and the reach + # grew with the rules corpus; the 1.4.0 roster above carries + # both, and says the same at the other two baselines. "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": _Claim(13, ('family', 'suffix'), "fb9c68f36d0b", None), "fix(#296) a credential-only comma string reads a name and its postnominal": @@ -3114,16 +3090,8 @@ def _claim(rule: dict) -> _Claim: "fix(#462) the facade keeps an initial-shaped conjunction letter": _Claim(18, ('_initials',), "3dd0e0276be6", ('DEFAULT',)), # The 2.3 title-run bundle's five rules, last in every - # ledger. All five are anchored on NAMES, so the reach IS the - # mover list: 2 names for the run keying, 1 for the esq drop, - # 4 for the peel floor, 16 for the trailing title and 1 for - # the trailing title on a native-script name, which is the - # same argument on the one name a script-classified - # alternation may not hold. Twenty-four in all, and every one - # of them is explained by the rule that names it -- these are - # the rare rows where reach and explanation coincide, which is - # what an anchored name list buys. A widening past those names - # moves the digest here before it can reach the gate. + # ledger, and the same reach at all four: the 1.4.0 roster + # above carries the argument. "fix(#489) a title run addresses by its last title": _Claim(2, ('family', 'given'), "e14159a4d48f", None), "change(suffix-acronym-collisions) esq leaves the acronym set": @@ -3134,7 +3102,7 @@ def _claim(rule: dict) -> _Claim: # digest are identical in all four ledgers and only the roles # move. "fix(#489) the title peel leaves a name word a suffix cannot be": - _Claim(4, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "ac7318881b28", None), + _Claim(3, ('_ambiguities', 'family', 'given', 'suffix', 'title'), "07b02286cd81", None), "fix(#316) a trailing period-marked title word reads as a title": _Claim(16, ('family', 'given', 'middle', 'suffix', 'title'), "562e0e82a22b", None), diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index a79a7779..acfa8c64 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -296,14 +296,13 @@ def test_the_shipped_given_name_titles_are_every_one_a_single_word() -> None: def test_a_run_matches_by_its_last_word_when_the_whole_run_does_not() -> None: - # The other arm, on the shipped vocabulary: 'dr sir' is no entry - # and never could be, and `sir` is the run's last word (#489). + # The other arm's premise, on the shipped vocabulary: 'dr sir' is + # no entry and never could be, and `sir` is the run's last word + # (#489). What the parse then reads is the `Dr. Sir John` case + # row's, and is not asserted twice. lex = Lexicon.default() assert "dr sir" not in lex.given_name_titles assert "sir" in lex.given_name_titles - parsed = Parser(lexicon=lex).parse("Dr. Sir John") - assert (parsed.title, parsed.given, parsed.family) == \ - ("Dr. Sir", "John", "") @pytest.mark.parametrize("entry", ["lt .", "lt . col", ". col", ". ."]) diff --git a/tools/differential/compare.py b/tools/differential/compare.py index 276b0f3c..2abc58be 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -1950,13 +1950,20 @@ class _ShapeMismatch(NamedTuple): #: entered it, so the scan went 52 -> 53 and the roster 50 -> 51. #: Recounted 2026-09-08 with the title-run bundle: one name entered the #: population, 'John Smith Rev.', the only one of that bundle's -#: movers no test literal names, and it took a row at each -#: of the four baselines. Re-checked 2026-09-09 in that bundle's second +#: movers no test literal names -- the others on radar carry a +#: cases.py row or a v1 test -- and it took a row at each of the four +#: baselines, its shape measured by the run and the same at all four. +#: Re-checked 2026-09-09 in that bundle's second #: review round, when its mover list went from twenty-one to #: twenty-four: none of the three arrivals joins the population, all #: three being named as case-row literals in tests/v2/cases.py #: ('Sir John Prof.', 'Dr. Smith Sir.', '毛 泽东 Dr.'), so the row -#: counts below are unmoved. It is named NOWHERE under tests/, so it +#: counts below are unmoved. The /simplify round the same day took +#: that list to twenty-three, 'Sir Jr' leaving rules.md and so the +#: rules corpus; it was a cases.py literal and never in this +#: population, so the row counts are unmoved by that too. +#: 'John Smith Rev.' is named NOWHERE under +#: tests/, so it #: counts in both scans -- the every-file figures in the RECOMPUTE #: paragraph below count it too. That commit moved the ROW counts and nothing #: else: it did not re-derive the population clause above, so the @@ -2034,14 +2041,6 @@ class _ShapeMismatch(NamedTuple): "Jack M.A.": ("family", "suffix"), "Jane van der Berg 旧姓 Jones": ("family", "maiden"), "Janey née Jones": ("family", "given", "maiden", "middle"), - # The one name of the 2.3 title-run bundle's nineteen - # movers that no test literal watches: the other four on - # radar carry a cases.py row or a v1 test. Shape measured - # by the run, 2026-09-08, and the same at all four - # baselines. The POPULATION paragraph above was last - # re-derived 2026-09-07 and this addition did not re-derive - # it -- the row counts below are updated by the four rows - # added, nothing else. "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "John V": ("family", "suffix"), @@ -2086,14 +2085,6 @@ class _ShapeMismatch(NamedTuple): "Jane van der Berg 旧姓 Jones": ("family", "maiden"), "Janey née Jones": ("family", "given"), "Joe E. Smith": ("_initials",), - # The one name of the 2.3 title-run bundle's nineteen - # movers that no test literal watches: the other four on - # radar carry a cases.py row or a v1 test. Shape measured - # by the run, 2026-09-08, and the same at all four - # baselines. The POPULATION paragraph above was last - # re-derived 2026-09-07 and this addition did not re-derive - # it -- the row counts below are updated by the four rows - # added, nothing else. "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "John, Smith, Dr.": ("_ambiguities",), @@ -2136,14 +2127,6 @@ class _ShapeMismatch(NamedTuple): "Jane van der Berg 旧姓 Jones": ("family", "maiden"), "Janey née Jones": ("family", "given"), "Joe E. Smith": ("_initials",), - # The one name of the 2.3 title-run bundle's nineteen - # movers that no test literal watches: the other four on - # radar carry a cases.py row or a v1 test. Shape measured - # by the run, 2026-09-08, and the same at all four - # baselines. The POPULATION paragraph above was last - # re-derived 2026-09-07 and this addition did not re-derive - # it -- the row counts below are updated by the four rows - # added, nothing else. "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "John, Smith, Dr.": ("_ambiguities",), @@ -2170,14 +2153,6 @@ class _ShapeMismatch(NamedTuple): "E Anne D,Leonardo": ("_initials",), "JOSE E MARIA SANTOS": ("_initials",), "Joe E. Smith": ("_initials",), - # The one name of the 2.3 title-run bundle's nineteen - # movers that no test literal watches: the other four on - # radar carry a cases.py row or a v1 test. Shape measured - # by the run, 2026-09-08, and the same at all four - # baselines. The POPULATION paragraph above was last - # re-derived 2026-09-07 and this addition did not re-derive - # it -- the row counts below are updated by the four rows - # added, nothing else. "John Smith Rev.": ("family", "middle", "title"), "John Smith, RAI": ("family", "given", "suffix"), "Jose E. Maria Santos": ("_initials",), diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 0ad6c230..b30b894f 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -182,7 +182,6 @@ "Sidorov Ivan Petrovich Jr." "Sir John" "Sir John Prof." -"Sir Jr" "Sir Ph. D. Van Johnson" "Sir abdul van der Berg" "Sir de Mesnil" From 23959b278ef841a7fa0a1ff4611c5af255c68e56 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 20:38:22 -0700 Subject: [PATCH 10/12] perf(assign): the nickname scan runs only where a one-piece segment reads it CI's 3.12-3.15 jobs failed the facade band at 452.0 against a ceiling of 451.86 after the /simplify commit: the frame it traded for the shared tail reading was a list comprehension in _effective_order, which is a frame on 3.11 alone (PEP 709 inlines comprehensions from 3.12), so on the newer interpreters the wrapper went unpaid. `has_nickname` was computed at the top of the positional read as `any(t.role is Role.NICKNAME for t in tokens)`, a generator that every interpreter resumes once per token -- seven call events on the reference name -- and read by exactly one branch, the one-piece segment beside a nickname (rules.md#N3). The scan now sits last in that branch's condition, so an ordinary name never starts it. Measured with tools/perf/call_count.py on `Dr. Juan0000 de la Vega III`: py3.11 416/453 -> 409/446 (baseline 410/447); py3.12 395/432 -> 388/425 (baseline 388/425); py3.14 413/450 -> 406/443 (baseline 406/443). Every interpreter is back at or one under its recorded row; no row moves. Parses are unchanged: the branch's answer is the same, only when it is computed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- nameparser/_pipeline/_assign.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 7ea9929a..9be48608 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -237,7 +237,6 @@ def _assign_main(seg_idx: int, state: ParseState, -- None on every path that returns before resolving one.""" pieces = state.pieces[seg_idx] ptags = state.piece_tags[seg_idx] - has_nickname = any(t.role is Role.NICKNAME for t in tokens) n = _peel_leading_titles(pieces, ptags, tokens) if n == len(pieces): return None @@ -264,7 +263,13 @@ def _assign_main(seg_idx: int, state: ParseState, # the WHOLE segment before any title peeling -- 'Xyz. (Bud) Smith' # has two pieces, so the title peel wins and Smith stays the given # name (pinned live 2026-07-17) - if len(pieces) == 1 and len(rest) == 1 and has_nickname: + # The nickname scan sits LAST in the test: it is a generator, which + # every interpreter resumes once per token (seven call events on + # the reference name), and only a one-piece segment ever reads its + # answer. Hoisting it to the top of the read, where it once stood, + # is what put 3.12-3.15 one call over the band (decisions.md#parse-cost). + if (len(pieces) == 1 and len(rest) == 1 + and any(t.role is Role.NICKNAME for t in tokens)): _set_roles(tokens, pieces[rest[0]], Role.FAMILY) return None # rules.md#S2's trailing peel and rules.md#H5's title chain, read From 3471057f44ac6619b6534174da4a9e126cd589bd Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 20:50:49 -0700 Subject: [PATCH 11/12] docs(design): the /simplify round's counts re-measured; the census names tail_reading Six review findings on the pushed branch, all re-measured on this tree rather than carried over. decisions.md#H5's class bullet stated one width for two defects and named the wrong shape for it. The 88 shapes the fix closes PARTITION: 56 carry one period-marked title word and a bound given-name word besides -- the subtraction model -- and 32 carry two, the truncated fixed point. The 156 movers over the 82,712-input sweep split 60/96 the same way, and the two-title population is 784, of which 164 differed before and 132 still do. The population bullet's disabling stub was wrong the moment the same commit made `trailing_titles` return the count it LEAVES STANDING: a 0 stub now takes everything and raises IndexError on 'John Smith'. The recipe says `lambda rest, *a: len(rest)`, bound over both the `_pieces` name and the one `_assign` imported, and says why the fix's own tree needs the other stub. Re-run that way the five and the seventeen hold. mechanisms.md's ONE-PREDICATE-PER-QUESTION census was one round out of date: an AST call-site census over `_pipeline/*.py` says the shared predicate is `tail_reading`, that `peel_trailing` has no caller outside `_pieces.py` at all, and that `trailing_titles` has exactly one. #H4's rosters lose `Sir Jr`, which left corpus_rules.jsonl in the same round: eleven names in fourteen rows, the peel half nine, six contract and three radar -- with the tier rule spelled as the README states it, contract files sorted first. The release bullet counts the set once, three corpus names plus `Sir Jr` as an input `Dr Jr`'s row pins, and the four ledgers' peel-floor comment no longer claims a reach of four. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- docs/design/decisions.md | 8 ++++---- docs/design/mechanisms.md | 2 +- docs/release_log.rst | 2 +- tools/differential/expected_since_1.4.0.toml | 11 ++++++++--- tools/differential/expected_since_2.0.0.toml | 11 ++++++++--- tools/differential/expected_since_2.1.0.toml | 11 ++++++++--- tools/differential/expected_since_2.2.0.toml | 11 ++++++++--- 7 files changed, 38 insertions(+), 18 deletions(-) diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 67f71c44..3718efab 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -426,13 +426,13 @@ Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparse ### H4 — an input that is nothing but vocabulary still has to name somebody - 2026-09-07 #491 — the reading is unchanged and the silence is what ends. Handed a string the title peel eats down to one last word which is itself title vocabulary, the parser reads that word as the name; decisions.md#v1-xfail-triage recorded it in the fourth of its NOT FIXED entries — "this is a name parser, not a title parser" — and said in the same breath that what is actually wrong is the guess being silent. It now reports `title-or-name`, with `detail` naming the word that was made into the name. - Six corpus names gain it when this is written: the Queen's Bench string, `Lord Chancellor`, `Dr. King`, `The Rt Hon`, `His Holiness` and `His Holiness the Dalai Lama`. Three read contract-tier and three radar, and the tier split is an artifact of the documentation rather than of the shape — the three contract ones are contract because this bundle made them rules.md examples, which puts them in corpus_rules.jsonl. (2026-09-08, the #316/#489 bundle: the peel half's population is TEN, and the count moved for two different reasons. `Dr King Jr` and `Dr. King MD` gained the report because the title run's floor now leaves a name word standing where the run used to swallow it (#H3), and `Dr Jr` and `Sir Jr` gained it as the same floor's residue; three of the four also ENTERED the corpus in that commit, as rules.md examples of the floor. Seven contract and three radar now. Recompute rather than trusting either number: parse every name in the `tools/differential/corpus*.jsonl` glob and collect the ones whose `ambiguities` carry the kind, keeping the tier the file it first appears in declares — the README's table is where the tiers are.) `Dr. King` is the one worth arguing about and it is deliberate: `king` is in the titles vocabulary for the addressing forms, which the triage entry above decided and did not reopen, so `Dr. King` IS an input whose last standing word is title vocabulary and the rule claims it. Reporting there is honest rather than noisy — the reading came from a convention, not from anything the input says — and a caller who wants only the exotic cases has `detail` to filter on. + Six corpus names gain it when this is written: the Queen's Bench string, `Lord Chancellor`, `Dr. King`, `The Rt Hon`, `His Holiness` and `His Holiness the Dalai Lama`. Three read contract-tier and three radar, and the tier split is an artifact of the documentation rather than of the shape — the three contract ones are contract because this bundle made them rules.md examples, which puts them in corpus_rules.jsonl. (2026-09-08, the #316/#489 bundle: the peel half's population was TEN, and the count moved for two different reasons. `Dr King Jr` and `Dr. King MD` gained the report because the title run's floor now leaves a name word standing where the run used to swallow it (#H3), and `Dr Jr` gained it as the same floor's residue; those three also ENTERED the corpus in that commit, as rules.md examples of the floor. `Sir Jr` gained the report too and reads exactly as `Dr Jr` does, but it is NO corpus member and is counted nowhere here: it entered corpus_rules.jsonl in the same commit as a rules.md example and left again on 2026-09-09, when the /simplify round dropped the example as a duplicate of `Dr Jr`'s (#H5). So the peel half is NINE from that round, six contract and three radar. Recompute rather than trusting any of these numbers: parse every name in the `tools/differential/corpus*.jsonl` glob and collect the ones whose `ambiguities` carry the kind, keeping the tier the FIRST file holding the name declares with the contract files sorted first, as `main()` loads them — the README's table is where the tiers are, and sorting the glob plainly instead reads `Dr. King`, `Dr King Jr` and the Queen's Bench string as radar and gives three and six.) `Dr. King` is the one worth arguing about and it is deliberate: `king` is in the titles vocabulary for the addressing forms, which the triage entry above decided and did not reopen, so `Dr. King` IS an input whose last standing word is title vocabulary and the rule claims it. Reporting there is honest rather than noisy — the reading came from a convention, not from anything the input says — and a caller who wants only the exotic cases has `detail` to filter on. ONE emitter, and it is at assign's lone-name-word site rather than at H1's retag, which is where the drafting put it. H1 is not the site: under a declared family-first order the assignment places the word in the family directly and H1 never runs, so an emitter there would report under one order and not the other for a reading that is the same either way. Measured under the default order, FAMILY_FIRST and FAMILY_FIRST_GIVEN_LAST, the six report identically. That placement is also why the `detail` names no field, unlike O5's: under the default order H1 retags the word after assign, so a field named at the emitter would be the one the word was placed in and not the one it ends in — and the fork the kind reports is title-versus-name, which no field answers either way. What is silent, all measured 2026-09-08. A lone title word: `Dr.`, `Sir`, `King` and the chained `Prince of Wales` are a title run with nothing behind it, the peel takes the whole string, no word is left standing to be read as a name, and nothing was chosen — mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE's "a branch that runs but changes nothing is not a decision". A title with an ordinary word behind it: `Dr. Smith` and `King Charles` leave a word standing, but not a title-vocabulary one, so H1 alone explains them. And a title followed by post-nominal vocabulary: `Dr King Jr` and `Dr. King MD` peel `Dr King` and `Dr. King` WHOLE, leaving a credential that the bare-suffix carve-out makes the name — a different convention, and its report is scoped to inputs no title stands in (spelled `n == 0` when this was written, one clause of the `field_undecided` predicate since the 2026-09-08 consolidation recorded under O5), so a title in front of the run takes the input out of it and leaves the reading H1's. **Corrected 2026-09-08 (the #316/#489 bundle):** the two example names no longer read that way and the sentence is wrong about them, though the SCOPE it describes is unchanged and still true of the suffix half. The title run's floor gives back a one-word last piece where everything behind the run is post-nominal (#H3), so `Dr King Jr` reads title `Dr`, family `King`, suffix `Jr` and `Dr. King MD` title `Dr.`, family `King`, suffix `MD`. Both now leave a NAME word standing, and both report through this rule's TITLE half rather than being silent — `king` being title vocabulary is what makes them the peel shape. What is still true, and is what the sentence was written to say, is that the SUFFIX half is reached only where no title was peeled first: `MD DDS` has a title peeled and reports nothing, `DDS` being no title. rules.md#H4's Accepted clause is rewritten to match, and `Dr. King MD` is an example line there now. The peeled titles are never tested for anything: H2 makes an unlisted abbreviation a title by SHAPE, and `Xyz. Smith` is not this input. What the rule turns on is the word left standing. A refinement of Derek's, made after the population was measured and widening the rule past the all-titles shape it was drafted for: a lone name word that is a JOIN (P3) carrying title vocabulary reports `title-or-name` too, the fork there being whether the title word inside the unit is a title at all rather than which field the unit takes. `John of Prince` and `Smith and Prince` are the measured inputs; no corpus name reaches that branch, because a join LED by a title word is chained into a title run by H3 (`Prince of Wales`), so the two are pinned as case rows rather than as rules.md examples. It sits on O5's branch and takes precedence there, which is why O5's statement says a title silences THIS kind and not every report at the site. The suffix half is the same argument on the other vocabulary and needed no new kind. An input whose every word is post-nominal vocabulary reads its first word as a name — assign's "everything suffix-shaped after titles: first one is the name" carve-out — and that is the doubt SUFFIX_OR_NAME already names, so it reports that. `Rinpoche` and `QC MP` are the corpus names; `PhD`, `MBA` and `III` are the same shape. `Jr.` alone is NOT the shape at all: H2's opening-abbreviation rule reads it as a title before the suffix vocabulary is consulted, which is that rule's stated precedence and is recorded here because the expectation going in was that `Jr.` alone read as a name. The guard carries two exclusions of its own. A maiden name beside the credential says the input is not post-nominal vocabulary and nothing else, so `abd née Jones` is out for the reason M4 keeps it out of O5's report. And the report is scoped to names no script order placed, so a lone glued CJK honorific — さん, 씨, 선생님 — reports nothing: that is the same shape read through the glued-honorific rules (W2, #271/#308) and the script's own order, and whether those readings should report is left to the arc that revisits them rather than settled here. An asymmetry on the boundary between this rule and O5, noted and deliberately not fixed. `MA` and `Ma` alone report `given-or-family`; `PhD` alone reports `suffix-or-name`. Both are a bare credential with nothing beside it, and what separates them is which gate reads them: `ma` is an AMBIGUOUS acronym, so S2's gate declines to peel it and the word stands as the one name word, which is O5's branch; `phd` is unambiguous, so it peels to suffix, leaves no name piece, and reaches the bare-suffix carve-out, which is this rule's. Two conventions, two kinds, and the reading each name gets is the same either way — the caller sees a report in both cases and only the kind differs. Fixing it would mean one of the two gates changing what it reads, which moves fields for a bundle that moves none. - Nothing moves but the report, and the rules go in the three 2.x ledgers only — two of them at first and a third after the 2026-09-08 round below, which is where the ten names and the join clause's own rule come from. None is a copy of a wordlist and all three are declared in `_NOT_A_VOCABULARY_COPY`: a member copying TITLES would reach `Dr. Smith`, `Rabbi Cohen` and every other title-plus-surname name, and a member copying the suffix sets would reach `John Smith MD` and `Smith Jr.`, none of which moves. What selects these names is a SHAPE — the peel leaving one unit, and vocabulary standing in it — which no wordlist expresses. The population is EIGHT distinct names when this is written and TWELVE after the #316/#489 bundle; the ten this entry says twice above are two OTHER quantities — the ten corpus ROWS the eight names occupy, `Dr. King` and the Queen's Bench string each appearing in two corpus files, and the peel half's ten after the bundle. Recompute all three by parsing every name in the `tools/differential/corpus*.jsonl` glob, collecting the ones whose `ambiguities` carry the kind, and counting rows and distinct names separately (2026-09-09: eight names in ten rows at a0b93f0, twelve names in fifteen rows on this branch). + Nothing moves but the report, and the rules go in the three 2.x ledgers only — two of them at first and a third after the 2026-09-08 round below, which is where the ten names and the join clause's own rule come from. None is a copy of a wordlist and all three are declared in `_NOT_A_VOCABULARY_COPY`: a member copying TITLES would reach `Dr. Smith`, `Rabbi Cohen` and every other title-plus-surname name, and a member copying the suffix sets would reach `John Smith MD` and `Smith Jr.`, none of which moves. What selects these names is a SHAPE — the peel leaving one unit, and vocabulary standing in it — which no wordlist expresses. The population is EIGHT distinct names when this is written, TWELVE after the #316/#489 bundle and ELEVEN from 2026-09-09, when the /simplify round on the bundle's PR took `Sir Jr` out of rules.md and so out of corpus_rules.jsonl (#H5); the ten this entry says twice above names two OTHER quantities — the ten corpus ROWS the eight names occupy, `Dr. King` and the Queen's Bench string each appearing in two corpus files, and the peel half's ten after the bundle, itself nine from that same round. Recompute all three by parsing every name in the `tools/differential/corpus*.jsonl` glob, collecting the ones whose `ambiguities` carry the kind, and counting rows and distinct names separately (2026-09-09: eight names in ten rows at a0b93f0, eleven names in fourteen rows on this branch, nine of the eleven the peel half and two the join half). - 2026-09-08 #518 review round — the join clause is HOISTED beside the peel clause instead of sitting under O5's. It had been written inside O5's `n == 0` leg, so a leading title, a maiden marker or a vocabulary claim on the word silenced it — and every one of those decides which FIELD the unit takes, which is not what this clause asks. Measured before and after on this branch: `Lord Chancellor née Jones` and `Dr. King née Jones` (maiden), `Dr. Smith and Prince` and `Mr. John and King` (a peeled title), `van and Prince` and `J. and Prince` (a claimed word) were all silent and all now report `title-or-name`, with no role moving on any of them. The maiden pair takes the peel half and the other four the join half. TWO CORPUS NAMES MOVE, which the drafting expected to be zero: `Attorney General of Minnesota` and `Deputy Secretary of State`, both a title peeled in front of a joined unit that stands last — and a title needs a following piece, so H3 cannot chain the join into a title run the way it chains `Prince of Wales`. "No corpus name reaches the branch" was true only of the guarded version, and the claim is corrected in rules.md#H4 as well. They take a third `feat(#491)` ledger rule of their own rather than joining the all-titles alternation, whose issue line describes an all-titles input, which neither of these is. Also this round: the emitter comment's carve-out list no longer names "a group-flagged credential" (no such condition is in the code, and `phd_split` pins the opposite), and the `Order` NamedTuple is `EffectiveOrder` with field `order`, `roles` having been the name of two different things three lines apart. @@ -447,14 +447,14 @@ Decided 2026-09-08 (was Open: [#316](https://github.com/derek73/python-nameparse - **A1, the peel order, decided by Derek: the first peel is PROVISIONAL and a trailing title is TRANSPARENT to the suffix reading.** `X Prof. Y` reads exactly as `X Y` reads, plus the title. SCOPED 2026-09-09, in review of the docs commit: that is a claim about inputs where a name word still stands on both sides of the chain — two or more name words. Where the chain leaves ONE name word there is no second reading for it to be transparent to, and #H1 decides the field instead: `Smith Prof.` reads family `Smith` where `Smith` alone reads given `Smith` and reports `given-or-family`, and `Smith Sir.` reads given `Smith` with an empty family, `sir` being a given-name title. All three measured on this tree; rules.md#H5's statement carries the scope and `_assign.py`'s excerpt of it was updated in the same edit. The problem is order: with ONE peel, `Prof.` standing behind `Jr.` stopped the suffix peel before `Jr.`, and the title walk then removed the very word that had been blocking it — `John Smith Jr. Prof.` read family `Jr.`, a generational suffix promoted to the family name, which is worse output than the reading it replaced. So the pieces the walk takes are spliced out and ONE peel runs over what stands, in original order, and that second answer alone places a piece or reports a fork. Measured: `John Smith Jr. Prof.` reads suffix `Jr.`; `John Prof. MA` reads the family `MA` that `John MA` reads, S2's reserve keeping a bare ambiguous acronym the family of a two-word name where a second peel laid over a first read family `John`, suffix `MA`; and `John Smith V Prof. VI` reads what `John Smith V VI` reads — middle `Smith V`, family `VI`, nothing reported — where two peels each reporting their own last piece reported twice. The choice is CORPUS-NEUTRAL: both variants move the same eight names, no more and no fewer, so it is a question about output quality alone. - **A2, the family-comma segment-1 path, decided: it gets the walk.** The spec's condition was "if the comma segment gate already routes those, say so and leave it". Measured, it does not: `Smith, John Prof.` read middle `Prof.` at 1.4.0, 2.0.0, 2.1.0, 2.2.0 and at this branch's parent — all five measured — while `Smith, John Prof. Dr.` read middle `Prof.`, suffix `Dr.` through 2.1.0 and middle `Prof. Dr.` from 2.2.0, `dr` having left the suffix vocabulary in #296. The eleven comma rows the spec calls "already routed" are the `Smith, Prof.` shape, where segment 1 holds NO name word and `segment_suffix_reading` reads it piece by piece — a different gate and a different mechanism. So the segment-1 walk was genuinely missing; with it, `Smith, John Prof.` reads title `Prof.`, given `John`, family `Smith`. It moved NO corpus name at the fix; `Smith, John Prof.` is a rules.md#H5 example, so it ENTERS the corpus in the docs commit that follows and is one of the seventeen in the population bullet below. Case rows pin it either way. The segment's walk follows the same transparency principle with its own lenient loop, which is why the candidates are the pieces that walk would not read as a suffix rather than the strict suffix test alone (`Smith, John Prof. Jr.` must reach past the post-nominal, `Smith, John Prof. V` past the numeral the lenient tail test claims, #144). - **What A2 does NOT reach, and why that is right.** `Smith, John Prof. MA` still reads middle `Prof. MA`: after a family comma a bare ambiguous acronym is a MIDDLE name and has been since 2.0 (the `Smith, Ed` cost S2 already accepted), so `MA` stands at the end of the segment, is not a period-marked title word, and stops the walk before it starts. The walk reads from the end; it does not hunt. -- **Measured population: FIVE corpus names at the fix, SEVENTEEN after the docs commit that follows it and the two review rounds on it.** The five are the four planted no-comma rows in corpus_issues.jsonl (`John Smith Dr.`, `John Smith Mr.`, `John Smith Prof.`, `John Smith Rev.`) plus `Andrew Perkins (Mgr.)`, which the drafting did not expect. Eleven of the other twelve are rules.md examples entering corpus_rules.jsonl, eight of them this rule's own — `Smith Prof.`, `Dr. John Smith Prof.`, `John Smith Prof. Dr.`, `John Smith Prof. Jr.`, `John Smith Jr. Prof.`, `John Prof. MA`, `Smith, John Prof.` and `Mary Jane King.`, the last of those added by the first 2026-09-09 review of that commit — plus three of #H1's, which reach this walk because the run each puts behind the name word is what the walk takes: `Smith Sir.`, and `Sir John Prof.` and `Dr. Smith Sir.` from the second review round the same day. The twelfth is `毛 泽东 Dr.`, which entered corpus_cjk_tolerated.jsonl in the FIRST review round and is the one member of the population in a native script: the walk runs before the positional read, so the pieces the script test sees are all Han and the family-first order stands. It is a corpus name the walk moves like any other, and the reason it took its own ledger rule rather than a seventeenth alternative is a guard about alternations, not a difference in the argument (the ledger comment beside it says which). So the growth is documentation rather than reach, and both numbers come off the same recipe run on the two trees. Re-measured on the docs tree with `_pieces.trailing_titles` stubbed to return 0, which disables the walk at both sites. That fifth is a genuine hit rather than a misfire: rules.md#S1 drops the brackets and reads the content exactly as if written bare, so `(Mgr.)` is a trailing period-marked title word and reads as one; its own test stays green. Recompute by parsing every name in `tools/differential/corpus*.jsonl` on the tree and on the branch's third commit's parent and diffing the seven name fields plus `ambiguities`. Baselines differ per name — `John Smith Dr.` read suffix `Dr.` at 1.4.0, 2.0.0 and 2.1.0 (`dr` left the suffix vocabulary in 2.2, #296) and family `Dr.` at 2.2.0 — so the ledger entries are per-baseline. +- **Measured population: FIVE corpus names at the fix, SEVENTEEN after the docs commit that follows it and the two review rounds on it.** The five are the four planted no-comma rows in corpus_issues.jsonl (`John Smith Dr.`, `John Smith Mr.`, `John Smith Prof.`, `John Smith Rev.`) plus `Andrew Perkins (Mgr.)`, which the drafting did not expect. Eleven of the other twelve are rules.md examples entering corpus_rules.jsonl, eight of them this rule's own — `Smith Prof.`, `Dr. John Smith Prof.`, `John Smith Prof. Dr.`, `John Smith Prof. Jr.`, `John Smith Jr. Prof.`, `John Prof. MA`, `Smith, John Prof.` and `Mary Jane King.`, the last of those added by the first 2026-09-09 review of that commit — plus three of #H1's, which reach this walk because the run each puts behind the name word is what the walk takes: `Smith Sir.`, and `Sir John Prof.` and `Dr. Smith Sir.` from the second review round the same day. The twelfth is `毛 泽东 Dr.`, which entered corpus_cjk_tolerated.jsonl in the FIRST review round and is the one member of the population in a native script: the walk runs before the positional read, so the pieces the script test sees are all Han and the family-first order stands. It is a corpus name the walk moves like any other, and the reason it took its own ledger rule rather than a seventeenth alternative is a guard about alternations, not a difference in the argument (the ledger comment beside it says which). So the growth is documentation rather than reach, and both numbers come off the same recipe — parse every corpus name twice, once on the tree and once with the walk disabled, and diff — run on the two trees. **The disabling stub is NOT the same on the two trees, corrected 2026-09-09 in review of the /simplify round below.** At the fix (`5568fdc`) `trailing_titles` returned the count the chain TAKES and both call sites were in assign, so a stub returning 0 took nothing and disabled the walk. Since the /simplify round it returns the count it LEAVES STANDING, so 0 now takes EVERYTHING and `John Smith` raises IndexError before any measurement happens; the disabling stub on this tree is `lambda rest, *a: len(rest)`, and it has to be bound over both `_pieces.trailing_titles`, which is what `tail_reading` calls, and the name `_assign` imported for the segment-1 walk, patching either alone leaving the other walk live. Re-run that way the two figures hold: FIVE corpus names at `5568fdc`, SEVENTEEN on this branch (2026-09-09). That fifth is a genuine hit rather than a misfire: rules.md#S1 drops the brackets and reads the content exactly as if written bare, so `(Mgr.)` is a trailing period-marked title word and reads as one; its own test stays green. Recompute by parsing every name in `tools/differential/corpus*.jsonl` on the tree and on the branch's third commit's parent and diffing the seven name fields plus `ambiguities`. Baselines differ per name — `John Smith Dr.` read suffix `Dr.` at 1.4.0, 2.0.0 and 2.1.0 (`dr` left the suffix vocabulary in 2.2, #296) and family `Dr.` at 2.2.0 — so the ledger entries are per-baseline. - **The inline frame-free gates at the two walk sites were REMOVED in review**, and this is the ONE-PREDICATE-PER-QUESTION half of the entry. Each site had a cheap inline test written to match the walk's own first condition; that is a second implementation of the question, and the measurement that justified it did not survive re-running. The band test runs early in a session where the facade sits at 453, so the "one frame of headroom" claim did not reproduce. With the gates gone the walk costs +1 frame on each entry point, inside the plan's target of two and inside `test_facade_cost_stays_within_its_band`. The walk's own cheapness is where the saving lives instead: the abbreviation-shape test is a compiled regex (a C call, no Python frame) and runs BEFORE the vocabulary call, and almost no name ends in a period-marked word, so the ordinary parse pays one match and stops. - **A pre-existing detail mismatch, recorded and NOT fixed.** #H4's join shape reports `title-or-name` with a `detail` that says the unit was "read as a given name by convention", while under the default order H1 retags the unit to the family — so `Dr. John of Prince` reports that text with the unit in `family`. This rule adds a second input with the same mismatch, `John of Prince Prof.`, and fixes neither: the wording predates this bundle, the fork the kind reports is title-versus-name which no field answers either way, and #H4 already records why the detail names no field for the peel shape. - 2026-09-09 (review round 2) — **A1's transparency holds for the ONE-name-word case too, and the scope written a day earlier was the defect rather than the boundary.** A1 was scoped that morning to inputs where a name word still stands on both sides of the chain, handing the one-word case to #H1 unconditionally: "the title behind that word decides its field as a title in front of it would". #H1 was then reading BOTH ends of the name as one title run, so `Sir John Prof.` read family `John` where `Sir John` reads given `John` — the trailing title changing what the leading one addressed by, which is exactly the non-transparency A1 denies. The clause is now conditioned on there being no run in FRONT of the word: where one stands it addresses and the chained trailing title only joins the title field, so `X Prof.` reads as `X` plus the title for one name word as for two. Where no run stands in front the trailing one still decides, unchanged — `Smith Prof.` reads family `Smith`, `Smith Sir.` given `Smith`. Nothing in the walk moved; the fix is entirely in which run H1 keys, and #H1's 2026-09-09 entry carries it, the measurement and the mutation. The #P5 clause below reads the same way afterwards and gains its second half: a bound given-name word behind a trailing title joins as it joins with the title absent AND lands in the same field. Said of the COMMA-LESS writing, and the review that found the H1 defect measured why the qualifier is needed: after a family comma the reserve reads no peel at all, so the join fires over the trailing title word and takes it into the given name — `Berg, abdul Prof.` reads given `abdul Prof.` where `Berg, John Prof.` reads title `Prof.`. That is PARITY (1.4.0 reads it the same, measured) and a gap in the segment path rather than a boundary of the rule, so it is recorded rather than fixed here: the clause carries the scope, and the case row `title_word_trailing_behind_a_bound_pair_after_a_comma` pins it without making it normative. Tracked as part of #316. - 2026-09-09 (the /simplify round on the bundle's PR) — **the trailing peel and the title chain are ONE reading run to a FIXED POINT, and A1's "ONE peel runs over what stands" was a truncation of it.** Two contrived readings contradicted clauses this section states, both found by an altitude review reading the code against the rule rather than by a test. (1) THE SUBTRACTION MODEL. #P5's reserve modelled assign's reading as the peel's count minus the chain's take, on each of the two views the join compares. That arithmetic equals assign's answer only while the chain's take does not change what the peel would do, and at #S2's bare-ambiguous reserve it does: `abdul rahman MA` reads given `abdul`, family `rahman`, suffix `MA` — the join declining because peeling the acronym unjoined and not joined is a suffix reading the join would change — while `abdul rahman MA Prof.` read given `abdul rahman`, family `MA`, the subtraction having counted `MA` a name word to spare. The Accepted clause below, "a bound given-name word behind one joins exactly as it joins with the title absent", was false as stated. (2) THE TRUNCATED FIXED POINT. The re-peel ran ONCE, so a second title was read half way: `John Prof. MA` reads title `Prof.`, given `John`, family `MA`, and `John Prof. MA Prof.` read title `Prof.`, given `John`, family `Prof.`, suffix `MA` — the re-peel un-peeled the acronym and re-exposed the FIRST title, which nothing then took. Both measured on the branch tip, 2026-09-09. - **The fix is one function, `_pieces.tail_reading`:** peel, chain, splice the chained pieces out, peel again over what is left, until the chain takes nothing. Assign calls it in place of its inline sequence and all its reports still come from the final peel; the reserve calls it on both views and compares the suffix lists it returns, and both subtractions are deleted. `abdul rahman MA Prof.` now reads given `abdul`, family `rahman`, suffix `MA` under title `Prof.`, and `John Prof. MA Prof.` reads title `Prof. Prof.`, given `John`, family `MA`. One further clause had to be said in the code rather than left to the arithmetic: the join must not take a piece the chain takes (#P5 — "a trailing title word the run takes is no word to spare"), which the subtraction had enforced by accident, by leaving that piece in the tail it compared. Without it `Sir abdul Prof.` joins the title into the given name; with it the reading is title `Sir Prof.`, given `abdul`, as before. `trailing_titles` returns the count it LEAVES STANDING in the same edit, the way `peel_trailing` counts, so the two compose at the call site without arithmetic at all. - **Cost, and what paid it.** The shared helper is one frame on the common path, where the inline sequence was none. It is paid by `_effective_order` taking the segment's pieces and the name-piece indices — the shape every piece-layer predicate takes — instead of a list built for it, which on 3.11 is a comprehension frame on every parse; the reference name stays at 416 parse / 453 facade. The helper returns a bare 3-tuple rather than a NamedTuple for the same budget: a NamedTuple's `__new__` is itself a frame, measured at +1 here. -- **How big the class is, measured as the rule states it.** The two readings named above are the shapes that made the defects visible; the class they belong to is every input carrying TWO trailing period-marked title words, and it is 88 shapes wide over a generated sweep. Recipe: take the alphabet `John Smith abdul rahman Prof. Dr. Sir MA Jr. V Xyz. King de née`, form every 3- and 4-word arrangement ending in `Prof.` or `Dr.` whose spelling WITHOUT that last word still fills two of given/middle/family (3,210 of them — the two-or-more-name-words scope the A1 clause is stated for), and compare the seven fields other than `title` against that shorter spelling. rules.md#H5 says they must be equal. On the branch tip 786 differ; on this commit 698 do, none of them new, and all 698 are the P2/M2 residue the rule already accepts as a boundary — a particle chain or a maiden marker took the trailing word before the H5 chain could see it (`John de Prof.`, `Mary née Prof.`). The 88 the fix closes are all the two-title shape. Over the same sweep's 82,712 inputs (both comma-less and family-comma spellings, 2 to 4 words) exactly 156 parses move, and every one ends in that shape. +- **How big each defect's class is, measured as the rule states it.** The two readings named above are the shapes that made the defects visible; the class both sit in is the TRANSPARENCY class A1 is stated over, and the two defects reach different parts of it — an earlier wording of this bullet flattened them into one width and named the wrong shape for it. Recipe: take the alphabet `John Smith abdul rahman Prof. Dr. Sir MA Jr. V Xyz. King de née`, form every 3- and 4-word arrangement ending in `Prof.` or `Dr.` whose spelling WITHOUT that last word still fills two of given/middle/family (3,210 of them — the two-or-more-name-words scope the A1 clause is stated for), and compare the seven fields other than `title` against that shorter spelling. rules.md#H5 says they must be equal. On the branch tip 786 differ; on this commit 698 do, none of them new, and all 698 are the P2/M2 residue the rule already accepts as a boundary — a particle chain or a maiden marker took the trailing word before the H5 chain could see it (`John de Prof.`, `Mary née Prof.`). The 88 that close PARTITION by defect, and that partition is what makes them two blast radii rather than one. 56 of them carry a SINGLE period-marked title word, and every one of those carries a bound given-name word as well — the subtraction model's reach, `abdul rahman MA Prof.` itself and `abdul John MA Prof.` beside it, where the arithmetic counted the acronym a name word to spare. The other 32 carry TWO, which is the truncated fixed point's reach: `John Prof. MA Prof.` and `John Dr. MA Dr.`, where the single re-peel re-exposed a title nothing then took. The two-title members of the 3,210 number 784, which is the population that could have witnessed the fixed point at all: 164 of them differed on the branch tip and 132 still do, every one of those 132 the same P2/M2 residue the 698 are, so the defect accounted for exactly the 32 that closed and no more. Over the same sweep's 82,712 inputs (both comma-less and family-comma spellings, 2 to 4 words) exactly 156 parses move, splitting the same way — 96 carry two period-marked title words, 60 carry one, and all 60 of those are bound-given — and not one mover is a comma spelling, the family-comma segment path reaching neither defect. Every figure in this bullet re-measured 2026-09-09 on this tree against `7ac2218`, the round's parent. - **Nothing else moved, and it was measured rather than argued.** Snapshot all seven fields, the ambiguity kinds and the recorded `order` for every name in `tools/differential/corpus*.jsonl` plus every string literal in `tests/v2/cases.py` (`ast.walk` over that file, which sweeps up the notes and the case ids too — harmless, they parse like anything else), on the branch tip and on the fix, and diff: 2682 strings before and 2681 after, 2674 of them on both trees, and NONE of those 2674 reads differently. The seven that differ are the case rows this round added and dropped, not parses that moved. The two readings above are inputs no corpus holds — as is every other member of the class the bullet below measures — and each takes a case row (`title_word_trailing_run_is_read_to_a_fixed_point`, `title_word_trailing_behind_a_bound_pair_at_the_peel_reserve`). The same round dropped the `Sir Jr` example from rules.md#S2 and its case row: it reads by the same branches as `Dr Jr`, the same roles and the same reported kind, the run being empty by then and no branch reading `vocab:given-title` — so the pair pinned one reading twice. Its leaving corpus_rules.jsonl moves two ledger rosters: the jr suffix-routing rule 7 → 6 corpus names in the 1.4.0 ledger, and the #489 peel-floor rule 4 → 3 in all four. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 53abef9b..159b54f1 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -55,7 +55,7 @@ Problem shape. "Which stage does X?" — asked before attributing behavior in pr ## ONE-PREDICATE-PER-QUESTION — one predicate answers it, and every other site calls that -Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles, peel_walk and peel_trailing are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` was that last shape for one day (2026-09-08, the #316/#489 bundle, rules.md#H5) and is a shared one since 2026-09-09: assign calls it at two sites — the main walk and the family-comma segment-1 walk — and group's bound-given reserve at a third, because that reserve counts the name words assign will leave and this walk is half of what leaves them (rules.md#P5; counting a trailing title word among them joined 'Prof. abdul rahman Prof.' where 'Prof. abdul rahman' does not). It is in the leaf rather than inline because each assign site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. The reserve's two calls cost the reference name nothing — it never enters that branch, having no bound given word — and the parse and facade frame counts did not move (measured 2026-09-09). Re-measured 2026-09-09 by call site over `_pipeline/*.py`, the rest of the census above holds unchanged: is_suffix_piece, leading_titles, peel_walk, peel_trailing and now trailing_titles shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. +Problem shape. Two stages need the same answer about the same input, and the one that does not own the decision is about to test for it. Contract statement. Where two sites ask the same question, exactly one predicate answers it and every other site calls that one — never a condition written to match it. The predicate belongs to the QUESTION, not to whichever stage decides: it may sit in a leaf both stages import, and for the leading-title test it must, since the deciding stage is assign and group cannot import assign. How it works. A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Five instances, every one found as a defect before it was found as a pattern — #319 lifted the wholly-suffix predicate into the vocabulary layer "so the comma decision and the honorific peel's segment test cannot drift apart"; #401/#421 lifted the trailing-numeral fork out of assign so the bound-given reserve stopped carrying a copy, its hand-written mirror having been falsified in review more than once — the lesson recorded there being that what must be mirrored is assign's WALK, not merely its condition; #425 replaced that reserve's hand re-derivation of the trailing peel with one function over the view the join would leave; #424 moved assign's leading-title test down because group's own `title()` does not see H2's unlisted abbreviations, so `Xyz. van Johnson` chained where `Dr. van Johnson` did not; #429 moved the no-name-segment test down because group asked by segment INDEX where assign asks by CONTENT. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`. Both are leaves the stages sit on. The piece layer got its own module only in #439 — until then those predicates collected in `_group`, not because grouping owned them but because `_assign` imports `_group` and cannot be imported back, so group was the one place both stages could reach; five had accumulated across four PRs before the module existed. Stage order is this mechanism's limit, and it forecloses the alternative: where the reader comes AFTER the decider, record the answer on the state instead — `ParseState.order` is that shape, "Recorded rather than recomputed downstream, because the two can differ" — which is unavailable whenever the EARLIER stage is the one asking. (The concrete assign→group import that forced the `_group` collection is gone since #439; what remains is the ordering it was a symptom of, and tests/v2/test_layering.py is where the leaf's contract is now written down.) The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other; recording that number was the right answer there over plumbing a state field the two sites would not otherwise share. Lives in. nameparser/_pipeline/_vocab.py over text (is_wholly_suffix; is_trailing_numeral_suffix — the #401/#421 instance, whose only caller since #439 is the shared peel rather than a stage; and maiden_marker_run, the #434 instance and the clearest two-stage case, called by classify over token texts and by extract over a clause's whitespace words, with group reading the tags classify recorded because it runs later; and delimiter_cores, the #436/#437 instance, read by group where a tail segment DROPS a configured delimiter core and by post_rules where the suffix view's entry boundary asks whether a dropped token was one, with a third reader inside this same module, is_wholly_suffix, where a configured core counts as suffix-shaped) and nameparser/_pipeline/_pieces.py over pieces: is_suffix_piece, leading_titles and peel_walk are called by both stages, while is_leading_title, is_title_piece and trailing_start are called by group alone (measured 2026-09-06 by call site: `is_leading_title` has no caller in `_assign.py`, which reads `leading_titles` instead — a first draft of this clause listed it among the shared ones) — `trailing_start` being the one to know, since it answers where the trailing run begins and is what P2's chain and M2's walk stop at — and segment_suffix_reading by assign alone since #436/#437, that last one being #430's instance, where THREE readers shared one answer until the render join, group's third, was replaced by a rule over the commas the writer typed (decisions.md#C1, 2026-09-06); it stays where it is, one call site being no reason to move a predicate that two sites will contest again. `trailing_titles` was that last shape for one day (2026-09-08, the #316/#489 bundle, rules.md#H5), and since the /simplify round of 2026-09-09 the SHARED predicate is `tail_reading` instead — the peel-and-chain fixed point that answers where the name pieces end (decisions.md#H5). Assign calls it at its main walk and group's bound-given reserve calls it twice, once per view the join compares, because that reserve reads the name words assign will leave and this walk is half of what leaves them (rules.md#P5; counting a trailing title word among them joined 'Prof. abdul rahman Prof.' where 'Prof. abdul rahman' does not). `peel_trailing` and `trailing_titles` are what that fixed point is BUILT from, and neither is a two-stage question any longer: `peel_trailing` has no caller outside `_pieces.py` at all, `trailing_start` and `tail_reading` being the only two and both in the leaf, and `trailing_titles` has exactly one, assign's family-comma segment-1 walk, which reads the chain without the re-peel; `_group.py` imports neither. The tail reading is in the leaf rather than inline because each assign site had been given a cheap frame-free gate written to match the walk's own first condition, which is a second implementation of the question and was removed in review; what the leaf costs is one frame per entry point, measured, and the walk's own first test is a compiled regex rather than a call, so an ordinary name pays a match and stops. The reserve's two calls cost the reference name nothing — it never enters that branch, having no bound given word — and the parse and facade frame counts did not move (measured 2026-09-09). Re-measured 2026-09-09 by an AST call-site census over `_pipeline/*.py` — every call node whose callee is one of these names, keyed by module and enclosing function, which is what caught the census claiming a share for `peel_trailing` that the round had just taken away — the rest of it holds unchanged: is_suffix_piece, leading_titles, peel_walk and now tail_reading shared, is_leading_title, is_title_piece and trailing_start group-only — assign still reads `leading_titles` and never `is_leading_title`, which is what keeps H2's shape inference out of the trailing slot. And nameparser/_pipeline/_post_rules.py over a state: suffix_entries, the #511 instance, the R1 entry pass as a function, the one instance living in a stage rather than in a leaf — it is a pass over a whole ParseState and no leaf takes one, and AGENTS.md names it as the exception — run by post_rules last in the stage (through its in-place worker) and by Parser.revise over a sub-parse whose roles it has forced, so a suffix value handed to revise() derives its entries by the rule a whole name uses rather than by a second reading of the value's commas (decisions.md#C1, 2026-09-06 #511). tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. Reach for it when. You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. ## RENDER-HONORS-THE-PARSE — the parse decides it, the views honor it diff --git a/docs/release_log.rst b/docs/release_log.rst index 67b2b31c..bff433aa 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -40,7 +40,7 @@ Release Log - **Add AmbiguityKind.GIVEN_OR_FAMILY, reported when a name of one name word had nothing to decide which field it is:** ``parse("Andrew")`` still gives given ``Andrew`` and now says that field was a convention rather than a reading -- one word gives the positional rule nothing to compare, so the library picks the given name under the default order and the family name under a declared family-first one, and ``detail`` names the field it picked. A trailing suffix does not decide it either: ``parse("Smith Jr.")`` reports it too, the suffix being peeled and the convention placing the one name word left. A name something DID decide stays silent -- ``"Dr. Smith"``, ``"Smith née Jones"``, ``"'Smitty' Jones"`` and ``"Smith, Andrew"`` -- and so do ``"abdul"`` and ``"de"``, where the bound given-name and particle vocabularies claimed the word, and ``"J."``, claimed by the initial's own shape. A name whose script settles the order is silent too: ``"毛泽东"`` reads family by convention of the writing system, not of this rule. Twenty-seven names in the differential corpora gain the report, and no field moves anywhere. See the ``O5`` entry of ``docs/design/decisions.md`` (closes #449) - - **Add AmbiguityKind.TITLE_OR_NAME, reported when an input that is nothing but honorifics had its last word read as the name:** ``parse("Lord Chancellor")`` still gives title ``Lord``, family ``Chancellor``, and now says so -- this is a name parser, not a title parser, so handed a string with no name in it, it reads the last title word as one. ``"The Right Hon. the President of the Queen's Bench Division"`` and ``"His Holiness"`` move the same way, and so does ``"Dr. King"``, ``king`` being title vocabulary for the addressing forms. A title with an ordinary word behind it is silent (``"Dr. Smith"``, ``"King Charles"``), and so is a lone title word: ``parse("Dr.")`` is a title with no name beside it, the peel having taken the whole string, so no word was left standing to be read as a name and nothing was chosen. The same convention on the other vocabulary reports the existing suffix-or-name -- ``parse("Rinpoche")`` gives given ``Rinpoche`` and flags it, as does ``"QC MP"`` -- while ``"Jr."`` is unchanged and unflagged, reading as a title on its shape. The same doubt inside a joined unit reports too -- ``parse("Attorney General of Minnesota")`` reads title ``Attorney``, family ``General of Minnesota``, and whether ``General`` is a title is the fork. Eight names in the differential corpora gain a report from this change -- ten rows, two of those names sitting in two corpus files each -- and no field moves. Four more inputs join the kind later in this cycle, from the title-peel fix above -- ``Dr King Jr``, ``Dr. King MD``, ``Dr Jr`` and ``Sir Jr``, where the run now leaves a title-vocabulary word standing; those move fields, for the reasons that bullet gives. See the ``H4`` entry of ``docs/design/decisions.md`` (closes #491) + - **Add AmbiguityKind.TITLE_OR_NAME, reported when an input that is nothing but honorifics had its last word read as the name:** ``parse("Lord Chancellor")`` still gives title ``Lord``, family ``Chancellor``, and now says so -- this is a name parser, not a title parser, so handed a string with no name in it, it reads the last title word as one. ``"The Right Hon. the President of the Queen's Bench Division"`` and ``"His Holiness"`` move the same way, and so does ``"Dr. King"``, ``king`` being title vocabulary for the addressing forms. A title with an ordinary word behind it is silent (``"Dr. Smith"``, ``"King Charles"``), and so is a lone title word: ``parse("Dr.")`` is a title with no name beside it, the peel having taken the whole string, so no word was left standing to be read as a name and nothing was chosen. The same convention on the other vocabulary reports the existing suffix-or-name -- ``parse("Rinpoche")`` gives given ``Rinpoche`` and flags it, as does ``"QC MP"`` -- while ``"Jr."`` is unchanged and unflagged, reading as a title on its shape. The same doubt inside a joined unit reports too -- ``parse("Attorney General of Minnesota")`` reads title ``Attorney``, family ``General of Minnesota``, and whether ``General`` is a title is the fork. Eight names in the differential corpora gain a report from this change -- ten rows, two of those names sitting in two corpus files each -- and no field moves. Three more corpus names join the kind later in this cycle, from the title-peel fix above -- ``Dr King Jr``, ``Dr. King MD`` and ``Dr Jr``, where the run now leaves a title-vocabulary word standing; those move fields, for the reasons that bullet gives. ``Sir Jr`` reads the same way -- first ``Sir``, suffix ``Jr``, the same report -- but no differential corpus holds it, so it is ``Dr Jr``'s row that pins the reading for both. See the ``H4`` entry of ``docs/design/decisions.md`` (closes #491) * 2.2.0 - August 31, 2026 diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 055e664e..23b54a47 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -3026,9 +3026,14 @@ issue = "fix(#489) the title peel leaves a name word a suffix cannot be" # declines and every one of which is a _MUST_NOT_MATCH probe, with # 'Dr Smith Jr' beside them for the ordinary titled name the floor # never sees. The members are a list of names and copy no wordlist, -# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins -# the reach at 4 with its digest. One set, identical in all four -# ledgers; only the `fields` differ, as the paragraph above says. +# which is what _NOT_A_VOCABULARY_COPY records. THREE of the four are +# corpus names, not four: 'Sir Jr' left corpus_rules.jsonl on +# 2026-09-09, when the rules.md example pinning it was dropped as a +# duplicate of 'Dr Jr' (decisions.md#H5), and it stays in the +# alternation because a name_regex names INPUTS rather than corpus +# rows. _CORPUS_CLAIMS pins the reach at 3 with its digest. One set, +# identical in all four ledgers; only the `fields` differ, as the +# paragraph above says. name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" fields = ["family", "given", "suffix", "title"] diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 93b7f23b..754da1a6 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -1892,9 +1892,14 @@ issue = "fix(#489) the title peel leaves a name word a suffix cannot be" # declines and every one of which is a _MUST_NOT_MATCH probe, with # 'Dr Smith Jr' beside them for the ordinary titled name the floor # never sees. The members are a list of names and copy no wordlist, -# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins -# the reach at 4 with its digest. One set, identical in all four -# ledgers; only the `fields` differ, as the paragraph above says. +# which is what _NOT_A_VOCABULARY_COPY records. THREE of the four are +# corpus names, not four: 'Sir Jr' left corpus_rules.jsonl on +# 2026-09-09, when the rules.md example pinning it was dropped as a +# duplicate of 'Dr Jr' (decisions.md#H5), and it stays in the +# alternation because a name_regex names INPUTS rather than corpus +# rows. _CORPUS_CLAIMS pins the reach at 3 with its digest. One set, +# identical in all four ledgers; only the `fields` differ, as the +# paragraph above says. name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" fields = ["_ambiguities", "family", "given", "suffix", "title"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index acec949e..93198433 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -1811,9 +1811,14 @@ issue = "fix(#489) the title peel leaves a name word a suffix cannot be" # declines and every one of which is a _MUST_NOT_MATCH probe, with # 'Dr Smith Jr' beside them for the ordinary titled name the floor # never sees. The members are a list of names and copy no wordlist, -# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins -# the reach at 4 with its digest. One set, identical in all four -# ledgers; only the `fields` differ, as the paragraph above says. +# which is what _NOT_A_VOCABULARY_COPY records. THREE of the four are +# corpus names, not four: 'Sir Jr' left corpus_rules.jsonl on +# 2026-09-09, when the rules.md example pinning it was dropped as a +# duplicate of 'Dr Jr' (decisions.md#H5), and it stays in the +# alternation because a name_regex names INPUTS rather than corpus +# rows. _CORPUS_CLAIMS pins the reach at 3 with its digest. One set, +# identical in all four ledgers; only the `fields` differ, as the +# paragraph above says. name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" fields = ["_ambiguities", "family", "given", "suffix", "title"] diff --git a/tools/differential/expected_since_2.2.0.toml b/tools/differential/expected_since_2.2.0.toml index 5f4d77a4..1a499ba8 100644 --- a/tools/differential/expected_since_2.2.0.toml +++ b/tools/differential/expected_since_2.2.0.toml @@ -452,9 +452,14 @@ issue = "fix(#489) the title peel leaves a name word a suffix cannot be" # declines and every one of which is a _MUST_NOT_MATCH probe, with # 'Dr Smith Jr' beside them for the ordinary titled name the floor # never sees. The members are a list of names and copy no wordlist, -# which is what _NOT_A_VOCABULARY_COPY records; _CORPUS_CLAIMS pins -# the reach at 4 with its digest. One set, identical in all four -# ledgers; only the `fields` differ, as the paragraph above says. +# which is what _NOT_A_VOCABULARY_COPY records. THREE of the four are +# corpus names, not four: 'Sir Jr' left corpus_rules.jsonl on +# 2026-09-09, when the rules.md example pinning it was dropped as a +# duplicate of 'Dr Jr' (decisions.md#H5), and it stays in the +# alternation because a name_regex names INPUTS rather than corpus +# rows. _CORPUS_CLAIMS pins the reach at 3 with its digest. One set, +# identical in all four ledgers; only the `fields` differ, as the +# paragraph above says. name_regex = "^(?:Dr Jr|Dr King Jr|Dr\\. King MD|Sir Jr)$" fields = ["_ambiguities", "family", "given", "suffix", "title"] From ed8ef121dab5602c6dfa91950dcd3bb34f160bd4 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson <derek73@gmail.com> Date: Wed, 9 Sep 2026 20:51:59 -0700 Subject: [PATCH 12/12] docs(differential): the 1.4.0 header no longer counts Sir Jr as corpus reach Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --- tools/differential/expected_since_1.4.0.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 23b54a47..02d425b0 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -2920,7 +2920,8 @@ orders = ["DEFAULT"] # ledgers: at 1.4.0 fix(#296)'s trailing `dr` reaches 'John Smith # Dr.' and 'John Smith Prof. Dr.', the two comma routings reach # 'Smith, John Prof.', and the two-token `jr` rule reaches 'Dr Jr' -# and 'Sir Jr'; at 2.0.0 and 2.1.0 only fix(#296) reaches any of +# (and reached 'Sir Jr' until that row left corpus_rules.jsonl on +# 2026-09-09); at 2.0.0 and 2.1.0 only fix(#296) reaches any of # them; at 2.2.0 none does. Not one of those rules ADMITS the diff # it reaches -- every shape is outside its field list, measured -- # so nothing below takes a name off a rule that was explaining it,