Skip to content

The issues corpus misses every name written in backticks — 104 of them, including Beethoven, Ludwig van #413

Description

@derek73

build_issues_corpus.py harvests name-like strings from the issue tracker, but its pattern only recognizes names inside "..." or '...':

_CANDIDATE = re.compile(
    r"""HumanName\(\s*["']([^"']{3,60})["']"""
    r"""|["']([A-Z][^"'\n]{4,60})["']"""
)

Issues here are markdown, and the house convention writes names in backticks. So the corpus whose whole purpose is "what users reported" has been systematically blind to the way this project reports things.

Measured

Adding a backtick branch to the pattern, over all 220 issues:

shipped regex:     291 names
with backticks:    394 names   (+104)
checked-in corpus: 200 names

All 104 are absent from the checked-in corpus. They are not junk — many are the headline example of the issue that fixed them:

Beethoven, Ludwig van        #379's example
Berg, Jan vd                 #380's example
Del Toro                     the family-first particle shape
Ursula von der Leyen geb. Albrecht    #399's example
Jane van der Berg née Jones           #399's example
Abd van der Berg née Jones            #411's example
Dr. Smith née Jones                   #410's example
Aishwarya Rai Bachchan
Baba Ramdev

Why it matters beyond the count

The three corpora exist to have different blind spots (build_issues_corpus.py's own docstring makes that argument). This is a blind spot none of them was meant to have: it is not "v1's authors had no reason to test this" or "the tracker had not seen it yet", it is a formatting convention silently filtering the harvest.

Concretely, #399 was closed noting its class was invisible to the differential gate — no corpus name had a particle ahead of a maiden marker. The name that would have made it visible was in #399's own title the entire time, in backticks.

Scope of the fix

The pattern change is small. The consequences are not, and that is the reviewable part:

  • 104 new names enter the differential gate, and each diff they produce needs classifying in up to three ledgers. Expect real work in expected_since_*.toml, and expect some of it to surface intended-but-unclassified behavior, which is what happened on this corpus's first run.
  • Over-collection grows too. _NOT_A_NAME screens {}<>=/\| but not parens, * or #, so a backtick branch also admits Constants.__init__(self, **state) and DEVIATION #364. The builder's docstring accepts over-collection on principle ("costs one parse and produces no diff"), so this may be fine as-is; worth deciding rather than assuming, since backticks in this tracker wrap code far more often than quotes do.
  • Regeneration is already an explicit, reviewable act by design, so this should land as its own PR rather than riding along with a parser fix.

Not doing this by hand

PR #409 briefly added a synthetic Jane van der Berg 旧姓 Jones case row to reach the CJK corpus builder — the only case-row-to-corpus route that exists — purely to make the #399 class gate-visible. That was removed: it used the CJK corpus's provenance as a back door, and it pinned nothing the Latin rows did not. Fixing the harvester is the honest version of the same goal.

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions