Skip to content

add_ranges_file appends a second header reference when the companion spells the range id in another normalization #33

Description

@imnasnainaec

Summary

Lexicon.add_ranges_file decides which ranges the header already references by exact string
comparison, so a header that spells a range id in NFC and a companion that spells it in NFD
are treated as unrelated ranges: a second <range> header reference is appended for the
same conceptual range, and save() writes both. FLEx mixes normalizations between an id and
the references to it (#14, #27), so a lexicon derived from a real export is exactly where
the two spellings meet.

Reproduction

import sil_lift, unicodedata

nfd = lambda s: unicodedata.normalize("NFD", s)
name = "Catégorie"

lex = sil_lift.Lexicon()
lex.header.ranges.append(sil_lift.Range(id=name, href="dup.lift-ranges"))  # NFC in the header
ranges = sil_lift.RangesFile()
ranges.add_range(nfd(name)).add_element("Nom")                             # NFD in the companion
lex.add_ranges_file(ranges, href="dup.lift-ranges")

print([ascii(r.id) for r in lex.header.ranges])
lex.save("dup.lift")
["'Cat\\xe9gorie'", "'Cate\\u0301gorie'"]

and in the saved .lift:

<ranges>
  <range id="Catégorie" href="dup.lift-ranges"/>
  <range id="Catégorie" href="dup.lift-ranges"/>
</ranges>

Two references, rendering identically, differing only by normalization — one of which the
caller never asked for.

Cause

src/sil_lift/_model.py:825-829:

referenced = {range_.id for range_ in self.header.ranges}
for range_ in ranges_file.ranges:
    if range_.id not in referenced:
        self.header.ranges.append(Range(id=range_.id, href=href))
        referenced.add(range_.id)

The membership test is exact, so the two spellings of one name are different keys.

Expected

Resolve referenced the way the validator resolves a name to an id — exact spelling first,
then NFC — and append nothing when the header already references the range under either
spelling. The existing header id must not be rewritten: whichever spelling the document came
with is the one it keeps.

Scope

Write path only. Validation of such a document is already correct as of #28, which resolves
a header range/@id against a companion's range id under NFC and reports the split as a
normalization-mismatch warning. This is about what save() then writes.

Distinct from #29: that is the merged read view (all_ranges()) dropping a range when two
companions define the same id. Same underlying theme — ids compared as exact strings — but a
different code path and a different symptom. See also the all_ranges() NFC-keying note on
that issue.

Notes

Pre-existing; not introduced by #28.

Reachability is narrow: it needs a header that already references the range under one
spelling plus a call to add_ranges_file with a companion spelling it the other way — the
documented "call again to reference ranges added later" flow on a FLEx-derived lexicon. No
corpus fixture exercises it, so this needs a hand-authored case.

One design decision comes with it: the package's only NFC machinery is three closures inside
_semantic_problems (src/sil_lift/_validate.py:432). Fixing this needs either a second
unicodedata.normalize call in _model.py or promoting nfc/resolve into a shared
private module. The latter is probably right if any other module ever needs it, but it is
worth deciding deliberately rather than inlining a call a later refactor has to undo.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions