Redirected CLI output crashes on characters the locale encoding cannot represent, truncating export

[imnasnainaec]

Summary

Every CLI command that prints lexicon content writes to sys.stdout with the platform's
locale encoding. When stdout is not a console — a pipe, a > redirect — that is cp1252 on
this Windows box and ASCII under a C/POSIX locale, so any character the codepage cannot
represent raises UnicodeEncodeError and the command dies mid-output. LIFT content is
Unicode by definition: NFD combining marks and IPA modifiers like ː (U+02D0) are ordinary
in real FLEx exports.

export is the worst of it. Redirected, it crashes partway and leaves a truncated file
whose bytes are not even UTF-8; the same export through -o is complete and UTF-8.

Reproduction

validate, on a committed fixture:

$ sil-lift validate tests/corpus/negative/nfd-range-ids.lift > out.txt
Traceback (most recent call last):
  ...
  File "src\sil_lift\_cli.py", line 89, in _cmd_validate
    print(problem)
UnicodeEncodeError: 'charmap' codec can't encode character '́' in position 90

The message that fails is range-parent, which interpolates ids with !r:

range 'grammatical-info': range-element 'Órfão' has parent 'Preposicao' ...

Not specific to Windows — any stdout encoding that cannot hold the content fails, so a
C/POSIX locale (ASCII stdout) fails too, and there on any non-ASCII content, precomposed
'\xe9' included:

$ PYTHONIOENCODING=ascii sil-lift validate tests/corpus/negative/nfd-range-ids.lift
UnicodeEncodeError: 'ascii' codec can't encode character '́' in position 90

export, same command twice, on tests/corpus/large/sango/sango.lift:

$ sil-lift export sango.lift > stdout.csv
  File "src\sil_lift\_cli.py", line 250, in _cmd_export
    writer.writerow(row)
UnicodeEncodeError: 'charmap' codec can't encode character 'ː' in position 158
$ sil-lift export sango.lift -o out_flag.csv     # exit 0

stdout.csv      202656 bytes   1118 lines   NOT UTF-8: invalid continuation byte
out_flag.csv    837883 bytes   4542 lines   valid UTF-8

So the redirected run loses 3424 of 4542 rows and writes cp1252 bytes for the ones it does
emit, with no indication in the file itself that it is incomplete.

Cause

The CLI forces an encoding when it opens a file and inherits the locale when it does not
(src/sil_lift/_cli.py:234-236):

if args.output is None:
    out_file = sys.stdout                                          # locale encoding
else:
    out_file = args.output.open("w", encoding="utf-8", newline="")  # explicit

Same asymmetry for the text-mode print sites: print(problem) in validate
(src/sil_lift/_cli.py:89), the trait/language lines in stats
(src/sil_lift/_cli.py:143-150), and the href lines in check-media
(src/sil_lift/_cli.py:166, src/sil_lift/_cli.py:185).

Not affected

• --format json — json.dump defaults to ensure_ascii=True, so validate's JSON output
  is pure ASCII and safe.
• export -o FILE, and the Lexicon.save() / sort writers — all explicitly UTF-8.
• An interactive Windows console, where Python writes through WriteConsoleW rather than
  the codepage. Only redirected runs fail, which is presumably why this has gone unnoticed:
  the failure appears exactly when the output is being captured for something else.

Options

1. Reconfigure the standard streams to UTF-8 at CLI entry when they are not already
   (sys.stdout.reconfigure(encoding="utf-8")). One place, matches what --output
   already does, and makes redirected output byte-stable across platforms and locales.
2. errors="backslashreplace" on the streams instead. Never crashes, but converts a
   loud failure into silently corrupted data in the export path, which is the one place
   the bytes are the product.
3. Leave it and document PYTHONIOENCODING=utf-8. No code change, but the CLI stays
   unusable in a pipeline by default on any non-UTF-8 platform.

1 looks right, with a test that runs each command with stdout redirected to a cp1252/ASCII
stream over non-ASCII content.

Notes

Pre-existing; not introduced by #28. On main, sil-lift validate tests/corpus/large/sango/sango.lift > out.txt exits 1 after 42 lines with the same
combining-acute failure: its two range-parent findings name an NFD Complément du lieu unescaped. #28 removes those two findings, and its own normalization-mismatch
message is ASCII-safe because it escapes both ids with !a, so nothing there changes the
shape of this in either direction.

Related but separate: #21 covers surrogate pairs and UTF-16 on the input side; this is
the output side.

Environment: Windows 11, Python 3.12, locale.getpreferredencoding() = cp1252.

[imnasnainaec]

Sharpening the link to #21, which is closer than "input side vs output side": its subject
matter both triggers this and constrains the fix.

Non-BMP characters trigger it. A "surrogate pair" in a UTF-16 source is just a non-BMP
character once decoded, and no legacy codepage can hold one. Adlam U+1E900 as a
grammatical-info value:

$ sil-lift validate nonbmp.lift > out.txt          # cp1252 stdout
exit 1  UnicodeEncodeError: 'charmap' codec can't encode character '\U0001e900'
$ PYTHONIOENCODING=utf-8 sil-lift validate nonbmp.lift > out.txt
exit 0  warning [undefined-range-value] ... value '\U0001e900' is not defined in the range

Option 1 covers this cleanly, and it means any Plane-1 script — not just NFD marks — puts a
lexicon in scope here.

Lone surrogates would defeat option 1. UTF-8 refuses them too, so reconfiguring the
encoding is not sufficient on its own:

cp1252:                   character maps to <undefined>
utf-8:                    surrogates not allowed
utf-8 backslashreplace:   b'\\ud800'
utf-8 surrogatepass:      b'\xed\xa0\x80'   (invalid UTF-8)

If #21 concludes the tooling should tolerate a lone surrogate rather than reject it, option
1 has to become option 1 plus errors="backslashreplace" — and surrogatepass is not a
candidate for export, where the bytes are the product.

That is bounded today: a lone surrogate cannot reach the model through the reader, because
lxml rejects it before the model sees it.

LiftParseError: not well-formed XML: xmlParseCharRef: invalid xmlChar value 55296

Only in-memory construction can produce one, so the errors= question is about API users,
not about files on disk.