Skip to content

fix(svm): decode the text, and close out the plan - #793

Merged
andiwand merged 1 commit into
mainfrom
fix/svm-text-encoding
Aug 30, 2026
Merged

fix(svm): decode the text, and close out the plan#793
andiwand merged 1 commit into
mainfrom
fix/svm-text-encoding

Conversation

@andiwand

Copy link
Copy Markdown
Member

🤖 Generated with Claude Code

Closes #772. This is the last of its ten defects, plus the doc pass that
retires the plan.

The defect

Defect 1 was raised as "text is not XML-escaped". Escaping landed in the first
stage; the other half did not. read_string_with_encoding handed back the
file's own bytes for every encoding but UCS2:

if (encoding == RTL_TEXTENCODING_UCS2) { return read_uint32_prefixed_utf16_string(in); }
return read_uint16_prefixed_ascii_string(in);   // bytes, undecoded

So a label in MS_1252 or an ISO-8859 part emitted invalid utf-8, and an xml
parser refuses that exactly as hard as the unescaped & the defect was raised
for — the whole image, not one label. README.md carried it as the "Known
defect" and svm/AGENTS.md as "escaping is not yet enough".

The charset a FONT action names now picks a decoder from
internal/encoding, which already has the single-byte tables. 29 of the
rtl_TextEncoding values map onto one; anything else is taken for MS_1252,
which is what a file of this age most likely means and what vcl falls back to
— a label in the wrong characters still draws, where the bytes cost the image.

Two things that came with it:

  • The font's own names are bytes too, and go straight into font-family.
    They are decoded with the charset the same action reads next. (vcl uses the
    stream charset for these, which a metafile never sets.)
  • A run offset counts utf-16 code units, because SvmReader decodes the
    string before it indexes the OUString. It was a byte substr, which was
    correct only while the string was the raw bytes; after decoding it would
    have cut a character in half.

The numbers are from
include/rtl/textenc.h
rather than memory — SYMBOL is 10 and ASCII_US 11, with 9 unused, so the
constant already in the enum was right.

The docs

svm/PLAN.md is spent — every stage is done — so it goes. Two parts had
forward value and moved to svm/AGENTS.md rather than going with it: the
corpus frequency table, which is the evidence behind every "occurs
nowhere" claim in the module and exists nowhere else, and the
FLOATTRANSPARENT shortcut.

svm/README.md's feature matrix was left behind by #784#791 — it still
advertised unchecked boxes for the poly-polygon fill rule, the primitives,
béziers, bitmaps, gradients, clipping and the state stack, all of which
shipped. It now says what is true (38 of the 54 action types we name are
drawn), lists what is not implemented and why, and names the two places the
drawing is deliberately approximate (SQUARE/RECT gradients as ellipses,
the triple hatch's diagonal spacing).

What is left in the module

None of it occurs in the corpus, and each is logged where it is skipped:
TEXTRECT and TEXTLINE (the reader has both, nothing draws them),
MOVECLIPREGION, WALLPAPER, FLOATTRANSPARENT, EPS, the MASK family,
ZCOMPRESSed dibs, and version-1 (pre-VCLMTF) files.

Verification

3 new tests (50 in the svm suite): the same bytes decode differently under
MS_1252 and latin-1 (0x92 is a curly apostrophe in one and a control
character in the other), an unknown encoding still yields valid utf-8, and a
run offset survives a character that is more than one utf-8 byte. Full suite
1326 passed, 6 skipped, no reference output moves — the corpus text is UCS2
or ascii, so nothing rendered changes.

Checked as CI compiles, -Wall -Wextra -Werror, not just as the local build
does.

Closes #772, whose last open defect this is. `read_string_with_encoding`
handed back the file's own bytes for every encoding but `UCS2`, so a label in
`MS_1252` or an ISO-8859 part emitted invalid utf-8 - and an xml parser
refuses that exactly as hard as the unescaped `&` the same defect was raised
for, costing the whole image rather than one label.

The charset a `FONT` action names now picks a decoder from
`internal/encoding`, which already has the single-byte tables. An encoding we
have no decoder for is taken for `MS_1252`, what a file of this age most
likely means and what vcl falls back to: a label in the wrong characters
still draws. The font's own family and style names go through it too - they
are bytes as well, and they go straight into `font-family`.

The run a text action names is measured in utf-16 code units, because vcl
decodes the string before it indexes it. That was a byte `substr` before,
which was right only while the string was the raw bytes.

Numbers checked against `include/rtl/textenc.h`: `SYMBOL` is 10 and
`ASCII_US` 11, 9 being unused.

The plan is spent, so `svm/PLAN.md` goes. What had forward value moved rather
than went: the corpus frequency table, which is the evidence behind every
"occurs nowhere" in the module, and the `FLOATTRANSPARENT` shortcut, both now
in `svm/AGENTS.md`. `svm/README.md`'s feature matrix was left behind by
it now says what is true, including where the drawing is deliberately
approximate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQpLmJpJ87qbKoG8B7kbLY
@andiwand
andiwand force-pushed the fix/svm-text-encoding branch from 680adb3 to fbe0bc9 Compare August 30, 2026 19:33
@andiwand
andiwand merged commit a990ce6 into main Aug 30, 2026
25 checks passed
@andiwand
andiwand deleted the fix/svm-text-encoding branch August 30, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve SVM to SVG conversion

1 participant