Compressed translations: escape rare characters instead of 16-bit tables - #11373
Open
dhalbert wants to merge 4 commits into
Open
Compressed translations: escape rare characters instead of 16-bit tables#11373dhalbert wants to merge 4 commits into
dhalbert wants to merge 4 commits into
Conversation
…timal parsing Rework the translation compressor in `py/maketranslationdata.py` and the decoder in `translate.c`: - Renumber a language's non-ASCII characters into 0x80.. by frequency and map them back through `alphabet[]`, so every symbol fits in 8 bits and the dictionary gets the remaining code points. Languages with more than 127 distinct non-ASCII characters (ja, ko) keep the 16-bit fallback. - Choose Huffman codes per class of the previously decoded byte (start or space, letter, other, ...) through `class_map[]`; the generator tries several class presets and keeps the smallest result. - Tokenize each string by dynamic programming for the fewest bits instead of greedy longest match, deciding per occurrence whether a qstr reference pays off. Dictionary words left unused are dropped. The format is documented in `compressed_string.h`. Same-commit gains on the tightest languages: metro_m4_express fr +1,208 bytes free, pl +1,160; metro_m0_express pl +672. The decoder grows 76-108 bytes. Verified on the unix coverage port (979 tests) and on a Metro M4 Express in fr and en_US. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add `translation_class_t` and `translation_symbol_t` enums to `compressed_string.h` and use the names in `base_class()`, the decode loop, and the matching Python constants in `maketranslationdata.py`, instead of bare 0..6 and 1. No change to generated data or firmware. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dhalbert
force-pushed
the
ja-ko-escapes
branch
from
September 14, 2026 03:09
0c6a91b to
cabc123
Compare
Write each preset as the list of Huffman tables to build, each naming the base classes that share it, and derive `class_map[]` with `class_map_for()` instead of spelling out index tuples. Generated output is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Translations with more than 127 distinct non-ASCII characters (ja, ko) used `uint16_t` for every `values[]` and `words[]` entry, doubling their tables for the sake of a few hundred kanji and hangul, most used once or twice. Keep only the most frequent non-ASCII characters in the dense 8-bit alphabet (the generator tries 32, 48, 64 and 80 and keeps the smallest result) and escape the rest: symbol 2 followed by an index into a small `uint16_t rare_chars[]` for characters used more than once, symbol 3 followed by a raw 16-bit code point for characters used once. Dictionary words cannot contain escaped characters. The 16-bit table mode and `translation_requires_uint16` are gone; `mchar_t` is replaced by `uint8_t`. Languages whose alphabet already fit are byte-identical to before, and the two decoder branches fold away for them. On metro_m4_express: ja +708 bytes free, ko +820, fr unchanged. Verified on the unix coverage port (979 tests) and on a Metro M4 Express in ja. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dhalbert
force-pushed
the
ja-ko-escapes
branch
from
September 14, 2026 03:25
cabc123 to
58613db
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude wrote the code and did the measurements in this PR, with @dhalbert directing and reviewing each step.
Stacked on #11369; targets
translation-shrink. Until #11369 merges, the diff shows both commits.Motivation
ja and ko have more than 127 distinct non-ASCII characters, so #11369 leaves them on the old 16-bit table mode: every entry of
values[]andwords[]costs two bytes, for the sake of a few hundred kanji and hangul that are mostly used once or twice. Their text is 77% and 89% ASCII, which the 8-bit scheme compresses well.What changes
py/maketranslationdata.py: only the most frequent non-ASCII characters get dense 8-bit codes. For languages that overflow, the generator tries 32, 48, 64 and 80 dense characters and keeps the smallest result; ja picks 48 and ko 32. Languages whose whole alphabet fits are unchanged.uint16_t rare_chars[]for characters used more than once. Symbol 3 is followed by a raw 16-bit code point for characters used once; those are cheaper without a table entry. Dictionary words cannot contain escaped characters.translation_requires_uint16are removed, andmchar_tis replaced byuint8_t.supervisor/shared/translate/translate.c: two branches next to the existing qstr escape. Each is guarded by a compile-time count of escaped characters, so for a language with none the compiler removes the branch and the decoder is the same size as before.supervisor/shared/translate/compressed_string.h: format comment updated.Results
Bytes free in flash on metro_m4_express, same commit, #11369 versus this PR:
fr and en_US generate byte-identical output to #11369. The generator takes about 4 seconds longer for ja and ko because of the four dense-alphabet tries; other languages are unaffected.
Testing
locale.getlocale(),ValueError,%q-formattedAttributeError, the errno text path,SyntaxError, and messages containing kanji, kana and fullwidth punctuation all decode correctly at the REPL.🤖 Generated with Claude Code