Fixed width char datasets - #29
Merged
Merged
Conversation
A text aux component was written as a variable-length HDF5 string dataset
whatever it was, and binsparse_to_ssmc_problem guessed the class to rebuild
from the matrix id:
use_cellstr = isfield(Problem, 'id') && Problem.id > 2776;
That rule is wrong in both directions. Across the collection there are 247
text aux components: 239 char matrices, ten of which have id > 2776, and eight
cellstr, all of which have id <= 2776. So every real cellstr came back as a
char matrix and every SNAP char matrix came back as a cellstr, and isequal
failed for 13 matrices.
The id was never what made this work for MM and RB either. sswrite only sends
a cellstr to a single .txt when sscellstring(X) && id > 2776, which no matrix
in the collection satisfies; every real cellstr takes the generic iscell branch
and is written one element per file, and it is that structure, not the
heuristic, that round trips.
Let the HDF5 string datatype record the class instead, so a file says what it
holds and no side channel is needed:
fixed-length (H5T_STR_NULLPAD) <-> char matrix, m-by-n
variable-length (H5T_VARIABLE) <-> cellstr, m-by-1
A char matrix is rectangular, so each element of a fixed-length dataset holds
one complete row including the blanks MATLAB pads it with, and the datatype
size carries the width. Nothing is deblanked on either side, which retires
strip_text_rows and its trick of keeping padding on the widest row so that
char() could recover the width. A cellstr is ragged, so it stays
variable-length and needs no padding convention.
Padding is NUL rather than blank because HDF5 sizes a fixed-length string in
bytes while MATLAB sizes a char matrix in characters. The two agree only for
ASCII; blank padding would widen any row holding a multi-byte character.
Stripping trailing NULs recovers a row's exact characters whatever it contains.
Trailing blanks are stripped as well when a foreign file declares
H5T_STR_SPACEPAD, the Fortran convention.
Text now converts through matlab_bsp_strings.h rather than mxArrayToString,
which goes by way of the local code page and replaces what it cannot represent
with SUB (0x1A). That silently mangled every non-ASCII aux row, 65,409 of them
in Pajek/IMDB alone. The same fix applies to the JSON descriptor in
binsparse_write.c, where it was corrupting Problem.notes for SNAP/wiki-RfA (en
dash) and Pajek/Journals (replacement character).
Text datasets are also chunked and deflated now. A variable-length payload
lives on the global heap, which no dataset filter reaches, and costs a 16-byte
descriptor per element; Pajek/IMDB colname alone took 27.6 MB of a 58.2 MB
file. The whole file is now 18.7 MB.
Every one of the 341 locally cached matrices round trips through sswrite and
ssread with isequal true, including all 13 that used to fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chunk-and-deflate decision keyed off the element count, so a text dataset of one very wide row was left uncompressed however large it was. Threshold on count * element_size instead, which covers both that case and the many-small- elements one, and record what compression actually buys: SNAP/wiki-topcats pagenames 372.6 MB -> 19.2 MB fixed-length Moqri/MISKnowledgeMap Abstract 3.25 MB -> 3.23 MB variable-length A fixed-length dataset compresses well, since blank padding is redundant. A variable-length one barely moves: HDF5 keeps the strings on the global heap, which the filter pipeline does not reach, so only the 16-byte-per-element descriptor array is filtered. That is an HDF5 limitation rather than a choice here, and it costs about 3.2 MB across the whole collection, which has only eight cellstr aux components. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.