Stream CSS tokens and resume unfinished input - #317
Merged
Conversation
This was referenced Sep 8, 2026
# Conflicts: # components/DataLiberation/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets a CSS token span input chunks and continue in a new PHP process without restarting the file.
Review order: #315 (merged) → #316 → #317 → #314.
The processor keeps unfinished input and parses it again after another read. Whole strings and streamed input use the same
next_token(), getters, and value setter.flush_processed_css()returns edited, completed input and releases its source bytes. The cursor contains the remaining bytes and whether more input is expected; the caller saves it with the source and output offsets after writing the output.This is layer 3 of the CSS stack, based on #316. It adds input, flushing, and the token cursor together. It contains no URL mappings or
rewrite_chunk(); those remain in #314. ASCII spans are scanned together so reparsing long identifiers and URLs does not walk every byte in PHP.A large token increases memory use, cursor size, and repeated parsing work. Token size is not capped. The README spells out that cost and links the real-file caller.
Testing
The streamed tokens match the whole-string corpus, including one-byte reads, escapes, and invalid UTF-8. Separate PHP processes edit a multi-chunk file, stop before or after a saved checkpoint, and resume with the unfinished token. The written file must match whole-string edits without missing or duplicated bytes. The full DataLiberation run completed with 2,618 tests and 16 skips. PHPCS passes.