Skip to content

Stream CSS tokens and resume unfinished input - #317

Merged
adamziel merged 26 commits into
trunkfrom
codex/css-token-stream
Sep 9, 2026
Merged

Stream CSS tokens and resume unfinished input#317
adamziel merged 26 commits into
trunkfrom
codex/css-token-stream

Conversation

@adamziel

@adamziel adamziel commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Lets a CSS token span input chunks and continue in a new PHP process without restarting the file.

Review order: #315 (merged) → #316#317#314.

The processor keeps unfinished input and parses it again after another read. Whole strings and streamed input use the same next_token(), getters, and value setter. flush_processed_css() returns edited, completed input and releases its source bytes. The cursor contains the remaining bytes and whether more input is expected; the caller saves it with the source and output offsets after writing the output.

This is layer 3 of the CSS stack, based on #316. It adds input, flushing, and the token cursor together. It contains no URL mappings or rewrite_chunk(); those remain in #314. ASCII spans are scanned together so reparsing long identifiers and URLs does not walk every byte in PHP.

A large token increases memory use, cursor size, and repeated parsing work. Token size is not capped. The README spells out that cost and links the real-file caller.

Testing

The streamed tokens match the whole-string corpus, including one-byte reads, escapes, and invalid UTF-8. Separate PHP processes edit a multi-chunk file, stop before or after a saved checkpoint, and resume with the unfinished token. The written file must match whole-string edits without missing or duplicated bytes. The full DataLiberation run completed with 2,618 tests and 16 skips. PHPCS passes.

Base automatically changed from codex/css-url-contexts to trunk September 9, 2026 16:03
@adamziel
adamziel merged commit 8e411e7 into trunk Sep 9, 2026
29 checks passed
@adamziel
adamziel deleted the codex/css-token-stream branch September 9, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant