Rewrite URL mappings in streamed CSS file chunks

components/DataLiberation/README.md

@@ -330,3 +330,92 @@ There is no token-size cap. A large comment, string, identifier, or embedded
 image increases memory use and the saved cursor size. Each read reparses the
 unfinished token. Small chunks therefore do not bound the largest token's
 memory use or parsing work.
+
+## Rewrite a CSS file in chunks and resume
+
+A read can end between `https://old.exa` and `mple/photo.png`. Supply the next
+part to `CSSURLProcessor::rewrite_chunk()` and it can rewrite the complete URL.
+To continue in a new process, save the parser state, called a cursor. It holds
+the unfinished CSS bytes and remembers where a string can be a URL, such as
+after `@import` or inside `image-set()`.
+
+The same CSS parser handles whole-string and chunked input. It finds URLs in
+`url()`, quoted `@import` values, and `image-set()` image strings. Comments,
+displayed text, and malformed string or URL tokens stay unchanged.
+
+<!-- snippet:
+filename: css-chunks.php
+runnable: true
+-->
+```php
+<?php
+require '/php-toolkit/vendor/autoload.php';
+
+use WordPress\DataLiberation\URL\CSSURLProcessor;
+
+$mapping = array( 'https://old.example' => 'https://new.example' );
+$processor = CSSURLProcessor::create_for_streaming( $mapping );
+foreach ( $processor->rewrite_chunk( 'a{src:url(https://old.exa', false ) as $bytes ) {
+	echo $bytes;
+}
+
+// The foreach loop has written all available output. The cursor keeps the
+// unfinished URL, so the new processor needs only the bytes after 'old.exa'.
+// This JSON round trip shows how to restore state; it does not save a file.
+$cursor = json_decode( json_encode( $processor->get_reentrancy_cursor() ), true );
+$processor = CSSURLProcessor::create_for_streaming( $mapping, $cursor );
+foreach ( $processor->rewrite_chunk( 'mple/photo.png)}', true ) as $bytes ) {
+	echo $bytes;
+}
+echo "\n";
+```
+
+<!-- expected-output -->
+```
+a{src:url(https://new.example/photo.png)}
+```
+
+For files, use a fixed input chunk size and save progress in this order:
+
+1. Read a source chunk and pass it to `rewrite_chunk()`.
+2. Write every output piece from the `foreach` loop. Each piece is at most
+   64 KiB. Finish the loop, then flush the output file.
+3. Save the source byte offset, output byte offset, and parser cursor together.
+   This saved state is a checkpoint. The source offset counts all bytes read,
+   including the unfinished bytes held in the cursor.
+
+On resume, seek the source to its saved offset. Remove output bytes after the
+saved output offset, then append there. Those extra bytes may have been written
+before the previous process stopped, but after its last checkpoint. Removing
+them prevents duplicate output when the corresponding source is read again.
+Keep the source file and URL mapping unchanged between runs. The cursor stores
+a hash of the prepared replacement rules; resume rejects different rules.
+
+If a write fails or the output loop stops early, discard the processor and
+resume from the last checkpoint. Do not continue the unfinished output loop.
+The [file-rewrite test caller](Tests/fixtures/css-stream/rewrite-file.php)
+shows how to save and restore both file positions and the parser state.
+
+Before matching, CSS escapes in a URL are decoded. Scheme and host matching
+ignores letter case; path matching uses letter case. The longest source base
+wins. For example, `/blog` matches `/blog/photo.png`, `/blog?x=1`, and `/blog#top`,
+but not `/blogger`. A match can also end at the URL end.
+
+URLs that start with `//` keep that form. Relative paths, data URLs, and unrelated
+hosts stay unchanged. Only the matched base is replaced. The remaining URL bytes
+and surrounding quotes, parentheses, and spaces keep their original spelling.
+Escapes inside the replaced base can change spelling. URLs read from CSS are
+not fully normalized: path parts such as `/a/../b` and alternate encoded host
+spellings are not resolved before matching.
+
+The [token-streaming limits above](#stream-css-tokens-and-resume) still apply.
+An unfinished token, such as a comment or URL, is kept and parsed again when
+more input arrives. There is no size limit for that token. Small input and output
+chunks therefore do not limit its memory use, cursor size, or parsing work.
+More than 128 open, nested `image-set()` functions causes an error before the
+file is complete. `rewrite_chunk()` releases completed source bytes itself;
+the caller does not need to call `flush_processed_css()`.
+
+Set `$is_last` to `true` only at the actual file end. If that is known only after
+the last nonempty read, call `rewrite_chunk('', true)`. A download that stops
+early has not reached the file end and must not be marked as complete.

components/DataLiberation/Tests/CSSURLContextProcessTest.php

@@ -32,6 +32,20 @@ public function test_file_rewrite_finds_only_url_contexts() {
 		$this->assertSame( $input, file_get_contents( $this->directory . '/source.css' ) );
 	}
 
+	/** A malformed import consumes its URL position; a later valid URL must still be found. */
+	public function test_file_rewrite_skips_malformed_urls_and_following_text() {
+		$input = "@import \"https://old.example/bad\n"
+			. '"https://old.example/text";a{src:url(https://old.example/bad(image),url(https://old.example/good)}'
+			. '@import/**/"https://old.example/theme.css";';
+		$expected = "@import \"https://old.example/bad\n"
+			. '"https://old.example/text";a{src:url(https://old.example/bad(image),url("https://new.example/good")}'
+			. '@import/**/"https://new.example/theme.css";';
+		file_put_contents( $this->directory . '/source.css', $input );
+		$this->assertSame( 0, $this->run_worker(), file_get_contents( $this->directory . '/worker.log' ) );
+		$this->assertSame( $expected, file_get_contents( $this->directory . '/target.css' ) );
+		$this->assertSame( $input, file_get_contents( $this->directory . '/source.css' ) );
+	}
+
 	/** NUL-containing URLs can be rewritten; other controls still make a URL invalid. */
 	public function test_file_rewrite_applies_nul_preprocessing_before_url_validation() {
 		$comment = "/* Keep this NUL: \x00 and these line endings: \r\n\f */";
@@ -68,6 +82,19 @@ public function test_whole_string_finder_recognizes_import_and_image_set_urls()
 		$this->assertSame( array( 'https://old.example/theme.css', 'https://old.example/a', 'https://old.example/b', 'https://old.example/c' ), $urls );
 	}
 
+	/** Reading a URL twice must not consume another token or lose an empty URL. */
+	public function test_url_reads_leave_the_iterator_on_the_current_url() {
+		$css = '@import/**/"theme.css";a{content:"text";src:url(""),url(a.png),image-set("b.png" type("image/png"),"c.png" 2x)}';
+		$processor = new CSSURLProcessor( $css );
+		foreach ( array( 'theme.css', '', 'a.png', 'b.png', 'c.png' ) as $url ) {
+			$this->assertTrue( $processor->next_url() );
+			$this->assertSame( $url, $processor->get_raw_url() );
+			$this->assertSame( $url, $processor->get_raw_url() );
+		}
+		$this->assertFalse( $processor->next_url() );
+		$this->assertFalse( $processor->next_url() );
+	}
+
 	/** Runs a whole-file caller without streamed input or a saved cursor. */
 	private function run_worker() {
 		$arguments = array( PHP_BINARY, __DIR__ . '/fixtures/css-context/rewrite-file.php', $this->directory );

components/DataLiberation/Tests/CSSURLStreamProcessTest.php

@@ -0,0 +1,124 @@
+<?php
+
+use PHPUnit\Framework\TestCase;
+
+/** Rewrites real CSS files in child PHP processes to check output and saved state after a stop. */
+class CSSURLStreamProcessTest extends TestCase {
+	/** @var string Temporary directory for one test's source, output, saved state, and process log. */
+	private $directory;
+
+	/** @before */
+	public function create_directory() {
+		$this->directory = sys_get_temp_dir() . '/css-stream-' . bin2hex( random_bytes( 8 ) );
+		mkdir( $this->directory );
+	}
+
+	/** @after */
+	public function remove_directory() {
+		foreach ( glob( $this->directory . '/*' ) as $path ) {
+			unlink( $path );
+		}
+		rmdir( $this->directory );
+	}
+
+	/**
+	 * Checks an uninterrupted rewrite and two runs that exit before or after saving state.
+	 * A new process must finish each stopped run with the exact expected file bytes.
+	 *
+	 * @dataProvider interruptions
+	 */
+	public function test_file_rewrite_resumes_after_process_death( $stop ) {
+		// The worker reads 32 KiB at a time. Its first read ends after '\6',
+		// inside the '\6f ' escape for 'o'. Its second read ends inside a comment.
+		// Both saved positions require unfinished CSS bytes to survive a restart.
+		$prefix = '/*' . str_repeat( 'a', 32743 ) . '*/a{src:url("https://\\6f ld.example/a.png")}';
+		$input = $prefix . '/*' . str_repeat( 'b', 32768 ) . '*/'
+			. '@import "https://old.example/theme.css";'
+			. 'a{src:url(https://old.example/' . str_repeat( 'c', 131072 ) . ')}';
+		$this->assertSame( '\\6', substr( $input, 32766, 2 ) );
+		$expected = strtr( $input, array( 'https://\\6f ld.example/' => 'https://old.example/moved/', 'https://old.example/' => 'https://old.example/moved/' ) );
+		file_put_contents( $this->directory . '/source.css', $input );
+		$this->assertSame( 'none' === $stop ? 0 : 99, $this->run_worker( $stop ), file_get_contents( $this->directory . '/worker.log' ) );
+		if ( 'none' !== $stop ) {
+			$state = json_decode( file_get_contents( $this->directory . '/state.json' ), true );
+			$this->assertGreaterThan( 0, $state['source_bytes'] );
+			$this->assertLessThan( strlen( $input ), $state['source_bytes'] );
+			$this->assertSame( 0, $this->run_worker( 'none' ), file_get_contents( $this->directory . '/worker.log' ) );
+		}
+		$this->assertSame( hash( 'sha256', $expected ), hash_file( 'sha256', $this->directory . '/target.css' ) );
+		$this->assertSame( $input, file_get_contents( $this->directory . '/source.css' ) );
+		$state = json_decode( file_get_contents( $this->directory . '/state.json' ), true );
+		$this->assertSame( strlen( $input ), $state['source_bytes'] );
+		$this->assertSame( strlen( $expected ), $state['output_bytes'] );
+	}
+
+	/**
+	 * Restores the URL position after @import, even when a comment spans the saved offsets.
+	 *
+	 * @dataProvider interruptions
+	 */
+	public function test_file_rewrite_resumes_between_import_keyword_and_url( $stop ) {
+		// The comment ends at the second 32 KiB read. Both stop modes save a
+		// position after @import but before the string that supplies its URL.
+		$prefix = 'a{src:url(https://old.example/first)}@import';
+		$comment = '/*' . str_repeat( ' ', 65536 - strlen( $prefix ) - 4 ) . '*/';
+		$input = $prefix . $comment . '"https://old.example/theme.css";'
+			. 'a{content:"https://old.example/text";src:url(https://old.example/bad(image),url(https://old.example/last)}';
+		$expected = 'a{src:url(https://old.example/moved/first)}@import' . $comment . '"https://old.example/moved/theme.css";'
+			. 'a{content:"https://old.example/text";src:url(https://old.example/bad(image),url(https://old.example/moved/last)}';
+		file_put_contents( $this->directory . '/source.css', $input );
+		$this->assertSame( 'none' === $stop ? 0 : 99, $this->run_worker( $stop ), file_get_contents( $this->directory . '/worker.log' ) );
+		if ( 'none' !== $stop ) {
+			$state = json_decode( file_get_contents( $this->directory . '/state.json' ), true );
+			$this->assertSame( 'before' === $stop ? 32768 : 65536, $state['source_bytes'] );
+			$this->assertSame( 'import', $state['css']['context']['expect'] );
+			$this->assertSame( 0, $this->run_worker( 'none' ), file_get_contents( $this->directory . '/worker.log' ) );
+		}
+		$this->assertSame( $expected, file_get_contents( $this->directory . '/target.css' ) );
+		$this->assertSame( $input, file_get_contents( $this->directory . '/source.css' ) );
+		$state = json_decode( file_get_contents( $this->directory . '/state.json' ), true );
+		$this->assertSame( strlen( $input ), $state['source_bytes'] );
+		$this->assertSame( strlen( $expected ), $state['output_bytes'] );
+	}
+
+	/**
+	 * Saves part of the file, then reaches 129 nested image-set() calls and fails.
+	 * A second process must report the same error and leave the saved state before the file end.
+	 */
+	public function test_file_rewrite_reports_a_nesting_limit_and_keeps_the_last_checkpoint() {
+		$input = '/*' . str_repeat( 'a', 65536 ) . '*/a{src:' . str_repeat( 'image-set(', 129 ) . '"https://old.example/a"' . str_repeat( ')', 129 ) . '}';
+		file_put_contents( $this->directory . '/source.css', $input );
+		for ( $attempt = 0; $attempt < 2; ++$attempt ) {
+			$this->assertNotSame( 0, $this->run_worker( 'none' ) );
+			$this->assertStringContainsString( 'nesting exceeds 128', file_get_contents( $this->directory . '/worker.log' ) );
+			$state = json_decode( file_get_contents( $this->directory . '/state.json' ), true );
+			$this->assertLessThan( strlen( $input ), $state['source_bytes'] );
+			$this->assertTrue( $state['css']['css']['expecting_more_input'] );
+			$this->assertSame( $input, file_get_contents( $this->directory . '/source.css' ) );
+		}
+	}
+
+	/**
+	 * Selects normal completion, or exit just before or after the second saved state.
+	 * The worker has written the second chunk's output before either exit point.
+	 */
+	public static function interruptions() {
+		return array( array( 'none' ), array( 'before' ), array( 'after' ) );
+	}
+
+	/**
+	 * Starts the file-rewrite script and waits for its exit code.
+	 * Code 0 means completion, 99 means a test stop, and other codes report failures.
+	 * The script loads any saved state from the previous process before reading more source bytes.
+	 */
+	private function run_worker( $stop ) {
+		$arguments = array( PHP_BINARY, __DIR__ . '/fixtures/css-stream/rewrite-file.php', $this->directory . '/source.css', $this->directory . '/target.css', $this->directory . '/state.json', $stop );
+		$command = implode( ' ', array_map( 'escapeshellarg', $arguments ) );
+		// Bypass cmd.exe on Windows because it strips the quotes around these
+		// paths. Keep a command string: proc_open() in PHP 7.2 cannot take an array.
+		$process = proc_open( $command, array( 0 => array( 'pipe', 'r' ), 1 => array( 'file', $this->directory . '/worker.log', 'w' ), 2 => array( 'file', $this->directory . '/worker.log', 'a' ) ), $pipes, null, null, array( 'bypass_shell' => true ) );
+		$this->assertIsResource( $process );
+		fclose( $pipes[0] );
+		return proc_close( $process );
+	}
+}