Skip to content

Add WebP image decoding - #976

Open
Bendr0id wants to merge 2 commits into
antirez:mainfrom
Bendr0id:webp-image-support
Open

Add WebP image decoding#976
Bendr0id wants to merge 2 commits into
antirez:mainfrom
Bendr0id:webp-image-support

Conversation

@Bendr0id

@Bendr0id Bendr0id commented Sep 4, 2026

Copy link
Copy Markdown

Screenshot tooling for mobile and browser debugging emits WebP by default, and clients transcode to it on their own initiative, but an image sent that way was refused by media type before anything looked at it. The request failed as a whole rather than the image simply being read, which is the worst outcome: the caller attached a screenshot and got an error about the format instead of an answer.

Decoding is a new dependency-free single header beside the vendored JPEG and PNG ones, covering both compression modes the format defines. A still lossy image is a single VP8 key frame, so the lossy path is intra only and needs no motion compensation or reference frames: boolean entropy decoder, frame, segment, filter and quantizer headers, 16x16, 4x4 and chroma intra prediction, token decoding, dequantization, the inverse DCT and the Walsh-Hadamard pass, both loop filters, and the conversion to RGB. The lossless path is a separate algorithm: prefix codes, backward references, the color cache and the four inverse transforms, including palettes packed several pixels to the byte. RIFF and VP8X parsing skips ICCP, EXIF and XMP, which is how real payloads arrive. Animation is refused with a message saying so rather than silently showing the first frame, since a caller who sent an animation did not ask for one arbitrary frame of it. The alpha chunk of a lossy image is ignored, so those decode to opaque RGB.

All three front ends reach that decoder through one path, so /read in the CLI and view_image in the agent read WebP without a change of their own, and neither gains a way to reject a format the decoder handles. Only the server needed its gate widened, because it alone filters on a media type the client declares rather than on the bytes themselves, and it accepts image/webp for both the OpenAI data URI form and the Anthropic form, where the media type travels beside the data. The /read help text offered PNG and JPEG only and now names what it actually takes.

The probability, quantizer and mode tables are values fixed by RFC 6386 and were taken from the reference implementation rather than retyped, and third_party/webp/LICENSE records that.

Output is bit exact with libwebp's own dwebp across 244 images spanning tiny and odd dimensions, flat and noisy content, every quality setting, alpha, and each lossless transform. That mattered: it is what caught the Walsh-Hadamard pass needing to scatter DCs across the whole coefficient array, and the inner edge filter flag depending on whether a macroblock decoded to all zero coefficients rather than on its skip bit. Because this parses untrusted network input, 14640 truncated and corrupted inputs were also run under the address and undefined behaviour sanitizers, which is what found a distance code tree built over 256 symbols instead of the 40 its alphabet actually has.

Tests are in tests/test_deepseek4_vision_image.c for the decoder and in the server unit tests for acceptance over both request forms. The build is warning free under -Wall -Wextra.

@Bendr0id
Bendr0id force-pushed the webp-image-support branch 2 times, most recently from badec74 to c2f6019 Compare September 4, 2026 18:06
Screenshot tooling for mobile and browser debugging emits WebP by default,
and clients transcode to it on their own initiative, but an image sent that
way was refused by media type before anything looked at it. The request
failed as a whole rather than the image simply being read, which is the
worst outcome: the caller attached a screenshot and got an error about the
format instead of an answer.

Decoding is a new dependency-free single header beside the vendored JPEG and
PNG ones, covering both compression modes the format defines. A still lossy
image is a single VP8 key frame, so the lossy path is intra only and needs no
motion compensation or reference frames: boolean entropy decoder, frame,
segment, filter and quantizer headers, 16x16, 4x4 and chroma intra
prediction, token decoding, dequantization, the inverse DCT and the
Walsh-Hadamard pass, both loop filters, and the conversion to RGB. The
lossless path is a separate algorithm: prefix codes, backward references, the
color cache and the four inverse transforms, including palettes packed
several pixels to the byte. RIFF and VP8X parsing skips ICCP, EXIF and XMP,
which is how real payloads arrive. Animation is refused with a message
saying so rather than silently showing the first frame, since a caller who
sent an animation did not ask for one arbitrary frame of it. The alpha chunk
of a lossy image is ignored, so those decode to opaque RGB.

All three front ends reach that decoder through one path, so /read in the CLI
and view_image in the agent read WebP without a change of their own, and
neither gains a way to reject a format the decoder handles. Only the server
needed its gate widened, because it alone filters on a media type the client
declares rather than on the bytes themselves, and it accepts image/webp for
both the OpenAI data URI form and the Anthropic form, where the media type
travels beside the data. The /read help text offered PNG and JPEG only and
now names what it actually takes.

The probability, quantizer and mode tables are values fixed by RFC 6386 and
were taken from the reference implementation rather than retyped, and
third_party/webp/LICENSE records that. Output is bit exact with libwebp's
own dwebp across 244 images spanning tiny and odd dimensions, flat and noisy
content, every quality setting, alpha, and each lossless transform. That
mattered: it is what caught the Walsh-Hadamard pass needing to scatter DCs
across the whole coefficient array, and the inner edge filter flag depending
on whether a macroblock decoded to all zero coefficients rather than on its
skip bit. Because this parses untrusted network input, 14640 truncated and
corrupted inputs were also run under the address and undefined behaviour
sanitizers, which is what found a distance code tree built over 256 symbols
instead of the 40 its alphabet actually has.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant