Add WebP image decoding - #976
Open
Bendr0id wants to merge 2 commits into
Open
Conversation
Bendr0id
force-pushed
the
webp-image-support
branch
2 times, most recently
from
September 4, 2026 18:06
badec74 to
c2f6019
Compare
Screenshot tooling for mobile and browser debugging emits WebP by default, and clients transcode to it on their own initiative, but an image sent that way was refused by media type before anything looked at it. The request failed as a whole rather than the image simply being read, which is the worst outcome: the caller attached a screenshot and got an error about the format instead of an answer. Decoding is a new dependency-free single header beside the vendored JPEG and PNG ones, covering both compression modes the format defines. A still lossy image is a single VP8 key frame, so the lossy path is intra only and needs no motion compensation or reference frames: boolean entropy decoder, frame, segment, filter and quantizer headers, 16x16, 4x4 and chroma intra prediction, token decoding, dequantization, the inverse DCT and the Walsh-Hadamard pass, both loop filters, and the conversion to RGB. The lossless path is a separate algorithm: prefix codes, backward references, the color cache and the four inverse transforms, including palettes packed several pixels to the byte. RIFF and VP8X parsing skips ICCP, EXIF and XMP, which is how real payloads arrive. Animation is refused with a message saying so rather than silently showing the first frame, since a caller who sent an animation did not ask for one arbitrary frame of it. The alpha chunk of a lossy image is ignored, so those decode to opaque RGB. All three front ends reach that decoder through one path, so /read in the CLI and view_image in the agent read WebP without a change of their own, and neither gains a way to reject a format the decoder handles. Only the server needed its gate widened, because it alone filters on a media type the client declares rather than on the bytes themselves, and it accepts image/webp for both the OpenAI data URI form and the Anthropic form, where the media type travels beside the data. The /read help text offered PNG and JPEG only and now names what it actually takes. The probability, quantizer and mode tables are values fixed by RFC 6386 and were taken from the reference implementation rather than retyped, and third_party/webp/LICENSE records that. Output is bit exact with libwebp's own dwebp across 244 images spanning tiny and odd dimensions, flat and noisy content, every quality setting, alpha, and each lossless transform. That mattered: it is what caught the Walsh-Hadamard pass needing to scatter DCs across the whole coefficient array, and the inner edge filter flag depending on whether a macroblock decoded to all zero coefficients rather than on its skip bit. Because this parses untrusted network input, 14640 truncated and corrupted inputs were also run under the address and undefined behaviour sanitizers, which is what found a distance code tree built over 256 symbols instead of the 40 its alphabet actually has.
Bendr0id
force-pushed
the
webp-image-support
branch
from
September 4, 2026 18:07
c2f6019 to
8fc7322
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Screenshot tooling for mobile and browser debugging emits WebP by default, and clients transcode to it on their own initiative, but an image sent that way was refused by media type before anything looked at it. The request failed as a whole rather than the image simply being read, which is the worst outcome: the caller attached a screenshot and got an error about the format instead of an answer.
Decoding is a new dependency-free single header beside the vendored JPEG and PNG ones, covering both compression modes the format defines. A still lossy image is a single VP8 key frame, so the lossy path is intra only and needs no motion compensation or reference frames: boolean entropy decoder, frame, segment, filter and quantizer headers, 16x16, 4x4 and chroma intra prediction, token decoding, dequantization, the inverse DCT and the Walsh-Hadamard pass, both loop filters, and the conversion to RGB. The lossless path is a separate algorithm: prefix codes, backward references, the color cache and the four inverse transforms, including palettes packed several pixels to the byte. RIFF and VP8X parsing skips ICCP, EXIF and XMP, which is how real payloads arrive. Animation is refused with a message saying so rather than silently showing the first frame, since a caller who sent an animation did not ask for one arbitrary frame of it. The alpha chunk of a lossy image is ignored, so those decode to opaque RGB.
All three front ends reach that decoder through one path, so
/readin the CLI andview_imagein the agent read WebP without a change of their own, and neither gains a way to reject a format the decoder handles. Only the server needed its gate widened, because it alone filters on a media type the client declares rather than on the bytes themselves, and it acceptsimage/webpfor both the OpenAI data URI form and the Anthropic form, where the media type travels beside the data. The/readhelp text offered PNG and JPEG only and now names what it actually takes.The probability, quantizer and mode tables are values fixed by RFC 6386 and were taken from the reference implementation rather than retyped, and
third_party/webp/LICENSErecords that.Output is bit exact with libwebp's own
dwebpacross 244 images spanning tiny and odd dimensions, flat and noisy content, every quality setting, alpha, and each lossless transform. That mattered: it is what caught the Walsh-Hadamard pass needing to scatter DCs across the whole coefficient array, and the inner edge filter flag depending on whether a macroblock decoded to all zero coefficients rather than on its skip bit. Because this parses untrusted network input, 14640 truncated and corrupted inputs were also run under the address and undefined behaviour sanitizers, which is what found a distance code tree built over 256 symbols instead of the 40 its alphabet actually has.Tests are in
tests/test_deepseek4_vision_image.cfor the decoder and in the server unit tests for acceptance over both request forms. The build is warning free under-Wall -Wextra.