perf(workbench): stream the tracked patch into the content digest - #256
Open
rohanpoudel2 wants to merge 4 commits into
Open
perf(workbench): stream the tracked patch into the content digest#256rohanpoudel2 wants to merge 4 commits into
rohanpoudel2 wants to merge 4 commits into
Conversation
`worktree_content_digest_for_context` read the working-tree patch through `git_bytes`, which is `subprocess.run(..., capture_output=True)`, so the whole `git diff --binary` output was materialised in memory before being hashed. A repository holding a large changed binary could therefore exhaust the workbench during setup inspection, which runs on every inspection rather than only at registration. On a fixture with a 20 MiB incompressible change, Git emitted a 51.5 MiB patch and the digest process peaked at 145.8 MiB RSS. `update_digest_field` frames every value with an 8-byte big-endian length, so the total byte count must be known before any content is hashed and stdout cannot simply be fed into the hash. `git_digest_field` spools Git's stdout straight to a private temporary file, takes the length from `fstat`, writes the same framing, and then hashes the file in 1 MiB chunks. A single Git invocation still produces the patch, so the snapshot stays atomic. The spool file lives in the process temporary directory, never in the scan directory or the scanned repository, is created owner-only, and is removed by the `with` block; on POSIX it is unlinked before Git writes to it, so the patch is never reachable by name. `git_bytes` keeps its buffered behaviour for its many small callers, including the untracked `ls-files` listing in the same function. Digests are unchanged. Old and new code produce identical digests for an empty diff, a text diff, a binary diff, a diff with untracked files, and a 20 MiB binary diff, and a clean worktree still hashes to the hardcoded `clean_worktree_content_digest` sentinel. Peak RSS on the 20 MiB fixture falls from 145.8 MiB to 27.4 MiB.
rohanpoudel2
added a commit
to rohanpoudel2/codex-security
that referenced
this pull request
Aug 4, 2026
committed_diff_content_digest read the patch through git_bytes, which is subprocess.run(capture_output=True), so the whole `git diff --binary` output sat in memory before a byte was hashed. Binary patches outgrow the files they describe because Git base85-encodes a forward and a reverse literal for every rewritten blob, so registering a scan over a large changed binary could exhaust the workbench before review began. Measured on a 20 MiB incompressible change (macOS 15, Python 3.14.5, git 2.50.1) the patch was 27.1 MiB and peak RSS was 88.9 MiB, against 27.4 MiB streaming. Spool Git's stdout into a private temporary file, take the value length from fstat, and hash the file in chunks, writing exactly the framing update_digest_field writes. Streaming directly from the pipe is not possible because the 8-byte value length precedes the value in the hashed material, and a second counting pass would not be atomic. git_command gained a keyword-only stdout so the streaming path keeps the existing environment scrubbing and core.fsmonitor=false hardening; capture_output became the equivalent explicit stdout/stderr pipes, so every other caller is unchanged. Digests are unchanged: buffered and streaming agree on an empty range, a text diff, a 64 KiB binary diff, a rename, a mode change, a root commit against the empty tree, and the 20 MiB binary. The failure path still exits 1 with "Could not snapshot the selected committed changes." git_digest_field and the git_command signature are byte for byte those of openai#256, which applies the same helper to the working-tree digest. Neither branch touches the other's call site, so the two merge cleanly in either order.
The streaming helper reports Git failure as False where the buffered helper returned None, so worktree_content_digest_for_context had to change its guard from `tracked is None` to `not tracked`. Nothing covered that line: restoring the `is None` form makes the digest path accept a failed `git diff` and record a snapshot digest computed over an empty patch, which is a wrong snapshot rather than a visible error. Add a probe over a repository on an unborn branch, where `git diff HEAD` exits non-zero and writes no patch, and assert the call raises the existing "Could not snapshot the selected working-tree changes." message and produces no digest. The shared probe runner also asserts the private TMPDIR is empty afterwards, which pins that the spool is removed on the Git-failure return.
…-digest # Conflicts: # sdk/typescript/_bundled_plugin/scripts/workbench_target.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #249.
Streams the tracked Git patch into the worktree content digest without buffering the full binary diff in Python memory, while preserving the existing digest framing.
Changes
Testing
bun test --timeout 30000 ./tests-ts/workbench-content-digest.test.ts: 3 passed, 0 failed.pnpm run types: passed.pnpm run format: passed.git diff --check: passed.Risk and rollout
The digest algorithm and framing are unchanged. The temporary spool is private and removed automatically; failures still reject the snapshot rather than recording an incomplete digest.
Public disclosure review