[Pi] Image fingerprinting tokenizes base64 and blocks the event loop for ~15s

[Qihuanxishini]

Short description

Pi image fingerprinting invokes the text tokenizer on the complete base64/data URL, although the image already has its own token estimate. A cold lookup blocked the event loop for approximately 15 seconds in an isolated reproduction.

What happened?

In a long retained Pi session containing an image, context processing intermittently paused for many seconds and Esc was slow to take effect. The slow transform stage was channelNudgeAccounting.

The Pi tail-hygiene walker already calculates image tokens through imageContentAndTokens(). Later, finalizeParts() requests a content hash through partHash(). That hash-only request enters memoizedContent(), which also synchronously calls estimateTokens(content) on a cache miss. The resulting text-token count is unused for the image: the final measurement uses draft.tokens instead.

Expected behavior: image fingerprinting should compute/cache the fingerprint without running the text tokenizer. Image accounting should continue to use the existing image-specific estimator, and ordinary text-token accounting should retain its current semantics.

Reproduction steps

1. Use Pi with @cortexkit/pi-magic-context@0.42.4 and the working ai-tokenizer backend.
2. Load an image of roughly 3 MiB, producing a nontrivial base64 payload of about 4.2 million characters, as a read tool result. User image attachments also reach the same hash helper through the file kind.
3. Trigger context processing with a cold content memo, for example after resuming that retained session in a fresh Pi process. Ensure the image is still present in the rendered tail.
4. Observe the synchronous pause in channelNudgeAccounting. Esc input during this work cannot be processed until the event loop is released. Repeating the same lookup with a warm memo can hide the problem.

Diagnostics

Environment

• Client: Pi TUI
• Pi: @earendil-works/pi-coding-agent@0.85.1
• Plugin: @cortexkit/pi-magic-context@0.42.4
• Node.js: v24.18.0
• Platform: Windows, win32 x64
• Tokenizer: ai-tokenizer@1.0.6, Claude encoding

The affected code is also present on upstream master at 55f7a8771d06fe2bcab4d749303a2276ea184e5f.

Isolated measurements

The unmodified installed partHash / memoizedContent helpers were evaluated in an isolated Node VM. The instrumented estimateTokens dependency used the installed tokenizer's same successful path: tokenizer.encode(text, "all").length. Tokenizer construction and reading the image/session data completed before the timed section. A zero-delay timer was scheduled immediately before the synchronous hash lookup.

Measurement	Result
Base64 payload length	4,232,852 characters
Data URL passed to the text tokenizer	4,232,874 characters
Cold partHash("toolOutput", dataUrl)	15,026.9 ms
Time inside the text-tokenizer call	14,430.5 ms
Zero-delay timer's observed delay	15,031.1 ms
Warm lookup of the same key	1.2 ms

The cold and warm lookups returned identical hashes. These are isolated helper measurements, not a before/after benchmark of a patched Pi application.

Relevant runtime log samples from the affected retained session (UTC; session ID sanitized):

[2026-09-15T00:42:09.177Z] [magic-context][<session-id>] transform stage: stage=channelNudgeAccounting elapsed=14401.5ms
[2026-09-15T00:45:26.640Z] [magic-context][<session-id>] transform stage: stage=channelNudgeAccounting elapsed=16155.9ms

The original image and conversation contents are omitted; the measurements contain only lengths, durations, and the affected code path.

Root cause

All source links below are pinned to the checked commit:

• imageContentAndTokens, lines 196–209 obtains the image-specific token estimate.
• Image drafts, lines 603–617 retain image.tokens and use kind toolOutput or file.
• finalizeParts, lines 456–478 uses draft.tokens for accounting while separately calling partHash.
• memoizedContent / memoizedTokens / partHash, lines 92–125 couple hash creation to eager text tokenization.
• estimateTokens, lines 340–355 calls the synchronous text encoder.

measurePiTailHygiene
  -> imageContentAndTokens: image token estimate already available
  -> finalizeParts
     -> partHash
        -> memoizedContent (cold miss)
           -> estimateTokens(entire image data URL)
              -> tokenizer.encode(..., "all")

The finalizer needs only the hash from this second path. The expensive text-token value is redundant for this image measurement.

Proposed narrow fix

Keep the existing bounded content memo and hash algorithm, but populate its text-token field lazily in memoizedTokens(). A hash-only lookup should leave that field uncomputed. Preserve the existing zero-token handling of excluded content.

Suggested diff against the checked source:

--- a/packages/pi-plugin/src/tail-hygiene-walk-pi.ts
+++ b/packages/pi-plugin/src/tail-hygiene-walk-pi.ts
@@ -16,7 +16,7 @@
 const MAX_CONTENT_MEMO_BYTES = 64 * 1024 * 1024;
 const contentMemo = new Map<
 	string,
-	{ hash: string; tokens: number; keyBytes: number }
+	{ hash: string; tokens: number | undefined; keyBytes: number }
 >();
 let contentMemoBytes = 0;
 const FNV1A_32_OFFSET = 0x811c9dc5;
@@ -92,13 +92,13 @@
 function memoizedContent(
 	kind: TailHygienePartKind,
 	content: string,
-): { hash: string; tokens: number } {
+): { hash: string; tokens: number | undefined } {
 	const key = `${kind}\0${content}`;
 	const cached = contentMemo.get(key);
 	if (cached) return cached;
 	const measured = {
 		hash: fnv1a32(key),
-		tokens: kind === "excluded" ? 0 : estimateTokens(content),
+		tokens: kind === "excluded" ? 0 : undefined,
 		keyBytes: key.length * 2 + 32,
 	};
 	contentMemo.set(key, measured);
@@ -117,7 +117,11 @@
 }
 
 function memoizedTokens(kind: TailHygienePartKind, content: string): number {
-	return memoizedContent(kind, content).tokens;
+	const measured = memoizedContent(kind, content);
+	if (measured.tokens === undefined) {
+		measured.tokens = estimateTokens(content);
+	}
+	return measured.tokens;
 }
 
 function partHash(kind: TailHygienePartKind, content: string): string {

Important invariants:

• Keep the same ${kind}\0${content} key, FNV hash, cache bounds, and eviction behavior.
• Keep image-specific token estimates in image drafts and normal text estimates in memoizedTokens.
• A later text-token request for a previously hash-only key must still calculate its proper text count. toolOutput is shared by text and image parts, so a blanket zero count for that kind would be incorrect.
• Cache a genuine result of zero normally; use undefined as the uncomputed state.

Validation and regression coverage

The proposed behavioral changes were applied in memory only to the installed helper functions and checked with the real installed tokenizer. These helper-level checks passed:

• Cold/warm hash-only calls for file and toolOutput invoke estimateTokens zero times and produce the original hashes.
• A subsequent text-token request for the same previously hashed value returns the original text count and computes it only once.
• Ordinary text and zero-token results remain cached correctly.
• excluded content retains zero tokens without text tokenization.
• Changed image content changes the fingerprint.

A repository-level patch should add regression coverage to tail-hygiene-walk-pi.test.ts:

1. Exercise the real walker with cold-cache user images and tool-result images, both raw base64 and already-prefixed data URLs. Assert that image payloads do not reach the text tokenizer. Prefer a call-count invariant over a machine-dependent timing assertion.
2. Verify unchanged image token totals, u/t, content signatures, protected-part handling, and pending-drop accounting.
3. Cover hash-first/text-count-later access, zero-token caching, and ordinary text counts.
4. Optionally retain a multi-megabyte image benchmark with an event-loop timer to observe the remaining hash cost independently of tokenization.

Full repository tests and an end-to-end patched Pi run have not been performed; the diff is a proposed fix, with helper-level validation as described above.

Related

#448 reports another synchronous tokenization hotspot in OpenCode's droppedTokens telemetry. This report identifies the separate Pi image-fingerprinting caller; its redundant text-token work can be removed while retaining the existing image and text accounting semantics.

[magic-alfonso]

Confirmed at source and fixed on master, exactly as you diagnosed: partHash went through memoizedContent, which tokenized the whole data URL on a cache miss for a count the image path never reads. The token field is now lazy — memoizedTokens computes it on first request and caches it, partHash never tokenizes — with the memo key, FNV hash, bounds and eviction unchanged, excluded still eagerly 0, and a genuine zero cached as 0.

One addition to your scope: the OpenCode walker (packages/plugin/src/hooks/magic-context/tail-hygiene-walk.ts) had the identical coupling — image drafts take their tokens from estimateImageTokensFromDataUrl, then finalize hashed the data URL through the same tokenizing memo — so both lanes are fixed in one change.

Coverage follows your list: cold-cache walks with user images and tool-result images, raw base64 and prefixed data URLs, asserting zero tokenizer calls receive an image payload (call-count invariants, no timing); image token totals, u/t, content signatures, protected handling and pending-drop accounting unchanged; hash-first then text-count-later computes once; zero-token and excluded semantics pinned. On a 3 MiB image fixture each lane went from two image-payload tokenizer calls per cold walk to zero.

Ships in the next release. Thanks for the isolated measurements and the diff — the pinned source links made the verification a few minutes.