feat(extensions): add host-owned file-view syntax highlighting

[benvinegar]

Summary

Closes #1023.

File-view extensions can now declare complete code documents and map symbolic spans to exact UTF-16 source slices. Hunk retains ownership of language detection, Shiki/Pierre tokenization, active-theme colors, worker scheduling, caching, terminal safety, projection, and plain-text fallback.

This enables source, generated-code, blame, Markdown-fence, guttered, and split old/new presentations without exposing token colors or renderer internals to extensions.

Approach

• Add the import-free API-v26 codeDocuments and span.syntax contract with bounded validation, normalization, immutable snapshots, and exact text matching.
• Generalize compact highlighting artifacts and the worker protocol for complete document jobs.
• Add one bounded shared document-highlight service with strong identities, single-flight work, subscriber-local cancellation, retry classification, and byte/entry-bounded caches.
• Migrate expanded-source highlighting to that shared service.
• Demand FileView documents only for mounted/halo rows, then project token ranges over retained authoritative span text.
• Preserve native OpenTUI word wrapping and fixed two-cell tabs; require native measurement so syntax paint cannot alter geometry.
• Add API documentation, a checked-in old/new code-document example, package-consumer coverage, and a minor hunkdiff Changeset.

Important invariants and non-goals

• Complete documents are tokenized so multiline lexical state remains correct.
• Highlighting is paint-only: it cannot change text, wrapping, row height, notes, navigation, selection backgrounds, layout generation, or scrolling.
• Unsupported languages, limits, cancellation, stale results, and failures retain ordinary symbolic FileView content.
• sourceRanges remain note/navigation bindings and are independent from syntax references.
• Extensions never provide colors or access Shiki, Pierre, HAST, workers, or terminal renderer types.

Validation

Passed after rebasing onto current main:

• bun run typecheck
• bun run lint
• bun run deps:check
• bun run check:pack
• bun run check:docs
• bun run website:check
• bun run changeset:status
• bun run test:integration — 174 passed, 1 platform skip
• bun run test:tty-smoke — 10 passed
• git diff --check origin/main...HEAD

Focused unit, React lifecycle, service/worker, compiled-worker, consumer, and FileView projection/geometry suites also passed throughout implementation. The new file-view PTY case verifies real syntax colors, multiline offscreen lexical state, theme changes, resize, refresh generations, cross-file navigation, and exact viewport stability.

bun run test retains one environment-dependent failure in unchanged packages/hunk/src/extensions/hostRuntimeModules.test.ts:179: this machine has an external /tmp/node_modules/react, so the test's deliberately outside-extension import resolves. The failing file is unchanged from main; the two shards otherwise reported 1,882 and 2,379 passing tests, respectively.

Platform and visual evidence

• Validated on Linux with Bun 1.4.2 and Node.js 22.
• macOS and Windows compiled-worker execution were not performed.
• Real PTY and TTY smoke coverage exercised the terminal UI. No standalone screenshot/video is attached; the deterministic PTY test covers syntax paint and layout stability across theme, resize, refresh, and file switching.

Follow-up adopter

A reviewed hunk-viewed migration exists locally and uses complete old/new documents for its unified and split full-file views. Its release is intentionally sequenced after the next minor hunkdiff release publishes API v26.

This PR description was generated by Pi using gpt-5.6-sol

[greptile-apps]

Too many files changed for review (57 files, 50 file limit).

Bypass the limit by tagging @greptile-apps to review.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hunk-web	Ready	Preview	Sep 10, 2026 1:12pm UTC

[jacegodk]

Tested #1053 with hunk-viewed's full-file view (unified and split, old and new documents, search spans interleaved): tokenization, exact-text validation and fallback all behave as documented. Thanks.

One gap for diff-presenting views: a token color always overrides tone, and file-view rows get only theme.selectedHunk or theme.panel as background. So once an added or removed row is syntax painted, nothing but the +/- marker distinguishes it from context. In the raw diff the row background carries that signal. Could a row (or span) opt into the raw diff's added/removed background, e.g. background: "added" | "removed", painted under the token colors? That is the "layer the tone on top" part of #1023. Until then hunk-viewed highlights context lines only and keeps changed lines in solid tone.

Two smaller points: a single span/text mismatch rejects the whole layout, so extensions must re-verify every reference; dropping syntax on that span would be more forgiving. And the 10,000-line cap is aggregate across documents, so an old+new view fits files up to about 5,000 lines while the row cap allows 10,000.