fix(parsers): structure-preserving knowledge base parsers with before/after benchmark - #7709
Merged
Conversation
Ground-truth corpus generator, real-world fetcher, bun harness over the production parseBuffer path, reference extractors and scorer, plus the plan and findings from the 2026-09-09 audit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`XlsxParser` converted sheets without `raw: false`, so the indexed text held stored values rather than what a user sees: dates as Excel serials (46085), 20% as 0.2, $1,250.00 as 1250, booleans as `true`, and ODS dates as `String(Date)` in the worker's local time zone. The Google Drive connector exports every Google Sheet through this parser while the Sheets and Excel connectors already request formatted text, so the same sheet indexed differently by path. The Files viewer had the same defect. Read with `cellDates` + `cellNF` and convert with `raw: false`, rewriting only the two cases the file's own text gets wrong inside the bounded window: dates become zone-free ISO text from the UTC fields SheetJS parsed, and General numbers print their full stored value instead of Excel's 11-char rendering (4111111111111111 -> 4.11111E+15). The shared pass handles dense and sparse sheets so the viewer reuses it without pulling `xlsx` into the client bundle. The parser-eval fixture used `0.#%`, which Excel renders as `20.%`; it now uses `0%` / `0.0%` so the spec strings match Excel. The `sheet-wide` row builder is also typed so the script type-checks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
DOCX now routes mammoth's HTML rendering through the shared HTML structured-text walker so tables keep their rows, lists keep their markers, and footnotes survive; the unread metadata.html field is gone. PPTX and ODT/ODP get dedicated XML walkers that render tables row by row, skip slide-number/date/header/footer placeholders, read presenter notes from the notes body placeholder only, and drop ODF annotations and tracked deletions. Legacy OLE .ppt is rejected as unsupported_type instead of scraping printable bytes from the container. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ad legacy .doc Text parsers decoded every buffer as UTF-8 and then stripped U+FFFD, so a Latin-1 or Windows-1252 file silently lost every accented character, a UTF-8 BOM leaked into content and broke JSON.parse, and UTF-16 only worked for ASCII. `decodeTextBuffer` (BOM > strict UTF-8 with a guarded truncated-tail retry > Windows-1252) now backs txt/md/csv/json/jsonl/yaml, the .doc plain-text fallback and the connectors' text decode, and records `encoding`/`warning` in metadata. `parseBuffer` routed on the caller-supplied extension alone. `sniff.ts` now identifies the bytes (PDF, OLE2, ZIP central-directory part names, ODF mimetype, UTF-16 layout, HTML head) and reconciles them with the extension's family: a sniffed kind with its own parser overrides the route and records `detectedType`; binary/unknown bytes under a mismatched family are a typed `invalid_format` instead of mojibake or placeholder prose. Legacy OLE .doc goes through word-extractor (body, headers, footers, footnotes, endnotes; Word 6/95 magic maps to `unsupported_type`); the byte scrape that returned ZIP part names as degraded prose is deleted. Legacy .ppt is dropped from the registry, upload and connector allowlists and Chat's parseable set so it is refused up front. Chat's file reader and the internal file tool now treat `degraded` output as a parse failure. pdf.js `InvalidPDFException`/`FormatError`/`PasswordException` are mapped to typed parser errors at the single `openPdfDocument` choke point, and the zip guard's `ArchiveIntegrityError` surfaces from `parseBuffer` as a typed `invalid_format`, so neither classifies as transient and retries forever. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…enchmark Wires decodeTextBuffer into the HTML parser, refreshes the degraded docblock now that legacy formats raise typed errors, and adds the large corpus harness plus the regression-gated comparer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…etry The PDF parser collapsed every page to a single line and concatenated items without separators, so the chunker fell back to sentence splits, words fused across Form XObject boundaries and backwards x-moves, and running headers/footers landed mid-sentence in most chunks. - Build positioned lines from pdf.js item transforms; derive separators from baseline shifts, backwards x-moves, and word-sized gaps, falling back to hasEOL when an item carries no geometry - Join lines per page with paragraph breaks from the median pitch and height changes, rejoin same-row and wrapped table cells, and dehyphenate line-end breaks unless the compound appears intact in the document - Suppress repeated header/footer furniture and page numbers across pages, keeping the first occurrence of each - Prefix short oversized lines with a heading marker - Replace the whitespace collapse with a structure-preserving normaliser and join pages with a paragraph break Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Ambiguous archives and binary layouts stay on the SheetJS and legacy Word routes instead of being refused; line-end hyphens are removed only when the document shows the joined word; page numbers printed inside a wide margin are dropped from a page's edge lines; time-of-day cells no longer carry the 1899 epoch; table cells with several paragraphs keep a space between them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Kubernetes manifests, Helm output and CI fixtures hold several YAML documents separated by ---; js-yaml's single-document load rejected them outright. A stream now becomes one item per document. JSON files with comments or trailing commas (tsconfig, editor settings) parse leniently after strict parsing fails, with a warning in metadata. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Elapsed formats (`[h]:mm`, `[mm]:ss`) are durations; `cellDates` still parses them into a Date, so the ISO rewrite fabricated `1900-01-01T06:00:00` where Excel shows `30:00`. Their SSF-rendered `w` is now kept. A time-only cell was decided by its epoch year, which breaks in a 1904 workbook where `h:mm:ss` landed on `1904-01-01T12:29:59`; the decision now comes from the format (no `y`/`d`, and every `m` run beside hours or seconds), verified for xlsx, xls, xlsb and ods in both epochs. General numbers round fractions to Excel's 15 significant digits (`=0.1+0.2` reads `0.3`) while integers stay exact. The Files viewer read its workbook without `cellDates`/`cellNF`, which left the normalizer overwriting every rendered `w`; the read now lives in `readXlsxWorkbook` with the display options, and its test builds the fixture through that read path. A tab or line break inside a cell no longer splits the row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A serial such as 45366.572916666664 parses to 13:44:59.999, and slicing the ISO string truncated it to 13:44:59 — one second early for three of twelve probed cells. The instant is rounded to the nearest second before either the date-time or time-only text is formatted, and a value that rounds up to midnight renders as a whole date. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sample reference lines across the whole document instead of its head, add count-aware word-depletion checks so a repeated table header that vanishes is visible, score noise symmetrically, and commit the corpus build scripts with a SHA-256 manifest so the 961-file benchmark can be rebuilt. Adds a README with requirements. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…acy formats
Routing and the KB pipeline:
- A non-PDF named .pdf (HTML error page, plain text) is rejected as a
permanent `invalid_file` in `readEmbeddedPdfText` instead of being indexed
as the "text layer" or sent to OCR to fail terminally; the document
processor logs `detectedType`/`warning` at info with the filename (no
document id is in scope in that module).
- The HTML override now applies only to `.txt` and `.md` (a `.md` opening
with `<!DOCTYPE html>` is deliberately treated as HTML); an HTML document
under csv/json/jsonl/yaml is `invalid_format`.
- RTF (`{\rtf` at offset 0) is a sniffed kind and is `unsupported_type`
under any extension, so control words are never indexed as prose.
- `%PDF-` is searched through the first KiB only under a declared `.pdf`;
elsewhere it must be at offset 0 (after BOM/whitespace), so a `.txt` that
mentions the magic string stays text.
- NUL bytes in a declared text file keep the text route: the decoder handles
UTF-16/Windows-1252 and the sanitizer strips stray NULs. Recognised
containers under a text extension are still refused.
- `resolveParserExtension` throws `FileParserError('unsupported_type')`, so
stored `.ppt` documents dead-letter as permanent instead of burning the
retry budget as transient.
- Workspace-file "get content" (`internal/file/operations.ts`) treats
`degraded` output as a parse failure like the other two tool paths.
Decoding:
- Windows-1252 uses the runtime `TextDecoder` when a module-init self-test
proves the label is real (Bun 1.3.14), else a one-pass table decode into
UTF-16 code units. 100 MB of C1 bytes: 379 ms / +200 MB native,
286 ms / +401 MB table (was ~4 s / +5.4 GB).
- Connectors sftp, s3, databricks, google-drive and bitbucket decode through
`decodeTextBuffer` (bitbucket previously skipped non-UTF-8 files; it now
indexes them decoded). Connectors hash source revisions (blob sha, etag,
rev), not decoded text, so there is no mass re-sync: documents indexed
earlier with mojibake stay as they are until the source changes.
Legacy .doc: text boxes are a sixth extracted section; word-extractor's raw
`RangeError` text is replaced by the stable "This .doc file could not be
read".
Behaviour notes: UTF-32 input is not recognised and decodes as Windows-1252;
BOM-less UTF-16 whose code units are mostly non-ASCII (CJK) does not match
the NUL-layout heuristic and also falls to Windows-1252 — the warning now
says the file may use another encoding. Stale `.ppt` mentions removed from
the files-audit OpenAPI description (regenerated), the Box representation
list, `OFFICE_REPAIR_EXTENSIONS` and two TSDoc blocks; the package.json
re-sort from the previous commit is reverted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Nested tables in HTML and DOCX were emitted once glued into the outer cell and again as rows of their own; the HTML walker now visits only a table's direct rows and renders a nested table inline as its cells joined with ' / '. The ODF walker does the same inside cells. An encrypted .docx/.pptx/.xlsx is an OLE container carrying EncryptedPackage and EncryptionInfo streams, so the sniff now reports it as encrypted-ooxml and every route maps it to encrypted_file instead of unsupported_type or a mojibake plaintext fallback. The presentation and ODF walkers bound each XML part at 16 MB before parsing, honor mc:AlternateContent, skip slidenum/datetime fields anywhere, clamp notes targets under ppt/notesSlides, emit picture alt text, and reject an archive with no content.xml as invalid_format. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
# Conflicts: # apps/sim/lib/file-parsers/sniff.ts
…arser modules The knowledge page reaches document-processor and therefore the file parsers; seven new parser modules plus word-extractor's dependency tree add server-graph modules beyond the allowed drift. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… markers Furniture suppression treated any band text repeating across pages as a running header, which deleted multi-page table headers (IRS tax tables, NIST recommendation tables, EIC tables, DFAST captions). A band group is now exempt when it runs into the body at line pitch, when its key also occurs in body positions, or when dropping it would orphan a hyphenated word; folio candidates get the same flow test so edge table cells survive, and roman numerals must parse and fit the page count. joinLines accumulated the page in one string and ran anchored regexes over it per line, which was quadratic (40k lines: 72 s, now 7 ms); the hyphen and compound scans are bounded to the line tail, assembly yields to the event loop and honours the abort signal, and the line count and word set are capped. Geometry separators no longer count against the character budget, and preview output that overflows after decoration sets the truncated flag. Heading markers are off by default: on documents dominated by table or footnote text the estimated body height turned prose into headings that the chunker then split per line. The estimator now weighs prose-like lines only and skips runs of same-height lines for when it is enabled. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ath plain cells Only the slide-number field is layout text; date and time fields outside a dt placeholder are content, and skipping them emptied a deck made of them. Image alternative text that is a bare file name or an auto caption is noise, so the HTML, PresentationML, and OpenDocument walkers share one filter. A table cell with no element children is read directly instead of running the block-spacing and nested-table queries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lity # Conflicts: # apps/sim/connectors/google-drive/google-drive.ts
…t text The PPTX walker sorted physical slide part names, but a deck reordered in PowerPoint keeps its old part names and changes only p:sldIdLst, so it was indexed out of order. Slides now follow the presentation's id list resolved through its rels, skipping ids whose part is missing and falling back to part numbering only when nothing resolves. Graphic frames that hold SmartArt or a chart were dropped entirely; the diagram data part's dgm:pt text bodies and a modest chart summary (title, axis titles, series, categories) are now emitted, with connector text included. Every relationship target is clamped under ppt/ and read through the per-part size cap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Collaborator
Author
Collaborator
Author
|
@cubic-dev-ai review this PR |
Contributor
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
…targets An ODF text:s element passed its text:c count straight to String.repeat, so one tiny element could request a multi-gigabyte allocation. Each run is now capped at 100 spaces, every emitted piece is charged against a 16 MiB document budget that throws complexity_limit before later parts are inflated, and both walkers assert the assembled text against the same ceiling. The line-end trim that followed was quadratic on a long whitespace run — a document inside the budget could still hang it — so it is now a linear per-line trimEnd. OPC relationship targets may be package-absolute (/ppt/slides/slide1.xml); the PPTX resolver joined them onto the base directory and skipped those parts. A leading slash now resolves from the package root under the same ppt/ clamp. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Collaborator
Author
Collaborator
Author
|
@cubic-dev-ai review this PR |
Contributor
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An audit of the knowledge-base file parsers found that content recall was fine but the text reaching the chunker was structurally noisy: PDFs arrived as one line with running headers and footers mid-sentence and words glued across line breaks; Office tables were exploded one cell per line; spreadsheet dates were indexed as Excel serials (including every Google Sheet synced through Drive); legacy
.doc/.pptfell through to a byte scrape that emitted ZIP names and XML; non-UTF-8 text lost its accents silently; a corrupt PDF raised an untyped error. Each finding was verified against the code and open-source precedent (unpdf, pdf.js, Marker, OmniDocBench, olmOCR-bench, MarkItDown, Docling, unstructured, Tika, SheetJS), fixed, benchmarked, then audited again by six independent reviewers whose findings were fixed in turn.Fixes
pdf-parser.ts, newpdf-lines.ts,pdf-furniture.ts): rebuild lines and paragraphs from item geometry instead of collapsing whitespace (the collapse was copied from unpdf 1.4.0, fixed upstream in 1.7.0); spaces on x-gaps and breaks on backwards moves so Form-XObject and table-cell fusions stop; dehyphenate only when the document shows the joined word; suppress running headers/footers that repeat across pages (frequency + streak rule, first copy kept, table headers exempt via a text-flow test) and page numbers; join pages with a blank line. The page join is linear; heading markers exist but are off by default.xlsx-parser.ts, newsheet-display-text.ts, file viewer): display text ($1,250.00,20%,2026-03-04,30:00,TRUE), ISO dates independent of timezone and epoch, elapsed-time formats and time-only cells rendered as Excel does, General numbers at 15 significant digits with integer IDs exact.docx-parser.ts,html-parser.ts): mammoth HTML through the shared structured-text walker so headings, lists, tables (nested tables once), footnotes and endnotes survive; the never-readmetadata.htmlis gone.ooxml-presentation.ts,odf-text.ts,office-text.ts): JSZip + htmlparser2 walkers that render table rows, skip slide-number/date/header/footer placeholders and notes-page chrome, keep speaker-note bodies and date fields, handlemc:AlternateContent, skip ODT annotations and tracked deletions, cap XML parts at 16 MiB, and detect encrypted packages.doc-parser.ts,pptx-parser.ts):.docviaword-extractor@1.0.4(body, headers, footers, footnotes, endnotes, text boxes); Word 6/95 →unsupported_type..pptremoved from every allowlist and refused with a typed error; the byte-scrape fallbacks are deleted. The Sim file reader, File block and workspace-file "get content" treatdegradedoutput as a parse failure instead of handing it to the model.utils.ts,sniff.ts,ooxml-encryption.ts,pdfjs-server.ts, text parsers, connectors): BOM → strict UTF-8 → Windows-1252 decoding with a warning (single-pass, bounded); magic-byte sniffing that overrides an obviously wrong extension, refuses RTF and encrypted packages with typed errors, and keeps ambiguous bytes on the declared route; pdf.jsInvalidPDFException/PasswordExceptionwrapped asFileParserErrors; HTML mislabelled as.pdfno longer indexed as a text layer;.pptrows already in a knowledge base fail permanently instead of burning retries.Evaluation framework and benchmark
apps/sim/scripts/parser-eval/holds the plan, a ground-truth corpus generator (pandoc + typst renders with known headers, footers and tables), a fetcher and SHA-256 manifest for a 961-file real-world corpus, a bun harness over the productionparseBufferpath, reference extractors, a scorer modelled on olmOCR-bench unit tests, and a before/after comparer with a regression gate (line recall sampled across the whole document, vocabulary recall, noise, glued tokens, count-aware word depletion).BENCHMARK.mdis rendered from the comparer's output and records the metric history.Final run, 961 files, staging vs this branch: PDF line recall 0.990 → 0.989 and vocabulary 0.977 → 0.976 while output goes from one line to real paragraphs and glued tokens fall 3.2 → 1.2 per file;
.docnoise 0.75 → 0.003 with recall 0.52 → 0.79; DOCX/ODT vocabulary recall 0.94 → 0.96 / 0.87 → 0.97; text-only spreadsheets byte-identical; latency flat (mean 30 → 31 ms, p99 303 → 329 ms, max 1.37 → 1.10 s). 56 gate flags remain, each traced to a reference artifact or an intended change (table inBENCHMARK.md). Ground-truth corpus: PDF paragraph retention 0.06 → 0.98, Office table adjacency 0.00 → 1.00, typed-cell presence 0.87 → 1.00, robustness 14/14.Behaviour changes to note
.pptand Word 6/95 files are refused with typed errors everywhere (File block and Chat previously received scraped text for them).Test plan
bun run type-check, Biome, andbun run check:audits(all 46 checks, tool-registry baseline re-recorded for the new parser modules) pass.apps/simvitest: 3404 files / 48,060 tests pass. New unit tests cover line joining (including a 50k-line join under 1 s), dehyphenation, furniture detection with repeated table headers, sheet display text under four timezones and both epochs, the OOXML/ODF walkers, encrypted packages, sniffing and decoding, YAML streams, JSONC, and the degraded-output gates.bench/build.sh,bench-run.ts,bench-compare.pyandbench-summary.py(seeREADME.md).🤖 Generated with Claude Code