Skip to content

v0.8.31: search connector hardening, document parsers improvements, org view refinements - #7723

Merged
waleedlatif1 merged 30 commits into
mainfrom
staging
Sep 10, 2026
Merged

v0.8.31: search connector hardening, document parsers improvements, org view refinements#7723
waleedlatif1 merged 30 commits into
mainfrom
staging

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

TheodoreSpeaks and others added 30 commits September 9, 2026 18:26
…7692)

The signed-in front door sent organization members whose organization has not
been rolled out to workspace settings, so opening the app dropped them on the
General settings form instead of their workspace. Send them to the workspace
picker, which is where the entry pointed before the organization surface
existed. The default landing never opens settings; the /o guards still fall
back to settings for viewers who explicitly asked for the organization surface.

Also stop the entry from bouncing a stale session cookie to /login. The proxy
treats /home as an app surface and redirects cookie-less requests to /login
before the route renders, and auth-disabled deployments always resolve an
anonymous session, so a null session here always means a present-but-invalid
cookie. Redirecting that to /login was bounced straight back by the proxy's
presence-only cookie check, looping until the browser gave up and leaving the
viewer unable to reach the login page at all. Hand off to the workspace loader
instead, the one identity-recovery surface, which clears the stale cookies
before navigating.


Claude-Session: https://claude.ai/code/session_01CGkraEeFNiPCU4jb784FE4

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(organizations): align sidebar loading and chat actions

* fix(organizations): preserve chat context during pagination
* feat(search): add GitHub installation indexing

* fix(search): scope GitHub access checks to requested results

* fix(search): clarify public setup documentation
* fix(billing): scope subscription limit syncs to the exact payer

* fix(billing): retry subscription limit reconciliation through webhook events

* chore(tests): remove timing-dependent search setup case
* fix(landing): prevent theme flashes and sharpen footer animation

* fix(landing): preserve the footer liquid morph
Co-authored-by: Sim Pi Agent <pi@sim.ai>
…7693)

* improvement(settings): remove personal connected accounts page

* improvement(search): simplify personal integrations and source setup

* fix(search): update source route test fixtures
* feat(workspace-sync): add portable imports and v2 fork workflows

* fix(workspace-sync): preserve activity attribution and update boundary tests
* fix(search): expose provider configuration updates in Sources

* chore(search): document provider refresh option mapping
* fix(search): harden connector setup and indexed access

* fix(search): persist verified Gmail size skips
* fix(search): clear cached content after disconnect

* fix(search): reset directly opened document caches
* fix(search): clarify source settings and permission updates

* fix(search): clarify recovery when Search is disabled
* improvement(docs): illustrate GitHub Search setup

* improvement(docs): update GitHub App setup walkthrough
…/after benchmark (#7709)

* chore(parsers): add parser quality evaluation framework

Ground-truth corpus generator, real-world fetcher, bun harness over the
production parseBuffer path, reference extractors and scorer, plus the
plan and findings from the 2026-09-09 audit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): index spreadsheet cells as display text

`XlsxParser` converted sheets without `raw: false`, so the indexed text held
stored values rather than what a user sees: dates as Excel serials (46085),
20% as 0.2, $1,250.00 as 1250, booleans as `true`, and ODS dates as
`String(Date)` in the worker's local time zone. The Google Drive connector
exports every Google Sheet through this parser while the Sheets and Excel
connectors already request formatted text, so the same sheet indexed
differently by path. The Files viewer had the same defect.

Read with `cellDates` + `cellNF` and convert with `raw: false`, rewriting
only the two cases the file's own text gets wrong inside the bounded window:
dates become zone-free ISO text from the UTC fields SheetJS parsed, and
General numbers print their full stored value instead of Excel's 11-char
rendering (4111111111111111 -> 4.11111E+15). The shared pass handles dense
and sparse sheets so the viewer reuses it without pulling `xlsx` into the
client bundle.

The parser-eval fixture used `0.#%`, which Excel renders as `20.%`; it now
uses `0%` / `0.0%` so the spec strings match Excel. The `sheet-wide` row
builder is also typed so the script type-checks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): walk office document structure instead of flattening cells

DOCX now routes mammoth's HTML rendering through the shared HTML
structured-text walker so tables keep their rows, lists keep their
markers, and footnotes survive; the unread metadata.html field is gone.
PPTX and ODT/ODP get dedicated XML walkers that render tables row by
row, skip slide-number/date/header/footer placeholders, read presenter
notes from the notes body placeholder only, and drop ODF annotations
and tracked deletions. Legacy OLE .ppt is rejected as unsupported_type
instead of scraping printable bytes from the container.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): decode text by encoding, sniff bytes before routing, read legacy .doc

Text parsers decoded every buffer as UTF-8 and then stripped U+FFFD, so a
Latin-1 or Windows-1252 file silently lost every accented character, a UTF-8
BOM leaked into content and broke JSON.parse, and UTF-16 only worked for
ASCII. `decodeTextBuffer` (BOM > strict UTF-8 with a guarded truncated-tail
retry > Windows-1252) now backs txt/md/csv/json/jsonl/yaml, the .doc plain-text
fallback and the connectors' text decode, and records `encoding`/`warning`
in metadata.

`parseBuffer` routed on the caller-supplied extension alone. `sniff.ts` now
identifies the bytes (PDF, OLE2, ZIP central-directory part names, ODF
mimetype, UTF-16 layout, HTML head) and reconciles them with the extension's
family: a sniffed kind with its own parser overrides the route and records
`detectedType`; binary/unknown bytes under a mismatched family are a typed
`invalid_format` instead of mojibake or placeholder prose.

Legacy OLE .doc goes through word-extractor (body, headers, footers,
footnotes, endnotes; Word 6/95 magic maps to `unsupported_type`); the byte
scrape that returned ZIP part names as degraded prose is deleted. Legacy .ppt
is dropped from the registry, upload and connector allowlists and Chat's
parseable set so it is refused up front. Chat's file reader and the internal
file tool now treat `degraded` output as a parse failure.

pdf.js `InvalidPDFException`/`FormatError`/`PasswordException` are mapped to
typed parser errors at the single `openPdfDocument` choke point, and the zip
guard's `ArchiveIntegrityError` surfaces from `parseBuffer` as a typed
`invalid_format`, so neither classifies as transient and retries forever.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): decode HTML by detected encoding and add before/after benchmark

Wires decodeTextBuffer into the HTML parser, refreshes the degraded
docblock now that legacy formats raise typed errors, and adds the large
corpus harness plus the regression-gated comparer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): rebuild PDF line and paragraph structure from item geometry

The PDF parser collapsed every page to a single line and concatenated
items without separators, so the chunker fell back to sentence splits,
words fused across Form XObject boundaries and backwards x-moves, and
running headers/footers landed mid-sentence in most chunks.

- Build positioned lines from pdf.js item transforms; derive separators
  from baseline shifts, backwards x-moves, and word-sized gaps, falling
  back to hasEOL when an item carries no geometry
- Join lines per page with paragraph breaks from the median pitch and
  height changes, rejoin same-row and wrapped table cells, and
  dehyphenate line-end breaks unless the compound appears intact in the
  document
- Suppress repeated header/footer furniture and page numbers across
  pages, keeping the first occurrence of each
- Prefix short oversized lines with a heading marker
- Replace the whitespace collapse with a structure-preserving normaliser
  and join pages with a paragraph break

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): tighten benchmark-found edge cases

Ambiguous archives and binary layouts stay on the SheetJS and legacy
Word routes instead of being refused; line-end hyphens are removed only
when the document shows the joined word; page numbers printed inside a
wide margin are dropped from a page's edge lines; time-of-day cells no
longer carry the 1899 epoch; table cells with several paragraphs keep a
space between them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(parsers): record the before/after parser benchmark

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* style(parsers): apply biome formatting

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): accept YAML document streams and JSON with comments

Kubernetes manifests, Helm output and CI fixtures hold several YAML
documents separated by ---; js-yaml's single-document load rejected them
outright. A stream now becomes one item per document. JSON files with
comments or trailing commas (tsconfig, editor settings) parse leniently
after strict parsing fails, with a warning in metadata.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): render elapsed, time-only and General cells as Excel does

Elapsed formats (`[h]:mm`, `[mm]:ss`) are durations; `cellDates` still
parses them into a Date, so the ISO rewrite fabricated `1900-01-01T06:00:00`
where Excel shows `30:00`. Their SSF-rendered `w` is now kept. A time-only
cell was decided by its epoch year, which breaks in a 1904 workbook where
`h:mm:ss` landed on `1904-01-01T12:29:59`; the decision now comes from the
format (no `y`/`d`, and every `m` run beside hours or seconds), verified for
xlsx, xls, xlsb and ods in both epochs. General numbers round fractions to
Excel's 15 significant digits (`=0.1+0.2` reads `0.3`) while integers stay
exact.

The Files viewer read its workbook without `cellDates`/`cellNF`, which left
the normalizer overwriting every rendered `w`; the read now lives in
`readXlsxWorkbook` with the display options, and its test builds the fixture
through that read path. A tab or line break inside a cell no longer splits
the row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): round float date serials to the nearest second

A serial such as 45366.572916666664 parses to 13:44:59.999, and slicing the
ISO string truncated it to 13:44:59 — one second early for three of twelve
probed cells. The instant is rounded to the nearest second before either the
date-time or time-only text is formatted, and a value that rounds up to
midnight renders as a whole date.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(parser-eval): harden the comparer and pin the benchmark corpus

Sample reference lines across the whole document instead of its head,
add count-aware word-depletion checks so a repeated table header that
vanishes is visible, score noise symmetrically, and commit the corpus
build scripts with a SHA-256 manifest so the 961-file benchmark can be
rebuilt. Adds a README with requirements.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): close audit findings on byte sniffing, decoding and legacy formats

Routing and the KB pipeline:
- A non-PDF named .pdf (HTML error page, plain text) is rejected as a
  permanent `invalid_file` in `readEmbeddedPdfText` instead of being indexed
  as the "text layer" or sent to OCR to fail terminally; the document
  processor logs `detectedType`/`warning` at info with the filename (no
  document id is in scope in that module).
- The HTML override now applies only to `.txt` and `.md` (a `.md` opening
  with `<!DOCTYPE html>` is deliberately treated as HTML); an HTML document
  under csv/json/jsonl/yaml is `invalid_format`.
- RTF (`{\rtf` at offset 0) is a sniffed kind and is `unsupported_type`
  under any extension, so control words are never indexed as prose.
- `%PDF-` is searched through the first KiB only under a declared `.pdf`;
  elsewhere it must be at offset 0 (after BOM/whitespace), so a `.txt` that
  mentions the magic string stays text.
- NUL bytes in a declared text file keep the text route: the decoder handles
  UTF-16/Windows-1252 and the sanitizer strips stray NULs. Recognised
  containers under a text extension are still refused.
- `resolveParserExtension` throws `FileParserError('unsupported_type')`, so
  stored `.ppt` documents dead-letter as permanent instead of burning the
  retry budget as transient.
- Workspace-file "get content" (`internal/file/operations.ts`) treats
  `degraded` output as a parse failure like the other two tool paths.

Decoding:
- Windows-1252 uses the runtime `TextDecoder` when a module-init self-test
  proves the label is real (Bun 1.3.14), else a one-pass table decode into
  UTF-16 code units. 100 MB of C1 bytes: 379 ms / +200 MB native,
  286 ms / +401 MB table (was ~4 s / +5.4 GB).
- Connectors sftp, s3, databricks, google-drive and bitbucket decode through
  `decodeTextBuffer` (bitbucket previously skipped non-UTF-8 files; it now
  indexes them decoded). Connectors hash source revisions (blob sha, etag,
  rev), not decoded text, so there is no mass re-sync: documents indexed
  earlier with mojibake stay as they are until the source changes.

Legacy .doc: text boxes are a sixth extracted section; word-extractor's raw
`RangeError` text is replaced by the stable "This .doc file could not be
read".

Behaviour notes: UTF-32 input is not recognised and decodes as Windows-1252;
BOM-less UTF-16 whose code units are mostly non-ASCII (CJK) does not match
the NUL-layout heuristic and also falls to Windows-1252 — the warning now
says the file may use another encoding. Stale `.ppt` mentions removed from
the files-audit OpenAPI description (regenerated), the Box representation
list, `OFFICE_REPAIR_EXTENSIONS` and two TSDoc blocks; the package.json
re-sort from the previous commit is reverted.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): dedupe nested tables, sniff encrypted OOXML, cap XML parts

Nested tables in HTML and DOCX were emitted once glued into the outer
cell and again as rows of their own; the HTML walker now visits only a
table's direct rows and renders a nested table inline as its cells
joined with ' / '. The ODF walker does the same inside cells. An
encrypted .docx/.pptx/.xlsx is an OLE container carrying
EncryptedPackage and EncryptionInfo streams, so the sniff now reports it
as encrypted-ooxml and every route maps it to encrypted_file instead of
unsupported_type or a mojibake plaintext fallback. The presentation and
ODF walkers bound each XML part at 16 MB before parsing, honor
mc:AlternateContent, skip slidenum/datetime fields anywhere, clamp notes
targets under ppt/notesSlides, emit picture alt text, and reject an
archive with no content.xml as invalid_format.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(parsers): re-record tool-registry module baseline for the new parser modules

The knowledge page reaches document-processor and therefore the file
parsers; seven new parser modules plus word-extractor's dependency tree
add server-graph modules beyond the allowed drift.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): keep table headers, bound PDF assembly, disable heading markers

Furniture suppression treated any band text repeating across pages as a
running header, which deleted multi-page table headers (IRS tax tables,
NIST recommendation tables, EIC tables, DFAST captions). A band group is
now exempt when it runs into the body at line pitch, when its key also
occurs in body positions, or when dropping it would orphan a hyphenated
word; folio candidates get the same flow test so edge table cells
survive, and roman numerals must parse and fit the page count.

joinLines accumulated the page in one string and ran anchored regexes
over it per line, which was quadratic (40k lines: 72 s, now 7 ms); the
hyphen and compound scans are bounded to the line tail, assembly yields
to the event loop and honours the abort signal, and the line count and
word set are capped. Geometry separators no longer count against the
character budget, and preview output that overflows after decoration
sets the truncated flag.

Heading markers are off by default: on documents dominated by table or
footnote text the estimated body height turned prose into headings that
the chunker then split per line. The estimator now weighs prose-like
lines only and skips runs of same-height lines for when it is enabled.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): keep date fields, drop file-name image alt text, fast-path plain cells

Only the slide-number field is layout text; date and time fields outside
a dt placeholder are content, and skipping them emptied a deck made of
them. Image alternative text that is a bare file name or an auto caption
is noise, so the HTML, PresentationML, and OpenDocument walkers share
one filter. A table cell with no element children is read directly
instead of running the block-spacing and nested-table queries.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(parsers): regenerate the benchmark from the final run

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): walk slides in display order and read SmartArt and chart text

The PPTX walker sorted physical slide part names, but a deck reordered in
PowerPoint keeps its old part names and changes only p:sldIdLst, so it
was indexed out of order. Slides now follow the presentation's id list
resolved through its rels, skipping ids whose part is missing and falling
back to part numbering only when nothing resolves. Graphic frames that
hold SmartArt or a chart were dropped entirely; the diagram data part's
dgm:pt text bodies and a modest chart summary (title, axis titles,
series, categories) are now emitted, with connector text included. Every
relationship target is clamped under ppt/ and read through the per-part
size cap.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(parsers): bound ODF whitespace expansion and accept absolute OPC targets

An ODF text:s element passed its text:c count straight to String.repeat,
so one tiny element could request a multi-gigabyte allocation. Each run
is now capped at 100 spaces, every emitted piece is charged against a
16 MiB document budget that throws complexity_limit before later parts
are inflated, and both walkers assert the assembled text against the
same ceiling. The line-end trim that followed was quadratic on a long
whitespace run — a document inside the budget could still hang it — so
it is now a linear per-line trimEnd.

OPC relationship targets may be package-absolute (/ppt/slides/slide1.xml);
the PPTX resolver joined them onto the base directory and skipped those
parts. A leading slash now resolves from the package root under the same
ppt/ clamp.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…#7711)

* feat(search): add personal integration inventory and connection cards

* feat(slack): stream Search tool progress

* fix(tests): isolate PitchBook error handling from network
* feat(mcp): add dynamic operations and workflow access controls

* improvement(mcp): simplify server and operation selection

* fix(mcp): preserve operation selection and discover managed connections

* fix(mcp): scope operation provenance to MCP delegations

* improvement(mcp): use literal tool IDs for Agent access controls

* fix(mcp): migrate portable references with operation fields

* fix(mcp): isolate shared query keys from discovery hooks
…story (#7717)

* fix(search): bound personal Gmail sources and sync them from Gmail history

A personal Gmail connection indexed the whole mailbox from all time, and
members mode removes the thread cap by design (a capped listing cannot tell
a thread that fell out of the window from one the person lost access to).
Search sources now start from the connector's declared defaults, so Gmail
indexes the last six months unless the source says otherwise. Explicit
settings still win and knowledge-base connectors are unchanged.

Every hourly Gmail sync relisted the entire mailbox because the connector
had no change feed. It now opens a cursor at the mailbox history id and
reads users.history.list, re-reading only threads that gained a message,
were relabelled, or were deleted, and evaluates the configured labels,
date range and category exclusions locally. A free-form search filter
cannot be evaluated locally, so such a source keeps relisting. An expired
history id reopens the feed from a full listing through the engine's
existing cursor-invalid path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(search): assert the Gmail date-range table as const

The lookup narrows an arbitrary config string through a type guard
instead of indexing a widened record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(slack): preserve text order around tool progress

* fix(slack): defer task progress behind withheld text
…ed invites (#7718)

* fix(search): index only meetings from Google Calendar and mark declined invites

The listing never filtered by event type, so working-location, out-of-office,
focus-time and birthday entries were indexed as documents. With recurring
events expanded to instances, a daily working-location entry alone produced
dozens of near-identical documents per member. The listing now asks Google
for default events only and the connector drops any other type it still
receives, including on a direct fetch.

A shared calendar the member can read only as free/busy returns time blocks
with no title or description, which were indexed as "Untitled Event". An
event with neither is no longer a document.

Invitations the connected account declined stay indexed, since the agenda
and links are still something the person was sent, but the content now
carries a "Response: declined" line and the metadata records the response.
The metadata-only hash gains a declined suffix so already-indexed
invitations pick the line up on the next sync without waiting for an edit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(search): keep untitled Calendar meetings that name a room or attendees

Only an event with no title, description, location, organizer and no
attendees is the bare time block a free/busy reader receives.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…uments for an hour (#7720)

* fix(knowledge): wait for embedding admission instead of deferring documents for an hour

Every embedding batch on the indexing path waited at most five seconds for
the deployment's shared admission bucket. Twenty concurrent documents fanning
out eight batches each queue for minutes behind the configured per-minute
budget, so under load most batches timed out, the document stopped, and it
was re-dispatched with a delay that started at a minute and doubled to an
hour. The bucket's own estimate of when capacity returns was only a floor
under that ladder. During a bulk sync this produced thousands of hour-long
deferrals for a limiter we run ourselves, while the provider was healthy.

The knowledge path now waits up to a minute for admission, which is cheaper
than the re-dispatch it replaces and still bounded by the per-request retry
budget. The request bucket admits 64 concurrent starts instead of 8, so
documents that begin together no longer lose a race for slots while the
token budget sits unused. When an admission wait still expires, the document
resumes after the bucket's stated wait, clamped to 10 to 60 seconds with
jitter, and the yield counts against the processing-slice budget rather than
the provider-failure attempts, since the provider did nothing wrong. The
deadline path now carries the bucket's last stated wait so that estimate is
available to the scheduler.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(knowledge): report only the admission wait still left at the deadline

The bucket's stated wait is stored as an absolute instant so a deadline hit
after a sleep carries the remainder, not the original duration. A test pins
the knowledge admission wait below the retry budget the processing deadline
reserves for each request.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(search): add central Google Workspace indexing

* fix(search): align Google source management and guides
…ons (#7722)

* fix(search): clarify Atlassian setup and validate Confluence permissions

* chore(docs): clarify Confluence setup permission sampling
@waleedlatif1
waleedlatif1 requested a review from a team as a code owner September 10, 2026 08:05
@vercel

vercel Bot commented Sep 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 10, 2026 8:06am UTC

Request Review

@greptile-apps

greptile-apps Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Too many files changed for review (830 files, 100 file limit).

@waleedlatif1
waleedlatif1 merged commit 03dffb4 into main Sep 10, 2026
70 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants