feat: build-generate AI retrieval layer (knowledge graph + index)

[eugenia-scandit]

Add a Docusaurus postBuild plugin that turns the rendered docs into an AI-consumable layer, generated at build time:

/assets/knowledge-retrieval-index.json  (fast lookup)
/assets/knowledge-graph.jsonld          (concept graph)

Parses the final rendered HTML (so imported partials/MDX are captured in full), splits each page of the version served at the root into ~1400-char knowledge modules, and derives per-chunk metadata: summary (from the frontmatter description), rule-based intents/audiences, framework, product, and the real URL. The graph mines real edges — BelongsToProduct, CitesApi, SeeAlso, and per-product AvailableOn / NotAvailableOn. Extraction failures are non-fatal so they never block a deploy.

Division of labour with docusaurus-plugin-llms

Both plugins export for AI consumers, so the boundary is worth stating. Figures are from a current build.

docusaurus-plugin-llms ships the prose, for context-stuffing: llms.txt as a link index (0.08 MiB) and llms-full.txt as the text (2.28 MiB).

This plugin ships typed metadata and edges, for deciding what to read: intents, audiences, channels, frameworks, products, cites-API, see-also, availability — none of which a flat text dump can express. It publishes only an excerpt of the prose (10.41 MiB index, 11.62 MiB graph) and a url; the full text is the page itself.

Neither replaces the other. Publishing the prose per module would add roughly 6 MiB to a 10 MiB artifact to restate text that is already published beside it, so CHUNK_TARGET_CHARS is a granularity knob rather than a payload size: it decides which links land in which module's references and api_refs, so changing it changes the edges.

Shared scope, one deliberate difference. Both read customFields.assistantIgnoreFiles — the curation half of the llms ignore list, covering deprecated platforms, features a platform lacks, redirect-only stubs and non-route partials — so what an assistant may see is decided once. They differ only on llmsDedupedToWeb: a flat corpus wants a single copy of prose repeated across frameworks, while a routing index needs the module that points at the framework the reader is actually on. Applying that dedup here removed ~1,400 modules along with the ability to answer "where in the Flutter docs does this live".

Known limits

• 10 MiB is large to fetch whole. Nothing consumes the index that way today; if something does, it needs a served index or a split by framework rather than a smaller excerpt.
• On a frozen-at-root build most modules share one last_verified — the snapshot commit's, because that is when those files were created. Accurate, but close to a constant.
• --auto self-retrieval is gated at 0.80, against a measured 0.8493 over 531 pages / 4,386 modules. Re-measure after anything that changes what is indexed.
• One glob divergence from minimatch, unreachable today: * and whole-segment ** here also match a segment beginning with ., where minimatch defaults to dot: false. No doc source is dot-prefixed.

[lucatorella]

🤖 Two build/deploy-integrity findings inline.

[lucatorella]

🤖 Re-reviewed 65676e09..d948727b. One finding on the new metric; notes on the two earlier threads.

[raffaelefarinaro]

🤖 This review was AI-assisted. I verified each blocking finding by hand against the code and a local build before posting.

Starting with what works, because it is genuinely careful. The exclusion logic is correct, and I checked it against the built output rather than reading it: zero URLs from the frozen versions, the external data-capture-sdk reference, or any of the 235 *.html redirect stubs make it into the index. Build cost is not a regression, Build Documentation on this branch ran 3m29s to 4m08s against 3m36s to 4m00s on recent main. And the eval gates sit far above chance, since a random retriever over this corpus would score success@3 = 0.158 against your floor of 0.8, with 0.900 actual.

Your three open threads are already fixed, so close them

You have probably been looking at three red threads for six days. They are stale, not outstanding:

• cheerio and gray-matter are both declared in dependencies as of 1748eaf, with matching entries in both lockfiles.
• The zero-modules floor check at index.ts:697 and the .tmp plus rename writes at index.ts:712 landed in the same commit, which is exactly what the thread asked for.
• Recall was removed and replaced with an honestly named MIN_PRECISION in 972e6cb, plus a comment explaining the algebra.

Luca's re-confirm was against d948727b, which is six commits behind the current head 2a6edd5. I checked the recall claim against the built corpus rather than taking it on trust, and he was right: every one of the 20 gold entries resolves to between 31 and 1075 relevant modules against K = 3, so capped recall did reduce to precision. The fix is the right one.

Rebase before responding to anything else here

The last four commits, which add the entire frontmatter-ingestion feature, have zero CI runs. Here is why. main deleted package-lock.json in f23a28f (#416), and this branch still edits it, so git merge-tree reports a modify/delete conflict on package-lock.json plus a content conflict in package.json. GitHub cannot build the refs/pull/418/merge ref for a conflicted PR, so no pull_request run gets created. Runs exist and are green for the first five commits and then stop dead on 2026-08-05 at 12:53Z. gh pr checks 418 reports "no checks" because it queries head 2a6edd5 specifically.

A rebase onto main plus deleting package-lock.json fixes the conflict, restores CI, and picks up docs-gate.yml, which does not exist on this branch at all. Do that first, because CI will probably surface more than any human reader can, and it would be wasteful to work through review comments against a head that has never been built.

Provenance of the numbers

The corpus statistics below (4523 records, 89% at the 400-character cap, 219 Titanium modules, 26.9 MB, the random-retriever baseline) come from my own local build on Node 26 with npm ci, not from CI on Node 18 with yarn. My counts came out at 4523 modules against CI's 4526 at 972e6cb, a 0.07% difference I attribute to the four intervening commits. Treat the figures as indicative of magnitude, not exact.

What is in the inline comments

Five blocking: the package-lock.json conflict, the Titanium trees, the chunk truncation, the contradictory core availability edges, and the division of labour with docusaurus-plugin-llms. Three smaller ones: a substring match in pickIntents, MIN_AUTO_SUCCESS defaulting to zero, and last_verified being the build timestamp.

Two of these are product decisions more than defects, so I have put them as questions about intent rather than verdicts. I also could not see who consumes these artifacts, so my point about truncation argues from the plugin's own stated purpose in its header comment, not from a requirement I can verify. If a consumer exists that only needs pointers plus a snippet, that finding weakens a lot and I would rather hear that from you than assume it.

I have left a longer writeup with the non-blocking observations out of this review to keep it readable. Happy to walk through it whenever suits.

[raffaelefarinaro]

Blocking, as a question rather than a defect: what is the division of labour between this and docusaurus-plugin-llms?

Seven lines above this, the same config registers docusaurus-plugin-llms, which already emits an AI-consumable layer from the same docs on the same build: llms.txt at 0.07 MB and llms-full.txt at 2.03 MB in my build. That plugin's setup here is deliberately curated, with shared-partial dedup, non-Web-SDK root trimming, and measured before/after numbers in the header comment.

This plugin adds a second layer with different curation rules and 26.9 MB of output. I am not assuming one should go, but two parallel AI exports with no stated boundary is the thing most likely to rot, and the next person to touch either one will not know which is authoritative. Could you add a line to the PR description saying what each is for, and whether the intent is for one to eventually replace the other?

[raffaelefarinaro]

Blocking: this exclusion set does not honour a curation decision made elsewhere in the same config.

docusaurus.config.ts:49 declares llmsIgnoredSdkTrees = ["docs/sdks/titanium/**"] with the comment "Entire platform omitted from llms export (deprecated / not needed for assistant context)". build/llms.txt has zero /titanium/ references. This index has 219 modules under /sdks/titanium/.

Nothing secret leaks, so this is not a security problem, but the repo has already decided which trees it wants an assistant reading, and this artifact overrides that decision silently and publishes the result to the public site.

Was that deliberate? If the new layer should have wider coverage than the llms export, that is a defensible answer and worth writing down. If not, reusing llmsIgnoredSdkTrees and the shared-partial dedup list would keep the two layers consistent for free.

[raffaelefarinaro]

Blocking: the index does not ship the content it chunks.

CHUNK_TARGET_CHARS is 1400 at line 30, but these two lines truncate to 400 and 300. In my build, 4031 of 4523 records (89%) sit exactly at the 400-character cap and 4475 (99%) sit exactly at the 300-character cap. So roughly 70% of every chunk gets parsed, chunked, held in memory, and then discarded, and each published record is a fragment cut mid-sentence.

The header comment describes this as the artifact "an assistant / in-docs search consume". Against that goal, a consumer gets a snippet and still has to fetch the URL, which llms-full.txt already covers.

I want to be careful here, because I could not see who actually consumes these files. If the design intent is a pointer index where a 400-character preview is enough, this is fine as built and I would just note it in the header. If the intent is self-contained retrieval modules, then either raise the caps or drop the chunking to page level, because the 1400-character chunking currently buys nothing that survives into the output. Which is it?

[raffaelefarinaro]

text.includes(" auth") fires on any word starting with "auth", not just authentication. Every one of the 31 modules tagged with the secure intent in my build matched on authorized, authenticity or authority.

A word-boundary regex fixes it, something like /\bauth(entication|orization|n)?\b/. Small blast radius, easy fix.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://Scandit.github.io/data-capture-documentation/pr-preview/pr-418/
Built to branch gh-pages at 2026-09-08 09:51 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[raffaelefarinaro]

🤖 Automated review, posted on Raffa's behalf and not yet read by him. Treat the findings as a starting point, not a verdict. Anything that looks wrong probably is, so push back.

Seven of the eight existing review threads are stale. Closing them saves the most time here, so start there.

Fixed, safe to close:

• cheerio undeclared dependency (two lucatorella threads) — cheerio and gray-matter have been in dependencies since 1748eaf, matching lockfile entries confirmed.
• Empty extraction not asserted (two lucatorella threads) — the plugin now throws when pagesProcessed === 0 || index.length === 0, plus a 0.5 partial-drift ratio guard, plus best-effort atomic tmp+rename with rollback on partial failure. docs-retrieval-evals.yml also gained a push: main trigger, closing the gap Luca flagged about nothing checking main.
• recall@k algebraically equal to precision@k — replaced with an honestly-named MIN_PRECISION threshold in 972e6cb.
• package-lock.json conflict blocking every pull_request run — resolved by the merge from main at 5d54997, which picked up main's deletion of the file (#416). git status on this worktree confirms it, and CI is green: 6/6 checks passing, mergeable: MERGEABLE.
• Contradictory core availability edges (AvailableOn + NotAvailableOn for the same product/framework pair) — fixed with the SYNTHETIC_PRODUCTS set (core, general) excluded from availability mining, landed in 0110d02.
• last_verified being the build timestamp for every module — fixed by reading each file's real last-commit date via one batched git log (0110d02), with fetch-depth: 0 added to build-docs.yml, docs-preview.yml, and docs-retrieval-evals.yml (284aaca) so the shallow-clone case degrades to "" instead of a fake constant.

What's genuinely good here beyond the fixes: the empty-extraction guard is layered correctly (total-failure throw, then a partial-drift ratio check that can't be gamed by measuring against a post-filter count), and the rollback logic on a failed graph rename actually restores the previous index rather than leaving a mismatched pair. That is real build-integrity engineering, not just a try/catch.

Still open, not stale: the substring auth-intent match, the chunk-truncation question, the docusaurus-plugin-llms division-of-labour question, and the Titanium exclusion mismatch are all inline below, unchanged since the last review. One new item on the latest commit (7ed5493) is also inline.

Non-blocking, left out of inline comments to stay under the cap: MIN_AUTO_SUCCESS in scripts/retrieval-evals/run.cjs:44 still defaults to 0, so --auto can never fail its own gate. Still latent rather than live, since no workflow passes --auto today — but worth a real default whenever that mode gets wired into CI.

[raffaelefarinaro]

Still text.includes(" auth"), unchanged since the Aug 11 thread on pickIntents. Any word starting with "auth" (authorized, authenticity, authority...) still gets classified into the secure intent — small blast radius, but a real defect. A word-boundary regex like /\bauth(entication|orization|n)?\b/ fixes it in one line.

[eugenia-scandit]

Fixed — now /\bauth(?:n|entication|orization)?\b/.

I measured the blast radius across the built HTML before touching it, and it is
the other way round from "small": the substring matched authenticity 12
times, authentic 10 and authority 5 — every one of them the ID-document
sense, about whether a passport is genuine — against authorization twice.
So the old test was filing roughly 27 document-verification mentions under the
security intent and 2 real ones.

Verb forms are deliberately left out of the alternation: authenticate does not
occur in this corpus at all, while authorized does — in "authorized dealer" —
so adding them would re-admit the false positives to catch nothing. The counts
are in the comment so the next person can re-measure instead of reasoning.

[raffaelefarinaro]

The truncation finding from Aug 11 is only partially addressed. assistantExcerpt's clip logic did improve (clips the chunk itself rather than the whole templated string, so it averages ~515 chars now instead of ~300), and KModule.content.docs_markdown internally carries the full chunk up to the 1400-char CHUNK_TARGET_CHARS target. But toIndexRecord never surfaces m.content — only this docs_excerpt (400 chars) and assistant_excerpt reach the published index.json. So most of every chunk is still parsed, chunked, held in memory, and discarded before publish. Same open question as before: is this meant to be a pointer index (fine as built, worth a header note) or a self-contained retrieval module (then ship content.docs_markdown, or shrink CHUNK_TARGET_CHARS to match what's actually published)?

[eugenia-scandit]

Answered as a pointer index, documented in the plugin header rather than
left implicit — and one part of the framing I want to push back on.

The push-back. "Most of every chunk is still parsed, chunked, held in
memory, and discarded before publish" — the chunk text is not discarded, it is
the input the published fields are derived FROM. On the current build the chunk
prose yields 2,823 api_refs, 1,053 internal references, 18,846
keywords, plus a summary and intents for all 4,386 modules. What is not
published is the prose itself.

That is also why CHUNK_TARGET_CHARS is a granularity knob and not a
payload size: it decides which links land in which module's references and
api_refs, so shrinking it to match the excerpt would change what a module IS —
more, smaller modules with different edges — which is a retrieval-quality call
to make against the evals, not a size cleanup.

The decision, and its cost. The index is already 10.41 MiB with an
11.62 MiB graph; publishing content.docs_markdown would add roughly
6 MiB more. For scale, llms-full.txt holds the site's prose in 2.28
MiB, so per-module text is an expensive way to store text.

One correction to my own earlier draft of this answer: I had written that each
module's prose "ships next to it" in llms-full.txt. I could not establish
that and have dropped it — the two artifacts cut the corpus differently, 452
page sections there against 4,386 modules here, so there is no per-module
correspondence to appeal to. The argument rests on the url (the page itself,
always reachable) and on the ~6 MiB.

Recorded as a known limit, prompted by your comment: 10 MiB is large for a
browser to fetch whole. Nothing consumes it that way today; if something does it
needs a served index or a split by framework, not a smaller excerpt.

docs_excerpt is also now documented honestly as averaging 376.5 with a max of
404, not "<=400" — clipMarkdown appends the fence closer after slicing, so
400 is the slice budget and not an enforced bound.

[raffaelefarinaro]

The Aug 11 question here — what's the division of labour between this and docusaurus-plugin-llms, registered a few lines up — doesn't seem to have been answered anywhere in the PR description, commits, or thread replies. The plugin has roughly tripled in size and hardening since that question was asked (500 to 1750 lines), so the two-parallel-AI-exports-with-no-stated-boundary concern is more relevant now, not less. Worth a line in the description either way.

[eugenia-scandit]

Answered in the plugin header and in a comment beside the registration here,
so it is next to whichever one you are reading — and it will go in the PR
description too.

The boundary. docusaurus-plugin-llms ships the prose for
context-stuffing (llms.txt as a link index, 0.08 MiB; llms-full.txt as the
text, 2.28 MiB). This plugin ships typed metadata and edges for deciding
what to read — intents, audiences, channels, frameworks, products, cites-API,
see-also, availability — which a flat text dump cannot express, with only an
excerpt of the prose (10.41 MiB index, 11.62 MiB graph). Neither replaces the
other.

You were right that the concern grew rather than shrank, and that it had already
bitten — the Titanium thread. But my first attempt at fixing it was wrong in
both directions, and the honest answer is narrower than "they cannot disagree":

• Exporting only llmsIgnoredSdkTrees left them disagreeing about 39
  modules (26 Linux stubs, 13 redirect-only root docs).
• Exporting the whole llms ignore list over-corrected: it also carries a
  corpus-shape decision — the prose repeated under every non-Web SDK root,
  deduped to Web — and applying that here removed ~1,400 modules along with
  the ability to answer "where in the Flutter docs does this live", which is
  exactly what a routing index is for.

So the list is split at the seam that was already implicit in it.
assistantIgnoreFiles is the curation half — what no AI consumer should be given
— and is shared. llmsDedupedToWeb is the corpus half and is llms-only.
llmsIgnoreFiles is their concatenation, so every path is still declared once
and the one deliberate difference has a name and a reason beside it.

[raffaelefarinaro]

This exclusion set still doesn't skip docs/sdks/titanium/**, which docusaurus.config.ts's llmsIgnoredSdkTrees (line 61) explicitly excludes from the sibling docusaurus-plugin-llms export as "deprecated / not needed for assistant context." The Aug 11 thread on this is still open — a local build still indexes Titanium modules that the repo has already decided an assistant shouldn't see. Is broader coverage here deliberate, or should this reuse llmsIgnoredSdkTrees?

[eugenia-scandit]

Fixed — and fixed by applying your decision rather than paraphrasing it.

Measured: 4,637 modules to 4,386, zero from /sdks/titanium/, and the 39
modules you identified elsewhere (26 Linux stubs, 13 redirect-only root docs)
gone with them. Broader coverage was not deliberate; it was an oversight.

Two things went wrong in my first attempt, both worth recording:

1. I reused llmsIgnoredSdkTrees, which is one of six entries in what the llms
   plugin is actually given — hence the 39. The plugin now reads a new
   assistantIgnoreFiles, the curation half of that list. (Not the whole list:
   see the division-of-labour thread.)
2. I derived route prefixes by string surgery — strip docs/, strip a trailing
   *. That understood exactly one glob shape and failed open on every
   other: a brace list, a leading **/, a bare file path, a mid-path * all
   matched nothing, silently. Adding a tree in any other form would have given a
   green build that kept feeding an assistant what the repo had just decided to
   hide.

It is now a small glob compiler that handles *, ** and literals and
throws on a shape it cannot express. Deliberately not a matcher dependency:
none is declared in this repo, and reaching for a transitive one is the mistake
this PR was already pulled up on with cheerio. Ten cases checked, including
the rejection of {a,b}.

A missing customFields.assistantIgnoreFiles now throws as well. Absent meant
"index everything", which is the wrong default for a curation list and invisible
to both output guards, because the page count goes up.

[raffaelefarinaro]

New in 7ed5493. This fix targets the beta-cycle case (a frozen version served at root, next holding the unreleased tree), but the commit's own verification numbers ("4,637 modules from 616 pages, 0 errors, every module version: 'current', 0 modules from /next/") are only reachable when servesCurrent is true — which is the branch that was already correct before this change, since both the version label and the next exclusion are no-ops when servesCurrent. Was the servesCurrent === false path (the actual bug being fixed) verified against a real frozen-at-root build, or only reasoned through?

[eugenia-scandit]

You were right, and it is worth saying that you were right twice.

First, on this comment: the numbers I quoted came from a
servesCurrent === true build, which is exactly the branch that was already
correct. So I verified the other one — with DOCS_LAST_VERSION temporarily set
to 7.6.14, the extractor produced 2,908 modules from 463 pages, every one
labelled version: "7.6.14", zero from /next/.

Second, that test was still not enough, and the next review round caught it:
it exercised the version LABEL only. That build's root pages were the docs/
tree, so the source lookup was never asked the hard question. On a real beta
build the root is served from versioned_docs/version-<served>/ while docs/
becomes the unreleased tree, and sourceCandidates hard-coded docs/ — so
every root module would have taken its frontmatter, dates, topic_type,
product, keywords and last_verified from the next version's file.
Correct on the day of the snapshot, wrong from the first edit after it:
comparing versioned_docs/version-7.6.14 against today's docs/ gives 388 of
449 shared pages with differing frontmatter.

Fixed in a5d6cf3: sourceCandidates takes the served version and prefixes
accordingly, threaded through resolveSource, contributingFiles, sourceDate
and readFrontMatter.

One honest footnote on the 7.6.14 test: that build then failed in
plugin-client-redirects, because 7.6.14 has no /sdks/ios/agent-skills/ or
/migrate-7-to-8/ to redirect to. That is a property of picking an old version
for the test — a real beta cycle freezes the newest release, which has those
pages — not of the change. The config is restored and a normal build succeeds.