feat(ai): retrieve over CopilotKit's source, not only its docs

[NathanTarbert]

Phase 2 of the response-quality work — CPK-8077, from the Agent's Output Doc.

Root cause 1 in that doc: for any question whose answer lives in the source — most of the hard ones — the agent had only the docs and was otherwise guessing from general React knowledge. A reporter asked whether Deep Agents supports subagents; the docs don't mention it, so the agent said it had no timeline and sent them to GitHub to ask. Subagents work today and one code search returns the proof.

This is wiring, not construction

Probed tools/list on https://mcp.copilotkit.ai/mcp rather than assuming. The server already exposes search-code, search-ag-ui-code and search-ag-ui-docs alongside search-docs — all four with an identical schema (query, limit, min_score, version). The client called one of them, and never called tools/list, so the names were hardcoded and could drift silently.

Two parsing defects had to be fixed for any of it to work

Neither was visible from reading the code — both turned up from calling the real tool and reading the real response.

1. Every code hit would have been dropped. parseSnippets accepted a block only if it matched /TITLE:/, and code results carry REPOSITORY and PATH with no TITLE. So searchCode would have returned [] on every call while looking like it worked. A retrieval source that silently contributes nothing is worse than one that errors — the answer just quietly has less to stand on.

2. The header regexes were unanchored, so the first title: / source: anywhere in a block won. A code block's body is source code, where title: "Chat" and source: 'user' are everyday object literals. Run against a realistic snippet:

const card = { title: "Chat", source: 'user' };
  →  title     = "Chat", source: 'user' };
  →  sourceUrl = 'user' };

That reached the prompt as [Source 1: "Chat", source: 'user' };] / URL: 'user' };, blobUrl never ran, and the file path the reply was supposed to cite was gone. Code blocks now take their title from PATH and their URL from a GitHub blob URL built off REPOSITORY + PATH — because "a repo link is a real answer", and without a URL a code-sourced answer has nothing to cite and the Phase 1 rules force it into a handoff.

Retrieval

Both tools run in parallel, merged with allSettled, not all — one index being down used to throw away the other source's results and answer from nothing. Code leads the interleave, matching the stated source-first precedence. The merged list is capped at defaultLimit: without it the prompt carried up to 2× the sources it did before, and code snippets are line-numbered file excerpts far larger than doc snippets, so input tokens per ticket roughly doubled — with a real path to a context-length error that lands in the generator's catch and publishes the apology fallback.

Results are also coerced to arrays. allSettled reports a non-promise or undefined return as fulfilled, so a client answering with anything else reached the merge and threw on .length, in the one class whose contract is that it never crashes. Four existing calibration tests caught that.

Both prompts had to move, and that's the subtle half

• GROUNDING_RULES opened with "You have NOT read CopilotKit's source code. Never write or imply otherwise." True while retrieval was docs-only; false the moment this lands. It's the instruction that told the model to disclaim its best evidence, and it's the mechanism behind the Deep Agents answer. Replaced with the narrower honest boundary — retrieved code is fair to cite, un-retrieved files aren't, a repro and a test run remain impossible — plus documentation silence is not evidence a feature is missing and where code and docs disagree, the code is what ships.
• CONFIDENCE_SYSTEM_PROMPT said the assistant "could not read CopilotKit's source" and to score LOW when a response names a file. A correct code-grounded answer was exactly the shape it marked down, and the pipeline takes min(generator, scorer) — so the scorer would have clawed back the win. Now exported, so the two prompts can be pinned against each other by test; a contradiction between them is otherwise invisible at runtime.

Deliberate choices, happy to be overruled

• AG-UI is not in the default path. The methods exist and are tested, but firing them on every CopilotKit question buys noise and spend with no way to tell when they're relevant. Strategy selection from the kind of question asked is the doc's step 5 and needs the classifier.
• main assumed for blob URLs. The tool response carries no ref. A wrong branch 404s rather than pointing somewhere misleading.
• searchDocs not folded into the shared helper. It falls back to a plain-text docs search on error; the code tools have no analogue and return [].

Verification

ai package 270 → 293. Full repo turbo run test 10/10. tsc --noEmit clean.

Mutations checked, each killing exactly the tests that name it: TITLE-only filter · dropped blob URL · concatenate instead of interleave · Promise.all semantics · no cap · the false line restored to the confidence prompt.

One mutation survived, and it's worth stating: anchoring PATH/REPOSITORY specifically is defence in depth, not load-bearing — a header always precedes CONTENT, so no test can distinguish anchored from unanchored there. The load-bearing half is preferring PATH over TITLE.

Two existing tests broke honestly and were rewritten rather than patched around: the GROUNDING_RULES test asserted the now-false "have NOT read" line (and gained a guard so it can't come back), and pipeline-groundedness stubbed only searchDocs.

Follow-ups filed, not fixed here

Three findings from the review that want their own change — see the linked issues. Briefly: the synthetic score: 1.0 on a marginal top code hit inflates retrieval confidence on any question returning code; search() collapsing to [] makes a code-index outage indistinguishable from "no hits"; and generateStreamingResponse still calls searchDocs only, so it will silently regress this fix the moment anyone streams.

Depends on nothing else in flight — branched from main, independent of #239, #240 and #241.

[jerelvelarde]

Reviewed deeply, then put every blocking finding through an adversarial round that tried to refute it. One survived. The retrieval plumbing itself is solid — the allSettled coercion, the cap and the interleave are all real, backed by tests that exercise the real AIPipeline against a stubbed client rather than asserting against their own mock, and every mutation the body names does kill the test it names.

Blocker — the anchoring fix is asymmetric, and opens the mirror-image hole in docs blocks

pathfinder.ts:307-317. /^\s*PATH:\s*(.+)$/im is matched against the entire block, body included. ^\s* allows code indentation and i allows lowercase, so a documentation snippet whose CONTENT contains a line-initial path: yields a truthy path. Then title = path ?? TITLE makes the code fragment beat the real header, and source = path ? blobUrl(repository, path) : SOURCE throws the docs URL away — repository is undefined on a docs block, so blobUrl returns undefined and there is no URL at all.

Run this through the real functions:

SNIPPET 1
TITLE: Self-hosting the CopilotKit Runtime
SOURCE: https://docs.copilotkit.ai/guides/self-hosting
CONTENT:
```ts
const handler = copilotRuntimeNextJSAppRouter({
  path: "/api/copilotkit",
});

→ `title = '"/api/copilotkit",'`, `sourceUrl = undefined`.

This is the same defect you fixed for code blocks — unanchored headers letting a body line win — reintroduced on the other side. And it lands where it hurts most given the rest of this PR: a docs page that loses its URL cannot be cited, and #241's `source-link-or-handoff` rule then collapses that answer into a two-sentence handoff. `copilotRuntimeNextJSAppRouter({ path: ... })` is not an exotic snippet; it is in the self-hosting docs.

The discriminator is already stated in your adjacent comment — a code block has no `TITLE`:

```ts
const titleHeader = block.match(/^\s*TITLE:\s*(.+)$/im)?.[1]?.trim();
const isCode = !titleHeader && !!path;
title = titleHeader ?? path ?? 'Documentation';
source = isCode ? blobUrl(repository, path) : sourceHeader;

Better still, split each block at the first CONTENT: and only ever match headers in the part above it.

Worth settling before merge, your call on each

blobUrl fails silently. pathfinder.ts:18-23 returns undefined with no log when REPOSITORY is missing or is not an https://github.com/ URL. If the server ever emits a bare slug (CopilotKit/CopilotKit), an SSH URL, or drops the header for one index, every code hit reaches the prompt with nothing to cite and leaves no trace. That is the same silent-degradation shape this PR exists to remove, one layer up.

The prompt cannot tell code from docs. generator.ts:236 — GROUNDING_RULES now asserts "Code entries are shown with their file path", but buildSystemPrompt renders both kinds as an identical [Source N: <title> (relevance: X)]. A docs page titled api-reference/components/CopilotKit and a code hit titled packages/react-core/src/index.ts are indistinguishable. So "where code and docs disagree, the code is what ships" is only resolvable by guessing, which is exactly the instruction you most need to be reliable now that the "you have NOT read the source" line is gone.

The cap over-fetches then discards. pipeline.ts:188 — with both tools at defaultLimit: 8 and a .slice(0, 8) on the interleave, a purely docs-answerable question that used to get 8 docs snippets now gets 4, with the other 4 going to code hits that only cleared min_score: 0.3. Every ticket pays for 16 and throws away 8. Lowering each tool's limit instead costs nothing and keeps docs recall intact on the majority path.

search()'s doc comment overclaims. It says keeping the tool name private removes "the silent no-retrieval failure mode this whole change exists to remove". A server-side rename of search-code produces exactly that: JSON-RPC error → one console.error → [] → indistinguishable from "no code matched", for as long as nobody reads the logs. SearchTool is a compile-time union; it cannot know what the server exposes. The tools/list probe you already do would make this checkable at startup.

Not a finding, but the one I'd want a follow-up on

You note generateStreamingResponse still calls searchDocs only. Given the groundedness gate is now the only mechanism that can withhold a response, and assessGroundedness excludes path-shaped tokens and blanks URLs before it looks, this capability ships with no fail-closed backstop on the streaming path. Worth a linked issue rather than a line in the body.

The prompt rewrites are the right call and the reasoning for both is sound — CONFIDENCE_SYSTEM_PROMPT marking down exactly the answers this PR makes possible was a genuinely subtle catch, and exporting it so the two prompts can be pinned against each other by test is the right shape.

[linear-code]

CPK-8077 Phase 2 — Turn on code search and AG-UI search (doc step 4)

CopilotKit/outpost#242 open. Confirmed against tools/list that all four search tools already exist server-side; the client called one. Both code-search tools and both AG-UI tools are now on the client, and the pipeline retrieves docs + code in parallel.

Two parsing defects had to be fixed for any of it to work, neither visible from reading the code:

1. parseSnippets required /TITLE:/ and code results carry REPOSITORY/PATH with no TITLE — every code hit would have been dropped, searchCode returning [] while looking like it worked.
2. The header regexes were unanchored, so title: "Chat" inside a retrieved file won over the real header. A real run-handler.ts parsed to title "Chat", source: 'user' }; and sourceUrl 'user' };, so blobUrl never ran and the citable path was lost.

Both prompts moved. GROUNDING_RULES said "You have NOT read CopilotKit's source code" — true when retrieval was docs-only, and the instruction that told the model to disclaim its best evidence. CONFIDENCE_SYSTEM_PROMPT said the same thing and scored LOW on responses naming a file, and the pipeline takes min(generator, scorer) — so the scorer would have clawed the win straight back. It is now exported so the two can be pinned against each other by test.

ai 270 → 293, repo 10/10, typecheck clean. Six mutations checked; one survived and is called out as defence-in-depth rather than claimed as verified.

AG-UI is deliberately not in the default retrieval path — the methods exist, but firing them on every CopilotKit question buys noise with no routing logic. That is step 5 and needs the classifier.

Three follow-ups filed, not fixed: #243 synthesized positional scores inflate retrieval confidence once code hits exist · #244 a Pathfinder outage is indistinguishable from an empty result set · #245 generateStreamingResponse is still docs-only and will regress this.

Doc step 4. Biggest single quality win, and smaller than it sounds.

Root cause #1 in the doc: for any question whose answer lives in the source — most of the hard ones — the agent is guessing from general React knowledge.

The capability is already live

config.ts:15 defaults pathfinderMcpUrl to https://mcp.copilotkit.ai. That server exposes search-code, explore-code, search-ag-ui-code and search-ag-ui-docs — confirmed against the live tool roster. Our client (pathfinder.ts) calls exactly two of them:

• :291 — search-docs
• :310 — explore-docs (and #124 wants this one deleted as dead code, so it's effectively one docs call)

It also never calls tools/list, so the tool names are hardcoded and drift silently. Worth adding discovery while here.

So this is wiring, not construction.

Ships with the prompt change, not after it

GROUNDING_RULES:21 currently says: "You have NOT read CopilotKit's source code, reproduced the user's problem, or run any test. Never write or imply otherwise." That is accurate today and becomes false the moment code search lands. Both go in the same PR, or the model is instructed to disclaim the evidence it now has.

The doc's decisions that land with this:

• Source code first, docs second. Code is truth; docs are the lagging indicator.
• Docs silence ≠ feature missing. May not say "not supported" unless code was searched and came back empty.
• If code and docs disagree, code wins — and the reply says so plainly: this works, the docs don't cover it yet, we're getting that fixed. No auto-filed ticket; a human decides.
• Repo file link when the answer only lives in code. A repo link is a real answer; hedging is not.

Fixes, measurably

Appendix case A directly. Also the doc's "zero escalations where the answer turned out to be sitting in the code all along" — which Phase 1's harness can score, since case A's answer is one code search away.

Related

Doesn't remove the need for #220 (the Pathfinder corpus audit — empty FAQ, thin troubleshooting, no pricing docs, untouched since 2026-08-17). Code search compensates for a thin docs corpus; it doesn't fix it. That's content work and can run in parallel.

[NathanTarbert]

Thanks Jerel — the blocker was real, and it was my own fix reintroducing the same defect on the other side. Fixed in fa12023, CI green.

The asymmetry

Reproduced it against the real function with your snippet from the self-hosting guide before changing anything:

before → title = '"/api/copilotkit",'   sourceUrl = undefined
after  → title = 'Self-hosting the CopilotKit Runtime'
         sourceUrl = 'https://docs.copilotkit.ai/guides/self-hosting'

Headers are now read only from the region above CONTENT:, which takes out the whole class rather than the two spellings that happened to get noticed. A block is code when it has a PATH and no TITLE, as you suggested. Your point about where it lands is the part that made it a blocker rather than a nit: a docs page with no URL can't be cited, and #241's citation rule then collapses that answer into a handoff.

The other four, all taken

• The prompt couldn't tell code from docs. Good catch, and the sharpest one here — GROUNDING_RULES asserts code entries are shown with their file path and that the code wins a conflict, while buildSystemPrompt rendered both as an identical [Source N: title]. SearchResult.kind now carries it through and sources render as SOURCE CODE / DOCS. An unlabelled source stays unlabelled rather than being asserted as either.
• The cap over-fetched. Each tool now gets half the budget instead of a full defaultLimit each, so a docs-answerable question keeps its docs recall rather than paying for 16 snippets to keep 8.
• blobUrl failed silently. It logs now. A bare slug or a missing REPOSITORY on one index would have left every code hit with nothing to cite and no trace.
• search()'s comment overclaimed. You're right that a compile-time union can't know what the server exposes; a server-side rename still yields one console.error and []. Comment corrected and pointed at A Pathfinder outage is indistinguishable from an empty result set #244, which tracks making that observable.

On mutations, stated plainly

Reverting the header region alone, or the isCode rule alone, kills nothing — three layers independently prevent this now (title reads TITLE before PATH, isCode requires no TITLE, headers come only from above CONTENT:). Reverting the two structural ones together fails two tests. Dropping the prompt label and restoring the full per-tool limit each fail the tests that name them.

ai 293 → 303, repo 10/10, typecheck clean.

Your generateStreamingResponse point is now #245, filed alongside #243 (synthesized scores inflating retrieval confidence) and #244. Worth noting #245 is the third issue describing the same underlying thing — that path is a second copy of the pipeline nobody exercises, so every improvement to the real one skips it. Parity or deletion is probably the actual decision.

[jerelvelarde]

Re-reviewed. The blocker is closed, and closed structurally rather than by patching the two spellings that happened to get noticed — headers are read only from block.slice(0, contentAt), and isCode = !titleHeader && !!path. ai/src is 303 tests.

I ran the exact case from my review through the suite. It is now a fixture (pathfinder.test.ts:252-288), and reverting both structural fixes reproduces the original defect precisely:

AssertionError: expected undefined to be 'https://docs.copilotkit.ai/guides/self-hosting'
AssertionError: expected 'code' to be 'docs'

That is the failure I described — a docs page losing its URL and being misclassified — asserted rather than argued.

Mutations, run:

Mutation	Result
headerRegion → whole block	303 passed (survives)
isCode → !!path	303 passed (survives)
both together	2 failed / 301 passed
code/docs prompt label removed	1 failed / 302 passed
per-tool limit → full defaultLimit	2 failed / 301 passed

Your statement of this was accurate, including the part that does not flatter the change — the two structural layers are redundant, so neither is independently pinned. I would take the follow-up rather than leave it: a test per layer, or an explicit comment that the redundancy is deliberate and untested. As it stands, someone simplifying one layer away in six months gets a green suite and a silently reduced defence.

Approving.

Three notes, none blocking

• The prompt labels are the fix I care most about here. SearchResult.kind reaching buildSystemPrompt as SOURCE CODE / DOCS is what makes "where code and docs disagree, the code is what ships" resolvable instead of guessable, and leaving an unlabelled source unlabelled rather than asserting a default is the right call.
• isCode is correct today and fragile tomorrow. It depends on code results never carrying a TITLE. That holds against the current tools/list, and it is a server-side contract you do not control — if a code hit ever gains a title, source falls back to header('SOURCE'), which is absent, and the file path silently stops being citable. A one-line comment naming the dependency would be enough.
• blobUrl logging closes the trace gap, and pinning the streaming-path question as #245 rather than a line in the PR body is the right disposition. Your observation that #243/#244/#245 all describe the same underlying thing — a second copy of the pipeline nobody exercises — reads as the real finding. Parity or deletion, but not a third improvement that skips it.