Sanitize and distill inbound bodies before docs search; stop double-processing Discord forum posts

[jpr5]

Two bugs on the inbound path: raw message bodies were forwarded verbatim as docs-search queries, and one Discord forum post was ingested twice.

Both are invisible today because SHADOW_MODE gates only the platform post-back (ai-response.ts:178) — retrieval and Claude generation run fully either way.

---

Bug 1 — raw inbound bodies reached the embedder

Chain: thread-create.ts → inbound.ts:180 (truncate(content, 8000)) → ai-response.ts:87 → pipeline.ts:72 → pathfinder.ts:291. Nothing sanitized or narrowed the text; the only sanitizer in the codebase (formatter.ts:149) runs on the OUTBOUND reply.

Three defects, all addressed:

• No platform-markup stripping. sanitizePlatformMarkup (shared/src/text.ts) removes Discord mentions / custom emoji / timestamps, Slack link and mention syntax, HTML comments, issue-template checklists and boilerplate headings, and pasted channel-sidebar rows. Deterministic, and it leaves prose and fenced code alone. It lives in shared so every platform gets it.
• No distillation. SearchQueryBuilder (ai/src/query.ts) distills the sanitized body into a focused query with Haiku, mirroring TicketClassifier's shape. Short bodies (≤200 chars) short-circuit with no LLM call; a failed distillation degrades to a heuristic that prefers the sentences carrying the question mark rather than dropping the search.
• Over-broad trigger. An empty MONITORED_CHANNEL_IDS now fails closed instead of monitoring every channel, and isSupportRequest gates ingestion on a question mark, help phrasing, an error signature, or a direct @-mention of the bot — considering the thread title as well as the body, since forum posts often put the question in the title.

Retrieval gets the distilled query; generation and confidence scoring get the sanitized body, because distillation is lossy on purpose and the answer still needs the reporter's full context and code.

apps/web/src/app/api/qa/route.ts:62 needed no change — it calls the same AIPipeline.generateSupportResponse, so it inherits the fix.

Red-green — captured at the real Pathfinder wire boundary

Driven through the real AIPipeline + real PathfinderClient against a local HTTP server speaking real MCP Streamable-HTTP (nothing touches mcp.copilotkit.ai); Anthropic calls go through aimock. Input is a 727-char Discord post containing <#1205139168783503400>, <@!284920412034990081>, <:copilotkit:1187213988392189962>, a pasted channel sidebar, and a GitHub issue template.

RED — before the change, the entire body crosses the wire:

=== OUTBOUND search-docs CALLS: 1 ===
--- session=stub-session-1 ---
query (727 chars):
"<:copilotkit:1187213988392189962> hey <@!284920412034990081> :wave:\n\nI posted this in <#1205139168783503400> already but reposting here since <#1379082175625953370> said this is the right place.\n\nChannels\n# ┃welcome\n# ┃announcements\n# ┃general\n# ┃support\n# ┃showcase\n# ┃ag-ui-general\nVoice Channels\n🔊 Lounge\n🔊 Office Hours\n\n## Pre-flight Checklist\n- [x] I have searched existing issues\n- [x] I have read the contributing guidelines\n- [ ] I am willing to submit a PR\n\n### ♻️ Reproduction Steps\n1. npx create-next-app\n2. install @copilotkit/react-core\n\nMy actual question: how do I render a custom React component from a tool call with useCopilotAction? Generative UI never renders, the tool just returns text. <t:1738000000:R>"
limit=8 min_score=0.3
=== END ===

GREEN — same repro, unchanged, on the fixed code:

=== A. distiller available | OUTBOUND search-docs CALLS: 1 ===
session=stub-session-1
query (81 chars):
"render a custom React component from a useCopilotAction tool call (generative UI)"
limit=8 min_score=0.3
=== END ===

=== B. degraded -> heuristic (distiller LLM 503) | OUTBOUND search-docs CALLS: 1 ===
session=stub-session-2
query (100 chars):
"My actual question: how do I render a custom React component from a tool call with useCopilotAction?"
limit=8 min_score=0.3
=== END ===

=== C. sanitizePlatformMarkup (what generation now sees) ===
"hey\n\nI posted this in already but reposting here since said this is the right place.\n\nChannels\nVoice Channels\n\nReproduction Steps\n1. npx create-next-app\n2. install @copilotkit/react-core\n\nMy actual question: how do I render a custom React component from a tool call with useCopilotAction? Generative UI never renders, the tool just returns text."
=== END ===

727 chars of markup → an 81-char query, and still 100 focused chars when the distiller LLM is down.

---

Bug 2 — root cause: two Discord events, one message

Not multiple replicas racing the queue, and not the retry path. Events.ThreadCreate and Events.MessageCreate both fire for a new forum post, and both are wired up (apps/discord-bot/src/index.ts:25,27):

1. handleThreadCreate ingests the starter message and enqueues AI_RESPONSE.
2. handleMessageCreate then fires for that same starter message, findTicketByThreadId finds the ticket that was just created, and InboundHandler.handleReply enqueues a second AI_RESPONSE for the same ticket.

Two jobs → two workers → two Pathfinder sessions → two identical query_log rows ~0.2s apart from two egress IPs. The shadow path duplicates identically (handleShadowThreadCreate then handleShadowMessage). createJob has no idempotency key, so nothing collapses them.

The queue's claim is not at fault — worker.ts uses UPDATE ... WHERE id IN (SELECT ... FOR UPDATE SKIP LOCKED), which is correct for concurrent replicas. The duplication is at enqueue.

Fix: a thread's starter message shares the thread's own ID, so handleMessageCreate skips it. Genuine replies carry their own message ID and are unaffected.

Red-green — real event handlers, real outbound calls

One forum post driven through the unmodified handleThreadCreate + handleMessageCreate, then the real PathfinderClient once per enqueued job against the MCP stub.

RED:

=== AI_RESPONSE JOBS ENQUEUED: 2 ===
AI_RESPONSE {"ticketId":"ticket-internal-id","threadId":"thread-999","source":"discord"}
AI_RESPONSE {"ticketId":"ticket-internal-id","threadId":"thread-999","source":"discord"}
=== OUTBOUND search-docs CALLS: 2 ===
  +0ms session=stub-session-1 query="How do I render a custom React component from a tool call with useCopilotAction?"
  +2ms session=stub-session-2 query="How do I render a custom React component from a tool call with useCopilotAction?"
=== END ===

GREEN — same repro, unchanged:

=== AI_RESPONSE JOBS ENQUEUED: 1 ===
AI_RESPONSE {"ticketId":"ticket-internal-id","threadId":"thread-999","source":"discord"}
=== OUTBOUND search-docs CALLS: 1 ===
  +0ms session=stub-session-1 query="How do I render a custom React component from a tool call with useCopilotAction?"
=== END ===

---

Tests

35 new regression tests, following the repo's conventions (aimock for every LLM call, no hand-stubbed responses):

• shared/src/__tests__/text.test.ts — 14 tests over the sanitizer and the support-request gate.
• ai/src/query.test.ts — 14 tests over the distiller and its heuristic fallback, incl. "never returns the raw body" and "sends the sanitized body, not the raw one".
• ai/src/pipeline.test.ts — searches with the distilled query, generates from the sanitized body, counts distiller tokens.
• discord-bot — starter-message dedupe (normal + shadow), genuine replies still processed, fail-closed channels, announcement rejection, title-only questions, shadow-mode gating.

Local gate, all green from the repo root: prettier (new files), eslint on every changed file, turbo run typecheck 10/10, turbo run test (825 in @copilotkit/outpost, up from 794; 64 in discord-bot, up from 56; 501 web; all others unchanged), turbo run build 10/10.

Deliberately not changed

• No idempotency key / dedupe column on Job. It would need a Prisma migration against the production DB, and it is not the root cause — the queue's claim is already correct and the duplicate came from a duplicate event path. Worth doing as defence in depth, but as its own change.
• Shadow-mode behaviour. ai-response.ts:178 still gates only the platform post-back, so retrieval and generation run in shadow mode. Flagging rather than touching it: it is why both bugs were invisible, but changing it is a separate decision.
• Repo-wide formatting. main is not prettier-clean (several bot files and tests fail --check at HEAD), so only the new files were formatted; the edited files' added lines are prettier-conformant and the untouched lines were left as-is to avoid diff noise.

[linear-code]

CPK-7931 4. PR #144 — rebase + land query hygiene (conflicting; feeds answer quality)

CopilotKit/outpost#144 — Sanitize and distill inbound bodies before docs search; stop double-processing Discord forum posts. Draft, +969/-18 across 14 files. CONFLICTING / DIRTY — needs a rebase before anything else.

Filed under response quality because it is the retrieval half of answer quality: it touches packages/outpost/ai/src/query.ts, pipeline.ts, config.ts, and their tests. A bad query into Pathfinder produces a confidently wrong answer no downstream gate can rescue — the groundedness work catches fabrication, this reduces the odds of needing to.

Two things to settle before rebasing

It overlaps #169. The inbound-dedupe half addresses the same duplicate-ticket class as #169 (redelivered events create duplicate tickets, so a conversation can get more than one AI answer). Decide whether #144 closes #169, partially addresses it, or should drop that half now that #191 arbitrates one-answer-per-ticket at the database. Doing this after #191 merges makes the answer much clearer.

It shares a file with #191 and one with #150. Rebase after #191 lands, not before.

Currently a draft, so it may also need a scope decision about whether it ships as one PR or splits the way #187 did.

[jerelvelarde]

Merged main in (5947c3b) and fixed one piece of fallout (de4beec). Merge rather than rebase, so your commits keep their hashes.

Three textual conflicts, all pure unions — ai/src/index.ts and ai/src/pipeline.ts (main widened the formatter.js exports; your query.js exports sit alongside), and the thread-create.test.ts imports (your shadow-mode mocks plus main's PlatformDiscordAdapter). Verified all three symbols are still used.

One semantic conflict with no textual conflict at all, which is the part worth knowing about. message-create.test.ts merged cleanly and then failed:

× still processes genuine replies in the same thread
  expected "vi.fn()" to be called with [ 'AI_RESPONSE', … ]
  Number of calls: 0

That test asserted a reply enqueues AI_RESPONSE. Since #172 and #191, InboundHandler never enqueues for a reply on any platform — its own docblock now says so: "Never enqueues AI_RESPONSE, whoever sent the reply — Outpost answers once per ticket, on the opening message only." The assertion was written before that rule existed, so nothing flagged it; the merge just made it false.

What the test is actually there to prove is that message.id === threadId tells the starter message apart from a reply. So it now asserts the reply still gets its Message record, plus an explicit expect(createJob).not.toHaveBeenCalled() — which pins the one-answer rule on this path too, rather than leaving a hole where it used to be asserted backwards.

Your dedup gate is untouched and still load-bearing. Neutering if (message.id === threadId) return; fails both starter-message tests. I checked that specifically, since the whole point of the change is that it can't be quietly weakened.

Verified: packages/outpost 64 files / 1106 tests, apps/discord-bot 8 files / 74 tests, apps/worker green. apps/web has 2 failing suites (@copilotkit/outpost/queue unresolved in postmark-webhook.test.ts and sync-api.test.ts) — I confirmed those fail identically on origin/main, so they're an unbuilt-workspace artifact, not this branch.

Two things I did not do:

1. Prettier. The file was already non-conformant before my edit, so --write would have reformatted your whole test file into the diff. CI runs neither lint nor prettier today (that's what docs: correct CI lint claims and apply review follow-ups from #135/#136 #159 corrected), so I left it alone rather than bury the change.
2. Split the PR. CPK-7931 suggested splitting the query-hygiene half from the Discord double-processing half, since only the second was queued behind feat(queue): one AI answer per ticket, arbitrated by the database #191. Now that feat(queue): one AI answer per ticket, arbitrated by the database #191 has landed there's nothing to gain from splitting, so this stays one PR.

Ready for review. I haven't approved it — worth someone else confirming the test-intent change above reads right to them, since I changed an assertion rather than making it pass.