feat(ai): lint a draft against the reply rules before it publishes

[NathanTarbert]

A draft is judged against the reply rules before it publishes, and in report mode the verdict is recorded without changing what goes out. Phase 3 of the response-quality work (CPK-8078), first slice.

Based on feat/response-quality-eval-harness (#241), not main — it consumes that rule module, so the diff here is only this work. Rebases onto whatever that review lands.

The doc's flowchart hangs on one arrow: "if the draft breaks a rule, it doesn't get cleaned up and posted — it collapses into the two-sentence version." This is that arrow, running the same rules the harness scores with, from the same module, so the thing measured and the thing enforced cannot drift.

The two prerequisites had to come first

Both would have made the linter withhold correct answers — the same direction as the groundedness gate suppressing one, which is what #234 was filed for.

• Laundering. The citation rule tested whether a link looked like ours, so a reply could put its invented hook name inside a docs.copilotkit.ai URL and satisfy the rule with a page that does not exist. assessGroundedness blanks URLs before it looks, so the identifier rule cannot catch it either. Now matched against the URLs actually retrieved.
• The plain-text fallback. textSearch sets sourceUrl: undefined on every result, so a correct answer built from it has nothing it could cite. Under a flat requirement it fails forever and, once rules gate publishing, collapses every time the fallback is in play.

Closing the second needed a third state, so RuleResult gained applicable: "did not cite" and "had nothing citable" are different facts. That also fixed a rule inapplicable everywhere reading as a clean sweep, since passed === total is trivially true at 0/0 — formatReport now prints n/a.

Two deliberate choices

Report mode is the default. lintDraft computes the verdict and changes nothing unless asked to enforce, carrying wouldCollapse as the counterfactual. Enforcing means a misfiring rule withholds a correct answer from a real person, so the order is: report mode → read what it would have collapsed against real traffic → enforce once the false-positive rate is measured rather than assumed.

It returns a verdict, never replacement copy. The caller substitutes its own — in the pipeline, the existing SUPPRESSED_RESPONSE_TEXT. That copy already promises a human, and #231 records what happens when two layers each add their own.

Not wired into pipeline.ts here. #242 is editing that file on another branch, and a report-mode linter changes nothing until someone reads its output, so wiring follows once #242 lands.

Review round, and the judgment call worth a second opinion

A review pass turned up seven items; all seven held under verification against the real functions. Two are worth calling out.

A hole that would have shipped. applicable treated "retrieval returned nothing" the same as "retrieval returned results with no URL". With sources: [] the citation rule went not-applicable and could not fail — so a long, uncited reply built on zero retrieval published under enforcement. That is Case A exactly, the one input where citing matters most, and the case-A fixture only avoided exposing it by sitting under the handoff cap at 48 words. Empty retrieval now still fails; the escape is scoped to results that carry no URL.

The metric and the gate are not the same thing, and that needed a decision. The linter failed grounded-identifiers at one unsourced identifier while groundedness.ts suppresses at SUPPRESS_AT_UNSOURCED_IDENTIFIERS = 2 — its own comment reasoning that "one could be a formatting artifact; two is a pattern of fabrication". So the linter was stricter than the pipeline beside it.

Matching production would have dropped the doc's actual criterion, which is zero invented names. So RuleResult now carries blocksPublish alongside passed: the harness scores the metric, one occurrence fails it and appears in the report; the linter gates on the pipeline's threshold, so one does not withhold the answer. Every other rule has the two agree.

That is the call I'd most like checked — it is a design decision rather than a fix, and the alternative (one threshold, whichever way) is defensible too.

Also from that round: the laundering had only moved a level deeper (includes accepted anything appended to a retrieved URL, and retrieval returns section URLs, so …/reference/hooks/useCopilotFabricated satisfied …/reference); the citation compare was case-sensitive on the host, which withholds a correctly-cited answer; the dead-package carve-out was a no-op on Case B's own failure, since naming a live package excused the dead one and naming both as current is version-mixing; and the index.ts comment claimed a dist guarantee that does not hold — tsc emits per file, so dist/eval/harness.js ships the fixtures regardless.

Verification

ai package 318 → 345. Full repo turbo run test 10/10. tsc --noEmit clean.

Mutations, each killing the tests that name it: laundering-permissive citation check · always-applicable citation rule · report mode withholding · enforce counting inapplicable rules · zero-retrieval treated as not-applicable · linter gating on passed · bare includes · dead-package excused by a live package alone · case-sensitive compare.

One mutation initially survived — nothing pinned formatReport's failing-cases output, so an unfiltered block printed n/a for a rule and then listed it as a failure. That test was added and the mutation now dies. Recording it because a surviving mutation is the only honest signal that a fix wasn't actually tested.

Two tests were updated rather than patched around: both cited arbitrary docs-shaped URLs absent from their fixture's sources, which is the laundering the new check closes.

[linear-code]

CPK-8078 Phase 3 — Let it say less: prompt rewrite plus a rule checker on every draft (doc steps 2, 3)

Doc steps 2 and 3, plus the brevity rules. Root cause #2: a short honest answer is currently impossible to produce.

The prompt as it stands

generator.ts:29-42, unchanged on main:

- Always include code examples when relevant (TypeScript/React preferred)
- Structure responses with **bold headers**, bullet points, and code blocks
- End with a relevant follow-up suggestion or "Was this helpful?"

Formatting is mandatory; having something to say is not. That is the whole of root cause #2. Add verdict-first, one approach not three, a real two-sentence route, and a word cap on handoffs (the doc suggests 60).

The linter has to land with or before the loosening

"A short honest reply is always the fallback" is only true if something enforces it. Nothing in the repo does today — no banned-phrase, hedged-name, or version-mixing check anywhere. The doc's flowchart hangs on one arrow: a draft that breaks a rule collapses into the two-sentence version rather than being cleaned up and posted.

Checks, all mechanical:

• Banned phrases — "Great question", "What I can't do from here", issue-writing advice, self-commentary about the agent's own limits, claims it can't see the thread.
• Hedged names — "or the equivalent hook" means it's guessing; drop the name.
• Version mixing — v1 and v2 hooks never in one snippet. Never mention @copilotkitnext at all; it merged into v2.
• Every named identifier appears in retrieved context, or it doesn't get written.
• A v1 question gets a real v1 answer plus "v2 is the path forward" — not a redirect, not a different question answered.

Fixes appendix cases B and D.

Step 2 — stop replying to our own team

Nothing in the AI path maps maintainer accounts. teamMember lookups exist only in discord-bot/src/commands/assign.ts:49 and lib/tickets.ts:44. The staff filter the doc mentions can't recognise maintainers whose Discord accounts were never linked to their team records — which is how appendix case C happened (replied to Nathan, complimented his community spirit, claimed it couldn't read the thread).

Needs maintainer account mapping plus a hard allowlist, and the reply gate: once a maintainer posts in a thread, the agent is out unless -mentioned.

Overlaps #174 — replace the one-response counter with a human-presence gate. Probably the same change; worth checking before building both.