[Chore]: disallow /llms-full.txt in robots.ts

[martyy-code]

Task Description

The docs site exposes AI-agent-friendly content via the llms-full.txt route at the URL prefix served by apps/web/src/app/llms-full.txt/. The current apps/web/src/app/robots.ts disallows /llms.txt and /llms.mdx/ but does not disallow /llms-full.txt (or its trailing-slash variant). Crawlers can index this route as duplicate content, diluting canonical signals on the human-facing docs.

Update apps/web/src/app/robots.ts so the disallow array covers every LLM surface the app ships:

disallow: ['/api/', '/llms.txt', '/llms.mdx/', '/llms-full.txt', '/llms-full.txt/'],

Motivation / Why

• AI/LLM content routes should never be indexed; they exist to serve agents and Markdown-preferred clients only.
• A single undislallowed route can be indexed, returned for "$$term$$" queries, and produce a duplicate-content canonicalization mess against the human-readable /docs/* pages.
• Two-line fix; aligns robots.ts with the routing table established in apps/web/src/app/proxy.ts.

In Scope

• apps/web/src/app/robots.ts — extend disallow to cover /llms-full.txt and /llms-full.txt/.

Out of Scope

• Changing the proxy / Markdown rewrite behaviour in apps/web/src/app/proxy.ts.
• Adding X-Robots-Tag HTTP headers (would be a separate consistency pass).
• Touching the human-facing /docs/* or (home)/, (home)/blog/ routes.

User-Facing Impact

• No user-facing impact - internal/tooling/maintenance only

Component(s) Affected

• Multiple Components (specifically apps/web/src/app/robots.ts; verification also touches apps/web/src/lib/shared.ts constants if docsContentRoute is referenced)

Priority

• p3: Low - Nice to have, can wait

Estimated Effort

• effort: xs - Few minutes

Related Issues / Pull Requests

• Related audit: see the in-repo audit summary at docs/internal/tasks/v1.4.x-seo-cleanup.md (this issue is part of that cleanup batch).

Relevant Documentation

• apps/web/src/app/proxy.ts — current rewrite table for /llms.mdx/* content routes.

Technical Approach

Implementation approach:

• Edit the disallow array literal in apps/web/src/app/robots.ts. No new imports or helpers needed.
• Verify locally with pnpm --filter web dev and curl http://localhost:3000/robots.txt to confirm both paths are listed under Disallow:.

Files that may need changes:

• apps/web/src/app/robots.ts

Constraints / considerations:

• None of the entries here affect production crawling today (Vercel doesn't serve /llms-full.txt as a dynamic route — it's a static asset under src/app/). The audit is preventative rather than fixing a live crawl issue.

Risk Assessment

Risk: accidentally disallowing a route that does need indexing.
Mitigation: compare each disallow entry against the actual file tree under apps/web/src/app/ before merging.
Rollback plan: revert the disallow array to its previous contents.

Pre-Submission Checklist

• Task description clearly defines the end state
• Scope (in/out) is clearly defined
• Component(s) affected are specified
• I have searched for existing issues covering this work
• I understand this issue will be labeled according to the project taxonomy
• Note: type: chore is a project-specific label for general maintenance tasks not covered by the standard type labels (bug, feature, refactor, docs, security)

[mikemikimike]

Implemented and opened PR #90 against staging to disallow both /llms-full.txt and /llms-full.txt/ in robots.txt. Local git diff --check passed. The repository Build, CI, Lint, Tests, and Type Check workflows were triggered for the fork head but are action_required pending maintainer approval; no code failure was reported. Package installation on Windows was attempted twice but hit registry slowness and pnpm EPERM copy errors, so formatter, lint, type-check, and build were not runnable locally.