Generate llms.txt, llms-full.txt and Markdown twins of every docs page (OPTIP-478)

[devin-ai-integration]

Summary

Makes the built site consumable by AI tools (ChatGPT, Claude, Cursor, Copilot, …), which today only get rendered HTML and sitemap.xml. The build now also emits:

Output	Content
/docs/llms.txt	Two-level root index: one section per product linking to its llms.txt / llms-full.txt (~4.4 KB)
/docs/llms-full.txt	Every page as Markdown (6.8 MB)
/docs/<product>/llms.txt	Page list with title + description, grouped like the sidebars: one section per top-level sidebar category, prefixed with the SDK for per-platform sidebars (## Android SDK: How-to guides, ## Roku SDK: Connectors, …); older versions last (## FAQ (10.14.0))
/docs/<product>/llms-full.txt	All pages of that product (theoplayer 4.4 MB, millicast 1.5, theolive 0.47, ads 0.28, open-video-ui 0.12, ad-engine 0.04)
/docs/<route>.md	Markdown twin of every page, e.g. /docs/theoplayer/how-to-guides/web/ads/google-ima.md

Products: theoplayer, open-video-ui, theolive, millicast, ads, ad-engine. Excluded: Player v4–v9, /contributing, /search, 404. Kept: Player v10, Ads v1, THEOlive v1.

How

Markdown is generated from the rendered HTML (not the MDX source) by @signalwire/docusaurus-plugin-llms-txt, so shared partials, usePlatforms() card lists, generated OpenAPI pages and versioned docs are all handled uniformly. Three local pieces around it:

• src/plugin/llmsTxt.ts wraps the SignalWire plugin and, in the same postBuild, re-groups its output into the per-product files above (Docusaurus runs plugin postBuild hooks in parallel, so a separate plugin could not rely on the .md files existing yet). It fails the build if a top-level route has no products entry configured. Descriptions come from each page's <meta name="description">; heading/markup fragments, title repeats and empty code spans are dropped, long ones are cut at ~200 chars.

• src/plugin/llmsTxtSidebars.ts maps every docs route to its version and its positions in the loaded sidebars (props.plugins[*].content.loadedVersions[*].sidebars). The product index uses this instead of URL paths:

  // docusaurus.config.ts
  { slug: 'theoplayer', …, platformSidebars: { web: 'Web SDK', android: 'Android SDK', ios: 'iOS & tvOS SDK', … } }

  page in exactly one platform sidebar  → "## <SDK>: <top-level category>"   e.g. "## Web SDK: Connectors"
  page in several sidebars              → "## <top-level category>"          e.g. "## FAQ", "## Knowledge base"
  page in no sidebar                    → "## Overview"                      (landing page first)
  page of an older version              → same, with " (<version label>)" suffix, after all current sections
  

  Within a section, pages keep their sidebar order.

• src/plugin/llmsTxtMarkdown.ts cleans up Docusaurus HTML before conversion: Prism token spans → fenced code with language, admonitions → blockquote with bold title, tabs (incl. OpenAPI response tabs) → label + panel in order, strips # hash-links and comments; and fixes the plugin's /docs/docs/ duplicated base URL and /route/.md link shape.

Two pages (web/hosted-player, getting-started/sdks/web/how-can-we-embed-iframe, current + v10) get an explicit description front matter field, because the Docusaurus auto-excerpt turned `<iframe>` into an empty code span in their <meta name="description">.

Preview builds use the preview origin

All links in llms.txt / .md are absolute (siteConfig.url + baseUrl). Preview builds only changed baseUrl to /pr-<n>/, so every absolute URL the build emits (canonical, og:url, and now the llms.txt links) pointed at optiview.dolby.com/pr-<n>/…, which does not exist. The pull-request workflow now passes the preview CloudFront origin to the build:

# pull-request.yml → _build.yml
docusaurus_url: ${{ vars.PREVIEW_CLOUDFRONT_URL }}   # new optional input → DOCUSAURUS_URL

// docusaurus.config.ts
url: process.env.DOCUSAURUS_URL || 'https://optiview.dolby.com/',

main.yml does not set it, so production output is unchanged (https://optiview.dolby.com/docs/…).

Verification

• npm run build (prod): 1,714 pages; all 1,733 links in the root and product llms.txt files resolve to a generated .md; no v4–v9 / contributing / search / pr-N / announcement-bar text in any output; build time unchanged (3m53–4m27 vs 4m24 baseline).
• /theoplayer/llms.txt: landing page is the first Overview entry; connectors and how-to guides are split per SDK (Web 15/136, Android 11/92, iOS 15/87, React Native 29/6, Roku 9/18, Flutter 0/13); the Roku API reference and React Native changelog sit in their SDK's Overview section; v10 sections come last. /open-video-ui/llms.txt is split into Web / Android / React / React Native the same way.
• Navigation test with an LLM that only had the index: six representative questions (Google IMA on Android, install the Roku SDK, Conviva connector for React Native, supported DRM systems, Chromecast getting started, OptiView Live token security on Web) were each resolved to the right .md page by picking the ## section and then the entry.
• DOCUSAURUS_URL=https://docs-preview.optiview.dolby.com DOCUSAURUS_BASE_URL=/pr-838/ DOCUSAURUS_PR_NUMBER=838 npm run build: all generated absolute links use docs-preview.optiview.dolby.com/pr-838/, canonical too; preview banner absent from .md.
• node serve.js: llms*.txt → text/plain, .md → text/markdown, HTML untouched. Deploy is the existing aws s3 sync (content type inferred from extension) — no deploy changes.
• npm run lint, npm run typecheck, npm run check-format pass; actionlint reports nothing new on the touched workflow lines.

Known limitation

OpenAPI reference pages are client-rendered, so their .md contains only the summary and response codes (parameters/schemas are not in the static HTML). Follow-up tickets are raised.

Link to Devin session: https://dolby.devinenterprise.com/sessions/0a1fa7b689f04b478e15356af0b76990
Open in Devin Desktop: https://dolby.devinenterprise.com/desktop/session/0a1fa7b689f04b478e15356af0b76990?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[MattiasBuelens]

/theoplayer/llms.txt has a lot of issues:

• The first link in "Overview" goes to a v10 page?

  • THEOplayer

• There are top-level sections for API references and changelog... but they point to only the Roku API and the React Native changelog.

  api-reference

  • Player API: Discover all properties and functions of THEOplayer.

  changelog

  • Changelog: Find out what's new in the OptiView Player.

• The connectors and how-to-guides sections list all connectors and guides of all SDKs. It would make more sense to separate them by SDK.
• There's a very tiny web section? Also with a weird empty code span.

  web

  • OptiView Hosted Player: Dolby hosts a special version of the OptiView video player for use with customers intended to be used for `` embedding.

Also, how have you tested this? Do LLMs actually manage to find their way around these files?

[devin-ai-integration]

Thanks for the review — all points from #838 (review) are addressed in a892109.

What changed. The product index no longer groups pages by the first URL path segment. It now reads the loaded Docusaurus sidebars (src/plugin/llmsTxtSidebars.ts) and groups pages the same way the site navigation does: one section per top-level sidebar category, prefixed with the SDK name when the page is in exactly one per-platform sidebar (platformSidebars in the product config). Pages shared between sidebars (FAQ, Knowledge base) get a plain section. Older versions (v10, Ads/THEOlive v1) come last with a version suffix.

/theoplayer/llms.txt now looks like this (1,031 entries, same set of pages as before):

## Overview                      ← landing page first, then Changelog and pages that are in no sidebar
## Web SDK: Overview
## Web SDK: Getting started
## Web SDK: How-to guides        (136)
## Web SDK: Examples
## Web SDK: Connectors           (15)
## Android SDK: Overview / Getting started / How-to guides (92) / Connectors (11)
## iOS & tvOS SDK: …
## React Native SDK: Overview    ← contains its Changelog, no separate "changelog" section anymore
## React Native SDK: Getting started / How-to guides / Connectors (29)
## Flutter SDK: …
## Chromecast SDK: …
## Roku SDK: Overview            ← contains "Player API", no separate "api-reference" section anymore
## Roku SDK: Getting started / How-to guides / Connectors
## FAQ
## How-to guides                 ← shared: index page + one guide linked from every SDK
## Knowledge base
## Overview (10.14.0) … Knowledge base (10.14.0)

• v10 landing page: moved to ## Overview (10.14.0), after all current-version sections.
• api-reference / changelog single-entry sections: the Roku API page and React Native changelog are now listed in their SDK's Overview section, like in the sidebar.
• Connectors and how-to guides: split per SDK (see above). The other SDKs' API references and changelogs are external links / client-rendered pages, so they are not in the index.
• Tiny web section: the hosted player page is in no sidebar, so it is now listed under Overview. The empty code span came from the Docusaurus auto-excerpt, which strips <iframe> from `<iframe>`. Fixed by adding a description to the two affected pages (this also fixes their <meta name="description"> on the site) and by dropping empty code spans in cleanDescription so it cannot happen again.
• Open Video UI gets the same treatment (Web: …, Android: …, React: …, React Native: …). The other products have a single sidebar, so their sections are simply the sidebar categories.

How it was tested.

• Production build; all 1,733 links in llms.txt and the six product llms.txt files resolve to a generated .md file; no v4–v9, /contributing or /search pages in any output.
• Navigation test with an LLM (myself, Devin) that only had the index, not the site: for six questions — "set up Google IMA ads on Android", "install the Roku SDK", "Conviva connector for React Native", "which DRM systems are supported", "getting started on Chromecast", "OptiView Live token security on Web" — pick the section from the ## headings, then find the page in that section. All six were found in one step (Android SDK: How-to guides → google-ima.md, Roku SDK: Getting started, React Native SDK: Connectors → conviva/getting-started.md, Knowledge base → drm-systems.md, Chromecast SDK: Getting started, Web SDK: How-to guides → theolive/token-based-security.md). With the old path-based grouping, the same lookups meant scanning 341 mixed how-to guides or 82 mixed connectors.
• Size: root llms.txt is 4.4 KB, /theoplayer/llms.txt is 215 KB (~55k tokens), so a tool can load the root index, then the product index, then only the .md pages it needs. The per-SDK sections also let a tool read just one section instead of the whole file.

I have not yet run this against an external agent (ChatGPT / Claude / Cursor) with the deployed preview URL; happy to do that once the preview for this commit is up if you want more evidence.

[Danesz]

Regarding testing:
This is a proposed standard to help agents navigate the websites when searching, but also when crawling. (so they don't need to parse HTML), but there is no "official" statement around it.

Companies (AI companies too) publish their doc already in this format, e.g.
Claude, Zapier, Stripe, Coinbase, Gemini
And IDEs (like Cursor, Cline, Windsurf) do find it, there are also skills that can use them.