feat: add llms markdown in doc pages#891
Conversation
4d75c03 to
5f883db
Compare
…Markdown button Signed-off-by: Shreya Sharma <shreyasharma.1510001@gmail.com>
Signed-off-by: Shreya Sharma <shreyasharma.1510001@gmail.com>
9a729fd to
7ed125a
Compare
There was a problem hiding this comment.
Pull request overview
Adds first-class “LLM-friendly” outputs for the docs site by generating per-page .md twins plus llms.txt / llms-full.txt, and exposes the per-page Markdown via UI actions under each doc title.
Changes:
- Add and configure
docusaurus-plugin-llmsto generatellms.txt,llms-full.txt, and per-page Markdown outputs at build time. - Add a
MarkdownPageActionsUI control to “Copy as Markdown” / “View as Markdown” for eligible doc pages. - Wire the new control into the DocItem theme and update lockfile for new dependencies.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| yarn.lock | Locks new transitive dependencies required by docusaurus-plugin-llms. |
| package.json | Adds docusaurus-plugin-llms dependency. |
| docusaurus.config.js | Configures the plugin to generate llms*.txt and Markdown per doc page. |
| src/theme/DocItem/index.js | Computes .md twin URL and renders Markdown actions under the doc title. |
| src/components/MarkdownPageActions.js | Implements “Copy as Markdown” / “View as Markdown” UI and clipboard logic. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…leak Signed-off-by: Shreya Sharma <shreyasharma.1510001@gmail.com>
…tripping Signed-off-by: Shreya Sharma <shreyasharma.1510001@gmail.com>
manasmanohar
left a comment
There was a problem hiding this comment.
Please address the comments below. Specifics left inline; this one isn't tied to a diff line:
Collides with the llms.txt we already ship.
We already have curated static/llms.txt and static/llms-full.txt (content team's — About Keploy, awards, positioning, curated links). The plugin generates the same two filenames, and its postBuild runs after static/ is copied, so it overwrites them: build/llms.txt is the generated version and the curated content is gone. Net effect — the content team's file is silently replaced, and editing static/llms.txt later would have no effect. The static file also drifts today since it's hand-maintained.
Option: merge both — feed the curated text into the plugin via rootContent/fullRootContent (+ title/description) so it stays on top and the auto-generated index below stays fresh. Needs the curated source moved out of static/ (so it isn't overwritten) and the static copies removed. Worth confirming the exact split with the content team.
| // and only for real doc pages, not root/category-index pages. | ||
| const showMarkdownActions = isLatestVersion && !suppressArticleSchema; | ||
| const markdownUrl = showMarkdownActions | ||
| ? `${permalink.replace(/\/+$/, "")}.md` |
There was a problem hiding this comment.
Copy/View as Markdown 404s on the 5 section landing pages (concepts, dependencies, operation, server, server/sdk-installation). For an index page the permalink is /docs/concepts/, so this produces /docs/concepts.md — but the plugin writes the file to /docs/concepts/index.md, so the link 404s and Copy shows "Copy failed". Normal pages are fine.
Options: special-case folder permalinks (trailing slash → append index.md instead of .md), or derive the .md URL from the plugin's output instead of reconstructing it from the permalink.
| + // Prop values can legitimately contain '<' or '>' inside quoted strings (e.g. | ||
| + // tools={["Linux kernel >= 5.10"]}), which would otherwise be mistaken for the |
There was a problem hiding this comment.
These two regexes run over the whole document including fenced code, so real code gets stripped. e.g. installation/windows.md: wsl --install -d <Distribution Name> → wsl --install -d (~5 files); quickstart/java-spring-boot-xml.md: the <User>...</User> wrapper is removed and the XML sample ends up malformed. This is new — those tags were preserved before (the plugin only stripped a fixed HTML allowlist).
Option: skip fenced code blocks when stripping (mask fences → strip prose → restore).
| "@tailwindcss/typography": "^0.5.0", | ||
| "autoprefixer": "^10.4.0", | ||
| "docusaurus": "^1.14.7", | ||
| "docusaurus-plugin-llms": "^0.4.0", |
There was a problem hiding this comment.
Plugin is ^0.4.0 but the patch is pinned to 0.4.0 — patch-package will fail the install if a 0.4.1 publishes. Consider pinning the exact version.
| [ | ||
| "docusaurus-plugin-llms", | ||
| { | ||
| docsDir: "versioned_docs/version-4.0.0", |
There was a problem hiding this comment.
docsDir is hardcoded to versioned_docs/version-4.0.0 — will need updating when we cut the next docs version, since the buttons show on whatever the latest version is (gated on isLatestVersion).
| ? "Copied" | ||
| : copyState === "error" | ||
| ? "Copy failed" | ||
| : "Copy as Markdown"; |
There was a problem hiding this comment.
add fixed width to prevent the div from shrinking when changing text
What has changed?
Every docs page is now also available as plain Markdown, and
llms.txt/llms-full.txtare auto-generated at build time — so AI agents (and humans) can grab clean per-page content instead of parsing HTML.docusaurus-plugin-llms, configured to generatellms.txt,llms-full.txt, and a.mdtwin for every doc page (same URL,.mdextension)..mdfile is actually discoverable, not just guessable. Styled with Tailwind utility classes to match this repo's existing convention.docusaurus-plugin-llms(viapatch-package, auto-reapplied on install) to fix two real content bugs found during review:import WhatIsKeploy from '../concepts/what-is-keploy.md'); the plugin only resolved these when the filename had an underscore, so ~7 pages were leaking that raw import/JSX line into their.mdoutput instead of the real text.<Tabs>,<ProductTier>,<HowTo>, etc.) into their.mdoutput. Fixed by stripping self-closing component tags entirely and unwrapping paired ones while preserving their real inner content — carefully excluding<YOUR_INGRESS_HOST>-style fill-in-your-own-value placeholders, which look similar but aren't components.setTimeoutresetting its label was never cleared, so it could fire after the component unmounted (React warning) or race itself on repeated clicks.Fixes https://github.com/keploy/enterprise/issues/2272
How Has This Been Tested?
Ran
npm run buildthennpm run serve— confirmed the build succeeds,llms.txt/llms-full.txtgenerate correctly, each page's.mdtwin loads at its matching URL, and the Copy/View buttons work in both light and dark mode.Preview.Video.mov