Serve llms.txt and llms-full.txt - #3994
Merged
Merged
Conversation
https://www.ruby-lang.org/llms.txt currently returns 404 and the Evil Martians LLM discoverability scorecard recommends serving one. The file follows the llmstxt.org convention and fills in the current stable version and recent news via Liquid, so it needs no manual updates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generated Markdown alternates under _site have their front matter stripped, so any lint run after a local build failed on them. CI never saw this because rake test lints before building. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Now that every page and news post is also served as .md, the links can target the Markdown routes directly instead of the HTML pages, which is what LLM tools prefer to fetch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
llms.txt is only an index. llms-full.txt is the llmstxt.org companion file for tools that want the whole site in one fetch, so it inlines the Markdown of all English pages (about 330 KB). News posts are left out because two decades of announcements would add over 1 MB of mostly archival text. llms.txt links the file in its Optional section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The hand-written llms.txt duplicated page titles and descriptions, so it would drift as pages change. The plugin now derives both llms.txt and llms-full.txt from the same source: every English page with a .md alternate, grouped by section, titled from its front matter, described by its first sentence. Only the tagline stays authored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying www-ruby-lang-org with
|
| Latest commit: |
e1ecfcd
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://119dc453.www-ruby-lang-org.pages.dev |
| Branch Preview URL: | https://claude-epic-brattain-763825.www-ruby-lang-org.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://www.ruby-lang.org/llms.txt returns 404, and the Evil Martians LLM discoverability scorecard at https://ruby.evilmartians.com/ recommends serving one. This adds
_plugins/llms.rb, which generates/llms.txtand/llms-full.txtat build time from the site itself so neither can drift from the published pages.llms.txtfollows the llmstxt.org convention. It lists every English page that has a.mdalternate, grouped by top-level section, titled from the front matter and described by the first sentence of the page, plus the ten latest English news posts.llms-full.txtinlines the full Markdown of the same pages (about 320 KB) for tools that want the whole site in one fetch. News posts are not inlined because two decades of announcements would add over 1 MB of archival text. Only the tagline in the header is authored.This also excludes
_sitefrom the Markdown linter. The generated.mdalternates have no front matter, sorake lintfailed after any local build.Generated with Claude Code