From 98fe9c20f865b158ca018b98c6115f8534f08ba5 Mon Sep 17 00:00:00 2001 From: Hiroshi SHIBATA Date: Wed, 5 Aug 2026 21:05:39 +0900 Subject: [PATCH 1/5] Add llms.txt for LLM discoverability https://www.ruby-lang.org/llms.txt currently returns 404 and the Evil Martians LLM discoverability scorecard recommends serving one. The file follows the llmstxt.org convention and fills in the current stable version and recent news via Liquid, so it needs no manual updates. Co-Authored-By: Claude Fable 5 --- llms.txt | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 llms.txt diff --git a/llms.txt b/llms.txt new file mode 100644 index 0000000000..9718df77f0 --- /dev/null +++ b/llms.txt @@ -0,0 +1,42 @@ +--- +permalink: /llms.txt +--- +{%- comment -%} +TODO: Once the Markdown-mirror plugin (rubygems/guides#524 style) lands and +pages expose markdown_url, switch these links to the .md routes, e.g. +https://www.ruby-lang.org/en/about.md. +{%- endcomment -%} +# Ruby Programming Language + +> Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. The current stable version is Ruby {{ site.data.downloads.stable | first }}. + +## Getting Started + +- [About Ruby]({{ site.url }}/en/about/): An introduction to Ruby's philosophy, history, and features +- [Download Ruby]({{ site.url }}/en/downloads/): Source code and installation options for the latest releases +- [Installing Ruby]({{ site.url }}/en/documentation/installation/): How to install Ruby with package managers or third-party tools +- [Ruby in Twenty Minutes]({{ site.url }}/en/documentation/quickstart/): A short hands-on tutorial for newcomers + +## Documentation + +- [Documentation]({{ site.url }}/en/documentation/): Guides, tutorials, and reference material for learning Ruby +- [Ruby from Other Languages]({{ site.url }}/en/documentation/ruby-from-other-languages/): What to expect when coming from C, Java, PHP, Python, or Perl +- [Official Ruby FAQ]({{ site.url }}/en/documentation/faq/): Frequently asked questions about Ruby +- [Libraries]({{ site.url }}/en/libraries/): How to find and install third-party libraries with RubyGems and Bundler + +## Releases + +- [Ruby Releases]({{ site.url }}/en/downloads/releases/): A list of all Ruby releases with download links +- [Ruby Maintenance Branches]({{ site.url }}/en/downloads/branches/): Maintenance status and end-of-life dates of each Ruby branch + +## Community + +- [Community]({{ site.url }}/en/community/): Mailing lists, user groups, conferences, and other ways to get involved +- [Ruby Community Conduct Guideline]({{ site.url }}/en/conduct/): The conduct guideline for the Ruby developer community +- [Security]({{ site.url }}/en/security/): How to report security vulnerabilities in Ruby and past advisories + +## Recent News +{% assign en_posts = site.posts | where: "lang", "en" %} +{%- for post in en_posts limit: 10 %} +- [{{ post.title }}]({{ site.url }}{{ post.url }}): {{ post.date | date: "%Y-%m-%d" }} +{%- endfor %} From afffdaf5c6d019e3bec0e076235343fbffd76bba Mon Sep 17 00:00:00 2001 From: Hiroshi SHIBATA Date: Thu, 6 Aug 2026 08:40:54 +0900 Subject: [PATCH 2/5] Exclude _site from the markdown linter The generated Markdown alternates under _site have their front matter stripped, so any lint run after a local build failed on them. CI never saw this because rake test lints before building. Co-Authored-By: Claude Fable 5 --- lib/linter.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/linter.rb b/lib/linter.rb index 4296e39c41..925b75ab9a 100644 --- a/lib/linter.rb +++ b/lib/linter.rb @@ -18,6 +18,7 @@ class Linter %r{\Aadmin/index\.md}, %r{\A[^/]*/examples/}, %r{\A_includes/}, + %r{\A_site/}, %r{\Atest/}, %r{\Anode_modules/} ].freeze From 556b5a5da7ac1c59cd170e6c636457b2db082efc Mon Sep 17 00:00:00 2001 From: Hiroshi SHIBATA Date: Thu, 6 Aug 2026 08:41:03 +0900 Subject: [PATCH 3/5] Point llms.txt at the Markdown alternates Now that every page and news post is also served as .md, the links can target the Markdown routes directly instead of the HTML pages, which is what LLM tools prefer to fetch. Co-Authored-By: Claude Fable 5 --- llms.txt | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/llms.txt b/llms.txt index 9718df77f0..afad791d39 100644 --- a/llms.txt +++ b/llms.txt @@ -1,42 +1,37 @@ --- permalink: /llms.txt --- -{%- comment -%} -TODO: Once the Markdown-mirror plugin (rubygems/guides#524 style) lands and -pages expose markdown_url, switch these links to the .md routes, e.g. -https://www.ruby-lang.org/en/about.md. -{%- endcomment -%} # Ruby Programming Language > Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. The current stable version is Ruby {{ site.data.downloads.stable | first }}. ## Getting Started -- [About Ruby]({{ site.url }}/en/about/): An introduction to Ruby's philosophy, history, and features -- [Download Ruby]({{ site.url }}/en/downloads/): Source code and installation options for the latest releases -- [Installing Ruby]({{ site.url }}/en/documentation/installation/): How to install Ruby with package managers or third-party tools -- [Ruby in Twenty Minutes]({{ site.url }}/en/documentation/quickstart/): A short hands-on tutorial for newcomers +- [About Ruby]({{ site.url }}/en/about.md): An introduction to Ruby's philosophy, history, and features +- [Download Ruby]({{ site.url }}/en/downloads.md): Source code and installation options for the latest releases +- [Installing Ruby]({{ site.url }}/en/documentation/installation.md): How to install Ruby with package managers or third-party tools +- [Ruby in Twenty Minutes]({{ site.url }}/en/documentation/quickstart.md): A short hands-on tutorial for newcomers ## Documentation -- [Documentation]({{ site.url }}/en/documentation/): Guides, tutorials, and reference material for learning Ruby -- [Ruby from Other Languages]({{ site.url }}/en/documentation/ruby-from-other-languages/): What to expect when coming from C, Java, PHP, Python, or Perl -- [Official Ruby FAQ]({{ site.url }}/en/documentation/faq/): Frequently asked questions about Ruby -- [Libraries]({{ site.url }}/en/libraries/): How to find and install third-party libraries with RubyGems and Bundler +- [Documentation]({{ site.url }}/en/documentation.md): Guides, tutorials, and reference material for learning Ruby +- [Ruby from Other Languages]({{ site.url }}/en/documentation/ruby-from-other-languages.md): What to expect when coming from C, Java, PHP, Python, or Perl +- [Official Ruby FAQ]({{ site.url }}/en/documentation/faq.md): Frequently asked questions about Ruby +- [Libraries]({{ site.url }}/en/libraries.md): How to find and install third-party libraries with RubyGems and Bundler ## Releases -- [Ruby Releases]({{ site.url }}/en/downloads/releases/): A list of all Ruby releases with download links -- [Ruby Maintenance Branches]({{ site.url }}/en/downloads/branches/): Maintenance status and end-of-life dates of each Ruby branch +- [Ruby Releases]({{ site.url }}/en/downloads/releases.md): A list of all Ruby releases with download links +- [Ruby Maintenance Branches]({{ site.url }}/en/downloads/branches.md): Maintenance status and end-of-life dates of each Ruby branch ## Community -- [Community]({{ site.url }}/en/community/): Mailing lists, user groups, conferences, and other ways to get involved -- [Ruby Community Conduct Guideline]({{ site.url }}/en/conduct/): The conduct guideline for the Ruby developer community -- [Security]({{ site.url }}/en/security/): How to report security vulnerabilities in Ruby and past advisories +- [Community]({{ site.url }}/en/community.md): Mailing lists, user groups, conferences, and other ways to get involved +- [Ruby Community Conduct Guideline]({{ site.url }}/en/conduct.md): The conduct guideline for the Ruby developer community +- [Security]({{ site.url }}/en/security.md): How to report security vulnerabilities in Ruby and past advisories ## Recent News {% assign en_posts = site.posts | where: "lang", "en" %} {%- for post in en_posts limit: 10 %} -- [{{ post.title }}]({{ site.url }}{{ post.url }}): {{ post.date | date: "%Y-%m-%d" }} +- [{{ post.title }}]({{ site.url }}{{ post.markdown_url }}): {{ post.date | date: "%Y-%m-%d" }} {%- endfor %} From 727f7bd8c38e893c0f6828eb369c6b80c9aba83a Mon Sep 17 00:00:00 2001 From: Hiroshi SHIBATA Date: Thu, 6 Aug 2026 08:41:13 +0900 Subject: [PATCH 4/5] Serve llms-full.txt with every English page inlined llms.txt is only an index. llms-full.txt is the llmstxt.org companion file for tools that want the whole site in one fetch, so it inlines the Markdown of all English pages (about 330 KB). News posts are left out because two decades of announcements would add over 1 MB of mostly archival text. llms.txt links the file in its Optional section. Co-Authored-By: Claude Fable 5 --- _plugins/llms_full.rb | 25 +++++++++++++++++++++++++ llms.txt | 4 ++++ 2 files changed, 29 insertions(+) create mode 100644 _plugins/llms_full.rb diff --git a/_plugins/llms_full.rb b/_plugins/llms_full.rb new file mode 100644 index 0000000000..17565a6186 --- /dev/null +++ b/_plugins/llms_full.rb @@ -0,0 +1,25 @@ +# frozen_string_literal: true + +# Writes /llms-full.txt: every English page (news posts excluded, they +# would add over 1 MB of archival announcements) concatenated into one +# Markdown file, for LLM tools that want the whole site in a single +# fetch. Rendering is shared with the .md alternates so both stay +# identical. llms.txt links this file in its Optional section. +Jekyll::Hooks.register :site, :post_write do |site| + pages = site.pages.select do |page| + page.url.start_with?("/en/") && page.data["markdown_url"] + end + + sections = pages.sort_by(&:url).map do |page| + body = MarkdownAlternates.markdown_for(site, page).strip + "# #{page.data["title"]}\n\nSource: #{site.config["url"]}#{page.data["markdown_url"]}\n\n#{body}\n" + end + + header = <<~HEADER + # Ruby Programming Language + + > Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. This file contains all English pages of www.ruby-lang.org as Markdown. The current stable version is Ruby #{site.data["downloads"]["stable"].first}. + HEADER + + File.write(site.in_dest_dir("llms-full.txt"), ([header] + sections).join("\n---\n\n")) +end diff --git a/llms.txt b/llms.txt index afad791d39..d3a0656ed6 100644 --- a/llms.txt +++ b/llms.txt @@ -35,3 +35,7 @@ permalink: /llms.txt {%- for post in en_posts limit: 10 %} - [{{ post.title }}]({{ site.url }}{{ post.markdown_url }}): {{ post.date | date: "%Y-%m-%d" }} {%- endfor %} + +## Optional + +- [llms-full.txt]({{ site.url }}/llms-full.txt): All English pages of this site concatenated into a single Markdown file From e1ecfcdab48214df91d7dc4b7352010fef7fb5b7 Mon Sep 17 00:00:00 2001 From: Hiroshi SHIBATA Date: Thu, 6 Aug 2026 08:47:31 +0900 Subject: [PATCH 5/5] Generate llms.txt from the site content The hand-written llms.txt duplicated page titles and descriptions, so it would drift as pages change. The plugin now derives both llms.txt and llms-full.txt from the same source: every English page with a .md alternate, grouped by section, titled from its front matter, described by its first sentence. Only the tagline stays authored. Co-Authored-By: Claude Fable 5 --- _plugins/llms.rb | 98 +++++++++++++++++++++++++++++++++++++++++++ _plugins/llms_full.rb | 25 ----------- llms.txt | 41 ------------------ 3 files changed, 98 insertions(+), 66 deletions(-) create mode 100644 _plugins/llms.rb delete mode 100644 _plugins/llms_full.rb delete mode 100644 llms.txt diff --git a/_plugins/llms.rb b/_plugins/llms.rb new file mode 100644 index 0000000000..fec7c0ab0d --- /dev/null +++ b/_plugins/llms.rb @@ -0,0 +1,98 @@ +# frozen_string_literal: true + +# Generates /llms.txt and /llms-full.txt (llmstxt.org) from the site +# itself, so neither can drift from the published pages. llms.txt lists +# every English page grouped by top-level section, with the title from +# the front matter and the first sentence of the page as description. +# llms-full.txt inlines the full Markdown of the same pages for tools +# that want the whole site in one fetch. News posts appear only as the +# ten latest links, because two decades of announcements would add over +# 1 MB of archival text. Rendering is shared with the .md alternates so +# the outputs stay identical to what those routes serve. +module LlmsTxt + SITE_NAME = "Ruby Programming Language" + TAGLINE = "Ruby is a dynamic, open source programming language " \ + "with a focus on simplicity and productivity. It has an " \ + "elegant syntax that is natural to read and easy to write." + + # Pages without a title (layout: null fragments such as the homepage + # code examples) are not standalone documents and are left out. + def self.pages(site) + site.pages.select do |page| + page.url.start_with?("/en/") && page.data["markdown_url"] && page.data["title"] + end.sort_by(&:url) + end + + def self.posts(site) + site.posts.docs.select {|post| post.data["lang"] == "en" }.last(10).reverse + end + + def self.header(site) + stable = site.data["downloads"]["stable"].first + "# #{SITE_NAME}\n\n> #{TAGLINE} The current stable version is Ruby #{stable}.\n" + end + + # First sentence of the first prose paragraph, with images, links, + # HTML tags, emphasis and kramdown attribute lines stripped. + def self.description(markdown) + markdown.split(/\n\s*\n/).each do |block| + block = block.strip + next if block.empty? || block.start_with?("<", "#", ">", "{%", "|", "*", "-", "!") + + text = block.lines.map(&:strip).reject {|line| line.start_with?("{:") }.join(" ") + text = text.gsub(/!\[[^\]]*\](\([^)]*\)|\[[^\]]*\])?/, "") + .gsub(/\[([^\]]*)\](\([^)]*\)|\[[^\]]*\])/, '\1') + .gsub(/<[^>]+>/, " ") + .gsub(/[*_`]/, "") + .squeeze(" ").strip + next if text.empty? + + return text[/\A.{10,}?[.!?](?=\s|\z)/m] || text[0, 200] + end + nil + end + + def self.index(site, rendered) + out = +header(site) + + pages(site).group_by {|page| page.url.split("/")[2] }.each do |slug, section| + front = section.find {|page| page.url == "/en/#{slug}/" } + out << "\n## #{front ? front.data["title"] : slug.capitalize}\n\n" + section.each do |page| + desc = description(rendered[page]) + out << "- [#{page.data["title"]}](#{site.config["url"]}#{page.data["markdown_url"]})" + out << ": #{desc}" if desc + out << "\n" + end + end + + out << "\n## Recent News\n\n" + posts(site).each do |post| + url = site.config["url"] + post.data["markdown_url"] + out << "- [#{post.data["title"]}](#{url}): #{post.date.strftime("%Y-%m-%d")}\n" + end + + out << "\n## Optional\n\n" + out << "- [llms-full.txt](#{site.config["url"]}/llms-full.txt): " \ + "All English pages of this site concatenated into a single Markdown file\n" + end + + def self.full(site, rendered) + sections = pages(site).map do |page| + "# #{page.data["title"]}\n\n" \ + "Source: #{site.config["url"]}#{page.data["markdown_url"]}\n\n" \ + "#{rendered[page]}\n" + end + + ([header(site)] + sections).join("\n---\n\n") + end +end + +Jekyll::Hooks.register :site, :post_write do |site| + rendered = LlmsTxt.pages(site).to_h do |page| + [page, MarkdownAlternates.markdown_for(site, page).strip] + end + + File.write(site.in_dest_dir("llms.txt"), LlmsTxt.index(site, rendered)) + File.write(site.in_dest_dir("llms-full.txt"), LlmsTxt.full(site, rendered)) +end diff --git a/_plugins/llms_full.rb b/_plugins/llms_full.rb deleted file mode 100644 index 17565a6186..0000000000 --- a/_plugins/llms_full.rb +++ /dev/null @@ -1,25 +0,0 @@ -# frozen_string_literal: true - -# Writes /llms-full.txt: every English page (news posts excluded, they -# would add over 1 MB of archival announcements) concatenated into one -# Markdown file, for LLM tools that want the whole site in a single -# fetch. Rendering is shared with the .md alternates so both stay -# identical. llms.txt links this file in its Optional section. -Jekyll::Hooks.register :site, :post_write do |site| - pages = site.pages.select do |page| - page.url.start_with?("/en/") && page.data["markdown_url"] - end - - sections = pages.sort_by(&:url).map do |page| - body = MarkdownAlternates.markdown_for(site, page).strip - "# #{page.data["title"]}\n\nSource: #{site.config["url"]}#{page.data["markdown_url"]}\n\n#{body}\n" - end - - header = <<~HEADER - # Ruby Programming Language - - > Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. This file contains all English pages of www.ruby-lang.org as Markdown. The current stable version is Ruby #{site.data["downloads"]["stable"].first}. - HEADER - - File.write(site.in_dest_dir("llms-full.txt"), ([header] + sections).join("\n---\n\n")) -end diff --git a/llms.txt b/llms.txt deleted file mode 100644 index d3a0656ed6..0000000000 --- a/llms.txt +++ /dev/null @@ -1,41 +0,0 @@ ---- -permalink: /llms.txt ---- -# Ruby Programming Language - -> Ruby is a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. The current stable version is Ruby {{ site.data.downloads.stable | first }}. - -## Getting Started - -- [About Ruby]({{ site.url }}/en/about.md): An introduction to Ruby's philosophy, history, and features -- [Download Ruby]({{ site.url }}/en/downloads.md): Source code and installation options for the latest releases -- [Installing Ruby]({{ site.url }}/en/documentation/installation.md): How to install Ruby with package managers or third-party tools -- [Ruby in Twenty Minutes]({{ site.url }}/en/documentation/quickstart.md): A short hands-on tutorial for newcomers - -## Documentation - -- [Documentation]({{ site.url }}/en/documentation.md): Guides, tutorials, and reference material for learning Ruby -- [Ruby from Other Languages]({{ site.url }}/en/documentation/ruby-from-other-languages.md): What to expect when coming from C, Java, PHP, Python, or Perl -- [Official Ruby FAQ]({{ site.url }}/en/documentation/faq.md): Frequently asked questions about Ruby -- [Libraries]({{ site.url }}/en/libraries.md): How to find and install third-party libraries with RubyGems and Bundler - -## Releases - -- [Ruby Releases]({{ site.url }}/en/downloads/releases.md): A list of all Ruby releases with download links -- [Ruby Maintenance Branches]({{ site.url }}/en/downloads/branches.md): Maintenance status and end-of-life dates of each Ruby branch - -## Community - -- [Community]({{ site.url }}/en/community.md): Mailing lists, user groups, conferences, and other ways to get involved -- [Ruby Community Conduct Guideline]({{ site.url }}/en/conduct.md): The conduct guideline for the Ruby developer community -- [Security]({{ site.url }}/en/security.md): How to report security vulnerabilities in Ruby and past advisories - -## Recent News -{% assign en_posts = site.posts | where: "lang", "en" %} -{%- for post in en_posts limit: 10 %} -- [{{ post.title }}]({{ site.url }}{{ post.markdown_url }}): {{ post.date | date: "%Y-%m-%d" }} -{%- endfor %} - -## Optional - -- [llms-full.txt]({{ site.url }}/llms-full.txt): All English pages of this site concatenated into a single Markdown file