From b54c9e74c3a2d271e2c9ac5d34c79f7137a3e4b4 Mon Sep 17 00:00:00 2001 From: Hiroshi SHIBATA Date: Wed, 5 Aug 2026 21:04:49 +0900 Subject: [PATCH] Add robots.txt explicitly allowing all crawlers https://www.ruby-lang.org/robots.txt returns 404 today. Serve a permissive robots.txt with a Sitemap directive, as recommended by the Evil Martians LLM discoverability scorecard (https://ruby.evilmartians.com/). Co-Authored-By: Claude Fable 5 --- robots.txt | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 robots.txt diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000000..acbc2ddf6b --- /dev/null +++ b/robots.txt @@ -0,0 +1,4 @@ +User-agent: * +Allow: / + +Sitemap: https://www.ruby-lang.org/sitemap.xml