[geo-optimizer] Add robots.txt to docs site to allow AI bot crawling

[github-actions]

GEO Improvement: Add robots.txt to docs site

Source audit: GEO Audit Report — 2026-07-23
Audit date: 2026-07-23
Run: §30026886630

Finding

Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot) — all 20 audited pages on github.github.com/gh-aw score 0/18 on the robots.txt check because no robots.txt file exists at that origin.

Why this matters

Without a robots.txt, AI crawlers (GPTBot, ClaudeBot, PerplexityBot) cannot confirm they have explicit permission to index the documentation site. This is the single largest score gap across the entire site: fixing it would add +18 points per page site-wide, raising the sitemap average from 38/100 to approximately 56/100. The README already allows 27 AI bots explicitly and scores 15/18 on this check — the docs site is the missing half.

Suggested fix

1. Add a robots.txt file to the static assets of the Astro docs site (served at https://github.github.com/gh-aw/robots.txt) with contents similar to the repo's own robots.txt:

User-agent: *
Allow: /

# AI crawlers — explicit allow
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: YouBot
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap.xml

2. Place the file at docs/public/robots.txt (or wherever the Astro project's static public directory is located).
3. Verify it is served correctly: curl https://github.github.com/gh-aw/robots.txt
4. Re-run gh aw run geo-optimizer-skill to confirm the score improves.

Generated by 🌍 GEO Optimizer Daily Audit · sonnet46 · 35.4 AIC · ⌖ 12.5 AIC · ⊞ 5K · ◷

• expires on Jul 30, 2026, 8:59 AM UTC-08:00