From 71cf9b9e282aaef2151d571ca778e7cb07147b74 Mon Sep 17 00:00:00 2001 From: danstotts-ops Date: Fri, 14 Aug 2026 22:51:46 -0500 Subject: [PATCH] robots.txt: scope the /_next/ disallow to Googlebot PR #785 intended a Googlebot-only rule but wrote it under User-agent: *, so the /_next/ disallows applied to every crawler, including GPTBot, ChatGPT-User, ClaudeBot and Claude Code. Mintlify serves client-side navigation payloads from under /_next/, so agents following those routes were blocked. Adds a Googlebot group carrying the asset disallows, and leaves * with only the shared directives. Content-Signal and Disallow: /cdn-cgi/ are repeated in both groups deliberately: a crawler obeys only its most specific matching group, so Googlebot would otherwise lose them. --- robots.txt | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/robots.txt b/robots.txt index ccf09e0d..3400ad08 100644 --- a/robots.txt +++ b/robots.txt @@ -1,8 +1,13 @@ -User-agent: * +User-agent: Googlebot Content-Signal: ai-train=yes, search=yes, ai-input=yes Disallow: /cdn-cgi/ Allow: /_next/image Allow: /mintlify-assets/_next/image Disallow: /_next/ Disallow: /mintlify-assets/_next/ + +User-agent: * +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ + Sitemap: https://docs.runpod.io/sitemap.xml