From fea72c1ede94cbc180c13e48bb12c9fb62c886e6 Mon Sep 17 00:00:00 2001 From: danstotts-ops Date: Thu, 13 Aug 2026 18:37:03 -0500 Subject: [PATCH] Block crawling of versioned Mintlify asset chunks Mintlify's default robots.txt disallows /_next/ but serves its build assets under /mintlify-assets/_next/, which the prefix rule does not cover. Every redeploy mints new deployment-versioned chunk URLs (...js?dpl=...) and kills the old ones, so Googlebot crawls a rolling population of dead JS files. GSC's "Not found (404)" bucket doubled from ~1.1K to 2.17K around 2026-07-19 to 07-30 from exactly these URLs, burying the real page-404 signal that #738 fixed. Reproduces the live default verbatim (Content-Signal line, /cdn-cgi/ block, /_next/image allowance, sitemap) and adds the same pattern one directory deeper: allow /mintlify-assets/_next/image, disallow /mintlify-assets/_next/. Page content, llms.txt, and AI-assistant access are unaffected. --- robots.txt | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 robots.txt diff --git a/robots.txt b/robots.txt new file mode 100644 index 00000000..ccf09e0d --- /dev/null +++ b/robots.txt @@ -0,0 +1,8 @@ +User-agent: * +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Allow: /mintlify-assets/_next/image +Disallow: /_next/ +Disallow: /mintlify-assets/_next/ +Sitemap: https://docs.runpod.io/sitemap.xml