Add agent-readiness surfaces (robots.txt, sitemap, llms.txt, markdown)#180
Open
sbddesign wants to merge 1 commit into
Open
Add agent-readiness surfaces (robots.txt, sitemap, llms.txt, markdown)#180sbddesign wants to merge 1 commit into
sbddesign wants to merge 1 commit into
Conversation
Make the site discoverable and consumable by AI agents and crawlers: - public/robots.txt: crawl rules, explicit AI crawler groups (GPTBot, ClaudeBot, Google-Extended, etc.), a Content-Signal line, and a Sitemap: reference. - app/sitemap.ts: /sitemap.xml generated from content. - app/llms.txt/route.ts: /llms.txt index linking every event/post/page to its plain-markdown mirror. - app/md/[...slug] + middleware.ts: serve a plain-markdown version of any content page, reached via /<section>/<slug>.md URLs or an Accept: text/markdown request. HTML stays the default for browsers. - next.config.js headers(): RFC 8288 Link headers pointing agents at llms.txt/sitemap and each page's .md alternate; metadata adds a <link rel="alternate" type="text/markdown">. - Footer with a subtle /llms.txt link. Scoped to a static content site: OAuth/MCP/WebMCP/DNS-AID/api-catalog items from the audit don't apply and were skipped.
✅ Deploy Preview for atlantabitdevs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the relevant items from the agent-readiness audit for a static content site. Each surface is generated from content, so it stays current as events/posts are added.
What's added
/robots.txt(public/robots.txt) — crawl rules, explicit AI-crawler groups (GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Applebot-Extended), aContent-Signalline, and aSitemap:reference. Plain text, returns 200./sitemap.xml(app/sitemap.ts) — canonical URLs for home, listings, events, posts, and pages; referenced from robots.txt./llms.txt(app/llms.txt/route.ts) — an llmstxt.org index linking every event/post/page to its.mdmirror.app/md/[...slug]+middleware.ts) — every content page is available as plain markdown. Two ways in:…/events/2026-05-20-bitcoin-socratic-seminar-50.mdAccept: text/markdownAccept: text/html).next.config.jsheaders()adds RFC 8288Linkheaders (homepage → llms.txt + sitemap; content pages → their.mdalternate), and content pages emit<link rel="alternate" type="text/markdown">./llms.txtlink in a new site footer.Content-Signal policy
Set to
search=yes, ai-input=yes, ai-train=yes— permissive, matching the community's free-and-open education mission. Easy to dial back inpublic/robots.txtif desired.Scope / skipped
Audit items that don't apply to a static content site (no APIs/auth/MCP) were intentionally skipped: OAuth/OIDC discovery, OAuth protected-resource, auth.md, API catalog, MCP Server Card, WebMCP, and DNS-AID.
Verification
yarn buildpasses; rannext startand confirmed status +Content-Typefor/robots.txt(text/plain),/sitemap.xml(application/xml),/llms.txt(text/plain),.mdURLs andAccept: text/markdown(text/markdown), HTML default still served, and theLinkheaders.