feat(ui): robots txt update

[MaxwellCohen]

🔗 Linked issue

Partially resolves: #2924

🧭 Context

Robots.txt helps keep websites safe from crawlers. npmx.dev uses a block list and no LLM crawling for robots.txt. This pr updats it so we have better alignment to those goals

📚 Description

This adds 3 changes to the robots.txt

1. Adds a Content-Signal: search=yes, ai-input=no, ai-train=no so npmx should be used for search indexing but not for AI input or training.
2. Adds an allow for the leaderboard page so it can be indexed for search
3. Adds an allow for the awesome new noodles section so it can be indexed for search

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
npmx.dev	Ready	Preview, Comment	Jul 17, 2026 5:28pm

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
docs.npmx.dev	Ignored	Preview	Jul 17, 2026 5:28pm
npmx-lunaria	Ignored		Jul 17, 2026 5:28pm

[coderabbitai]

📝 Walkthrough

Walkthrough

public/robots.txt now declares search access while disabling AI input and training, and permits crawlers to access the leaderboard likes route and /noodles.

Changes

Robots crawler directives

Layer / File(s)	Summary
Crawler signals and allow-list public/robots.txt	Adds the Content-Signal directive and permits /leaderboard/likes$ and /noodles under the wildcard crawler rules.

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	This PR does not address the core #2924 requirements about redundant or malformed robots.txt rules, especially the /opensearch.xml access issue.	Update robots.txt to remove the redundant rules and handle /opensearch.xml as requested, then keep the change focused on #2924.
Out of Scope Changes check	⚠️ Warning	The new /leaderboard/likes and /noodles allowances, plus Content-Signal, are not requested by #2924 and expand the scope.	Remove the unrelated path allowances or link them to a separate issue if they are intended.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and points to the robots.txt update, though the scope label is a bit broad.
Description check	✅ Passed	The description matches the robots.txt changes and linked issue.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!