Allow crawlers on package landing pages and biocViews

[seandavi]

Refs #398.

Removes three Disallow: lines from assets/robots.txt:

-Disallow: /packages/devel/
-Disallow: /packages/release/
-Disallow: /biocViews/

These block crawling of the package landing pages, the biocViews browse tree, and every rendered vignette — the largest content set on the site.

Why

GA4 data for the property, measured over 2024-07 → 2025-07 (a window chosen to predate a large bot-traffic influx starting 2025-08 that would skew more recent figures):

• 70.1% of sessions land directly on a /packages/* page
• 74.3% of sessions come from Organic Search

Search is the dominant way people reach this site, and package pages are where they arrive — while robots.txt tells crawlers to ignore exactly those pages. The observable result is that searches for package documentation surface third-party mirrors on older releases instead of the origin.

What is deliberately left blocked

• /packages/3.23/ and /packages/3.24/ — the versioned aliases of current release and devel. Leaving these disallowed means the canonical /packages/release/… URLs don't compete with duplicate versioned copies for the same content. Removing them too would be defensible if paired with rel=canonical, but that's a separate change.
• Numbered archival releases 1.8–3.22 — genuinely superseded.
• /packages/bioc/, /packages/data/, /repository/, /checkResults/, /stats/ and the rest — repository internals and generated reports, not user-facing documentation.

Caveat worth naming

This helps crawlability but does not fully fix discoverability on its own. sitemap.xml still serves 20 bytes containing the literal string <%= xml_sitemap %> (#57 — I've commented there with what appears to be the root cause, an empty compile '/sitemap/' rule in Rules:12-14 that never applies filter :erb). Package pages also emit no rel=canonical while resolving at five or more URLs. Those are worth doing too; this is the one-line prerequisite.

Nothing here is verifiable by a build — robots.txt is copied verbatim to the site root — so the check is just reading the diff.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YBSvPwEk6sH1rhGmjPmsFq