docs(ruby-gem): sync MCP envelope and post-0.26 gem contracts

src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx

@@ -12,8 +12,9 @@ import { Code } from "@astrojs/starlight/components";
 Without MCP, agents must invoke shell commands, manage output buffers, and parse unstructured text. With MCP:
 
 - The agent automatically discovers available tools, arguments, and return types.
-- Responses are structured (JSON Feed objects, configuration schemas, and RSS XML).
+- Responses are structured envelopes (`ok`, `next_step`, `guidance`, `payload`) — articles in `payload.items`, YAML in `payload.yaml`, RSS in `payload.rss`.
 - Agents can inspect, capture, validate, and verify feeds iteratively in a closed feedback loop.
+- `html2rss://runtime` reports whether Botasaurus is configured in this MCP process (boolean only).
 
 ## Client Setup
 
@@ -41,7 +42,10 @@ Add `html2rss` to your Cursor MCP settings (`~/.cursor/mcp.json` or `.cursor/mcp
     "mcpServers": {
       "html2rss": {
         "command": "mise",
-        "args": ["exec", "--", "html2rss", "mcp"]
+        "args": ["exec", "--", "html2rss", "mcp"],
+        "env": {
+          "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010"
+        }
       }
     }
   }
@@ -61,7 +65,10 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp
     "mcpServers": {
       "html2rss": {
         "command": "mise",
-        "args": ["exec", "--", "html2rss", "mcp"]
+        "args": ["exec", "--", "html2rss", "mcp"],
+        "env": {
+          "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010"
+        }
       }
     }
   }
@@ -71,23 +78,25 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp
 
 ## Autonomous Workflow Patterns
 
+**Budget:** `scrape_url` is one call. Durable config is `capture_config` → `validate_config` → `apply_config`. Call `inspect_url` only when the result is weak or you need recon (final URL, status, https→http, native RSS/Atom).
+
 ### Pattern A: One-Shot Content Scraping
 
 When an agent needs articles immediately without saving a feed configuration:
 
-1. The agent calls `scrape_url` with the target URL.
-2. `html2rss` runs auto-source extraction (Schema.org, JSON state, semantic HTML) and returns a JSON Feed items array.
-3. If the page is protected or rendered with JavaScript, the agent calls `inspect_url` to diagnose the structure, then retries `scrape_url` with `strategy: "botasaurus"`.
+1. Call `scrape_url` with `strategy: "auto"`. Auto already runs Faraday then Botasaurus.
+2. An empty item list is still success for articles-now. Read `payload.items`. Follow `next_step` (including `read_runtime` / `html2rss://runtime` if Botasaurus is unset).
+3. Call `inspect_url` only if the result is weak or you need recon. Do not retry `scrape_url` with explicit `faraday` after auto.
 
 ### Pattern B: Iterative Feed Config Authoring
 
 When an agent is tasked with creating a durable YAML feed configuration:
 
-1. **Inspect:** The agent calls `inspect_url` to check content type, SST node counts, and eligible scrapers.
-2. **Capture:** The agent runs `capture_config` to derive CSS selectors for items, title, link, and description.
-3. **Refine:** The agent reviews the derived selectors or asks the human user for domain-specific adjustments.
-4. **Validate:** The agent passes the configuration to `validate_config` to verify schema conformance.
-5. **Apply:** The agent tests the final configuration with `apply_config` to produce and inspect live RSS XML.
+1. **Capture:** Call `capture_config`. YAML is `payload.yaml` (`items` + `enhance: true`). Check `payload.articles_count` and `has_selectors`.
+2. **Recon (optional):** Call `inspect_url` for `final_url`, `status`, `scheme_downgrade`, and `alternate_feeds` if the draft is weak.
+3. **Rewrite:** If the destination is html2rss-configs, add `directory.topics` and explicit channel `title`/`url`. Strive to keep `enhance: true` (set `false` only when chrome leaks into items).
+4. **Validate:** Pass `yaml` (or `config`) to `validate_config`. Exactly one of those arguments.
+5. **Apply:** Call `apply_config`. `isError` plus `payload.item_count: 0` means the config is not shippable — channel title in the XML is not success.
 
 ## JavaScript-Rendered Sites (Botasaurus)
 
@@ -100,4 +109,8 @@ For dynamic JavaScript single-page applications or sites protected by anti-bot m
   lang="bash"
 />
 
-Ensure `BOTASAURUS_SCRAPER_URL` is accessible (typically `http://127.0.0.1:4010`) in the environment where the MCP server runs. Agents can then pass `strategy: "botasaurus"` to `scrape_url`, `inspect_url`, and `capture_config`.
+Put `BOTASAURUS_SCRAPER_URL` (typically `http://127.0.0.1:4010`) in the MCP server `env` block above — a shell export does not reach the Cursor/Claude MCP process. Confirm with `html2rss://runtime` (`botasaurus_configured: true`). AutoFallback then hops without a second tool call.
+
+## Daemon logs (stderr)
+
+`html2rss mcp` uses stdout for JSON-RPC, so the daemon logs to **stderr**. Unless you set `LOG_LEVEL`, the MCP process uses `info` (the gem library default stays `warn`). A foreground watcher sees the start banner, each tool call, and pipeline fallbacks. Use `LOG_LEVEL=debug` for more detail or `LOG_LEVEL=warn` to quiet it.

src/content/docs/ruby-gem/guides/backward-compatibility.mdx

@@ -7,6 +7,17 @@ import { Code } from "@astrojs/starlight/components";
 
 This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations.
 
+## Current Botasaurus request schema
+
+`request.botasaurus` matches the scrape-API OpenAPI 2.0 `ScrapeRequest` (unknown keys rejected):
+
+1. **`scroll`** — boolean. There is no `scroll_to_bottom` key; use `scroll: true` for infinite lists.
+2. **`window_size`** — object `{ width: 1920, height: 1080 }`, not a two-item array.
+3. **`wait_timeout_seconds`** — integer `1..20` (omit to use the scrape-API default of `15`).
+4. **`max_retries`** — integer `0..3` (omit to use the scrape-API default of `2`).
+
+Run `html2rss validate config.yml` after editing Botasaurus options.
+
 ## 0.26.0
 
 Upgrade notes for configs and integrators targeting gem **0.26.0**:
@@ -97,4 +108,5 @@ When upgrading to modern `html2rss` releases:
 4. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file.
 5. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS `<enclosure>`; select podcast/media URLs with `enclosure`.
 6. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`.
-7. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema.
+7. **Align `request.botasaurus` with OpenAPI 2.0**: Replace `scroll_to_bottom` with `scroll`, use `{ width, height }` for `window_size`, and keep `wait_timeout_seconds` in `1..20`.
+8. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema.

src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx

@@ -5,11 +5,11 @@ description: Derive a reusable YAML feed config from a URL with html2rss capture
 
 import { Aside, Code } from "@astrojs/starlight/components";
 
-`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable feed config with derived CSS selectors. Use it when you want a first draft faster than hand-writing selectors from scratch.
+`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable YAML draft with an **items selector plus `enhance: true`** — not title/url/description selector soup. `enhance: true` fills missing article fields at feed-build time.
 
 <Aside type="tip" title="Treat capture as a draft">
-  Selector quality depends on page structure. Complex layouts often need a manual `items_selector` hint or
-  small edits before you ship the config.
+  Selector quality depends on page structure. Catalog feeds still need `directory.topics` and explicit channel
+  `title`/`url`. Strive to keep `enhance: true`; set it `false` only when page chrome leaks into items.
 </Aside>
 
 ## When to Capture
@@ -36,6 +36,8 @@ Print YAML to stdout:
   html2rss capture https://example.com/articles --items_selector ".post-card"
   # Analyze a local HTML fixture
   html2rss capture https://example.com/articles --input ./page.html
+  # Quality JSON on stderr; YAML stays on stdout
+  html2rss capture https://example.com/articles --explain
   # Save the draft
   html2rss capture https://example.com/articles > my-feed.yml
 `}
@@ -46,9 +48,10 @@ Common options:
 
 - `--strategy` — `auto`, `faraday`, `botasaurus`, or `local_file` (default `auto`)
 - `--items_selector` — CSS selector hint for item cards
-- `--limit` — maximum articles kept while deriving selectors
+- `--limit` — maximum articles kept while deriving selectors (default `25`)
 - `--max-redirects` / `--max-requests` — request budget overrides
 - `--input` — local HTML file (sets `local_file` strategy)
+- `--explain` — print capture quality JSON on stderr (`articles_count`, `channel_title`, `has_selectors`, `segment_strategy`, `selected_strategy`, `admission_drops`); YAML stays on stdout
 
 See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full flag list.
 
@@ -57,56 +60,47 @@ See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full
 <Code
   code={`
   require 'html2rss'
-  require 'yaml'
-  # Derive a config hash (:channel and :selectors)
+  # Derive a config hash (channel + items selector with enhance: true)
   config = Html2rss.capture('https://example.com/articles')
   # Pin strategy or provide an items hint
   config = Html2rss.capture(
     'https://spa-site.com',
     strategy: :botasaurus,
     items_selector: '.article-card'
   )
-  # Serialize with string keys (same wire form as hand-written YAML)
-  File.write(
-    'my-feed.yml',
-    YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config))
-  )
-  # Use immediately
+  File.write('my-feed.yml', Html2rss::Config.to_yaml(config))
   feed = Html2rss.feed(config)
 `}
   lang="ruby"
 />
 
 ## How It Works
 
-1. **Request** — fetches the page with the chosen strategy
-2. **Discover** — runs AutoSource to extract articles
-3. **Analyze** — normalizes the page into an SST document and maps segment positions back to articles
-4. **Derive** — builds CSS selectors from SST tag paths for items, title, link, and description
-5. **Assemble** — returns a config hash ready for YAML or `Html2rss.feed`
+1. **Request** — `FeedPipeline` (AutoFallback when `:auto`)
+2. **Discover** — AutoSource extracts admitted articles
+3. **Segment** — SST Segmenter strategies `:list` → `:cluster` → `:semantic`
+4. **Gate** — emit an items selector only when enough articles match
+5. **Assemble** — `{ items: { selector:, enhance: true } }` plus channel. When AutoFallback selects a concrete transport (or you pin one), Capture **stamps** `strategy:` into the YAML so later `html2rss feed` / `Html2rss.feed` replay the same transport.
 
-Capture segment discovery currently uses the list Segmenter strategy only (not AutoSource cluster/semantic heuristics). When the draft is weak, pass `--items_selector` or refine selectors by hand.
+When the quality gate fails, selectors are omitted (`has_selectors: false`) rather than inventing attribute selectors. Hint with `--items_selector` or refine by hand.
 
 ## What Capture Derives
 
 Capture focuses on:
 
 - `channel.url` (and related channel defaults)
-- `selectors.items`
-- `selectors.title`
-- `selectors.url` (derived href selector)
-- `selectors.description` when a distinct description root exists
+- `selectors.items` with `enhance: true`
 
-It does not invent author, `published_at`, categories, or enclosure selectors. Add those manually when the page exposes them reliably.
+It does not invent per-field title/url/description selectors, author, `published_at`, categories, or enclosure selectors. `enhance: true` fills missing article fields via the HTML article extractor at feed-build time.
 
-Description is omitted when it would resolve to the invalid CSS selector `.` (item root equals description root).
+MCP `capture_config` returns that YAML in `payload.yaml`. `validate_config` / `apply_config` accept the YAML string (XOR a config hash). `apply_config` is `isError` when the feed has zero items (`payload.item_count`).
 
 ## Recommended Follow-Up
 
 1. Validate: `html2rss validate my-feed.yml`
 2. Render: `html2rss feed my-feed.yml`
-3. Tighten selectors, strategy, or `request.botasaurus` options if needed
-4. For Feed Directory contributions, add `directory.topics` (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config))
+3. Tighten the items selector, strategy, or `request.botasaurus` options if needed
+4. For Feed Directory contributions, add `directory.topics` and keep `enhance: true` unless chrome leaks (see [Creating Custom Feeds](/creating-custom-feeds/#sharing-your-config))
 
 ## Related
 

src/content/docs/ruby-gem/guides/custom-http-requests.mdx

@@ -73,7 +73,7 @@ Request budgets are configured under `request`, not as top-level keys:
 
 - `request.max_redirects` limits redirect hops
 - `request.max_requests` limits the total request budget for the feed build
-- `request.botasaurus.*` is reserved for Botasaurus-only behavior such as navigation mode, retries, header/cookie forwarding, and scroll options
+- `request.botasaurus.*` is reserved for Botasaurus-only behavior such as navigation mode, retries, header/cookie forwarding, and `scroll`
 
 ## Common Use Cases
 

src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx

@@ -22,7 +22,7 @@ When a site requires browser rendering or anti-bot bypass by default, you can ex
     botasaurus:
       navigation_mode: google_get_bypass
       wait_for_selector: ".article-card"
-      scroll_to_bottom: true
+      scroll: true
   channel:
     url: https://example.com/app
   selectors:
@@ -44,7 +44,7 @@ Botasaurus extraction is ideal when:
 - **Content loads after page load** - JavaScript fetches data dynamically
 - **Single Page Applications (SPAs)** - React, Vue, Angular apps
 - **Anti-bot interstitials** - Cloudflare, Google search referrers, or challenge pages
-- **Scroll-triggered content** - Infinite scroll lists (`scroll_to_bottom: true`)
+- **Scroll-triggered content** - Infinite scroll lists (`scroll: true`)
 
 ## Botasaurus Request Controls
 
@@ -70,7 +70,7 @@ Configure browser actions under `request.botasaurus`:
   strategy: botasaurus
   request:
     botasaurus:
-      scroll_to_bottom: true
+      scroll: true
   `}
   lang="yaml"
 />

src/content/docs/ruby-gem/guides/managing-feed-configs.mdx

@@ -76,7 +76,9 @@ Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON F
 - `to_rss` / `to_json_feed(feed_url:)` — render formats
 - `status` — scrape telemetry (`Html2rss::Status`)
 
-`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`).
+`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`), and `admission_drops` (Cleanup reason → count).
+
+For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same `FeedResult` (CLI `html2rss auto --explain` prints `status.to_h` on stderr).
 
 <Code
   code={`