diff --git a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx index df090b5b..c3064ce6 100644 --- a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx +++ b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx @@ -12,8 +12,9 @@ import { Code } from "@astrojs/starlight/components"; Without MCP, agents must invoke shell commands, manage output buffers, and parse unstructured text. With MCP: - The agent automatically discovers available tools, arguments, and return types. -- Responses are structured (JSON Feed objects, configuration schemas, and RSS XML). +- Responses are structured envelopes (`ok`, `next_step`, `guidance`, `payload`) — articles in `payload.items`, YAML in `payload.yaml`, RSS in `payload.rss`. - Agents can inspect, capture, validate, and verify feeds iteratively in a closed feedback loop. +- `html2rss://runtime` reports whether Botasaurus is configured in this MCP process (boolean only). ## Client Setup @@ -41,7 +42,10 @@ Add `html2rss` to your Cursor MCP settings (`~/.cursor/mcp.json` or `.cursor/mcp "mcpServers": { "html2rss": { "command": "mise", - "args": ["exec", "--", "html2rss", "mcp"] + "args": ["exec", "--", "html2rss", "mcp"], + "env": { + "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010" + } } } } @@ -61,7 +65,10 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp "mcpServers": { "html2rss": { "command": "mise", - "args": ["exec", "--", "html2rss", "mcp"] + "args": ["exec", "--", "html2rss", "mcp"], + "env": { + "BOTASAURUS_SCRAPER_URL": "http://127.0.0.1:4010" + } } } } @@ -71,23 +78,25 @@ Add `html2rss` to your Claude Desktop configuration (`~/Library/Application Supp ## Autonomous Workflow Patterns +**Budget:** `scrape_url` is one call. Durable config is `capture_config` → `validate_config` → `apply_config`. Call `inspect_url` only when the result is weak or you need recon (final URL, status, https→http, native RSS/Atom). + ### Pattern A: One-Shot Content Scraping When an agent needs articles immediately without saving a feed configuration: -1. The agent calls `scrape_url` with the target URL. -2. `html2rss` runs auto-source extraction (Schema.org, JSON state, semantic HTML) and returns a JSON Feed items array. -3. If the page is protected or rendered with JavaScript, the agent calls `inspect_url` to diagnose the structure, then retries `scrape_url` with `strategy: "botasaurus"`. +1. Call `scrape_url` with `strategy: "auto"`. Auto runs Faraday then Botasaurus when `BOTASAURUS_SCRAPER_URL` is set on the MCP process. +2. An empty item list is still success for articles-now. Read `payload.items`. Follow `next_step` (including `read_runtime` / `html2rss://runtime` when Botasaurus is unset). +3. Call `inspect_url` when the result is weak or you need recon (`final_url`, `status`, `scheme_downgrade`, native RSS/Atom). ### Pattern B: Iterative Feed Config Authoring When an agent is tasked with creating a durable YAML feed configuration: -1. **Inspect:** The agent calls `inspect_url` to check content type, SST node counts, and eligible scrapers. -2. **Capture:** The agent runs `capture_config` to derive CSS selectors for items, title, link, and description. -3. **Refine:** The agent reviews the derived selectors or asks the human user for domain-specific adjustments. -4. **Validate:** The agent passes the configuration to `validate_config` to verify schema conformance. -5. **Apply:** The agent tests the final configuration with `apply_config` to produce and inspect live RSS XML. +1. **Capture:** Call `capture_config`. YAML is `payload.yaml` (`items` + `enhance: true`). Check `payload.articles_count` and `has_selectors`. +2. **Recon (optional):** Call `inspect_url` for `final_url`, `status`, `scheme_downgrade`, and `alternate_feeds` if the draft is weak. +3. **Rewrite:** If the destination is html2rss-configs, add `directory.topics` and explicit channel `title`/`url`. Strive to keep `enhance: true` (set `false` only when chrome leaks into items). +4. **Validate:** Pass `yaml` (or `config`) to `validate_config`. Exactly one of those arguments. +5. **Apply:** Call `apply_config`. `isError` plus `payload.item_count: 0` means the config is not shippable — channel title in the XML is not success. ## JavaScript-Rendered Sites (Botasaurus) @@ -100,4 +109,8 @@ For dynamic JavaScript single-page applications or sites protected by anti-bot m lang="bash" /> -Ensure `BOTASAURUS_SCRAPER_URL` is accessible (typically `http://127.0.0.1:4010`) in the environment where the MCP server runs. Agents can then pass `strategy: "botasaurus"` to `scrape_url`, `inspect_url`, and `capture_config`. +Put `BOTASAURUS_SCRAPER_URL` (typically `http://127.0.0.1:4010`) in the MCP server `env` block above — a shell export does not reach the Cursor/Claude MCP process. Confirm with `html2rss://runtime` (`botasaurus_configured: true`). AutoFallback then hops without a second tool call. + +## Daemon logs (stderr) + +`html2rss mcp` uses stdout for JSON-RPC, so the daemon logs to **stderr**. Unless you set `LOG_LEVEL`, the MCP process uses `info` (the gem library default stays `warn`). A foreground watcher sees the start banner, each tool call, and pipeline fallbacks. Use `LOG_LEVEL=debug` for more detail or `LOG_LEVEL=warn` to quiet it. diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 39ebd2b4..4e79df5b 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -5,7 +5,22 @@ description: "Migration guide and backward-compatibility reference for html2rss import { Code } from "@astrojs/starlight/components"; -This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations. +This page outlines breaking changes, purged legacy aliases, and migration steps when upgrading the gem. + +## 0.27.0 + +Upgrade notes for configs and integrators targeting gem **0.27.0** (current `master` after `0.26.0`): + +1. **MCP result envelope** — Every MCP tool returns `{ ok, next_step, guidance, payload }` in text and `structuredContent`. Read `payload.items`, `payload.yaml`, and `payload.rss`; scrape text is not a raw item array. +2. **MCP AutoFallback** — `scrape_url` and `capture_config` with `strategy: "auto"` run Faraday → Botasaurus in one call. `inspect_url` with `auto` stays on Faraday (pin `botasaurus` when you need browser rendering for inspect). +3. **Capture output** — `html2rss capture` and MCP `capture_config` emit an items selector plus `enhance: true` only (no per-field title/url/description selector soup). Capture stamps `strategy:` when AutoFallback selects a concrete transport. +4. **CLI `--explain`** — `html2rss auto --explain` and `html2rss capture --explain` print telemetry JSON on stderr; stdout stays the feed or YAML. Status includes `admission_drops` when Cleanup dropped items. +5. **`Html2rss.auto_feed_result`** — URL-only auto discovery returns `FeedResult` with the same `status` telemetry as config builds. +6. **`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..20`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus). +7. **Botasaurus wire format** — Client expects OpenAPI 2.0 `ScrapeSuccess` / `ScrapeError` envelopes from the scrape API. +8. **`--limit` default** — Auto discovery default is `25` (`AutoSource::DEFAULT_LIMIT`). + +Run `html2rss validate config.yml` after editing Botasaurus options. ## 0.26.0 @@ -19,7 +34,7 @@ Upgrade notes for configs and integrators targeting gem **0.26.0**: Upgrade notes for configs and integrators targeting gem **0.25.0**: 1. **Removed `link` selector alias** — Custom selector configurations must use the canonical `url` key. The legacy `link` selector alias and exclusivity checks have been removed. -2. **`auto` article limit (`--limit N`)** — Discovery now supports a configurable article limit (`--limit N` CLI flag, default `20`) and `Html2rss.auto_source(url, limit: 20)` in Ruby. +2. **`auto` article limit (`--limit N`)** — Discovery supports a configurable limit (`--limit N` CLI flag). Default was `20` in 0.25.0; **0.27.0** default is `25` (`Html2rss.auto_source(url, limit: 25)`). 3. **Channel metadata in `auto`** — `auto_source` automatically extracts channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page. 4. **Actionable `NoFeedItemsExtracted` error** — `auto` mode fails loud with diagnostic surface classifications (blocked surface, app-shell, unsupported extraction surface) when zero items are found. @@ -89,12 +104,17 @@ Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` bl ## Migration Checklist -When upgrading to modern `html2rss` releases: +When upgrading to **0.27.0** or later: + +1. **MCP integrators**: Read tool results from the envelope `payload`; configure `BOTASAURUS_SCRAPER_URL` on the MCP process env. +2. **Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..20` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)). +3. **Validate configurations**: Run `html2rss validate config.yml`. + +When upgrading from releases before **0.26.0**: -1. **Migrate Browserless to Botasaurus**: Update any feeds using `strategy: browserless` or `request.browserless` to `strategy: botasaurus` or `strategy: auto` with `BOTASAURUS_SCRAPER_URL`. -2. **Rename URL selectors**: Ensure article URL selectors use `url` rather than `link`. -3. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`. -4. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file. -5. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS ``; select podcast/media URLs with `enclosure`. -6. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`. -7. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema. +4. **Migrate Browserless to Botasaurus**: Use `strategy: botasaurus` or `strategy: auto` with `BOTASAURUS_SCRAPER_URL`. +5. **Rename URL selectors**: Use `url` rather than `link`. +6. **Rename date selectors**: Use `published_at` rather than `updated` or `pubDate`. +7. **Move channel-level transport keys**: Put `strategy` and `headers` at the top level of the YAML file. +8. **Use `enclosure` for RSS media**: Select podcast/media URLs with `enclosure`; `image` does not become RSS ``. +9. **Prefer `feed_result` for dual-format / cache**: Use `Html2rss.feed_result` / `Html2rss.auto_feed_result` and `status.to_h`. diff --git a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx index 9b8bb7ce..8f11d655 100644 --- a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx @@ -5,11 +5,11 @@ description: Derive a reusable YAML feed config from a URL with html2rss capture import { Aside, Code } from "@astrojs/starlight/components"; -`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable feed config with derived CSS selectors. Use it when you want a first draft faster than hand-writing selectors from scratch. +`html2rss capture` analyzes a page through the auto-source pipeline and prints a reusable YAML draft: an **items selector plus `enhance: true`**. At feed-build time, `enhance: true` fills missing article fields from each matched card. ## When to Capture @@ -36,6 +36,8 @@ Print YAML to stdout: html2rss capture https://example.com/articles --items_selector ".post-card" # Analyze a local HTML fixture html2rss capture https://example.com/articles --input ./page.html + # Quality JSON on stderr; YAML stays on stdout + html2rss capture https://example.com/articles --explain # Save the draft html2rss capture https://example.com/articles > my-feed.yml `} @@ -46,9 +48,10 @@ Common options: - `--strategy` — `auto`, `faraday`, `botasaurus`, or `local_file` (default `auto`) - `--items_selector` — CSS selector hint for item cards -- `--limit` — maximum articles kept while deriving selectors +- `--limit` — maximum articles kept while deriving selectors (default `25`) - `--max-redirects` / `--max-requests` — request budget overrides - `--input` — local HTML file (sets `local_file` strategy) +- `--explain` — print capture quality JSON on stderr (`articles_count`, `channel_title`, `has_selectors`, `segment_strategy`, `selected_strategy`, `admission_drops`); YAML stays on stdout See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full flag list. @@ -57,8 +60,7 @@ See the [CLI reference](/ruby-gem/reference/cli-reference/#capture) for the full diff --git a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx index d346afa0..e1a635f2 100644 --- a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx @@ -76,7 +76,9 @@ Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON F - `to_rss` / `to_json_feed(feed_url:)` — render formats - `status` — scrape telemetry (`Html2rss::Status`) -`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`). +`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`), and `admission_drops` (Cleanup reason → count). + +For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same `FeedResult` (CLI `html2rss auto --explain` prints `status.to_h` on stderr). ` tag used by WordPress and pulls posts from the REST API without parsing article HTML. See [WordPress API](/ruby-gem/reference/wordpress-api/). -2. **`sitemap`:** Automatically locates XML sitemap documents (``, `/sitemap.xml`, or `/robots.txt`), filtering entries by priority and recency, with support for Google News tags (``). -3. **`meta_oembed`:** Extracts single-item articles, video pages, and media updates from OpenGraph/Twitter meta tags and resolves JSON oEmbed endpoints (``). -4. **`schema`:** Parses `