html2rss · gildesmarais · Aug 22, 2026 · Aug 22, 2026 · Aug 22, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -14,8 +14,8 @@ What this repo owns:
 What this repo does not own:
 
 - runtime extractor behavior and CLI semantics (`html2rss/`)
-- catalog metadata, YAML configs, or catalog serialization (`html2rss-configs/` → `Html2rss::Configs::Catalog`)
-- catalog HTTP API (`html2rss-web/` → `GET /api/v1/configs`)
+- feed YAML and signed registry bundle publishing (`html2rss-configs/` → `configs/`, `tool/registry-build`)
+- catalog HTTP API and registry sync (`html2rss-web/` → `GET /api/v1/configs`, `Registry::Sync`)
 
 When docs describe behavior from other repos, treat those repos as source-of-truth and update docs to match them.
 
@@ -30,9 +30,10 @@ Before substantial edits, state cross-repo context in your notes:
 Common contracts:
 
 - Feed Directory browse data comes from `{instance}/api/v1/configs` on a running `html2rss-web` instance (see OpenAPI in `html2rss-web`).
+- Catalog wire rows include `id`, `path`, `directory`, `channel`, `parameters`, plus `source` (`registry` | `local`) and `registry` (registry id for bundle entries). The browse UI ignores `source`/`registry` today — wire parsing stays in `adapters/catalog-api.ts`.
 - Instance URL persistence: default public instance, `#!url=` hash deep link from the web app, browser localStorage, and filter state in URL query params (`q`, `topic`, `lang`, `sort`, `page`).
 - Deep link from `html2rss-web`: `https://html2rss.github.io/feed-directory/#!url={encodedInstanceUrl}` must keep working.
-- Catalog metadata in YAML (`directory.title`, `directory.summary`, `directory.topics`) is authored in `html2rss-configs` only.
+- Catalog metadata in YAML (`directory.*`, `registry.id`) is authored in `html2rss-configs` only; instances load verified bundles via registry sync.
 - Ruby gem docs should match `html2rss` behavior and CLI output.
 - Web application docs should match `html2rss-web` behavior and published OpenAPI.
 

diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx
@@ -16,7 +16,7 @@ Upgrade notes for configs and integrators targeting gem **0.27.0** (current `mas
 3. **Capture output** — `html2rss capture` and MCP `capture_config` emit an items selector plus `enhance: true` only (no per-field title/url/description selector soup). Capture stamps `strategy:` when AutoFallback selects a concrete transport.
 4. **CLI `--explain`** — `html2rss auto --explain` and `html2rss capture --explain` print telemetry JSON on stderr; stdout stays the feed or YAML. Status includes `admission_drops` when Cleanup dropped items.
 5. **`Html2rss.auto_feed_result`** — URL-only auto discovery returns `FeedResult` with the same `status` telemetry as config builds.
-6. **`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..20`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus).
+6. **`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..30`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus).
 7. **Botasaurus wire format** — Client expects OpenAPI 2.0 `ScrapeSuccess` / `ScrapeError` envelopes from the scrape API.
 8. **`--limit` default** — Auto discovery default is `25` (`AutoSource::DEFAULT_LIMIT`).
 
@@ -107,7 +107,7 @@ Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` bl
 When upgrading to **0.27.0** or later:
 
 1. **MCP integrators**: Read tool results from the envelope `payload`; configure `BOTASAURUS_SCRAPER_URL` on the MCP process env.
-2. **Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..20` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)).
+2. **Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..30` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)).
 3. **Validate configurations**: Run `html2rss validate config.yml`.
 
 When upgrading from releases before **0.26.0**:

diff --git a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx
@@ -63,6 +63,8 @@ Configure browser actions under `request.botasaurus`:
   lang="yaml"
 />
 
+`wait_timeout_seconds` applies to selector wait on the scraper **after** the browser is ready. Slow Chrome or Xvfb boot consumes the scrape **total** wall (`SCRAPE_TIMEOUT_SECONDS`, default **45s**), not this work budget (`SCRAPE_WORK_TIMEOUT_SECONDS`, default **30s**).
+
 ### Scroll for Infinite Content
 
 <Code

diff --git a/src/content/docs/ruby-gem/reference/strategy.mdx b/src/content/docs/ruby-gem/reference/strategy.mdx
@@ -41,7 +41,7 @@ Requirements:
 - `BOTASAURUS_SCRAPER_URL` set to your Botasaurus scrape API base URL (for example `http://localhost:4010`)
 - pin `strategy: botasaurus` when you want to skip Faraday and force this transport directly
 
-html2rss enforces local request policy preflight and timeout budget. When a total request timeout remains, Botasaurus `max_retries` and `wait_timeout_seconds` are clamped so upstream work fits the remaining budget.
+html2rss enforces local request policy preflight and a feed-build timeout budget. The Botasaurus scrape API splits **total** scrape time (queue, boot, navigate, wait) from **work** time (navigate, selector wait, and scroll after the browser is ready). Feed YAML `wait_timeout_seconds` is validated against the work cap (`1..30` by default); the gem forwards `request.botasaurus` options to the scrape API without shrinking retries or waits to fit a remaining feed budget. Faraday transport timeout for POST `/scrape` is the lesser of the remaining feed-build budget and `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` plus a small buffer (default **45s** scrape total, **47s** transport cap).
 
 During the **browser** execution tier, the Botasaurus scrape API always captures JSON XHR/fetch response bodies (`xhr_responses`). html2rss forwards them as `Response#captured_responses` so AutoSource `xhr_articles` can extract articles without extra requests. The HTTP-request tier returns an empty capture list. Caps and filtering live in the scrape API (see its README); article-likeness filtering is a client concern.
 
@@ -70,7 +70,7 @@ Supported `request.botasaurus` options (unknown keys are rejected):
 - `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`, `organic_get`; default `auto`)
 - `max_retries` (`0..3`; omit to use the scrape-API default of `2`)
 - `wait_for_selector` (string)
-- `wait_timeout_seconds` (integer `1..20`; omit to use the scrape-API default of `15`)
+- `wait_timeout_seconds` (integer `1..30`; omit to use the scrape-API default of `15`; counts against post-boot **work** budget on the scraper, not browser boot)
 - `scroll` (boolean)
 - `block_images` (boolean)
 - `block_images_and_css` (boolean)

diff --git a/src/content/docs/web-application/reference/env-variables.mdx b/src/content/docs/web-application/reference/env-variables.mdx
@@ -5,36 +5,42 @@ description: "Configuration reference for html2rss-web environment variables."
 
 ## Supported ENV variables
 
-| Name                              | Description                                                                                                                                                  |
-| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `HTML2RSS_SECRET_KEY`             | required in production; development/test gets a temporary default                                                                                            |
-| `HTML2RSS_ACCESS_TOKEN`           | token for the main feed-creation flow; paste the same value into the web UI when prompted                                                                    |
-| `HEALTH_CHECK_TOKEN`              | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks |
-| `BUILD_TAG`                       | release metadata used in logs; published Docker images set this to the release version                                                                       |
-| `GIT_SHA`                         | deployed commit metadata used in logs; published Docker images set this to the released commit                                                               |
-| `SENTRY_DSN`                      | optional; enables Sentry errors/logs when set                                                                                                                |
-| `BOTASAURUS_SCRAPER_URL`          | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass                                           |
-| `AUTO_SOURCE_ENABLED`             | `true` by default in development/test, `false` otherwise                                                                                                     |
-| `ASYNC_FEED_REFRESH_ENABLED`      | optional boolean; default `false`                                                                                                                            |
-| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3`                                                                                                                         |
-| `FEEDS_CACHE_MAX_SIZE`            | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction                                                           |
-| `RATE_LIMIT_ENABLED`              | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting                                                     |
-| `RATE_LIMIT_MAX_REQUESTS`         | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window                                                                            |
-| `RATE_LIMIT_WINDOW_SECONDS`       | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds                                                                       |
-| `RETRY_AFTER_TIMEOUT_SECONDS`     | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts                                                             |
-| `PORT`                            | app listen port; compose uses `4000`                                                                                                                         |
-| `RACK_ENV`                        | Rack environment; quickstart uses `development`, production examples use `production`                                                                        |
-| `RACK_TIMEOUT_SERVICE_TIMEOUT`    | optional; total web-request timeout in seconds; default `30`                                                                                                 |
-| `HTML2RSS_TOTAL_TIMEOUT_SECONDS`  | optional; internal gem request timeout in seconds; default `25`                                                                                              |
+| Name                                     | Description                                                                                                                                                  |
+| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `HTML2RSS_SECRET_KEY`                    | required in production; development/test gets a temporary default                                                                                            |
+| `HTML2RSS_ACCESS_TOKEN`                  | token for the main feed-creation flow; paste the same value into the web UI when prompted                                                                    |
+| `HEALTH_CHECK_TOKEN`                     | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks |
+| `BUILD_TAG`                              | release metadata used in logs; published Docker images set this to the release version                                                                       |
+| `GIT_SHA`                                | deployed commit metadata used in logs; published Docker images set this to the released commit                                                               |
+| `SENTRY_DSN`                             | optional; enables Sentry errors/logs when set                                                                                                                |
+| `BOTASAURUS_SCRAPER_URL`                 | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass                                           |
+| `AUTO_SOURCE_ENABLED`                    | `true` by default in development/test, `false` otherwise                                                                                                     |
+| `ASYNC_FEED_REFRESH_ENABLED`             | optional boolean; default `false`                                                                                                                            |
+| `ASYNC_FEED_REFRESH_STALE_FACTOR`        | optional integer `>= 1`; default `3`                                                                                                                         |
+| `FEEDS_CACHE_MAX_SIZE`                   | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction                                                           |
+| `RATE_LIMIT_ENABLED`                     | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting                                                     |
+| `RATE_LIMIT_MAX_REQUESTS`                | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window                                                                            |
+| `RATE_LIMIT_WINDOW_SECONDS`              | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds                                                                       |
+| `RETRY_AFTER_TIMEOUT_SECONDS`            | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts                                                             |
+| `PORT`                                   | app listen port; compose uses `4000`                                                                                                                         |
+| `RACK_ENV`                               | Rack environment; quickstart uses `development`, production examples use `production`                                                                        |
+| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS`      | optional; Faraday client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api                          |
+| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api                          |
+| `SCRAPE_WORK_TIMEOUT_SECONDS`            | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose      |
+| `RACK_TIMEOUT_SERVICE_TIMEOUT`           | optional; total web-request timeout in seconds; default `55`                                                                                                 |
+| `HTML2RSS_TOTAL_TIMEOUT_SECONDS`         | optional; internal gem feed-build timeout in seconds; default `50`                                                                                           |
 
 ## Timeout Chain
 
-When fetching feeds from complex websites, multiple components work together. To prevent premature request termination, ensure your timeouts are configured in a "chain" where the outer layers wait longer than the inner layers.
+When fetching feeds from complex websites, multiple components work together. Configure timeouts so outer layers wait longer than inner layers and the scraper exhausts its budget before the web tier aborts the request.
 
-For an overall 30-second budget:
+Default compose aligns **scrape total (45) ≤ feed build (50) ≤ Rack (55)**. The **work** budget (**30s**) applies on botasaurus-scrape-api only after the browser is ready; browser boot and queue time count against the **total** scrape wall (**45s**), not against `wait_timeout_seconds`.
 
-1.  **Botasaurus Scraper (`SCRAPE_TIMEOUT_SECONDS`)**: Set to **20s**. This is the time allowed for the browser to fetch and render the page.
-2.  **html2rss Gem (`HTML2RSS_TOTAL_TIMEOUT_SECONDS`)**: Set to **25s**. This allows the gem enough time to wait for Botasaurus and process the results.
-3.  **html2rss-web (`RACK_TIMEOUT_SERVICE_TIMEOUT`)**: Set to **30s**. This is the total time the web server will wait before killing the request.
+1. **botasaurus-scrape-api (`SCRAPE_TIMEOUT_SECONDS`)**: **45s** handler wall (queue, boot, navigate, wait).
+2. **botasaurus-scrape-api (`SCRAPE_WORK_TIMEOUT_SECONDS`)**: **30s** post-boot navigate, `wait_for_selector`, and scroll.
+3. **html2rss client (`BOTASAURUS_SCRAPE_TIMEOUT_SECONDS`)**: **45s** Faraday cap for POST `/scrape` (mirrors scrape total).
+4. **Feed YAML (`wait_timeout_seconds`)**: **1..30**, capped by `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` (default **30**).
+5. **html2rss-web gem budget (`HTML2RSS_TOTAL_TIMEOUT_SECONDS`)**: **50s** for scrape plus extraction.
+6. **html2rss-web (`RACK_TIMEOUT_SERVICE_TIMEOUT`)**: **55s** Rack outer wall.
 
-Older environment-variable examples from previous docs revisions are obsolete. Use only the supported table above and the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md).
+Set matching `SCRAPE_*` on botasaurus-scrape-api and `BOTASAURUS_SCRAPE_*` on html2rss-web when you override defaults. See the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md) for compose examples.
diff --git a/src/content/docs/web-application/reference/monitoring.mdx b/src/content/docs/web-application/reference/monitoring.mdx
@@ -7,11 +7,11 @@ import { Code } from "@astrojs/starlight/components";
 
 ## Health Endpoints
 
-| Endpoint | Auth | Purpose |
-| --- | --- | --- |
-| `GET /api/v1/health/live` | none | Process liveness |
-| `GET /api/v1/health/ready` | none | Config readiness |
-| `GET /api/v1/health` | bearer | Operator and uptime checks |
+| Endpoint                   | Auth   | Purpose                    |
+| -------------------------- | ------ | -------------------------- |
+| `GET /api/v1/health/live`  | none   | Process liveness           |
+| `GET /api/v1/health/ready` | none   | Config readiness           |
+| `GET /api/v1/health`       | bearer | Operator and uptime checks |
 
 Set `HEALTH_CHECK_TOKEN`, then call the authenticated endpoint:
 
@@ -31,9 +31,9 @@ Set `SENTRY_DSN` to enable Issue capture in html2rss-web (Rack middleware and op
 
 Use separate Sentry projects for html2rss-web and botasaurus-scrape-api. Never share a DSN.
 
-| Env var | Service |
-| --- | --- |
-| `SENTRY_DSN` | html2rss-web |
+| Env var                 | Service               |
+| ----------------------- | --------------------- |
+| `SENTRY_DSN`            | html2rss-web          |
 | `BOTASAURUS_SENTRY_DSN` | botasaurus-scrape-api |
 
 In `docker-compose.yml`, the botasaurus service maps `BOTASAURUS_SENTRY_DSN` into its own `SENTRY_DSN`, requires it at startup, and does not fall back to the web project's DSN.
-Original file line number
+Diff line change
@@ Expand Up / @@ -63,6 +63,8 @@ Configure browser actions under `request.botasaurus`: @@
       lang="yaml"
     />
+    `wait_timeout_seconds` applies to selector wait on the scraper **after** the browser is ready. Slow Chrome or Xvfb boot consumes the scrape **total** wall (`SCRAPE_TIMEOUT_SECONDS`, default **45s**), not this work budget (`SCRAPE_WORK_TIMEOUT_SECONDS`, default **30s**).
     ### Scroll for Infinite Content
     <Code
@@ Expand Down @@