diff --git a/AGENTS.md b/AGENTS.md index 86245d79..1d783489 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,8 +14,8 @@ What this repo owns: What this repo does not own: - runtime extractor behavior and CLI semantics (`html2rss/`) -- catalog metadata, YAML configs, or catalog serialization (`html2rss-configs/` → `Html2rss::Configs::Catalog`) -- catalog HTTP API (`html2rss-web/` → `GET /api/v1/configs`) +- feed YAML and signed registry bundle publishing (`html2rss-configs/` → `configs/`, `tool/registry-build`) +- catalog HTTP API and registry sync (`html2rss-web/` → `GET /api/v1/configs`, `Registry::Sync`) When docs describe behavior from other repos, treat those repos as source-of-truth and update docs to match them. @@ -30,9 +30,10 @@ Before substantial edits, state cross-repo context in your notes: Common contracts: - Feed Directory browse data comes from `{instance}/api/v1/configs` on a running `html2rss-web` instance (see OpenAPI in `html2rss-web`). +- Catalog wire rows include `id`, `path`, `directory`, `channel`, `parameters`, plus `source` (`registry` | `local`) and `registry` (registry id for bundle entries). The browse UI ignores `source`/`registry` today — wire parsing stays in `adapters/catalog-api.ts`. - Instance URL persistence: default public instance, `#!url=` hash deep link from the web app, browser localStorage, and filter state in URL query params (`q`, `topic`, `lang`, `sort`, `page`). - Deep link from `html2rss-web`: `https://html2rss.github.io/feed-directory/#!url={encodedInstanceUrl}` must keep working. -- Catalog metadata in YAML (`directory.title`, `directory.summary`, `directory.topics`) is authored in `html2rss-configs` only. +- Catalog metadata in YAML (`directory.*`, `registry.id`) is authored in `html2rss-configs` only; instances load verified bundles via registry sync. - Ruby gem docs should match `html2rss` behavior and CLI output. - Web application docs should match `html2rss-web` behavior and published OpenAPI. diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 4e79df5b..1a69217b 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -16,7 +16,7 @@ Upgrade notes for configs and integrators targeting gem **0.27.0** (current `mas 3. **Capture output** — `html2rss capture` and MCP `capture_config` emit an items selector plus `enhance: true` only (no per-field title/url/description selector soup). Capture stamps `strategy:` when AutoFallback selects a concrete transport. 4. **CLI `--explain`** — `html2rss auto --explain` and `html2rss capture --explain` print telemetry JSON on stderr; stdout stays the feed or YAML. Status includes `admission_drops` when Cleanup dropped items. 5. **`Html2rss.auto_feed_result`** — URL-only auto discovery returns `FeedResult` with the same `status` telemetry as config builds. -6. **`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..20`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus). +6. **`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..30`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus). 7. **Botasaurus wire format** — Client expects OpenAPI 2.0 `ScrapeSuccess` / `ScrapeError` envelopes from the scrape API. 8. **`--limit` default** — Auto discovery default is `25` (`AutoSource::DEFAULT_LIMIT`). @@ -107,7 +107,7 @@ Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` bl When upgrading to **0.27.0** or later: 1. **MCP integrators**: Read tool results from the envelope `payload`; configure `BOTASAURUS_SCRAPER_URL` on the MCP process env. -2. **Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..20` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)). +2. **Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..30` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)). 3. **Validate configurations**: Run `html2rss validate config.yml`. When upgrading from releases before **0.26.0**: diff --git a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx index 89e930b8..503c9d14 100644 --- a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx +++ b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx @@ -63,6 +63,8 @@ Configure browser actions under `request.botasaurus`: lang="yaml" /> +`wait_timeout_seconds` applies to selector wait on the scraper **after** the browser is ready. Slow Chrome or Xvfb boot consumes the scrape **total** wall (`SCRAPE_TIMEOUT_SECONDS`, default **45s**), not this work budget (`SCRAPE_WORK_TIMEOUT_SECONDS`, default **30s**). + ### Scroll for Infinite Content = 1`; default `3` | -| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction | -| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting | -| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window | -| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds | -| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts | -| `PORT` | app listen port; compose uses `4000` | -| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` | -| `RACK_TIMEOUT_SERVICE_TIMEOUT` | optional; total web-request timeout in seconds; default `30` | -| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem request timeout in seconds; default `25` | +| Name | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `HTML2RSS_SECRET_KEY` | required in production; development/test gets a temporary default | +| `HTML2RSS_ACCESS_TOKEN` | token for the main feed-creation flow; paste the same value into the web UI when prompted | +| `HEALTH_CHECK_TOKEN` | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks | +| `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version | +| `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit | +| `SENTRY_DSN` | optional; enables Sentry errors/logs when set | +| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass | +| `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise | +| `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` | +| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` | +| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction | +| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting | +| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window | +| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds | +| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts | +| `PORT` | app listen port; compose uses `4000` | +| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` | +| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; Faraday client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api | +| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api | +| `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose | +| `RACK_TIMEOUT_SERVICE_TIMEOUT` | optional; total web-request timeout in seconds; default `55` | +| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` | ## Timeout Chain -When fetching feeds from complex websites, multiple components work together. To prevent premature request termination, ensure your timeouts are configured in a "chain" where the outer layers wait longer than the inner layers. +When fetching feeds from complex websites, multiple components work together. Configure timeouts so outer layers wait longer than inner layers and the scraper exhausts its budget before the web tier aborts the request. -For an overall 30-second budget: +Default compose aligns **scrape total (45) ≤ feed build (50) ≤ Rack (55)**. The **work** budget (**30s**) applies on botasaurus-scrape-api only after the browser is ready; browser boot and queue time count against the **total** scrape wall (**45s**), not against `wait_timeout_seconds`. -1. **Botasaurus Scraper (`SCRAPE_TIMEOUT_SECONDS`)**: Set to **20s**. This is the time allowed for the browser to fetch and render the page. -2. **html2rss Gem (`HTML2RSS_TOTAL_TIMEOUT_SECONDS`)**: Set to **25s**. This allows the gem enough time to wait for Botasaurus and process the results. -3. **html2rss-web (`RACK_TIMEOUT_SERVICE_TIMEOUT`)**: Set to **30s**. This is the total time the web server will wait before killing the request. +1. **botasaurus-scrape-api (`SCRAPE_TIMEOUT_SECONDS`)**: **45s** handler wall (queue, boot, navigate, wait). +2. **botasaurus-scrape-api (`SCRAPE_WORK_TIMEOUT_SECONDS`)**: **30s** post-boot navigate, `wait_for_selector`, and scroll. +3. **html2rss client (`BOTASAURUS_SCRAPE_TIMEOUT_SECONDS`)**: **45s** Faraday cap for POST `/scrape` (mirrors scrape total). +4. **Feed YAML (`wait_timeout_seconds`)**: **1..30**, capped by `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` (default **30**). +5. **html2rss-web gem budget (`HTML2RSS_TOTAL_TIMEOUT_SECONDS`)**: **50s** for scrape plus extraction. +6. **html2rss-web (`RACK_TIMEOUT_SERVICE_TIMEOUT`)**: **55s** Rack outer wall. -Older environment-variable examples from previous docs revisions are obsolete. Use only the supported table above and the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md). +Set matching `SCRAPE_*` on botasaurus-scrape-api and `BOTASAURUS_SCRAPE_*` on html2rss-web when you override defaults. See the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md) for compose examples. diff --git a/src/content/docs/web-application/reference/monitoring.mdx b/src/content/docs/web-application/reference/monitoring.mdx index ddc31083..ebb0195c 100644 --- a/src/content/docs/web-application/reference/monitoring.mdx +++ b/src/content/docs/web-application/reference/monitoring.mdx @@ -7,11 +7,11 @@ import { Code } from "@astrojs/starlight/components"; ## Health Endpoints -| Endpoint | Auth | Purpose | -| --- | --- | --- | -| `GET /api/v1/health/live` | none | Process liveness | -| `GET /api/v1/health/ready` | none | Config readiness | -| `GET /api/v1/health` | bearer | Operator and uptime checks | +| Endpoint | Auth | Purpose | +| -------------------------- | ------ | -------------------------- | +| `GET /api/v1/health/live` | none | Process liveness | +| `GET /api/v1/health/ready` | none | Config readiness | +| `GET /api/v1/health` | bearer | Operator and uptime checks | Set `HEALTH_CHECK_TOKEN`, then call the authenticated endpoint: @@ -31,9 +31,9 @@ Set `SENTRY_DSN` to enable Issue capture in html2rss-web (Rack middleware and op Use separate Sentry projects for html2rss-web and botasaurus-scrape-api. Never share a DSN. -| Env var | Service | -| --- | --- | -| `SENTRY_DSN` | html2rss-web | +| Env var | Service | +| ----------------------- | --------------------- | +| `SENTRY_DSN` | html2rss-web | | `BOTASAURUS_SENTRY_DSN` | botasaurus-scrape-api | In `docker-compose.yml`, the botasaurus service maps `BOTASAURUS_SENTRY_DSN` into its own `SENTRY_DSN`, requires it at startup, and does not fall back to the web project's DSN.