diff --git a/docker-compose.yml b/docker-compose.yml index 3690a0a6..c098fb03 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,8 +20,10 @@ services: # Web Sentry (Project A): SENTRY_DSN; logs opt-in via SENTRY_ENABLE_LOGS=true SENTRY_DSN: ${SENTRY_DSN:-} SENTRY_ENABLE_LOGS: ${SENTRY_ENABLE_LOGS:-false} - HTML2RSS_TOTAL_TIMEOUT_SECONDS: 25 - RACK_TIMEOUT_SERVICE_TIMEOUT: 30 + HTML2RSS_TOTAL_TIMEOUT_SECONDS: 50 + RACK_TIMEOUT_SERVICE_TIMEOUT: 55 + BOTASAURUS_SCRAPE_TIMEOUT_SECONDS: 45 + BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS: 30 BOTASAURUS_SCRAPER_URL: http://botasaurus:4010 # Trial runs use the image's bundled config/feeds.yml. # Uncomment the block below when you want to replace it with your own file. @@ -47,5 +49,7 @@ services: SENTRY_DSN: ${BOTASAURUS_SENTRY_DSN:?set BOTASAURUS_SENTRY_DSN} SENTRY_ENVIRONMENT: ${ENVIRONMENT:-production} SENTRY_RELEASE: ${GIT_SHA:-unknown} + SCRAPE_TIMEOUT_SECONDS: 45 + SCRAPE_WORK_TIMEOUT_SECONDS: 30 ports: - "127.0.0.1:4010:4010" diff --git a/docs/README.md b/docs/README.md index 7a552c4b..480eb526 100644 --- a/docs/README.md +++ b/docs/README.md @@ -226,6 +226,21 @@ Use separate Sentry projects for html2rss-web and botasaurus-scrape-api. Never s Compose requires `BOTASAURUS_SENTRY_DSN` for botasaurus and does not fall back to web `SENTRY_DSN`. +### Compose timeout ladder + +Default `docker-compose.yml` aligns botasaurus-scrape-api, the html2rss gem client, and html2rss-web so the scraper exhausts its budget before the web tier aborts the request. Keep **scrape total (45) ≤ feed build (50) ≤ Rack (55)**; the **work** budget (30) applies only after the browser is ready on the scraper. + +| Variable | Service | Default | Role | +| --- | --- | --- | --- | +| `SCRAPE_TIMEOUT_SECONDS` | botasaurus-scrape-api | `45` | Handler wall (queue, boot, navigate, wait) | +| `SCRAPE_WORK_TIMEOUT_SECONDS` | botasaurus-scrape-api | `30` | Post-boot navigate, selector wait, scroll | +| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | html2rss (web) | `45` | Faraday POST `/scrape` cap (mirrors scrape total) | +| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | html2rss | `30` | Max `wait_timeout_seconds` in feed YAML | +| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | html2rss-web | `50` | Feed build budget (scrape + extraction) | +| `RACK_TIMEOUT_SERVICE_TIMEOUT` | html2rss-web | `55` | Rack outer wall | + +When triaging `GATEWAY_TIMEOUT` or `error_category:timeout`, confirm both projects use this ladder. Scraper terminal timeouts near **45s** with web failures near **50–55s** indicate aligned budgets; scraper failures near **20–25s** while web waits longer usually mean stale `SCRAPE_*` / `BOTASAURUS_*` env on one side. + ## Sentry Runbook With `SENTRY_DSN` set, html2rss-web sends unhandled exceptions (Rack middleware) and P0 operational failures (`SentryOps`) to Sentry Issues. Release is `BUILD_TAG+GIT_SHA`; environment is `RACK_ENV`. @@ -242,7 +257,7 @@ Start triage from the newest `feed.create`, `feed.render`, and `request.error` e - `SCRAPER_UNAVAILABLE` / `navigation_error` → scraper down or unreachable - `challenge_block` (scraper) or `BLOCKED_SURFACE` (web) → site blocked automation (product signal) - `EXTRACTION_EMPTY` → selectors/config (product signal) - - `GATEWAY_TIMEOUT` / `timeout` → slow or unreachable target + - `GATEWAY_TIMEOUT` / `timeout` → slow target, or timeout ladder mismatch (see **Compose timeout ladder**) ### Alert baselines