From 54ed5c71a80e5ab871ec910266cf33df4196c844 Mon Sep 17 00:00:00 2001 From: hmishra2250 Date: Tue, 14 Jul 2026 18:05:47 +0530 Subject: [PATCH 1/2] Document scrape freshness and cache metadata --- skills/firecrawl-scrape/SKILL.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/skills/firecrawl-scrape/SKILL.md b/skills/firecrawl-scrape/SKILL.md index 6c6612756c..4ded790c94 100644 --- a/skills/firecrawl-scrape/SKILL.md +++ b/skills/firecrawl-scrape/SKILL.md @@ -51,8 +51,18 @@ firecrawl scrape "https://example.com/pricing" --query "What is the enterprise p | `--include-tags ` | Only include these HTML tags | | `--exclude-tags ` | Exclude these HTML tags | | `--redact-pii` | Redact personally identifiable information from output | +| `--max-age ` | Maximum age of cached content. Use `0` for final freshness checks | +| `--json` | Emit JSON so metadata such as `metadata.cache` can be inspected | | `-o, --output ` | Output file path | + +## Freshness and cache provenance + +- Use `--max-age 0` when the scrape is a final verification before an action that depends on current state. It bypasses Firecrawl cache and is slower/more failure-prone than allowing cache. +- Use `--json` when you need to inspect metadata. An attested Firecrawl-index cache hit includes `metadata.cache.source = "firecrawl-index"` and `metadata.cache.cachedAt`. +- If `metadata.cache` is absent, do not infer miss, fresh scrape, bypass, or source-page liveness. It only means cache provenance was not attested. +- For job postings, event pages, inventory, and other stateful records, verify the business object itself before acting: prefer an ATS/source API when available, or check for active apply buttons, closed labels, canonical IDs, and redirects in the fresh content. HTTP 200 and rendered HTML are not enough. + ## Tips - **Prefer plain scrape over `--query`.** Scrape to a file, then use `grep`, `head`, or read the markdown directly — you can search and reason over the full content yourself. Use `--query` only when you want a single targeted answer without saving the page (costs 5 extra credits). From b9e42db6dd4b3b384e93e497a96d84a9ed0cf0bc Mon Sep 17 00:00:00 2001 From: hmishra2250 Date: Tue, 14 Jul 2026 23:08:26 +0530 Subject: [PATCH 2/2] Clarify CLI freshness semantics --- skills/firecrawl-scrape/SKILL.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/skills/firecrawl-scrape/SKILL.md b/skills/firecrawl-scrape/SKILL.md index 4ded790c94..a6814a8a52 100644 --- a/skills/firecrawl-scrape/SKILL.md +++ b/skills/firecrawl-scrape/SKILL.md @@ -51,17 +51,16 @@ firecrawl scrape "https://example.com/pricing" --query "What is the enterprise p | `--include-tags ` | Only include these HTML tags | | `--exclude-tags ` | Exclude these HTML tags | | `--redact-pii` | Redact personally identifiable information from output | -| `--max-age ` | Maximum age of cached content. Use `0` for final freshness checks | +| `--max-age ` | Maximum age of cached content. Use `0` to request fresh retrieval | | `--json` | Emit JSON so metadata such as `metadata.cache` can be inspected | | `-o, --output ` | Output file path | - ## Freshness and cache provenance -- Use `--max-age 0` when the scrape is a final verification before an action that depends on current state. It bypasses Firecrawl cache and is slower/more failure-prone than allowing cache. +- Use `--max-age 0` to request a fresh retrieval path before an action that depends on current state. It skips Firecrawl's index cache and can be slower or more failure-prone than allowing cache. - Use `--json` when you need to inspect metadata. An attested Firecrawl-index cache hit includes `metadata.cache.source = "firecrawl-index"` and `metadata.cache.cachedAt`. - If `metadata.cache` is absent, do not infer miss, fresh scrape, bypass, or source-page liveness. It only means cache provenance was not attested. -- For job postings, event pages, inventory, and other stateful records, verify the business object itself before acting: prefer an ATS/source API when available, or check for active apply buttons, closed labels, canonical IDs, and redirects in the fresh content. HTTP 200 and rendered HTML are not enough. +- For job postings, event pages, inventory, and other stateful records, verify the business object itself before acting: prefer an ATS/source API when available, or check the returned content for active apply buttons, closed labels, canonical IDs, and redirects. HTTP 200 and rendered HTML are not enough. ## Tips