Skip to content

feat: add scrapeunblocker plugin#2732

Open
Kontuzijus wants to merge 1 commit into
langgenius:mainfrom
ScrapeUnblocker:scrapeunblocker-plugin
Open

feat: add scrapeunblocker plugin#2732
Kontuzijus wants to merge 1 commit into
langgenius:mainfrom
ScrapeUnblocker:scrapeunblocker-plugin

Conversation

@Kontuzijus

Copy link
Copy Markdown

Plugin Information

  • Author: scrapeunblocker
  • Plugin name: scrapeunblocker
  • Version: 0.0.1
  • Source repository: source is included in this PR under scrapeunblocker/scrapeunblocker/
  • Contact: https://www.scrapeunblocker.com

Submission type

  • New plugin

Description

ScrapeUnblocker renders web pages in a real browser behind anti-bot protections
(Cloudflare, DataDome, PerimeterX, Akamai) and returns the result to a Dify
workflow. It is useful when an ordinary HTTP request returns a block page, a
captcha, or an empty shell instead of content.

Two tools:

  • Get Page Source - fetch a single page. Returns rendered HTML, or AI-parsed
    structured JSON when parsed_data is set. Supports proxy_country for
    geo-restricted content.
  • Search Google - scrape a Google SERP for a keyword and return the organic
    results (title, url, description, position).

Risk level

  • High risk

This plugin proxies and fetches arbitrary user-supplied URLs, so it falls under
the high-risk category by your review guidelines. Specifically:

  • Arbitrary URL fetching is the plugin's purpose. Requests are made by
    ScrapeUnblocker's infrastructure, not by the Dify host, so the target site
    sees a ScrapeUnblocker IP.
  • Timeouts are explicit on every request (requests.post(..., timeout=...),
    default 180s, configurable via the timeout_seconds credential). Nothing can
    hang indefinitely.
  • The API key is never returned in errors. Network failures raise only the
    exception type and the path; HTTP failures raise only the status code and
    path. Request headers, which carry the key, are never interpolated into
    messages.
  • Data flow is documented in PRIVACY.md: the target URL (including its
    query string), the optional country code, the search keyword, and the API key
    are what leave the Dify instance; the fetched page content passes through
    ScrapeUnblocker's servers. It also warns against putting secrets in target
    URLs.

Verification

The client was exercised against the live ScrapeUnblocker API before packaging:
get_page_source returned a fully rendered page (58,917 chars, correct title),
search_google returned 20 organic results with the documented fields, and
validate_credentials succeeded. An invalid key correctly raised a credential
error, and the test asserted the key does not appear in the error text.

Provider and tool classes were imported against dify_plugin 0.9.x to confirm
the SDK surface used here exists. Package built with dify plugin package
(CLI v0.6.4); the archive contains 14 files and no build artefacts.

Scrape pages behind anti-bot protections and read Google search
results via the ScrapeUnblocker API.
@github-actions github-actions Bot added the risk: high High-risk Marketplace submission label Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: high High-risk Marketplace submission

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant