#

webscraping

Web scraping is the process of programmatically retrieving web pages and extracting structured data from them for analysis, storage, or reuse. It powers price monitoring, search indexing, market research, and training-data collection.

Scraping ranges from plain HTTP requests and HTML parsing to full browser automation for JavaScript-rendered and anti-bot-protected pages. The ecosystem includes request libraries, HTML parsers, headless browsers, and cloud extraction platforms.

Here are 10,456 public repositories matching this topic...

firecrawl

firecrawl / firecrawl

The context API to search, scrape, and interact with the web at scale. 🔥

markdown crawler scraper ai html-to-markdown web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction ai-agents web-search ai-search web-data llm ai-crawler ai-scraping

Updated Sep 7, 2026
TypeScript

Scrapling

D4Vinci / Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Updated Sep 4, 2026
Python

huginn / huginn

Create agents that monitor and act on your behalf. Your agents are standing by!

notifications agent rss scraper automation twitter monitoring huginn feed feedgenerator webscraping twitter-streaming

Updated Sep 7, 2026
Ruby

CloakHQ / CloakBrowser

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 30/30 tests passed.

Updated Sep 6, 2026
Python

ScrapeGraphAI / Scrapegraph-ai

Python scraper based on AI

Updated Sep 7, 2026
Python

gpt-researcher

assafelovic / gpt-researcher

An autonomous agent that conducts deep research on any data using any LLM providers

python search agent automation research ai mcp webscraping llms mcp-server deepresearch

Updated Aug 27, 2026
Python

getmaxun / maxun

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

api crawler scraper automation crawling web-scraper self-hosted web-scraping data-extraction webscraping agents browser-automation no-code web-search rpa robotic-process-automation nocode playwright

Updated Sep 7, 2026
TypeScript

pystardust / ani-cli

A cli tool to browse and play anime

windows linux shell cli mac terminal anime fzf posix rofi termux webscraping syncplay steamdeck

Updated Sep 5, 2026
Shell

seleniumbase / SeleniumBase

APIs for browser automation, testing, and bypassing bot-detection. Includes CDP Mode: A stealthy configuration for chromium that passes every bot detection test.

Updated Sep 3, 2026
Python

daijro / camoufox

🦊 Anti-detect browser

firefox networking scraping fingerprint webscraping playwright antidetect antidetect-browser

Updated Sep 6, 2026
C++

lorien / awesome-web-scraping

List of libraries, tools and APIs for web scraping and data processing.

crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python

Updated Sep 2, 2026
JavaScript

autoscraper

alirezamika / autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

Updated Jul 29, 2026
Python

niespodd / browser-fingerprinting

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

bot crawler scraper automation recaptcha detection fingerprinting browser-fingerprinting stealth webscraping perimeterx bot-detection chromium-browser abck x-kpsdk-ct arkose reese84 aws-waf-token

Updated Jul 27, 2026
JavaScript

jaypyles / Scraperr

Self-hosted webscraper.

python docker kubernetes opensource helm scraping webscraper web-scraper self-hosted web-scraping web-scrapers webscraping playwright

Updated Oct 12, 2025
TypeScript

patchright

Kaliiiiiiiiii-Vinyzu / patchright

Undetected version of the Playwright testing and automation library.

Updated Sep 4, 2026
TypeScript

itsOwen / CyberScraper-2077

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

scraper web-scraper openai webscraping gemini-api llm llm-scraper ai-scraping

Updated Aug 20, 2026
Python

GodsScion / Auto_job_applier_linkedIn

Make your job hunt easy by automating your application process with this Auto Applier

python automation linkedin selenium python3 job-application webscraping job-search automation-selenium selenium-python easy-apply auto-apply linkedin-job-scraper undetected-chromedriver linkedin-jobs-scraper automatic-job-applier easy-apply-jobs

Updated Aug 10, 2026
Python

anaskhan96 / soup

Web Scraper in Go, similar to BeautifulSoup

go golang webscraper web-scraper beautifulsoup webscraping html-node

Updated Aug 1, 2026
Go

reworkd / tarsier

Vision utilities for web interaction agents 👀

python ocr selenium webscraping pypi-package playwright llms gpt4v

Updated Nov 25, 2024
Jupyter Notebook

TheWebScrapingClub / webscraping-from-0-to-hero

The web scraping open project repository aims to share knowledge and experiences about web scraping with Python

python scrapy-spider scrapy webscraping scrapysplash playwright

Updated May 27, 2024

Followers: 342 followers
Website: github.com/topics/web-scraping

Related topics

browser-automation headless-browser playwright puppeteer selenium