web-scraping repositories
Trending repositories tagged web-scraping, ranked by trending score. Updated daily.
-
1
firecrawl/firecrawl
Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.
GitHub · TypeScript · 204,946 stars · 10,743 forks · +1,470 stars in 24 hours · Trending score 5.69
-
2
D4Vinci/Scrapling
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
GitHub · Python · 104,598 stars · 10,941 forks · +468 stars in 24 hours · Trending score 5.28
-
3
browser-use/browser-use
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
GitHub · Python · 121,599 stars · 13,596 forks · +196 stars in 24 hours · Trending score 4.43
-
4
lexmount/moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
GitHub · Rust · 24,146 stars · 694 forks · +161 stars in 24 hours · Trending score 4.19
-
5
ProxyShard/ShardBrowser
Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ShardX team (formerly ProxyShard). Engine-level fingerprint spoofing in Chromium 157 (WebGL / WebGPU / Client Hints / fonts / TLS), 220+ device profiles bundled, stable QUIC + WebRTC over SOCKS5.
GitHub · TypeScript · 1,641 stars · 220 forks · +207 stars in 24 hours · Trending score 4.14
-
6
JCodesMore/ai-website-cloner-template
Clone any website with one command using AI coding agents
GitHub · TypeScript · 39,542 stars · 5,894 forks · +134 stars in 24 hours · Trending score 4.10
-
7
unclecode/crawl4ai
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
GitHub · Python · 87,596 stars · 9,143 forks · +87 stars in 24 hours · Trending score 3.84
-
8
cutetailbud/ff-ruyijs
This is a lightweight, headless browser purpose-built for AI agents (such as Codex or Claude Code) to perform browser automation tasks; it is designed to navigate through various web protection systems, login processes, and reverse-engineering scenarios
GitHub · C++ · 1,567 stars · 365 forks · +109 stars in 24 hours · Trending score 3.83
-
9
3441293738/creatorhub
多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手 / 视频号 / TikTok / 哔哩哔哩 / Instagram / 海外小红书(rednote)
GitHub · Python · 3,708 stars · 726 forks · +50 stars in 24 hours · Trending score 3.80
-
10
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
GitHub · Python · 65,511 stars · 12,162 forks · +47 stars in 24 hours · Trending score 3.40
-
11
jo-inc/camofox-browser
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
GitHub · JavaScript · 12,424 stars · 1,270 forks · +54 stars in 24 hours · Trending score 3.34
-
12
dgtlmoon/changedetection.io
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
GitHub · Python · 35,920 stars · 2,265 forks · +39 stars in 24 hours · Trending score 3.25
-
13
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
GitHub · TypeScript · 26,505 stars · 1,774 forks · +40 stars in 24 hours · Trending score 3.23
-
14
tiliondev/fortress
Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.
GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22
-
15
getmaxun/maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
GitHub · TypeScript · 18,532 stars · 1,596 forks · +28 stars in 24 hours · Trending score 3.20
-
16
CloakHQ/CloakBrowser
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
GitHub · Python · 32,998 stars · 2,741 forks · +22 stars in 24 hours · Trending score 2.85
-
17
browserbase/stagehand
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
GitHub · TypeScript · 26,101 stars · 1,835 forks · +13 stars in 24 hours · Trending score 2.63
-
18
Manavarya09/design-extract
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.
GitHub · HTML · 5,147 stars · 427 forks · +18 stars in 24 hours · Trending score 2.63
-
19
spider-rs/spider
Foundational low latency web data collecting in Rust
GitHub · Rust · 2,855 stars · 330 forks · +24 stars in 24 hours · Trending score 2.48
-
20
firecrawl/firecrawl-mcp-server
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
GitHub · JavaScript · 7,860 stars · 939 forks · +13 stars in 24 hours · Trending score 2.43
-
21
Vinyzu/cursory
Generate 100% human-realistic Mouse Trajectories with Timings.
GitHub · Python · 192 stars · 21 forks · +29 stars in 24 hours · Trending score 2.30
-
22
fastcrw/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
GitHub · Rust · 1,279 stars · 102 forks · +14 stars in 24 hours · Trending score 2.30
-
23
oss-invisible/playwright
invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
GitHub · Python · 2,610 stars · 279 forks · +16 stars in 24 hours · Trending score 2.26
-
24
vsmutok/FuckCaptcha
Open field guide to 39 CAPTCHA types: identification signals, free and open-source bypass methods, and paid alternatives. Includes a searchable web version.
GitHub · HTML · 158 stars · 17 forks · +14 stars in 24 hours · Trending score 2.24