web-scraping repositories

Trending repositories tagged web-scraping, ranked by trending score. Updated daily.

  1. 1
    firecrawl/firecrawl

    Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.

    GitHub · TypeScript · 204,946 stars · 10,743 forks · +1,470 stars in 24 hours · Trending score 5.69

  2. 2
    D4Vinci/Scrapling

    🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

    GitHub · Python · 104,598 stars · 10,941 forks · +468 stars in 24 hours · Trending score 5.28

  3. 3
    browser-use/browser-use

    Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

    GitHub · Python · 121,599 stars · 13,596 forks · +196 stars in 24 hours · Trending score 4.43

  4. 4
    lexmount/moli

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    GitHub · Rust · 24,146 stars · 694 forks · +161 stars in 24 hours · Trending score 4.19

  5. 5
    ProxyShard/ShardBrowser

    Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ShardX team (formerly ProxyShard). Engine-level fingerprint spoofing in Chromium 157 (WebGL / WebGPU / Client Hints / fonts / TLS), 220+ device profiles bundled, stable QUIC + WebRTC over SOCKS5.

    GitHub · TypeScript · 1,641 stars · 220 forks · +207 stars in 24 hours · Trending score 4.14

  6. 6
    JCodesMore/ai-website-cloner-template

    Clone any website with one command using AI coding agents

    GitHub · TypeScript · 39,542 stars · 5,894 forks · +134 stars in 24 hours · Trending score 4.10

  7. 7
    unclecode/crawl4ai

    Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

    GitHub · Python · 87,596 stars · 9,143 forks · +87 stars in 24 hours · Trending score 3.84

  8. 8
    cutetailbud/ff-ruyijs

    This is a lightweight, headless browser purpose-built for AI agents (such as Codex or Claude Code) to perform browser automation tasks; it is designed to navigate through various web protection systems, login processes, and reverse-engineering scenarios

    GitHub · C++ · 1,567 stars · 365 forks · +109 stars in 24 hours · Trending score 3.83

  9. 9
    3441293738/creatorhub

    多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手 / 视频号 / TikTok / 哔哩哔哩 / Instagram / 海外小红书(rednote)

    GitHub · Python · 3,708 stars · 726 forks · +50 stars in 24 hours · Trending score 3.80

  10. 10
    scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    GitHub · Python · 65,511 stars · 12,162 forks · +47 stars in 24 hours · Trending score 3.40

  11. 11
    jo-inc/camofox-browser

    Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

    GitHub · JavaScript · 12,424 stars · 1,270 forks · +54 stars in 24 hours · Trending score 3.34

  12. 12
    dgtlmoon/changedetection.io

    Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!

    GitHub · Python · 35,920 stars · 2,265 forks · +39 stars in 24 hours · Trending score 3.25

  13. 13
    apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    GitHub · TypeScript · 26,505 stars · 1,774 forks · +40 stars in 24 hours · Trending score 3.23

  14. 14
    tiliondev/fortress

    Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

    GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22

  15. 15
    getmaxun/maxun

    Turn any website into a structured API. Extract, automate, search and monitor the web.

    GitHub · TypeScript · 18,532 stars · 1,596 forks · +28 stars in 24 hours · Trending score 3.20

  16. 16
    CloakHQ/CloakBrowser

    Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.

    GitHub · Python · 32,998 stars · 2,741 forks · +22 stars in 24 hours · Trending score 2.85

  17. 17
    browserbase/stagehand

    The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.

    GitHub · TypeScript · 26,101 stars · 1,835 forks · +13 stars in 24 hours · Trending score 2.63

  18. 18
    Manavarya09/design-extract

    Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.

    GitHub · HTML · 5,147 stars · 427 forks · +18 stars in 24 hours · Trending score 2.63

  19. 19
    spider-rs/spider

    Foundational low latency web data collecting in Rust

    GitHub · Rust · 2,855 stars · 330 forks · +24 stars in 24 hours · Trending score 2.48

  20. 20
    firecrawl/firecrawl-mcp-server

    Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

    GitHub · JavaScript · 7,860 stars · 939 forks · +13 stars in 24 hours · Trending score 2.43

  21. 21
    Vinyzu/cursory

    Generate 100% human-realistic Mouse Trajectories with Timings.

    GitHub · Python · 192 stars · 21 forks · +29 stars in 24 hours · Trending score 2.30

  22. 22
    fastcrw/crw

    Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

    GitHub · Rust · 1,279 stars · 102 forks · +14 stars in 24 hours · Trending score 2.30

  23. 23
    oss-invisible/playwright

    invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.

    GitHub · Python · 2,610 stars · 279 forks · +16 stars in 24 hours · Trending score 2.26

  24. 24
    vsmutok/FuckCaptcha

    Open field guide to 39 CAPTCHA types: identification signals, free and open-source bypass methods, and paid alternatives. Includes a searchable web version.

    GitHub · HTML · 158 stars · 17 forks · +14 stars in 24 hours · Trending score 2.24