scraping repositories

Trending repositories tagged scraping, ranked by trending score. Updated daily.

  1. 1
    firecrawl/firecrawl

    Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.

    GitHub · TypeScript · 204,667 stars · 10,729 forks · +1,470 stars in 24 hours · Trending score 5.69

  2. 2
    D4Vinci/Scrapling

    🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

    GitHub · Python · 104,536 stars · 10,933 forks · +468 stars in 24 hours · Trending score 5.28

  3. 3
    soxoj/maigret

    🕵️‍♂️ Collect a dossier on a person by username from 6K websites

    GitHub · Python · 41,080 stars · 3,322 forks · +193 stars in 24 hours · Trending score 4.41

  4. 4
    scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    GitHub · Python · 65,507 stars · 12,163 forks · +47 stars in 24 hours · Trending score 3.40

  5. 5
    jo-inc/camofox-browser

    Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.

    GitHub · JavaScript · 12,420 stars · 1,268 forks · +54 stars in 24 hours · Trending score 3.34

  6. 6
    apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    GitHub · TypeScript · 26,505 stars · 1,774 forks · +40 stars in 24 hours · Trending score 3.23

  7. 7
    tiliondev/fortress

    Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

    GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22

  8. 8
    spider-rs/spider

    Foundational low latency web data collecting in Rust

    GitHub · Rust · 2,855 stars · 330 forks · +24 stars in 24 hours · Trending score 2.48

  9. 9
    oss-invisible/playwright

    invisible playwright: Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.

    GitHub · Python · 2,607 stars · 279 forks · +16 stars in 24 hours · Trending score 2.26

  10. 10
    soxoj/socid-extractor

    ⛏️ The extraction engine behind Maigret: turn any profile URL into a structured OSINT record across 150+ sites

    GitHub · Python · 1,141 stars · 133 forks · +15 stars in 24 hours · Trending score 2.15

  11. 11
    proxifly/free-proxy-list

    🚀 Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)

    GitHub · 7,069 stars · 804 forks · +11 stars in 24 hours · Trending score 1.91

  12. 12
    apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

    GitHub · Python · 9,680 stars · 837 forks · +3 stars in 24 hours · Trending score 1.59

  13. 13
    ymstar/dyhub

    抖音直播弹幕中台 (Douyin Live Danmaku Hub):双采集内核(浏览器+CDP / 轻量直连)、多房间并发、WS/SSE/Webhook 三通道分发、可视化控制台、Docker 一键部署

    GitHub · TypeScript · 60 stars · 20 forks · +5 stars in 24 hours · Trending score 1.52

  14. 14
    symfony/panther

    A browser testing and web crawling library for PHP and Symfony

    GitHub · PHP · 3,074 stars · 237 forks · +6 stars in 24 hours · Trending score 1.46

  15. 15
    h4ckf0r0day/awesome-ai-web-scraping

    A curated list of AI-powered web scraping tools, LLM-friendly crawlers, MCP servers, and infrastructure for turning the web into data.

    GitHub · 131 stars · 71 forks · 0 stars in 24 hours · Trending score 1.41

  16. 16
    jikan-me/jikan-rest

    The REST API for Jikan

    GitHub · PHP · 571 stars · 300 forks · +5 stars in 24 hours · Trending score 1.26

  17. 17
    adbar/trafilatura

    Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

    GitHub · Python · 7,138 stars · 510 forks · 0 stars in 24 hours · Trending score 1.14

  18. 18
    iiab/iiab

    Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !

    GitHub · Jinja · 2,040 stars · 154 forks · 0 stars in 24 hours · Trending score 1.07

  19. 19
    Jieyab89/OSINT-Cheat-sheet

    OSINT cheat sheet, list OSINT tools, wiki, dataset, article, book , red team OSINT for hackers and OSINT tips and OSINT branch. This repository will grow every time will research, there is a research, science and technology, tutorial. Please use it wisely.

    GitHub · HTML · 2,327 stars · 400 forks · 0 stars in 24 hours · Trending score 1.06

  20. 20
    yaroslav2901/OE_OUTAGE_DATA

    Репозиторій містить JSON-дані про відключення електроенергії та згенеровані зображення розкладів для кожної групи та загальне зображення для всіх груп.

    GitHub · 40 stars · 2 forks · +3 stars in 24 hours · Trending score 1.05

  21. 21
    alex-on-ai/WebReaper

    AI-native web scraper. Single binary with a bundled Claude Code skill. MIT-licensed alternative to Firecrawl.

    GitHub · C# · 153 stars · 33 forks · +4 stars in 24 hours · Trending score 1.05

  22. 22
    sqdshguy/wreq-js

    HTTP client for Node.js with browser TLS fingerprint impersonation

    GitHub · TypeScript · 460 stars · 20 forks · 0 stars in 24 hours · Trending score 1.03

  23. 23
    TulipFarm/huntsman

    Self-hosted web scraper that turns any web page into clean Markdown. Firecrawl-compatible API, powered by Lightpanda.

    GitHub · Go · 15 stars · 1 forks · +2 stars in 24 hours · Trending score 0.96

  24. 24
    any4ai/AnyCrawl

    AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

    GitHub · TypeScript · 3,528 stars · 372 forks · 0 stars in 24 hours · Trending score 0.96