crawler repositories

Trending repositories tagged crawler, ranked by trending score. Updated daily.

  1. 1
    firecrawl/firecrawl

    Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.

    GitHub · TypeScript · 204,667 stars · 10,729 forks · +1,470 stars in 24 hours · Trending score 5.69

  2. 2
    D4Vinci/Scrapling

    🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

    GitHub · Python · 104,536 stars · 10,933 forks · +468 stars in 24 hours · Trending score 5.28

  3. 3
    unclecode/crawl4ai

    Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

    GitHub · Python · 87,585 stars · 9,140 forks · +87 stars in 24 hours · Trending score 3.84

  4. 4
    3441293738/creatorhub

    多平台内容监控·采集·搬运 一个 Web 面板管起抖音 / 小红书 / 快手 / 视频号 / TikTok / 哔哩哔哩 / Instagram / 海外小红书(rednote)

    GitHub · Python · 3,694 stars · 723 forks · +50 stars in 24 hours · Trending score 3.80

  5. 5
    scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    GitHub · Python · 65,507 stars · 12,163 forks · +47 stars in 24 hours · Trending score 3.40

  6. 6
    apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    GitHub · TypeScript · 26,505 stars · 1,774 forks · +40 stars in 24 hours · Trending score 3.23

  7. 7
    tiliondev/fortress

    Stealth Chromium engine that stops scrapers and browser agents from getting blocked, with one line of code change.

    GitHub · Python · 810 stars · 55 forks · +86 stars in 24 hours · Trending score 3.22

  8. 8
    getmaxun/maxun

    Turn any website into a structured API. Extract, automate, search and monitor the web.

    GitHub · TypeScript · 18,528 stars · 1,596 forks · +28 stars in 24 hours · Trending score 3.20

  9. 9
    CharlesPikachu/videodl

    Videodl: A lightweight video downloader written in pure python. (轻量级视频下载器,优先高清无水印,支持抖音,快手,小红书,B站,TikTok,YouTube,FIFA+,优酷,腾讯,爱奇艺,1905电影网,乐视,芒果,咪咕,PPTV,搜狐,Facebook,Twitter,新浪微博,今日头条,网易公开课,全民K歌,CCTV央视频,酷狗音乐MV,新片场,知乎,百度贴吧,TED等海量流媒体平台)

    GitHub · Python · 3,288 stars · 534 forks · +51 stars in 24 hours · Trending score 2.92

  10. 10
    projectdiscovery/katana

    A next-generation crawling and spidering framework.

    GitHub · Go · 17,876 stars · 1,237 forks · +15 stars in 24 hours · Trending score 2.56

  11. 11
    spider-rs/spider

    Foundational low latency web data collecting in Rust

    GitHub · Rust · 2,855 stars · 330 forks · +24 stars in 24 hours · Trending score 2.48

  12. 12
    fastcrw/crw

    Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

    GitHub · Rust · 1,279 stars · 102 forks · +14 stars in 24 hours · Trending score 2.30

  13. 13
    orangecoding/fredy

    ❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

    GitHub · JavaScript · 1,643 stars · 265 forks · +9 stars in 24 hours · Trending score 1.90

  14. 14
    CharlesPikachu/freeproxy

    FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理,支持爬取数十个免费代理分享源,支持自定义规则代理筛选,爬虫与数据分析必备,每日更新海量免费代理。)

    GitHub · Python · 592 stars · 103 forks · +12 stars in 24 hours · Trending score 1.86

  15. 15
    cv-cat/catbus

    一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用

    GitHub · TypeScript · 631 stars · 195 forks · +3 stars in 24 hours · Trending score 1.62

  16. 16
    apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

    GitHub · Python · 9,680 stars · 837 forks · +3 stars in 24 hours · Trending score 1.59

  17. 17
    0x676e67/wreq

    An ergonomic, privacy-aware Rust HTTP Client

    GitHub · Rust · 1,116 stars · 148 forks · +3 stars in 24 hours · Trending score 1.41

  18. 18
    SolitudeZY/OpinionLens

    舆见 OpinionLens · 社媒舆情监测与 AI 报告桌面端(Electron + FastAPI sidecar + MediaCrawler 运行时包装)。设置中心:邮件/企微/Bark 通知、多账号号池、OpenAI 兼容/DeepSeek/Anthropic 模型 API。

    GitHub · Python · 28 stars · 16 forks · +12 stars in 24 hours · Trending score 1.30

  19. 19
    PhiFever/AfdianToMarkdown

    爱发电爬虫(afdian.com)

    GitHub · Go · 81 stars · 16 forks · +4 stars in 24 hours · Trending score 1.30

  20. 20
    adbar/trafilatura

    Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

    GitHub · Python · 7,138 stars · 510 forks · 0 stars in 24 hours · Trending score 1.14

  21. 21
    NaiboWang/EasySpider

    A visual no-code/code-free web crawler/spider易采集:一个可视化浏览器自动化测试/数据采集/网页爬虫软件,可以无代码图形化的设计和执行爬虫任务。别名:ServiceWrapper面向Web应用的智能化服务封装系统。

    GitHub · JavaScript · 44,798 stars · 5,394 forks · +2 stars in 24 hours · Trending score 1.11

  22. 22
    alex-on-ai/WebReaper

    AI-native web scraper. Single binary with a bundled Claude Code skill. MIT-licensed alternative to Firecrawl.

    GitHub · C# · 153 stars · 33 forks · +4 stars in 24 hours · Trending score 1.05

  23. 23
    DedSecInside/TorBot

    Dark Web OSINT Tool

    GitHub · Python · 5,197 stars · 780 forks · 0 stars in 24 hours · Trending score 1.03

  24. 24
    dengmeiluan/ticket-monitoring

    ✈️ 自托管的多渠道机票价格监控:航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线(五洲目的地)同一套采集-判定-推送链路;数据留在本机,配置全在网页,开源免费。

    GitHub · Python · 526 stars · 70 forks · 0 stars in 24 hours · Trending score 1.02