crawling repositories

Trending repositories tagged crawling, ranked by trending score. Updated daily.

  1. 1
    D4Vinci/Scrapling

    ๐Ÿ•ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

    GitHub ยท Python ยท 104,598 stars ยท 10,941 forks ยท +468 stars in 24 hours ยท Trending score 5.28

  2. 2
    scrapy/scrapy

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    GitHub ยท Python ยท 65,511 stars ยท 12,162 forks ยท +47 stars in 24 hours ยท Trending score 3.40

  3. 3
    apify/crawlee

    Crawleeโ€”A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    GitHub ยท TypeScript ยท 26,505 stars ยท 1,774 forks ยท +40 stars in 24 hours ยท Trending score 3.23

  4. 4
    getmaxun/maxun

    Turn any website into a structured API. Extract, automate, search and monitor the web.

    GitHub ยท TypeScript ยท 18,532 stars ยท 1,596 forks ยท +28 stars in 24 hours ยท Trending score 3.20

  5. 5
    apify/crawlee-python

    Crawleeโ€”A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

    GitHub ยท Python ยท 9,680 stars ยท 837 forks ยท +3 stars in 24 hours ยท Trending score 1.59

  6. 6
    ai-robots-txt/ai.robots.txt

    A list of AI agents and robots to block.

    GitHub ยท Python ยท 4,264 stars ยท 204 forks ยท 0 stars in 24 hours ยท Trending score 1.08

  7. 7
    dondai44423/donsetch

    Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

    GitHub ยท Rust ยท 1,119 stars ยท 74 forks ยท 0 stars in 24 hours ยท Trending score 0.86

  8. 8
    janreges/siteone-crawler

    SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimizationโ€”ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

    GitHub ยท Rust ยท 1,017 stars ยท 98 forks ยท 0 stars in 24 hours ยท Trending score 0.75

  9. 9
    ArchiveBox/abx-dl

    โฌ‡๏ธ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). ๐ŸŽญ Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

    GitHub ยท Python ยท 153 stars ยท 10 forks ยท +1 stars in 24 hours ยท Trending score 0.63

  10. 10
    scrapfly/scrapfly-scrapers

    Scalable Python web scraping scripts for +80 popular domains

    GitHub ยท Python ยท 1,131 stars ยท 208 forks ยท 0 stars in 24 hours ยท Trending score 0.55

  11. 11
    wildovortex96622/Youtube-resocket-viewbot

    ๐ŸฆŠ Fast views botter based on proxies

    GitHub ยท 50 stars ยท 0 forks ยท 0 stars in 24 hours ยท Trending score 0.50

  12. 12
    adbar/courlan

    Clean, filter and sample URLs to optimize data collection โ€“ Python & command-line โ€“ Deduplication, spam, content and language filters

    GitHub ยท Python ยท 179 stars ยท 19 forks ยท 0 stars in 24 hours ยท Trending score 0.42

  13. 13
    xberg-io/crawlberg

    Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

    GitHub ยท Rust ยท 197 stars ยท 30 forks ยท 0 stars in 24 hours ยท Trending score 0.42

  14. 14
    hardkoded/puppeteer-sharp

    Headless Chrome .NET API

    GitHub ยท C# ยท 3,936 stars ยท 482 forks ยท 0 stars in 24 hours ยท Trending score 0.20

  15. 15
    MarshalX/telegram-crawler

    Tracks changes to Telegram's sites, beta apps and servers

    GitHub ยท Python ยท 369 stars ยท 51 forks ยท 0 stars in 24 hours ยท Trending score 0.10

  16. 16
    elixir-crawly/crawly

    Crawly, a high-level web crawling & scraping framework for Elixir.

    GitHub ยท Elixir ยท 1,129 stars ยท 127 forks ยท โˆ’2 stars in 24 hours ยท Trending score 0.10

  17. 17
    apify/actor-rag-web-browser

    RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text content scraped from the web.

    GitHub ยท TypeScript ยท 82 stars ยท 23 forks ยท 0 stars in 24 hours ยท Trending score 0.09

  18. 18
    apache/nutch

    Apache Nutch is an extensible and scalable web crawler

    GitHub ยท Java ยท 3,309 stars ยท 1,282 forks ยท 0 stars in 24 hours ยท Trending score 0.04

  19. 19
    webrecorder/browsertrix-crawler

    Run a high-fidelity browser-based web archiving crawler in a single Docker container

    GitHub ยท TypeScript ยท 1,190 stars ยท 155 forks ยท 0 stars in 24 hours ยท Trending score 0.03

  20. 20
    lorien/grab

    Web Scraping Framework

    GitHub ยท Python ยท 2,463 stars ยท 271 forks ยท 0 stars in 24 hours ยท Trending score 0.00

  21. 21
    kkrugler/flink-crawler

    Continuous scalable web crawler built on top of Flink and crawler-commons

    GitHub ยท Java ยท 53 stars ยท 18 forks ยท 0 stars in 24 hours ยท Trending score 0.00

  22. 22
    codelucas/newspaper

    newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

    GitHub ยท Python ยท 15,183 stars ยท 2,102 forks ยท 0 stars in 24 hours ยท Trending score 0.00

  23. 23
    pzaino/thecrowler

    an open, self-hosted framework for discovering, collecting, understanding, correlating, transforming, and acting on information from heterogeneous environments.

    GitHub ยท Go ยท 62 stars ยท 10 forks ยท 0 stars in 24 hours ยท Trending score 0.00

  24. 24
    bitmakerla/estela

    estela, an elastic web scraping cluster ๐Ÿ•ธ

    GitHub ยท TypeScript ยท 202 stars ยท 17 forks ยท 0 stars in 24 hours ยท Trending score 0.00