web-crawler repositories

Trending repositories tagged web-crawler, ranked by trending score. Updated daily.

  1. 1
    firecrawl/firecrawl

    Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

    GitHub · TypeScript · 205,170 stars · 10,743 forks · +2,082 stars in 24 hours · Trending score 5.92

  2. 2
    lexmount/moli

    Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

    GitHub · Rust · 24,158 stars · 694 forks · +161 stars in 24 hours · Trending score 4.19

  3. 3
    unclecode/crawl4ai

    Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

    GitHub · Python · 87,599 stars · 9,140 forks · +87 stars in 24 hours · Trending score 3.84

  4. 4
    apify/crawlee

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

    GitHub · TypeScript · 26,505 stars · 1,774 forks · +40 stars in 24 hours · Trending score 3.23

  5. 5
    spider-rs/spider

    Foundational low latency web data collecting in Rust

    GitHub · Rust · 2,855 stars · 331 forks · +24 stars in 24 hours · Trending score 2.48

  6. 6
    firecrawl/firecrawl-mcp-server

    Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

    GitHub · JavaScript · 7,860 stars · 939 forks · +13 stars in 24 hours · Trending score 2.43

  7. 7
    fastcrw/crw

    Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

    GitHub · Rust · 1,281 stars · 102 forks · +14 stars in 24 hours · Trending score 2.30

  8. 8
    YOLOW92/pagesift

    Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.

    GitHub · Rust · 284 stars · 37 forks · +5 stars in 24 hours · Trending score 1.71

  9. 9
    apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

    GitHub · Python · 9,680 stars · 837 forks · +3 stars in 24 hours · Trending score 1.59

  10. 10
    MarginaliaSearch/MarginaliaSearch

    Internet search engine for text-oriented websites. Indexing the small, old and weird web.

    GitHub · Java · 2,241 stars · 68 forks · +2 stars in 24 hours · Trending score 1.18

  11. 11
    Goldziher/basemind

    Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

    GitHub · Rust · 124 stars · 24 forks · +1 stars in 24 hours · Trending score 1.02

  12. 12
    KnockOutEZ/wigolo

    The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

    GitHub · TypeScript · 5,559 stars · 462 forks · 0 stars in 24 hours · Trending score 0.82

  13. 13
    Ayyouboss0011/SherlockMaps

    Powerful Google Maps Crawler / Scraper tool with REST API, Docker support & multi-format export

    GitHub · JavaScript · 96 stars · 12 forks · 0 stars in 24 hours · Trending score 0.60

  14. 14
    wojcikdominik/openspider

    The crawler for the agentic era – open-source website crawler & technical SEO auditor for AI agents and humans (macOS, Windows). Rust + Tauri, MCP-native.

    GitHub · Rust · 47 stars · 1 forks · 0 stars in 24 hours · Trending score 0.58

  15. 15
    andalabx/ember

    Lightweight headless browser for AI agents

    GitHub · Python · 10 stars · 2 forks · 0 stars in 24 hours · Trending score 0.57

  16. 16
    netweir/netweir

    An ultra-fast, self-healing, undetectable web scraper. Rust does the work; you write Python.

    GitHub · Rust · 8 stars · 4 forks · +1 stars in 24 hours · Trending score 0.56

  17. 17
    scrapfly/scrapfly-scrapers

    Scalable Python web scraping scripts for +80 popular domains

    GitHub · Python · 1,131 stars · 207 forks · 0 stars in 24 hours · Trending score 0.55

  18. 18
    gildas-lormeau/single-file-cli

    CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

    GitHub · JavaScript · 1,697 stars · 147 forks · 0 stars in 24 hours · Trending score 0.53

  19. 19
    0xMassi/webclaw

    Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

    GitHub · Rust · 2,441 stars · 245 forks · −4 stars in 24 hours · Trending score 0.53

  20. 20
    Adityavardhanjain/WikiCrawl

    Interactive Wikipedia knowledge graph - crawl topics, visualize connections, and explore unexpected paths.

    GitHub · TypeScript · 29 stars · 3 forks · 0 stars in 24 hours · Trending score 0.50

  21. 21
    realsirjoe/relinos

    Organizing the web

    GitHub · Python · 56 stars · 5 forks · 0 stars in 24 hours · Trending score 0.47

  22. 22
    xberg-io/crawlberg

    Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

    GitHub · Rust · 197 stars · 30 forks · 0 stars in 24 hours · Trending score 0.42

  23. 23
    Sriram-PR/doc-scraper

    Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

    GitHub · Go · 110 stars · 25 forks · 0 stars in 24 hours · Trending score 0.38

  24. 24
    ilien-dev/svipall

    Local-first MCP server and CLI in Rust: any page as LLM-ready Markdown, whole-site crawls, keyless search, and local captcha solving. No cloud, no API keys.

    GitHub · Rust · 22 stars · 2 forks · 0 stars in 24 hours · Trending score 0.36