crawlers repositories

Trending repositories tagged crawlers, ranked by trending score. Updated daily.

  1. 1
    mintlify/agent-traffic

    Tell agents, AI crawlers, search crawlers and bots apart from people

    GitHub · TypeScript · 87 stars · 4 forks · +4 stars in 24 hours · Trending score 1.43

  2. 2
    ai-robots-txt/ai.robots.txt

    A list of AI agents and robots to block.

    GitHub · Python · 4,264 stars · 204 forks · 0 stars in 24 hours · Trending score 1.07

  3. 3
    D4vRAM369/annas-archive-hub

    Self-updating domain tracker for Anna's Archive with real-time verification and IPFS publishing. Never lose access again.

    GitHub · Python · 11 stars · 0 forks · +2 stars in 24 hours · Trending score 0.82

  4. 4
    edwardtay/awesome-scrapers

    100+ web scraping tools sorted by job: AI extraction, stealth browsers, MCP servers for agents, proxies and CAPTCHA solving. Dead tools listed with what to use instead.

    GitHub · 24 stars · 20 forks · 0 stars in 24 hours · Trending score 0.54

  5. 5
    scrapfly/Antibot-Detector

    Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

    GitHub · JavaScript · 632 stars · 74 forks · 0 stars in 24 hours · Trending score 0.45

  6. 6
    Norconex/crawler

    Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

    GitHub · Java · 205 stars · 71 forks · 0 stars in 24 hours · Trending score 0.09

  7. 7
    Intuned/cookbook

    A cookbook of examples for building browser automations with Intuned

    GitHub · Python · 32 stars · 4 forks · 0 stars in 24 hours · Trending score 0.01

  8. 8
    mascanho/RustySEO-Releases

    Repo for RustySEO Releases, A FREE SE0/GEO Toolkit

    GitHub · 14 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00

  9. 9
    Potelo/laravel-block-bots

    Block crawlers and high traffic users on your site by IP using Redis

    GitHub · PHP · 55 stars · 18 forks · 0 stars in 24 hours · Trending score 0.00

  10. 10
    ArchiveTeam/wget-lua

    Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

    GitHub · C · 139 stars · 19 forks · 0 stars in 24 hours · Trending score 0.00