crawlers repositories
Trending repositories tagged crawlers, ranked by trending score. Updated daily.
-
1
mintlify/agent-traffic
Tell agents, AI crawlers, search crawlers and bots apart from people
GitHub · TypeScript · 87 stars · 4 forks · +4 stars in 24 hours · Trending score 1.43
-
2
ai-robots-txt/ai.robots.txt
A list of AI agents and robots to block.
GitHub · Python · 4,264 stars · 204 forks · 0 stars in 24 hours · Trending score 1.07
-
3
D4vRAM369/annas-archive-hub
Self-updating domain tracker for Anna's Archive with real-time verification and IPFS publishing. Never lose access again.
GitHub · Python · 11 stars · 0 forks · +2 stars in 24 hours · Trending score 0.82
-
4
edwardtay/awesome-scrapers
100+ web scraping tools sorted by job: AI extraction, stealth browsers, MCP servers for agents, proxies and CAPTCHA solving. Dead tools listed with what to use instead.
GitHub · 24 stars · 20 forks · 0 stars in 24 hours · Trending score 0.54
-
5
scrapfly/Antibot-Detector
Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.
GitHub · JavaScript · 632 stars · 74 forks · 0 stars in 24 hours · Trending score 0.45
-
6
Norconex/crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
GitHub · Java · 205 stars · 71 forks · 0 stars in 24 hours · Trending score 0.09
-
7
Intuned/cookbook
A cookbook of examples for building browser automations with Intuned
GitHub · Python · 32 stars · 4 forks · 0 stars in 24 hours · Trending score 0.01
-
8
mascanho/RustySEO-Releases
Repo for RustySEO Releases, A FREE SE0/GEO Toolkit
GitHub · 14 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00
-
9
Potelo/laravel-block-bots
Block crawlers and high traffic users on your site by IP using Redis
GitHub · PHP · 55 stars · 18 forks · 0 stars in 24 hours · Trending score 0.00
-
10
ArchiveTeam/wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
GitHub · C · 139 stars · 19 forks · 0 stars in 24 hours · Trending score 0.00