robots-txt repositories
Trending repositories tagged robots-txt, ranked by trending score. Updated daily.
-
1
scrapy/protego
A pure-Python robots.txt parser with support for modern conventions.
GitHub · DIGITAL Command Language · 98 stars · 43 forks · +4 stars in 24 hours · Trending score 1.44
-
2
crawler-commons/crawler-commons
A set of reusable Java components that implement functionality common to any web crawler
GitHub · Java · 260 stars · 93 forks · 0 stars in 24 hours · Trending score 0.28
-
3
temoto/robotstxt
The robots.txt exclusion protocol implementation for Go language
GitHub · Go · 289 stars · 61 forks · 0 stars in 24 hours · Trending score 0.24
-
4
JosephDoUrden/vercel-seo-audit
If you're using Vercel and Google hates your site, this is for you.
GitHub · TypeScript · 80 stars · 3 forks · 0 stars in 24 hours · Trending score 0.19
-
5
samclarke/robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
GitHub · JavaScript · 168 stars · 19 forks · 0 stars in 24 hours · Trending score 0.10
-
6
nuxt-modules/robots
Tame the robots crawling and indexing your Nuxt site.
GitHub · TypeScript · 529 stars · 54 forks · 0 stars in 24 hours · Trending score 0.09
-
7
codevibe-technology/ai-bot-access
Can AI crawlers, search engines and link-preview bots (GPTBot, ClaudeBot, PerplexityBot, Googlebot, LinkedInBot and more) really read your site? robots.txt plus a real request per bot, share-preview check, bot IP verification and access-log reports.
GitHub · TypeScript · 6 stars · 5 forks · 0 stars in 24 hours · Trending score 0.07
-
8
healsdata/ai-training-opt-out
Known tags and settings suggested to opt out of having your content used for AI training.
GitHub · HTML · 160 stars · 5 forks · 0 stars in 24 hours · Trending score 0.06
-
9
ekalinin/robots.js
Parser for robots.txt for node.js
GitHub · JavaScript · 70 stars · 21 forks · 0 stars in 24 hours · Trending score 0.00
-
10
t1gor/Robots.txt-Parser-Class
Php class for robots.txt parse
GitHub · PHP · 85 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00
-
11
thedaviddias/llms-txt-hub
🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard
GitHub · TypeScript · 915 stars · 953 forks · 0 stars in 24 hours · Trending score 0.00
-
12
spatie/robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
GitHub · PHP · 258 stars · 44 forks · 0 stars in 24 hours · Trending score 0.00
-
13
rubenmarcus/aeo.js
Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.
GitHub · TypeScript · 149 stars · 17 forks · 0 stars in 24 hours · Trending score 0.00
-
14
samber/the-great-gpt-firewall
🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs
GitHub · Python · 101 stars · 7 forks · 0 stars in 24 hours · Trending score 0.00
-
15
AjnasNB/cockroach-crawler
Give AI agents bounded eyes on the public web: crawl, search, and normalize evidence without an unrestricted browser.
GitHub · JavaScript · 8 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00
-
16
Alpha-Park/genpark-robots-txt-crawl-delay-policy-evaluator-skill
RFC 9309 robots.txt parser with path permissions, Disallow/Allow rules, and crawl delay compliance
GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
17
alphaparkinc/genpark-robots-txt-crawl-delay-policy-evaluator-skill
RFC 9309 robots.txt parser with path permissions, Disallow/Allow rules, and crawl delay compliance
GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00
-
18
Serfe/serfe-ai-discovery
Open-source WordPress plugin for AI discovery: manage robots.txt and llms.txt, serve public content as Markdown, and validate discovery signals.
GitHub · PHP · 13 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00