robots-txt repositories

Trending repositories tagged robots-txt, ranked by trending score. Updated daily.

  1. 1
    scrapy/protego

    A pure-Python robots.txt parser with support for modern conventions.

    GitHub · DIGITAL Command Language · 98 stars · 43 forks · +4 stars in 24 hours · Trending score 1.44

  2. 2
    crawler-commons/crawler-commons

    A set of reusable Java components that implement functionality common to any web crawler

    GitHub · Java · 260 stars · 93 forks · 0 stars in 24 hours · Trending score 0.28

  3. 3
    temoto/robotstxt

    The robots.txt exclusion protocol implementation for Go language

    GitHub · Go · 289 stars · 61 forks · 0 stars in 24 hours · Trending score 0.24

  4. 4
    JosephDoUrden/vercel-seo-audit

    If you're using Vercel and Google hates your site, this is for you.

    GitHub · TypeScript · 80 stars · 3 forks · 0 stars in 24 hours · Trending score 0.19

  5. 5
    samclarke/robots-parser

    NodeJS robots.txt parser with support for wildcard (*) matching.

    GitHub · JavaScript · 168 stars · 19 forks · 0 stars in 24 hours · Trending score 0.10

  6. 6
    nuxt-modules/robots

    Tame the robots crawling and indexing your Nuxt site.

    GitHub · TypeScript · 529 stars · 54 forks · 0 stars in 24 hours · Trending score 0.09

  7. 7
    codevibe-technology/ai-bot-access

    Can AI crawlers, search engines and link-preview bots (GPTBot, ClaudeBot, PerplexityBot, Googlebot, LinkedInBot and more) really read your site? robots.txt plus a real request per bot, share-preview check, bot IP verification and access-log reports.

    GitHub · TypeScript · 6 stars · 5 forks · 0 stars in 24 hours · Trending score 0.07

  8. 8
    healsdata/ai-training-opt-out

    Known tags and settings suggested to opt out of having your content used for AI training.

    GitHub · HTML · 160 stars · 5 forks · 0 stars in 24 hours · Trending score 0.06

  9. 9
    ekalinin/robots.js

    Parser for robots.txt for node.js

    GitHub · JavaScript · 70 stars · 21 forks · 0 stars in 24 hours · Trending score 0.00

  10. 10
    t1gor/Robots.txt-Parser-Class

    Php class for robots.txt parse

    GitHub · PHP · 85 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00

  11. 11
    thedaviddias/llms-txt-hub

    🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard

    GitHub · TypeScript · 915 stars · 953 forks · 0 stars in 24 hours · Trending score 0.00

  12. 12
    spatie/robots-txt

    Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

    GitHub · PHP · 258 stars · 44 forks · 0 stars in 24 hours · Trending score 0.00

  13. 13
    rubenmarcus/aeo.js

    Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.

    GitHub · TypeScript · 149 stars · 17 forks · 0 stars in 24 hours · Trending score 0.00

  14. 14
    samber/the-great-gpt-firewall

    🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

    GitHub · Python · 101 stars · 7 forks · 0 stars in 24 hours · Trending score 0.00

  15. 15
    AjnasNB/cockroach-crawler

    Give AI agents bounded eyes on the public web: crawl, search, and normalize evidence without an unrestricted browser.

    GitHub · JavaScript · 8 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00

  16. 16
    Alpha-Park/genpark-robots-txt-crawl-delay-policy-evaluator-skill

    RFC 9309 robots.txt parser with path permissions, Disallow/Allow rules, and crawl delay compliance

    GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  17. 17
    alphaparkinc/genpark-robots-txt-crawl-delay-policy-evaluator-skill

    RFC 9309 robots.txt parser with path permissions, Disallow/Allow rules, and crawl delay compliance

    GitHub · Python · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.00

  18. 18
    Serfe/serfe-ai-discovery

    Open-source WordPress plugin for AI discovery: manage robots.txt and llms.txt, serve public content as Markdown, and validate discovery signals.

    GitHub · PHP · 13 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00