crawl repositories

Trending repositories tagged crawl, ranked by trending score. Updated daily.

  1. 1
    tavily-ai/tavily-python

    The Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. Easily integrate smart search, content extraction, and research capabilities into your applications, harnessing Tavily's powerful features.

    GitHub · Python · 1,432 stars · 190 forks · +14 stars in 24 hours · Trending score 1.76

  2. 2
    scrapfly/Antibot-Detector

    Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

    GitHub · JavaScript · 632 stars · 74 forks · 0 stars in 24 hours · Trending score 0.45

  3. 3
    any4ai/AnyCrawl

    AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

    GitHub · TypeScript · 3,528 stars · 372 forks · 0 stars in 24 hours · Trending score 0.04

  4. 4
    coder-hxl/x-crawl

    Flexible Node.js crawler library

    GitHub · TypeScript · 1,890 stars · 116 forks · 0 stars in 24 hours · Trending score 0.00

  5. 5
    tavily-ai/tavily-js

    Official TypeScript/JavaScript library for Tavily.

    GitHub · TypeScript · 95 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00

  6. 6
    datablueprint/api

    数巨蓝图(Datablueprint)是专注于海外主流社交媒体平台数据采集与分析的技术服务商。我们通过自研API接口,为全球企业提供合规、实时、结构化的多平台数据服务,覆盖 Facebook、Instagram、YouTube、Twitter(X)、TikTok、LinkedIn、Quora七大核心平台,提供最稳定的API接口,爬虫我们最在行,视频下载我们最拿手

    GitHub · 27 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00

  7. 7
    spatie/laravel-site-search

    Create a full-text search index by crawling your site

    GitHub · PHP · 310 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00

  8. 8
    ArchiveTeam/wget-lua

    Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

    GitHub · C · 139 stars · 19 forks · 0 stars in 24 hours · Trending score 0.00