spider repositories

Trending repositories tagged spider, ranked by trending score. Updated daily.

  1. 1
    spider-rs/spider

    Foundational low latency web data collecting in Rust

    GitHub · Rust · 2,855 stars · 330 forks · +24 stars in 24 hours · Trending score 2.48

  2. 2
    CharlesPikachu/freeproxy

    FreeProxy: Collecting free proxies from internet. (全球海量高质量免费代理,支持爬取数十个免费代理分享源,支持自定义规则代理筛选,爬虫与数据分析必备,每日更新海量免费代理。)

    GitHub · Python · 592 stars · 103 forks · +12 stars in 24 hours · Trending score 1.86

  3. 3
    cv-cat/catbus

    一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用

    GitHub · TypeScript · 631 stars · 195 forks · +3 stars in 24 hours · Trending score 1.62

  4. 4
    PhiFever/AfdianToMarkdown

    爱发电爬虫(afdian.com)

    GitHub · Go · 81 stars · 16 forks · +4 stars in 24 hours · Trending score 1.30

  5. 5
    NaiboWang/EasySpider

    A visual no-code/code-free web crawler/spider易采集:一个可视化浏览器自动化测试/数据采集/网页爬虫软件,可以无代码图形化的设计和执行爬虫任务。别名:ServiceWrapper面向Web应用的智能化服务封装系统。

    GitHub · JavaScript · 44,798 stars · 5,394 forks · +2 stars in 24 hours · Trending score 1.11

  6. 6
    DedSecInside/TorBot

    Dark Web OSINT Tool

    GitHub · Python · 5,197 stars · 780 forks · 0 stars in 24 hours · Trending score 1.03

  7. 7
    niubihu1/tvbox

    🐯 牛虎 ┃ TVBox 旗舰聚合接口 · 4K原画秒播 · 纯净零广告 · 2000+高清电视直播(含EPG台标) · 影音壳单仓与多仓精选 · 双核爬虫驱动 · 大数据共识优选 · 每日自动巡检保活与去死链

    GitHub · JavaScript · 237 stars · 53 forks · 0 stars in 24 hours · Trending score 0.74

  8. 8
    signal18/replication-manager

    Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server

    GitHub · Go · 747 stars · 177 forks · +1 stars in 24 hours · Trending score 0.63

  9. 9
    scrapfly/scrapfly-scrapers

    Scalable Python web scraping scripts for +80 popular domains

    GitHub · Python · 1,131 stars · 208 forks · 0 stars in 24 hours · Trending score 0.55

  10. 10
    scrapfly/Antibot-Detector

    Real-time detection of anti-bot systems, captchas & fingerprinting techniques. Identifies Cloudflare, Akamai, DataDome, reCAPTCHA, hCaptcha, Shape Security & more with confidence scoring and advanced capture tools.

    GitHub · JavaScript · 632 stars · 74 forks · 0 stars in 24 hours · Trending score 0.50

  11. 11
    ndgigliotti/shopify-spy

    Python library and CLI for scraping products, images and collections from Shopify and WooCommerce stores.

    GitHub · Python · 124 stars · 54 forks · 0 stars in 24 hours · Trending score 0.48

  12. 12
    alltheplaces/alltheplaces

    A set of spiders and scrapers to extract location information from places that post their location on the internet.

    GitHub · Python · 852 stars · 311 forks · 0 stars in 24 hours · Trending score 0.38

  13. 13
    BlessedRebuS/Krawl

    Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.

    GitHub · Python · 841 stars · 77 forks · 0 stars in 24 hours · Trending score 0.34

  14. 14
    polyrabbit/hacker-news-digest

    :newspaper: Let LLM Summarize Hacker News for You

    GitHub · Python · 754 stars · 94 forks · 0 stars in 24 hours · Trending score 0.21

  15. 15
    qnxg/hnu_query

    湖南大学校内系统查询库,将湖南大学的校内系统(如个人门户,教务系统等)的接口进行包装,提供一套获取结构化数据的能力。

    GitHub · Rust · 21 stars · 6 forks · 0 stars in 24 hours · Trending score 0.15

  16. 16
    postmodern/spidr

    A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

    GitHub · Ruby · 837 stars · 105 forks · 0 stars in 24 hours · Trending score 0.14

  17. 17
    yutao8/starred

    GitHub Star 收藏资源库,自动同步个人收藏项目。收录 2000+ 个精选开源项目,覆盖开发框架、组件、SDK、模板、API、脚本、工具及各类实用资源,帮助开发者快速发现优质项目。

    GitHub · PHP · 176 stars · 22 forks · 0 stars in 24 hours · Trending score 0.10

  18. 18
    elixir-crawly/crawly

    Crawly, a high-level web crawling & scraping framework for Elixir.

    GitHub · Elixir · 1,129 stars · 127 forks · −2 stars in 24 hours · Trending score 0.10

  19. 19
    Norconex/crawler

    Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

    GitHub · Java · 205 stars · 71 forks · 0 stars in 24 hours · Trending score 0.09

  20. 20
    hupo376787/HelloCrab

    A smart tool to download social media files. For learning only.

    GitHub · C# · 34 stars · 6 forks · 0 stars in 24 hours · Trending score 0.09

  21. 21
    RavelloH/PSGameSpider

    自动爬取所有PlayStationStore中的所有游戏信息,包括封面、描述、价格、评分等,生成网页并索引 # # # Automatically crawl all game infos in all playstationstore, automatically generate web pages and index them

    GitHub · JavaScript · 73 stars · 6 forks · 0 stars in 24 hours · Trending score 0.08

  22. 22
    yagami-hue/MisaTV

    轻量、本地优先的 TVBox 客户端(Flutter 重写自 Win-box):支持 jar / js / py 三类蜘蛛、网盘取流、磁力 BT、弹幕、外挂字幕、直播 EPG 与 DLNA 投屏

    GitHub · Dart · 111 stars · 24 forks · 0 stars in 24 hours · Trending score 0.07

  23. 23
    lulu38m/spider-crawler

    🕷️ A literal web crawler: drop a document or a link, watch a spider read it, get the gist. Procedural animation, 100% in your browser.

    GitHub · TypeScript · 5 stars · 1 forks · 0 stars in 24 hours · Trending score 0.07

  24. 24
    WuKongSecurity/SpiderBOX

    SpiderBox - 虫盒 - 爬虫逆向资源导航站

    GitHub · CSS · 159 stars · 29 forks · 0 stars in 24 hours · Trending score 0.07