deduplication repositories

Trending repositories tagged deduplication, ranked by trending score. Updated daily.

  1. 1
    restic/restic

    Fast, secure, efficient backup program

    GitHub · Go · 37,314 stars · 1,987 forks · +32 stars in 24 hours · Trending score 3.12

  2. 2
    kopia/kopia

    Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.

    GitHub · Go · 14,752 stars · 810 forks · +23 stars in 24 hours · Trending score 2.89

  3. 3
    PlakarKorp/plakar

    plakar is a backup solution powered by Kloset and ptar

    GitHub · Go · 2,178 stars · 104 forks · +50 stars in 24 hours · Trending score 2.74

  4. 4
    Ephemeral-AI-Lab/layerfs

    Ephemeral Workspaces, durable shared history. LayerFS gives every agent a complete, writable filesystem Workspace over shared committed history, without copying the base. Agents work in parallel, commit what changed as deduplicated checkpoints, then branch, rewind or merge. Try it with LayerFS × DSH.

    GitHub · Rust · 2,823 stars · 161 forks · +3 stars in 24 hours · Trending score 1.90

  5. 5
    rustic-rs/rustic

    rustic - fast, encrypted, and deduplicated backups powered by Rust

    GitHub · Rust · 3,427 stars · 151 forks · 0 stars in 24 hours · Trending score 1.52

  6. 6
    karanhudia/borg-ui

    Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.

    GitHub · Python · 1,690 stars · 62 forks · +7 stars in 24 hours · Trending score 1.45

  7. 7
    NVIDIA-NeMo/Curator

    Scalable data pre processing and curation toolkit for LLMs

    GitHub · Python · 1,866 stars · 379 forks · +3 stars in 24 hours · Trending score 1.41

  8. 8
    windirstat/windirstat

    Windows disk usage analyzer and cleanup tool with interactive treemaps, large-file discovery, duplicate detection, and file-system monitoring.

    GitHub · C++ · 4,579 stars · 290 forks · 0 stars in 24 hours · Trending score 1.15

  9. 9
    sahib/rmlint

    Extremely fast tool to remove duplicates and other lint from your filesystem

    GitHub · C · 2,472 stars · 161 forks · 0 stars in 24 hours · Trending score 0.99

  10. 10
    fcorbelli/zpaqfranz

    Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix

    GitHub · C++ · 515 stars · 40 forks · +1 stars in 24 hours · Trending score 0.97

  11. 11
    borgbackup/borg

    Deduplicating archiver with compression and authenticated encryption.

    GitHub · Python · 13,936 stars · 884 forks · 0 stars in 24 hours · Trending score 0.83

  12. 12
    NevermindNilas/TheAnimeScripter

    Anime Video Toolkit: AI upscale, RIFE interpolation and restore in one pass. CLI, Standalone and After Effects.

    GitHub · Python · 348 stars · 14 forks · 0 stars in 24 hours · Trending score 0.60

  13. 13
    mhx/dwarfs

    A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows

    GitHub · C++ · 2,633 stars · 91 forks · 0 stars in 24 hours · Trending score 0.59

  14. 14
    curlewlabs-com/apfs-dedupe

    Safe APFS clone deduplication for macOS — fclones detection + a windowless, ACL-complete apply

    GitHub · Shell · 8 stars · 1 forks · 0 stars in 24 hours · Trending score 0.47

  15. 15
    kdeldycke/mail-deduplicate

    📧 CLI to deduplicate mails from mail boxes

    GitHub · Python · 203 stars · 47 forks · 0 stars in 24 hours · Trending score 0.46

  16. 16
    benseverndev-oss/goldenmatch

    Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.

    GitHub · Python · 138 stars · 18 forks · 0 stars in 24 hours · Trending score 0.44

  17. 17
    maxkhim/laravel-storage-dedupler

    Laravel package to prevent file duplications

    GitHub · PHP · 58 stars · 3 forks · 0 stars in 24 hours · Trending score 0.39

  18. 18
    STEXS-Technologies/shardline

    Self-hosted, content-addressed storage infrastructure for AI models, datasets, containers, and build artifacts, with deduplication, integrity verification, and resilient data operations.

    GitHub · Rust · 22 stars · 5 forks · 0 stars in 24 hours · Trending score 0.36

  19. 19
    FerrLabs/LFSX

    A fast, lightweight, secure Git LFS server

    GitHub · Rust · 7 stars · 1 forks · 0 stars in 24 hours · Trending score 0.32

  20. 20
    rknightion/paperless-ngx-dedupe

    Find and remove duplicate documents in paperless-ngx. MinHash/LSH fuzzy matching catches near-duplicates that exact hashing misses, with optional AI metadata suggestions.

    GitHub · TypeScript · 28 stars · 2 forks · 0 stars in 24 hours · Trending score 0.29

  21. 21
    someplaceguy/hashsync

    Run a command over SSH, deduplicating its output or input against files stored on the receiving host, to save bandwidth.

    GitHub · Rust · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.24

  22. 22
    eldertek/duplicatefinder

    Discover and declutter! Find and eliminate duplicate files on Nextcloud.

    GitHub · JavaScript · 59 stars · 15 forks · 0 stars in 24 hours · Trending score 0.23

  23. 23
    shootthesound/tensorvillage

    One organized drive for every AI model you own - however many disks they are scattered across. Windows.

    GitHub · 11 stars · 1 forks · 0 stars in 24 hours · Trending score 0.14

  24. 24
    alex09x/mitts

    Rust-first build system with a shared, content-addressed build cache across projects and worktrees, plus explicit input tracking.

    GitHub · Rust · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10