deduplication repositories
Trending repositories tagged deduplication, ranked by trending score. Updated daily.
-
1
restic/restic
Fast, secure, efficient backup program
GitHub · Go · 37,314 stars · 1,987 forks · +32 stars in 24 hours · Trending score 3.12
-
2
kopia/kopia
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
GitHub · Go · 14,752 stars · 810 forks · +23 stars in 24 hours · Trending score 2.89
-
3
PlakarKorp/plakar
plakar is a backup solution powered by Kloset and ptar
GitHub · Go · 2,178 stars · 104 forks · +50 stars in 24 hours · Trending score 2.74
-
4
Ephemeral-AI-Lab/layerfs
Ephemeral Workspaces, durable shared history. LayerFS gives every agent a complete, writable filesystem Workspace over shared committed history, without copying the base. Agents work in parallel, commit what changed as deduplicated checkpoints, then branch, rewind or merge. Try it with LayerFS × DSH.
GitHub · Rust · 2,823 stars · 161 forks · +3 stars in 24 hours · Trending score 1.90
-
5
rustic-rs/rustic
rustic - fast, encrypted, and deduplicated backups powered by Rust
GitHub · Rust · 3,427 stars · 151 forks · 0 stars in 24 hours · Trending score 1.52
-
6
karanhudia/borg-ui
Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.
GitHub · Python · 1,690 stars · 62 forks · +7 stars in 24 hours · Trending score 1.45
-
7
NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
GitHub · Python · 1,866 stars · 379 forks · +3 stars in 24 hours · Trending score 1.41
-
8
windirstat/windirstat
Windows disk usage analyzer and cleanup tool with interactive treemaps, large-file discovery, duplicate detection, and file-system monitoring.
GitHub · C++ · 4,579 stars · 290 forks · 0 stars in 24 hours · Trending score 1.15
-
9
sahib/rmlint
Extremely fast tool to remove duplicates and other lint from your filesystem
GitHub · C · 2,472 stars · 161 forks · 0 stars in 24 hours · Trending score 0.99
-
10
fcorbelli/zpaqfranz
Deduplicating archiver with encryption and paranoid-level tests. Swiss army knife for the serious backup and disaster recovery manager. Ransomware neutralizer. Win/Linux/Unix
GitHub · C++ · 515 stars · 40 forks · +1 stars in 24 hours · Trending score 0.97
-
11
borgbackup/borg
Deduplicating archiver with compression and authenticated encryption.
GitHub · Python · 13,936 stars · 884 forks · 0 stars in 24 hours · Trending score 0.83
-
12
NevermindNilas/TheAnimeScripter
Anime Video Toolkit: AI upscale, RIFE interpolation and restore in one pass. CLI, Standalone and After Effects.
GitHub · Python · 348 stars · 14 forks · 0 stars in 24 hours · Trending score 0.60
-
13
mhx/dwarfs
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows
GitHub · C++ · 2,633 stars · 91 forks · 0 stars in 24 hours · Trending score 0.59
-
14
curlewlabs-com/apfs-dedupe
Safe APFS clone deduplication for macOS — fclones detection + a windowless, ACL-complete apply
GitHub · Shell · 8 stars · 1 forks · 0 stars in 24 hours · Trending score 0.47
-
15
kdeldycke/mail-deduplicate
📧 CLI to deduplicate mails from mail boxes
GitHub · Python · 203 stars · 47 forks · 0 stars in 24 hours · Trending score 0.46
-
16
benseverndev-oss/goldenmatch
Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.
GitHub · Python · 138 stars · 18 forks · 0 stars in 24 hours · Trending score 0.44
-
17
maxkhim/laravel-storage-dedupler
Laravel package to prevent file duplications
GitHub · PHP · 58 stars · 3 forks · 0 stars in 24 hours · Trending score 0.39
-
18
STEXS-Technologies/shardline
Self-hosted, content-addressed storage infrastructure for AI models, datasets, containers, and build artifacts, with deduplication, integrity verification, and resilient data operations.
GitHub · Rust · 22 stars · 5 forks · 0 stars in 24 hours · Trending score 0.36
-
19
FerrLabs/LFSX
A fast, lightweight, secure Git LFS server
GitHub · Rust · 7 stars · 1 forks · 0 stars in 24 hours · Trending score 0.32
-
20
rknightion/paperless-ngx-dedupe
Find and remove duplicate documents in paperless-ngx. MinHash/LSH fuzzy matching catches near-duplicates that exact hashing misses, with optional AI metadata suggestions.
GitHub · TypeScript · 28 stars · 2 forks · 0 stars in 24 hours · Trending score 0.29
-
21
someplaceguy/hashsync
Run a command over SSH, deduplicating its output or input against files stored on the receiving host, to save bandwidth.
GitHub · Rust · 9 stars · 0 forks · 0 stars in 24 hours · Trending score 0.24
-
22
eldertek/duplicatefinder
Discover and declutter! Find and eliminate duplicate files on Nextcloud.
GitHub · JavaScript · 59 stars · 15 forks · 0 stars in 24 hours · Trending score 0.23
-
23
shootthesound/tensorvillage
One organized drive for every AI model you own - however many disks they are scattered across. Windows.
GitHub · 11 stars · 1 forks · 0 stars in 24 hours · Trending score 0.14
-
24
alex09x/mitts
Rust-first build system with a shared, content-addressed build cache across projects and worktrees, plus explicit input tracking.
GitHub · Rust · 5 stars · 0 forks · 0 stars in 24 hours · Trending score 0.10