Recursively remove duplicate files in a filesystem.
-
Updated
Aug 18, 2022 - Go
Entity resolution (also known as data matching, data linkage, record linkage, and many other terms) is the task of finding entities in a dataset that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Entity resolution is necessary when joining different data sets based on entities that may or may not share a common identifier (e.g., database key, URI, National identification number), which may be due to differences in record shape, storage location, or curator style or preference.
Recursively remove duplicate files in a filesystem.
HardLink Deduplicator - Detect and Manage Duplicate Files with Hard Links
deduplicate data via hardlinks, analyze drive status, prepare & optimize - filesystems (app/lib/api)
A simple hash based photo collation and merge program for UNIX systems.
Find and obliterate duplicate files, but only the ones you don't care about.
Batch your call. Easily backpressure. Enjoy the performance.
Analyse 2 paths to found identical files and hard link them to save space
DaBaDee is a simple deduplication tool/storage for files. It uses SHA256* to hash the files and store them in the storage, replacing the original path with a hardlink to the storage location.
Fast and cheap partial file hashing provided as a CLI tool and a zero-dependency Go library.
Just some durrdy code to move files around and organize shit...mostly photos and videos.
Golang Deduplication Utility Library
Run client side encrypted backups using AWS S3. Tools to backup, restore and manage job configurations.
A tool that deduplicates lines of a textfile with the speed of ram and scales nicely on all cores concurrently.
A command line tool to create hard links for duplicate files
used to get the dirs/files tree on the disk, including meta, sha1, and record to the sqlite database, then deduplications, make and sync virtual links for dir and files, etc.
An opinionated library that combines Ent and Watermill into a set of powerful utilities to transactionally handle events.
🔃 用于提取文件间差异数据,并且用于在两个端点之间进行差异化的文件同步。核心采用 rsync 算法,并且支持多轮同步以及就地构造文件。
S3 compatible data deduplication and client side encryption program
Created by Halbert L. Dunn
Released 1946