It can produce a report of
which files are unique and which files are duplicates,
or it can hardlink duplicates - or both.
It does not attempt to hold all metadata in memory. That is, it is not limited by available RAM or Virtual Memory.
Instead, it stores metadata in files on disk, and uses GNU
sort
in combination with some Python to detect duplicates. GNU sort is very well optimized, BTW, so even though
drs-dedup's
algorithm is O(c*n*log(n) + d*n), d is much larger than c making its performance practically limited by d*n in
actual use, not c*n*log(n).
GNU Sort is an "external sort", so if it runs
low on memory, it is smart enough to switch to a disk-based algorithm.
It does hold all filenames of a given hash in memory at a time, but that's pretty small, comparatively speaking.
FYI: although drs-dedup can link many files, there is a practical limit on the number of files a given file can
be hardlinked to:
Filesystem
Limit on number of hardlinks to a single file allowed by the filesystem's on-disk representation
XFS
2^31 - 1
EXT4
2^16 - 1
NTFS
2^10 - 1
On a large, spinning, magnetic disk, drs-dedup
is pretty slow because of a poor locality of reference and large
number of
track-to-track seeks. For this reason, on a large, spinning, magnetic disk, you should try
fclones first, and only use drs-dedup if that runs out of
memory.
On a large SSD, drs-dedup is not all that slow, so it
might even be a tool of first-resort if you need to
deduplicate hundreds of millions or billions of files.
drs-dedup does not compare files; it only compares hashes. There is an infinitesimal chance of a rogue
hash collision. However, the odds of this happening are lower than the odds of a magnetic tape going bad.
Please note that the report has a limitation: if there are preexisting hardlinked files, only one of them will
be reported!
For the latest specifics on how to use it, run "drs-dedup --help". Or for your convenience, see below:
Usage: /usr/local/bin/drs-dedup
--input-dir /dir/to/dedeup
--work-dir /dir/for/temp/data/and/output
--sort-buffer-size 20%
--report
--hardlinks
--skip-cleanup
--help
Replace duplicate regular files under /dir/to/dedup with hardlinks to each other (or merely report).
Output will be left in /dir/for/temp/data/and/output. You will be given the precise filenames when the script
completes.
Note that --work-dir is used for GNU sort temporary files, not just your output.
Performance tip: if you put --input-dir and --work-dir in different filesystems, that should be slightly faster.
--sort-buffer-size can be anything GNU sort understands, EG 1G or 25%. Only one sort will be run at a time.
