File Deduplication

dedup.flexfs identifies files with identical content within a flexFS volume and optionally replaces duplicates with hard links, reclaiming storage space without altering the visible file structure.

How It Works

1. Candidate discovery: The metadata server identifies files that share the same size and block count — a fast, metadata-only operation. All paths are returned, including multiple hard links to the same inode.
2. Checksum grouping: For groups of 3 or more unique inodes, dedup.flexfs computes xxhash64 checksums in parallel to sub-group candidates. Pairs skip this step.
3. Byte verification: Each candidate inode is compared byte-for-byte against the retained inode to eliminate false positives. Verification runs once per unique inode, not per path.
4. Retention heuristic: The inode with the oldest birth time is retained. Ties are broken by the highest hard link count.
5. Hard link replacement: With --fix, each path for a verified duplicate inode is atomically replaced with a hard link to the retained inode — after a final per-path safety check that skips any file that changed since verification or whose owner or permissions differ from the retained file (see below).

Basic Usage

Dry Run (Report Only)

dedup.flexfs /mnt/flexfs/data

Output shows duplicate groups with file paths and a summary:

Bytes: 4194304, Blocks: 1
  /mnt/flexfs/data/file-a.bin (primary)
  /mnt/flexfs/data/backup/file-a.bin (duplicate)
1 duplicate file found in 1 group
Reclaimable space: 4.00 MiB
Run with --fix to deduplicate with hard links

Apply Deduplication

sudo dedup.flexfs --fix /mnt/flexfs/data

Root privileges are required for --fix because replacing files with hard links modifies inode link counts.

In fix mode, each path is labeled with what happened to it:

Bytes: 4194304, Blocks: 1
  /mnt/flexfs/data/file-a.bin (retained)
  /mnt/flexfs/data/backup/file-a.bin (fixed)
1 duplicate file found in 1 group
Space reclaimed: 4.00 MiB

• retained — the file that is kept; duplicates become links to it.
• fixed — replaced with a hard link to the retained file.
• already linked — already a link to the retained file; nothing to do. Re-running --fix on the same directory is safe: files deduplicated by an earlier run are no longer duplicates of anything and are simply passed over.
• skipped — not replaced; the reason is printed as an error line. A file is skipped when it changed after its contents were verified, when it is no longer a regular file, or when its owner or permissions differ from the retained file and --force was not given.

To also replace duplicates whose owner or permissions differ from the retained file, add --force:

sudo dedup.flexfs --fix --force /mnt/flexfs/data

Filtering by Size and Blocks

Focus on large files to maximize space savings:

# Only files 1 MiB or larger
dedup.flexfs --minSize 1048576 /mnt/flexfs/data

# Only files with 2-100 blocks
dedup.flexfs --minBlocks 2 --maxBlocks 100 /mnt/flexfs/data

Space Savings

When --fix is applied, all paths referencing a duplicate inode are replaced with hard links to the retained inode. Once all paths are replaced, the duplicate inode’s link count reaches zero and its data blocks are freed.

The summary’s Space reclaimed (or Reclaimable space in a dry run) only counts files whose every hard link lies inside the scanned directory. A file that keeps a hard link elsewhere still occupies its data blocks even after its in-scope paths are replaced, so the summary reports how many such files were found instead of counting their size.

Important Considerations for --fix

• Always dry-run first: Run without --fix to review which files will be affected before making changes.
• Permissions and ownership change: When a duplicate is replaced with a hard link, it inherits the retained file’s ownership, permissions, and timestamps. Because of this, files whose owner or permissions differ from the retained file are reported and skipped unless --force is given. Use --force only when adopting the retained file’s metadata is acceptable — especially in multi-user environments where files owned by different users may happen to have identical content.
• Shared inode side effects: After deduplication, all hard-linked paths point to the same inode and the same data blocks. A write to any path modifies the data seen by all paths. Similarly, chmod, chown, and truncate affect every linked path. If independent copies are needed, copy the file to a new path rather than linking.

Limitations

• Requires a flexFS mount: The path must be within an active flexFS mount. The tool discovers the metadata server by reading .flexfs/volume at the mount root.
• Time-travel mounts: --fix is not supported on time-travel mounts (read-only).
• Cross-volume: Deduplication operates within a single volume. Cross-volume deduplication is not supported.
• Content changes: If a file is modified while the tool runs, it is detected and skipped — the byte comparison catches changes made before verification, and a final per-path check catches files that changed between verification and replacement.

Scheduling

For ongoing deduplication, run dedup.flexfs periodically via cron:

/etc/cron.weekly/flexfs-dedup

#!/bin/bash
/usr/sbin/dedup.flexfs --fix --minSize 1048576 /mnt/flexfs/data >> /var/log/flexfs-dedup.log 2>&1

Complete Flag Reference

Flag	Type	Default	Description
--fix	bool	false	Replace duplicates with hard links (requires root)
--force	bool	false	With --fix, replace duplicates even when their owner or permissions differ from the retained file
--limit	uint64	0	Maximum number of duplicate groups (0 = unlimited). Largest groups first.
--maxBlocks	uint64	0	Maximum blocks filter (0 = no limit)
--maxSize	uint64	0	Maximum byte size filter (0 = no limit)
--minBlocks	uint64	0	Minimum blocks filter
--minSize	uint64	0	Minimum byte size filter

See the dedup.flexfs CLI reference for the complete flag listing including internal flags.