dedup.flexfs

dedup.flexfs identifies duplicate files within a flexFS volume and optionally replaces them with hard links to reclaim storage. It fetches duplicate candidates from the metadata server’s /duplicates endpoint, then verifies them through checksum comparison and byte-for-byte validation before making any changes.

Usage

dedup.flexfs [flags] [path]

If no path is specified, the current directory is used. The path must be within a flexFS mount.

Flags

Flag	Type	Default	Description	Visibility
--fix	bool	false	Replace duplicates with hard links (requires root)	Public
--maxBlocks	uint64	0	Maximum blocks filter (0 = no limit)	Public
--maxSize	uint64	0	Maximum byte size filter (0 = no limit)	Public
--minBlocks	uint64	0	Minimum blocks filter	Public
--minSize	uint64	0	Minimum byte size filter	Public
--noMetaSSL	bool	false	Disable SSL for metadata server connections	Internal

How It Works

1. Candidate Discovery: Queries the metadata server for files that share the same size and block count.
2. Checksum Grouping: For groups of 3+ files, computes xxhash64 checksums concurrently to sub-group candidates. For pairs, this step is skipped.
3. Byte Verification: Performs a byte-for-byte comparison of each candidate against the retained file.
4. Retention Heuristic: The file with the lowest birth time (oldest) is retained. Ties are broken by highest hard link count.
5. Hard Link Replacement (with --fix): Atomically replaces each duplicate with a hard link to the retained file.

Examples

Scan a directory for duplicates (dry run):

dedup.flexfs /mnt/flexfs/data

Fix duplicates by replacing them with hard links:

sudo dedup.flexfs --fix /mnt/flexfs/data

Only scan files with at least 2 blocks and at most 100 blocks:

dedup.flexfs --minBlocks 2 --maxBlocks 100 /mnt/flexfs/data

Requirements

• The path must be within an active flexFS mount.
• --fix requires root privileges.
• --fix is not supported on time-travel mounts.