dedup.flexfs

dedup.flexfs identifies duplicate files within a flexFS volume and optionally replaces them with hard links to reclaim storage. It fetches duplicate candidates from the metadata server’s /v1/duplicates endpoint, then verifies them through checksum comparison and byte-for-byte validation before making any changes.

Usage

dedup.flexfs [flags] [path]

If no path is specified, the current directory is used. The path must be a directory within a flexFS mount. The directory may be reached through symbolic links; links that stay within the volume are followed automatically.

Flags

Flag	Type	Default	Description	Visibility
--fix	bool	false	Replace duplicates with hard links (requires root)	Public
--force	bool	false	With --fix, replace duplicates even when their owner or permissions differ from the retained file. Without it, such files are reported and skipped.	Public
--limit	uint64	0	Maximum number of duplicate groups to return (0 = unlimited). Largest groups are returned first.	Public
--maxBlocks	uint64	0	Maximum blocks filter (0 = no limit)	Public
--maxSize	uint64	0	Maximum byte size filter (0 = no limit)	Public
--minBlocks	uint64	0	Minimum blocks filter	Public
--minSize	uint64	0	Minimum byte size filter	Public
--noMetaSSL	bool	false	Disable SSL for metadata server connections	Internal

How It Works

1. Candidate Discovery: Queries the metadata server for files that share the same size and block count. All paths are returned, including multiple hard links to the same inode.
2. Checksum Grouping: For groups of 3+ unique inodes, computes xxhash64 checksums concurrently to sub-group candidates. For pairs, this step is skipped.
3. Byte Verification: Performs a byte-for-byte comparison of each candidate inode against the retained inode (once per unique inode, not per path).
4. Retention Heuristic: The inode with the lowest birth time (oldest) is retained. Ties are broken by highest hard link count.
5. Hard Link Replacement (with --fix): Atomically replaces each duplicate path with a hard link to the retained inode, after a final safety check on every path (see below).

How —fix Protects Your Data

The duplicate list is a snapshot, and files can change while the tool runs. Immediately before each replacement, dedup.flexfs re-checks the file at that path and skips it — reporting the reason — if any of the following is true:

• The file was replaced, recreated, or modified after its contents were verified.
• The path is no longer a regular file (for example, it became a symbolic link).
• The file’s owner or permissions differ from the retained file’s, and --force was not given. Hard-linking makes both names share one owner and permission set, so this is never done silently.

A file that already links to the retained file is recognized as such and left alone, reported as already linked. Re-running --fix on the same directory is therefore safe: files deduplicated by an earlier run are no longer duplicates of anything and are simply passed over.

Each path in the output is labeled: retained (the file that is kept), fixed (replaced with a hard link), already linked, or skipped. In a dry run the labels are primary and duplicate.

The summary reports Space reclaimed (or Reclaimable space in a dry run). Space is only counted for files whose every hard link lies inside the scanned directory — a file that keeps a hard link elsewhere still occupies its blocks, and the summary calls out how many such files were found. Skipped paths are also totaled at the end.

Examples

Scan a directory for duplicates (dry run):

dedup.flexfs /mnt/flexfs/data

Fix duplicates by replacing them with hard links:

sudo dedup.flexfs --fix /mnt/flexfs/data

Only scan files with at least 2 blocks and at most 100 blocks:

dedup.flexfs --minBlocks 2 --maxBlocks 100 /mnt/flexfs/data

Warnings

Run without --fix first. Always perform a dry run to review which files will be deduplicated before applying changes.

Hard links share a single inode. When --fix replaces a duplicate with a hard link, the duplicate’s original ownership, permissions, and timestamps are replaced by those of the retained file. Files whose owner or permissions differ from the retained file are therefore skipped unless --force is given. Use --force only when you have reviewed the dry-run output and adopting the retained file’s metadata is acceptable, particularly in multi-user environments.

Hard links share data. After deduplication, all linked paths point to the same inode and the same data blocks. A write to any path modifies the data seen by all paths. Similarly, operations like chmod, chown, and truncate affect all linked paths. If independent copies are needed, the file must be copied (not linked) to a new path.

Requirements

• The path must be a directory within an active flexFS mount.
• --fix requires root privileges.
• --fix is not supported on time-travel mounts.