dedup.flexfs

dedup.flexfs identifies duplicate files within a flexFS volume and optionally replaces them with hard links to reclaim storage. It fetches duplicate candidates from the metadata server’s /v2/duplicates endpoint, then verifies them through checksum comparison and byte-for-byte validation before making any changes.

Persistent Flags

These flags apply to every subcommand.

Flag	Type	Default	Description	Visibility
--reportErrors	bool	false	Report errors and panics to Paradigm4	Public

Subcommands

Subcommand	Description	Visibility
deinit creds	Remove the credentials for a volume	Public
init creds	Store the credentials for a volume	Public
license	Print license information	Public
version	Print the build version	Public

Deduplicate

The root command scans a directory for duplicates and, with --fix, replaces them with hard links.

dedup.flexfs [flags] [path]

If no path is specified, the current directory is used. The path must be a directory within a flexFS mount. The directory may be reached through symbolic links; links that stay within the volume are followed automatically.

A path argument whose name matches a subcommand, such as a directory called init, is taken as the subcommand. Separate it with -- to use the directory instead:

# Runs the init subcommand
dedup.flexfs init

# Deduplicates the directory named init
dedup.flexfs -- init

Flags

Flag	Type	Default	Description	Visibility
--adminAddr	string	""	Admin server address. Only needed to override the one stored in the credentials, or alongside --token when there are none.	Internal
--credsFile	string	~/.flexfs/util/creds/<name>	Credentials file path	Public
--fix	bool	false	Replace duplicates with hard links (requires root)	Public
--force	bool	false	With --fix, replace duplicates even when their owner or permissions differ from the retained file. Without it, such files are reported and skipped.	Public
--limit	uint64	0	Maximum number of duplicate groups to return (0 = unlimited). Largest groups are returned first.	Public
--maxBlocks	uint64	0	Maximum blocks filter (0 = no limit)	Public
--maxSize	string	0	Maximum byte size filter, given as a size (10M, 1G) or bytes (1048576) (0 = no limit)	Public
--minBlocks	uint64	0	Minimum blocks filter	Public
--minSize	string	0	Minimum byte size filter, given as a size (10M, 1G) or bytes (1048576)	Public
--noAdminSSL	bool	false	Disable SSL for admin server connections	Internal
--noMetaSSL	bool	false	Disable SSL for metadata server connections	Internal
--token	string	""	Volume token to use, instead of reading one from a credentials file	Internal

How it works

1. Candidate Discovery: Queries the metadata server for files that share the same size and block count. All paths are returned, including multiple hard links to the same inode.
2. Checksum Grouping: For larger candidate groups, computes content checksums concurrently and sub-groups candidates by checksum comparison. For pairs, this step is skipped.
3. Byte Verification: Performs a byte-for-byte comparison of each candidate inode against the retained inode (once per unique inode, not per path).
4. Retention Heuristic: The inode with the lowest birth time (oldest) is retained. Ties are broken by highest hard link count, and any remaining ties by lowest inode number so the choice is deterministic across runs.
5. Hard Link Replacement (with --fix): Atomically replaces each duplicate path with a hard link to the retained inode, after a final safety check on every path (see below).

How —fix protects your data

The duplicate list is a snapshot, and files can change while the tool runs. Immediately before each replacement, dedup.flexfs re-checks the file at that path and skips it — reporting the reason — if any of the following is true:

• The file was replaced, recreated, or modified after its contents were verified.
• The path is no longer a regular file (for example, it became a symbolic link).
• The file’s owner or permissions differ from the retained file’s, and --force was not given. Hard-linking makes both names share one owner and permission set, so this is never done silently.

If a duplicate path already points at the retained file at the moment the tool goes to link it, dedup.flexfs leaves it untouched and labels it already linked. This makes re-running --fix on the same directory safe: once a set of files shares a single inode, those paths are one inode rather than several, so they no longer form a duplicate group and the scan passes over them. Where such a group still holds another distinct duplicate, the paths that share the retained inode are reported as retained.

Each path in the output is labeled: retained (the file that is kept), fixed (replaced with a hard link), already linked, or skipped. In a dry run the labels are primary and duplicate.

The summary reports Space reclaimed (or Reclaimable space in a dry run). Space is only counted for files whose every hard link lies inside the scanned directory — a file that keeps a hard link elsewhere still occupies its blocks, and the summary calls out how many such files were found. Skipped paths are also totaled at the end.

Examples

Scan a directory for duplicates (dry run):

dedup.flexfs /mnt/flexfs/data

Fix duplicates by replacing them with hard links:

sudo dedup.flexfs --fix /mnt/flexfs/data

Only scan files with at least 2 blocks and at most 100 blocks:

dedup.flexfs --minBlocks 2 --maxBlocks 100 /mnt/flexfs/data

Warnings

Run without --fix first. Always perform a dry run to review which files will be deduplicated before applying changes.

Hard links share a single inode. When --fix replaces a duplicate with a hard link, the duplicate’s original ownership, permissions, and timestamps are replaced by those of the retained file. Files whose owner or permissions differ from the retained file are therefore skipped unless --force is given. Use --force only when you have reviewed the dry-run output and adopting the retained file’s metadata is acceptable, particularly in multi-user environments.

Hard links share data. After deduplication, all linked paths point to the same inode and the same data blocks. A write to any path modifies the data seen by all paths. Similarly, operations like chmod, chown, and truncate affect all linked paths. If independent copies are needed, the file must be copied (not linked) to a new path.

Requirements

• The path must be a directory within an active flexFS mount.
• A volume token carrying the admin flag, stored with dedup.flexfs init creds.
• --fix requires root privileges.
• --fix is not supported on time-travel mounts.

init creds

Store the volume token dedup.flexfs uses for a volume.

dedup.flexfs init creds --adminAddr <admin-host:port> [flags]

The token is prompted for unless --token is given. The volume’s name is taken from the admin server, so the credentials are always filed under the name the mount reports, and the token is checked for the admin flag before anything is written.

Flag	Type	Default	Description	Visibility
--adminAddr	string	""	Admin server address (required)	Public
--credsFile	string	~/.flexfs/util/creds/<name>	Credentials file path	Public
--force	bool	false	Overwrite existing credentials file	Public
--noAdminSSL	bool	false	Disable SSL for admin server connections	Internal
--token	string	""	Volume token (prompted for if omitted)	Public
--volume	string	""	Volume name, checked against the token	Public

init on its own does the same thing as init creds.

Credentials

dedup.flexfs queries the metadata server, which requires a volume token carrying the admin flag. Store one per volume:

dedup.flexfs init creds --adminAddr admin.example.com:443

The command prompts for the token, confirms with the admin server that it allows reporting queries, and files it under the volume’s name. See Volume Tokens for how to obtain one.

Credentials are looked for in this order:

1. ~/.flexfs/util/creds/<volume-name>, written by dedup.flexfs init creds.
2. ~/.flexfs/mount/creds/<volume-name>, written by mount.flexfs init creds for the same user.

analyze.flexfs, dedup.flexfs and find.flexfs share the first of these, so one init creds serves all three — and one deinit creds removes the credentials all three were using.

Both files must be readable only by their owner (chmod 600); a file readable by others is refused.

--credsFile moves the first of these. Give the same value to init creds and to the query, or credentials will be written somewhere the query does not look — and to the other two utilities as well, if they are to go on sharing one file.

Because --fix requires root, credentials for it must be created as root:

sudo dedup.flexfs init creds --adminAddr admin.example.com:443

Credentials created as an ordinary user are kept under that user’s home directory and are not visible to a root run.

Unlike find.flexfs and analyze.flexfs, dedup.flexfs always runs inside a mount and takes the volume from it, so it has no --volume flag of its own. It does take --adminAddr: every run asks the admin server what its token is scoped to, and the flag overrides the address the credentials file records.

Tokens scoped to a subdirectory

A volume token can be issued for one subdirectory of a volume rather than the whole of it (see Volume Tokens). Every query made with such a token is answered from that subdirectory, whichever part of the volume the mount itself shows.

dedup.flexfs accounts for the difference, so paths are always given the way they look on this machine. With a volume mounted whole at /flexfs-1 and a token scoped to /sub-1:

dedup.flexfs /flexfs-1/sub-1        # the token's whole subtree
dedup.flexfs /flexfs-1/sub-1/data   # part of it
dedup.flexfs /flexfs-1              # refused: outside the token's subtree

To make the adjustment, dedup.flexfs asks the admin server what the token is scoped to — once per volume, on every run. Nothing about the token is cached on disk, so a token that is reissued or re-scoped needs no re-initialization. A token passed with --token is treated exactly like a stored one, but it must be paired with --adminAddr to say where to ask, since a token supplied on the command line brings no address with it. The admin server must therefore be reachable for a query made inside a mount.

deinit creds

Remove the credentials for a volume.

dedup.flexfs deinit creds <volume-name>

Removing credentials that are not there succeeds, so this is safe to run twice.

Flag	Type	Default	Description	Visibility
--credsFile	string	~/.flexfs/util/creds/<name>	Credentials file path	Public