Common Issues

Mount Failures

“mount point is already mounted”

The mount point is already in use by a working mount, so the client will not disturb it.

Solution: Check for existing processes with ps aux | grep mount.flexfs. If the mount is stale rather than working, see the stale mount section below.

“mount point is mounted but not usable”

Something is mounted here but it is not responding, and the client cannot tell whether it is safe to remove.

Solution: Unmount it and mount again:

sudo umount -l /mnt/flexfs
sudo mount /mnt/flexfs

If the message adds that the mount lock is held by a live process, a mount.flexfs process still owns the mount point. Find it with ps aux | grep mount.flexfs, stop it with sudo kill <pid>, then unmount and mount again as above.

“mount point is not empty and —nonEmpty flag has not been set”

The target directory contains files or subdirectories.

Solution: Either empty the directory or add the --nonEmpty flag (or nonEmpty in fstab options).

“invalid mount point: … (does not exist)”

The mount point directory does not exist.

Solution: Create the directory: mkdir -p /mnt/flexfs.

“invalid admin addr: empty”

The credentials file is missing or does not contain an adminAddr field.

Solution: Run mount.flexfs init creds --adminAddr <admin-addr> to initialize credentials, or verify the credentials file at the path shown in the error.

“invalid volume token: empty” or “invalid volume token: not a UUID”

The volume token is missing or malformed in the credentials file.

Solution: Re-run mount.flexfs init creds with a valid volume token.

ACL mount refuses to start

An --acl mount exits at startup, logging that it requires kernel POSIX ACL support and naming the running kernel.

Cause: the kernel is 4.9 or newer, so flexFS asked it to enforce ACLs, but its FUSE module was compiled without POSIX ACL support (CONFIG_FUSE_FS_POSIX_ACL). This is unusual — mainstream distribution kernels enable it.

FlexFS refuses rather than continuing, because at that point the kernel has already taken over basic permission checking and would ignore any ACL entry granting access beyond the standard owner/group/other bits. The mount would look healthy while quietly denying access that getfacl says should be allowed.

Resolution: use a kernel with POSIX ACL support enabled, or mount without --acl if extended ACLs are not required. Verify with:

grep CONFIG_FUSE_FS_POSIX_ACL /boot/config-$(uname -r)

A kernel older than 4.9 is not affected: there flexFS enforces ACLs itself and the mount starts normally.

FUSE Errors

“/dev/fuse: No such file or directory”

Also reported as open /dev/fuse: no such file or directory when the mount client opens the device itself. The FUSE kernel module is not loaded. A mount started as root runs modprobe fuse first and logs a warning if that fails; a mount started without root cannot load it.

Solution:

sudo modprobe fuse

For persistent loading, add fuse to /etc/modules-load.d/fuse.conf.

“the platform FUSE helper is not installed”

Error: the platform FUSE helper is not installed: neither fusermount3 nor fusermount was found

The FUSE3 userspace tools are not installed. A mount started as root, or with CAP_SYS_ADMIN, on a current system does not need them: it talks to the kernel directly. The helper is required for mounts started by an unprivileged user and on legacy systems whose /etc/mtab is a regular file rather than a symlink.

Solution:

# Debian/Ubuntu
sudo apt install fuse3

# RHEL/Rocky/Amazon Linux 2023
sudo dnf install fuse3

# CentOS 7 and Amazon Linux 2 (fuse3 is in the archived EPEL 7 repository)
sudo yum install https://dl.fedoraproject.org/pub/archive/epel/7/x86_64/Packages/e/epel-release-7-14.noarch.rpm
sudo yum install fuse3

See Prerequisites for the CentOS 7 repository requirement.

Stale Mounts

The examples in this section use sudo for a mount run as root. For a mount started by another user, run mount.flexfs start and fusermount3 -u (fusermount3 -uz in place of umount -l) as that user.

“Transport endpoint is not connected”

This indicates a stale FUSE mount where the daemon process has died.

Solution: Mount again. Any of these will clean up the stale mount first and report an error if they cannot:

sudo mount /mnt/flexfs                              # fstab entry
sudo mount.flexfs start <name> /mnt/flexfs          # direct

If that reports a failure, unmount by hand and retry:

sudo umount -l /mnt/flexfs

Mount appears in /proc/mounts but is not responding

Solution: Force unmount and remount:

sudo fusermount3 -u /mnt/flexfs   # or 'fusermount -u' on older FUSE 2 systems
sudo mount.flexfs start <name> /mnt/flexfs

SSL/TLS Errors

“x509: certificate signed by unknown authority” or “tls: failed to verify certificate”

A tool such as curl or a browser does not trust the certificate presented by a flexFS server, typically because the server uses its auto-generated self-signed certificate.

Solution: Use curl -k, add the server’s certificate to the system trust store, or install a certificate issued by a trusted CA (see TLS).

Credential Errors

“creds file already exists” during init

Solution: Use --force to overwrite: mount.flexfs init creds --force --adminAddr <admin-addr>

Credentials file not found

Solution: Run the appropriate init creds subcommand for the component, as the user that runs it. The default credential paths are under that user’s home folder; systemd services and fstab mounts run as root and use /root/.flexfs:

• Mount: ~/.flexfs/mount/creds/<volume-name>
• Meta: ~/.flexfs/meta/creds
• Admin: ~/.flexfs/admin/creds
• Proxy: ~/.flexfs/proxy/creds
• Configure: ~/.flexfs/configure/creds
• Find, analyze, dedup: ~/.flexfs/util/creds/<volume-name>

“creds file is readable by other users”

Error: creds file is readable by other users: "/root/.flexfs/meta/creds" is mode 0644 (run "chmod 600 /root/.flexfs/meta/creds")

A credentials file holds a bearer token, so flexFS refuses to read one that group or others can read.

Solution: Run the chmod the message names. Every init creds command writes the file correctly, so this means the permissions were changed afterwards — commonly by restoring from a backup or copying the file with a tool that does not preserve modes.

“no credentials for volume”

Error: no credentials for volume "research". Looked in:
  /home/ops/.flexfs/util/creds/research
  /home/ops/.flexfs/mount/creds/research
Run: find.flexfs init creds --adminAddr <admin-host:port>

Solution: Run the init creds command the message names. Credentials are per user, so each person running find.flexfs, analyze.flexfs or dedup.flexfs needs their own; a mount created by root keeps its credentials where other users cannot read them.

“the volume token … does not allow reporting queries”

Solution: The token is valid but was not created with the admin flag. Ask an administrator for one:

configure.flexfs create volume-token --volumeID <volume> --flags admin

See Volume Tokens.

Connectivity Issues

Admin server unreachable

Mount clients and metadata servers require connectivity to the admin server. While it is unreachable, mounts that are already connected keep working with their last validated volume settings, and new or reconnecting mounts wait. Token revocations and volume changes take effect once it is reachable again.

Solution:

1. Verify the admin server is running: sudo manage.flexfs status admin (on a Community install, sudo manage.flexfs status free)
2. Check network connectivity: curl -k https://<admin-addr>/status
3. Verify firewall rules allow traffic on port 443 (or the configured --bindAddr port)

“admin server refused the request, not the token”

The metadata server asked the admin server to validate a mount’s volume token, and the admin server answered with an error other than an invalid token. New and reconnecting mounts wait and retry, with this message in their log and in the metadata server’s log, and resume on their own once the cause is fixed. Mounts that are already connected keep working. A new mount started while this persists waits as well, and mount.flexfs start gives up after 10 minutes with an error that names the mount log.

Solution: Check what stands between the metadata server and the admin server:

1. The admin server’s access rules (--accessFile) must allow the metadata server’s address, including after the metadata server moves to a new host or address.
2. Any proxy, load balancer or firewall in front of the admin server must pass /v1/volumes/for-token/* requests through unchanged.
3. The admin server must be at least as new as the metadata server.

When the message comes from mount.flexfs start itself, or from a mount’s reconnect errors followed by “check its access rules for this host”, the admin server is refusing the mount host rather than the metadata server: allow the mount host’s address in the admin server’s access rules.

Mounts pause after “stopped accepting changes after a metadata read error”

The metadata server could not read part of that volume’s database and has paused the volume’s mounts. Current mount clients log the metadata server asked this mount to reconnect later and it will retry. Applications wait instead of receiving errors. In most cases the volume resumes on its own once its database reads cleanly again. See Metadata read errors for details.

Proxy unreachable

When proxy groups are configured but unreachable, mount clients fall back to direct object storage access. This is normal behavior, not an error.

Solution: If you expect proxy acceleration, verify:

1. The proxy server is running: sudo manage.flexfs status proxy
2. Network connectivity between mount client and proxy
3. The proxy group is associated with the volume in the admin server

Permission Errors

“Operation not permitted” on a file you own, even as root

Ordinary permission failures report Permission denied (EACCES). Operation not permitted (EPERM) on a write, delete, rename, chmod, or chown that root should be allowed to perform usually means the inode carries the immutable or append-only flag.

Diagnosis: lsattr <path> — an i or a in the flag field is the cause. Use lsattr -d <dir> for a directory, since a bare lsattr lists its entries instead. An immutable directory also refuses new entries, so this can surface as a failure to create a file rather than to change one.

Solution: sudo chattr -i <path> or sudo chattr -a <path> to clear it.

“chattr: Operation not permitted while setting flags”

Setting these flags requires root as the mount sees it. A --rootSquash mount remaps UID and GID 0, and an --allSquash mount remaps every caller, before the request reaches the filesystem, so chattr cannot be used at all there — including by real root. Reading flags with lsattr is unaffected. See Inode flags.

“chattr: Operation not supported while setting flags”

Only immutable (+i) and append-only (+a) are implemented. Any other flag is rejected outright rather than silently ignored, so a request never appears to succeed without taking effect.

File Operation Errors

“Directory not empty” from rmdir or mv on an older mount client

Mount clients older than v1.9.51 report Directory not empty when the metadata server refuses an rmdir or rename because the target is a directory, is not a directory, or the move would place a directory inside its own subtree. The metadata server refuses these when another host changed the same directories at about the same time, for example when two hosts move two directories into each other at once. Newer mount clients report Is a directory, Not a directory, or Invalid argument. Without a change from another host, the kernel normally refuses these operations itself with the correct error, whatever the mount client version.

Solution: Upgrade the mount client. See Deploying mount client updates.

Performance Issues

Slow sequential reads

Solution: Ensure block prefetching is not disabled. Check that the metadata server is responsive (monitor via /metrics). A larger block size can help workloads with large sequential reads, but block size is fixed at volume creation, so it can only be chosen when creating a new volume.

High write latency

Solution: For remote object storage, enable local disk writeback caching with --diskWriteback on the mount client. Writeback runs on top of the on-disk block cache, so you must also set --diskQuota <size> to allocate that cache; without it, --diskWriteback has no effect. For Enterprise deployments, deploy a proxy group in the same region as the compute.

Service Management

Services not starting after reboot

Solution: For servers run as systemd services, ensure the units are enabled:

sudo systemctl enable flexfs-admin.service
sudo systemctl enable flexfs-meta.service

On a Community install, the admin service is flexfs-free.service:

sudo systemctl enable flexfs-free.service
sudo systemctl enable flexfs-meta.service

Or use manage.flexfs start to manually start all services.

manage.flexfs upgrade stops with a download error for update.flexfs

update.flexfs is not part of flexFS. A manage.flexfs that includes update.flexfs in its binary list upgrades it along with the other flexFS binaries it finds in /sbin; that download fails, and the upgrade stops before installing anything.

Solution: Delete it and run the upgrade again:

sudo rm -f /sbin/update.flexfs
sudo manage.flexfs upgrade

A manage.flexfs without update.flexfs in its binary list removes /sbin/update.flexfs whenever it installs or upgrades binaries.

“this command must be run as root”

Most manage.flexfs operations require root privileges.

Solution: Use sudo or run as root.