POSIX Compliance

FlexFS implements a comprehensive set of POSIX filesystem operations through Linux FUSE (Filesystem in Userspace). Applications interact with flexFS exactly as they would with a local filesystem — no code changes or special APIs are required.

Supported FUSE operations

The mount client (mount.flexfs) implements the following FUSE operations:

Namespace operations

Operation	FUSE Op	Description
Lookup	FUSE_LOOKUP	Resolve a file name within a directory to its inode and attributes
Create	FUSE_CREATE	Create and open a new regular file
MkDir	FUSE_MKDIR	Create a directory
MkNod	FUSE_MKNOD	Create a filesystem node (regular file, device special file, FIFO)
Symlink	FUSE_SYMLINK	Create a symbolic link
Link	FUSE_LINK	Create a hard link to an existing inode
Unlink	FUSE_UNLINK	Remove a file
RmDir	FUSE_RMDIR	Remove a directory
Rename	FUSE_RENAME / FUSE_RENAME2	Rename or move a file or directory

File data operations

Operation	FUSE Op	Description
Open	FUSE_OPEN	Open a file
Read	FUSE_READ	Read data from an open file
Write	FUSE_WRITE	Write data to an open file
Flush	FUSE_FLUSH	Flush file data (called on each close())
Fsync	FUSE_FSYNC	Synchronize file data to storage
Release	FUSE_RELEASE	Close a file handle
Fallocate	FUSE_FALLOCATE	Preallocate or deallocate space for a file
Lseek	FUSE_LSEEK	Seek for data or holes (SEEK_DATA, SEEK_HOLE)
Ioctl	FUSE_IOCTL	Inode-flag ioctls only — the chattr/lsattr interface (see Inode flags). Every other command returns ENOTTY

Directory operations

Operation	FUSE Op	Description
OpenDir	FUSE_OPENDIR	Open a directory for reading
ReadDir	FUSE_READDIR	Read directory entries
ReadDirPlus	FUSE_READDIRPLUS	Read directory entries with pre-fetched attributes
ReleaseDir	FUSE_RELEASEDIR	Close a directory handle
Ioctl	FUSE_IOCTL	Inode-flag ioctls on a directory — chattr/lsattr apply to directories as well as files, where they govern the directory’s entries (see Inode flags)

Attribute operations

Operation	FUSE Op	Description
GetAttr	FUSE_GETATTR	Get file attributes (stat)
SetAttr	FUSE_SETATTR	Set file attributes (chmod, chown, truncate, utimes)
Access	FUSE_ACCESS	Check file access permissions
StatFs	FUSE_STATFS	Get filesystem statistics (df)
ReadLink	FUSE_READLINK	Read the target of a symbolic link

Extended attribute operations

Operation	FUSE Op	Description
GetXAttr	FUSE_GETXATTR	Get an extended attribute value
SetXAttr	FUSE_SETXATTR	Set an extended attribute
ListXAttr	FUSE_LISTXATTR	List all extended attribute names
RemoveXAttr	FUSE_REMOVEXATTR	Remove an extended attribute

Locking operations

Operation	FUSE Op	Description
GetLk	FUSE_GETLK	Test whether a lock could be placed
SetLk	FUSE_SETLK	Acquire or release a lock (non-blocking)
SetLkW	FUSE_SETLKW	Acquire or release a lock (blocking, with retry)

Lifecycle operations

Operation	FUSE Op	Description
Init	FUSE_INIT	Initialize the FUSE session and negotiate capabilities
Destroy	FUSE_DESTROY	Tear down the FUSE session
Forget	FUSE_FORGET	Release a cached inode reference
BatchForget	FUSE_BATCH_FORGET	Release multiple cached inode references

File locking

FlexFS supports both POSIX (fcntl) and BSD (flock) file locking semantics. Locks are coordinated through the metadata server, making them effective across all mount clients for a volume.

POSIX locks (fcntl)

POSIX-style byte-range locks are supported via fcntl(F_SETLK), fcntl(F_SETLKW), and fcntl(F_GETLK). These support shared (read) and exclusive (write) locks on arbitrary byte ranges.

Block alignment: When coordinated concurrent I/O is not available (see below), flexFS cannot guarantee atomic operations within a single block across concurrent mounts, so POSIX byte-range locks are aligned to block boundaries. A lock on bytes 100-200 of a file with a 4 MiB block size will effectively lock the entire first block (bytes 0 through 4,194,303). This is a best-effort approach that trades strict byte-range precision for correctness in the distributed case.

Coordinated concurrent I/O (default): When both the mount client and the metadata server support it (negotiated automatically at connection time), byte-range locks are passed through with their exact requested bounds — no block-boundary expansion. This is safe because block writes use a compare-and-swap against the metadata server: a mount that modifies part of a block records the block version it started from, and the server accepts the new version only if that version is still current. If another mount changed the block in the meantime, the writer re-fetches the current block, re-applies only the byte ranges it authored, and retries. Two mounts writing disjoint byte ranges within the same block therefore both succeed, and a lost update is never silently committed. The uncontended write path is unaffected; the re-merge cost is paid only under genuine contention. Older clients or servers that do not support the feature transparently fall back to block-aligned locking.

BSD locks (flock)

BSD-style whole-file locks are supported via flock(). These are implemented as full-range POSIX locks under the hood, with the lock owner identified by the file handle rather than by the fcntl owner field.

Blocking locks

SetLkW (the blocking variant of SetLk) retries the lock acquisition in a polling loop with a 100 ms interval until the lock is granted or an error occurs.

Cross-mount coordination

All lock state is stored on the metadata server. When mount client A holds a lock, mount client B will see the conflict via GetLk and will block (or receive EAGAIN) on SetLk. Locks are released when the file handle is closed or explicitly unlocked.

Concurrent appends

Files opened with O_APPEND are handled specially. The Linux kernel resolves an append offset from the inode size it has cached, which under close-to-open consistency can be stale on other mounts. To make appends correct across mounts, flexFS re-resolves the append offset in userspace.

When coordinated concurrent I/O is negotiated (the default; see File locking), each O_APPEND write reserves its region with a single atomic fetch-and-add on the file size at the metadata server, which returns the exact offset to write at. Because every appender receives a distinct, server-assigned region, concurrent appends from different mounts never overwrite one another. This costs one additional metadata round trip per append write and affects only files opened O_APPEND; all other writes are unchanged. When the feature is unavailable, the mount falls back to resolving the offset from its local (per-mount) view of the size, which is safe for a single writer but does not coordinate appends across mounts.

Hard links

FlexFS fully supports hard links via the Link operation. Multiple directory entries can reference the same inode, and the file’s link count (nlink) is maintained by the metadata server. The file’s data blocks are shared across all links; deleting a link decrements the link count, and the data blocks are only freed when the link count reaches zero and no file handles remain open.

Hard link semantics are consistent across mount clients — creating a hard link on one mount client is immediately visible to all other clients.

Symbolic links

Symbolic links are created via the Symlink operation and read via ReadLink. The link target is stored as a metadata field on the inode. When encryption is enabled, the link target is encrypted using AES-256-GCM with a deterministic nonce (SHAKE-256 hash of the target string).

Special files

The MkNod operation supports creating:

• Regular files
• FIFO (named pipe) nodes
• Character and block device special files

The device major/minor numbers and file mode are preserved in the inode metadata.

Inode flags (chattr / lsattr)

FlexFS implements the inode-flag ioctls that chattr(1) and lsattr(1) use, so the two flags below can be set and read exactly as on ext4 or xfs. No mount option is required — the flags are always available, including on volumes without --xAttr.

Flag	chattr	Effect
Immutable	chattr +i	The file cannot be modified, deleted, renamed, hard-linked to, or have its metadata or extended attributes changed.
Append-only	chattr +a	The file can only be opened for writing with O_APPEND, and cannot be deleted, renamed, hard-linked to, truncated, or have its metadata or extended attributes changed.

sudo chattr +i /mnt/flexfs/data/important.dat
lsattr /mnt/flexfs/data/important.dat
# ----i--------------- /mnt/flexfs/data/important.dat
sudo chattr -i /mnt/flexfs/data/important.dat

Both the legacy FS_IOC_GETFLAGS/FS_IOC_SETFLAGS pair and the newer FS_IOC_FSGETXATTR/FS_IOC_FSSETXATTR (fsxattr) interface are supported, so the flags work through the kernel’s fileattr path as well as through older chattr builds.

Which operations enforce the flags

Enforcement is not limited to writes. Thirteen operations consult the flags: Open, SetAttr (chmod, chown, truncate, utimes), Fallocate, SetXAttr, RemoveXAttr, Unlink, RmDir, Rename, Link, Create, MkNod, MkDir, and Symlink. A refused operation returns EPERM.

The flags also apply to directories, where they govern the directory’s entries: an immutable directory rejects new entries and removals alike, while an append-only directory permits new entries but refuses to remove or rename existing ones.

Append-only is not simply a stricter immutable — growing the file is the one thing it exists to permit. So an append-only file accepts fallocate that extends it but rejects a hole punch, whereas an immutable file rejects every fallocate mode.

Limits worth knowing

Caution

Setting a flag requires UID 0 as the mount sees it, and --rootSquash makes that impossible. Linux gates these flags on CAP_LINUX_IMMUTABLE, which FUSE does not forward; flexFS approximates it as UID 0. On a --rootSquash mount the caller’s UID is remapped to --rootSquashUID before the request reaches the filesystem, so even a genuine root caller is refused with EPERM and chattr cannot be used at all. Reading flags with lsattr is unrestricted and works normally.

• Enforcement is client-side. The flags are stored on the inode and checked by the mount client; the metadata server does not interpret them. They protect a file against callers going through a flexFS mount, which is every ordinary path to the data — but they are not a server-side guarantee, and a caller with direct credentialed access to the metadata server or the object store is not bound by them. Treat them as the accident-and-mistake protection they are on a local filesystem, not as a retention or compliance control.

• Only i and a are implemented. chattr requests that set any other flag are rejected with EOPNOTSUPP rather than silently ignored, so a request never appears to succeed without taking effect. lsattr reports the other positions as unset.

• An already-open file descriptor is not revoked. Setting a flag blocks new write opens; a descriptor opened for writing beforehand can still write() through it. Linux behaves the same way — the immutable check lives in the open path, not in write() — so this matches ext4 and xfs rather than diverging from them. ftruncate on such a descriptor is refused, and here flexFS is stricter than xfs: on xfs an immutable file can still be truncated through a descriptor opened before the flag was set, because the kernel’s ftruncate path tests only for append-only. FlexFS refuses it for both flags.

• Cross-mount visibility follows attribute caching. A flag set on one mount becomes visible to others on the same terms as any other attribute change: normally within milliseconds via the metadata server’s invalidation notifications, and bounded by --attrValid if a notification is missed. A peer mount that has the old attributes cached can briefly still permit an operation the flag now forbids. See Attribute and directory-entry caching.

• --ro mounts refuse to change flags, returning EROFS. Reading them still works.

• The flags are invisible in the xattr namespace. They are persisted in a private extended attribute that getxattr, listxattr, setxattr, and removexattr all filter out, so getfattr -d does not show it and it cannot be forged or cleared by writing an xattr directly — the ioctls are the only interface. On an encrypted volume, an entry that cannot be decrypted is read as both flags set, failing closed rather than silently unprotecting the file.

Extended attributes (xattrs)

Extended attributes are stored as key-value pairs on each inode in the metadata server. They are enabled with the --xAttr flag (or implicitly by --acl or --rootSquash).

Operation	Behavior
getxattr	Returns the value for a named attribute
setxattr	Sets or replaces an attribute. Supports XATTR_CREATE (fail if exists) and XATTR_REPLACE (fail if absent) flags
listxattr	Lists all attribute names on an inode
removexattr	Removes a named attribute

When encryption is enabled, both attribute names and values are encrypted. Names use deterministic encryption (for server-side matching); values use random nonces.

Access control lists (ACLs)

FlexFS supports POSIX extended ACLs (also known as POSIX.1e draft ACLs), which are stored as extended attributes (system.posix_acl_access and system.posix_acl_default). ACLs are enabled with the --acl flag.

The ACL implementation includes:

• Standard POSIX permission checks (user, group, other) for all operations.
• Extended ACL evaluation when ACL xattrs are present, supporting named user and group entries.
• SUID/SGID handling: SUID/SGID bits influence the effective user and group for permission checks.
• Sticky bit: The sticky bit on directories restricts deletion to the file owner, directory owner, or root.
• Root squashing: When --rootSquash is enabled, operations by uid 0 / gid 0 are remapped to a configurable uid/gid (default: 65534/65534, i.e., nobody). Root squashing implies ACL support.

Where ACLs are enforced

On Linux 4.9 and later the kernel evaluates ACLs itself, and an --acl mount performs like any other — including full attribute and directory-entry caching. On older kernels (in practice RHEL/CentOS 7), and whenever --rootSquash is set, the mount client evaluates every operation instead; that is equally correct but slower, because the kernel’s metadata caches have to be turned off for the checks to be reliable. mount.flexfs picks the mode at mount time and records it in the log as ACL enforcement: kernel or ACL enforcement: client.

If a kernel is new enough but its FUSE module was built without POSIX ACL support, an --acl mount fails to start with an error naming the kernel, rather than starting in a state where ACLs that grant extra access would be silently ignored. See Caching Architecture for the detail.

ACL-related mount options

Flag	Default	Description
--acl	false	Enable extended ACL support (implies --xAttr)
--noExec	false	Disable execution of files
--noSUID	false	Disable SUID/SGID special permissions
--rootSquash	false	Enable root squashing (implies --acl)
--rootSquashGID	65534	GID to map root to when root squashing is enabled
--rootSquashUID	65534	UID to map root to when root squashing is enabled
--umask	(none)	Explicit umask override in octal notation (e.g., 0002)

These flags can be set locally on the mount command line or centrally via volume flags and volume token flags in configure.flexfs.

Mount options

FlexFS supports standard mount options that affect POSIX behavior:

Flag	Effect
--atTime <RFC3339>	Mount the filesystem at a historical point in time (read-only).
--noATime	Do not update access time on file opens.
--nonEmpty	Allow mounting over a non-empty directory.
--ro	Read-only mount. All write operations return EROFS. Implies --noATime.

Known limitations

Note

These are inherent trade-offs in a distributed FUSE filesystem over object storage, not bugs. They apply to all network FUSE filesystems to varying degrees.

• Byte-range lock granularity (legacy / fallback only): When coordinated concurrent I/O is negotiated — the default for current clients and servers — POSIX byte-range locks use their exact requested bounds and concurrent sub-block writes from different mounts are safe (see File locking). Only when talking to an older client or server that does not support the feature are byte-range locks aligned to block boundaries, so sub-block locking effectively locks the whole block.

• Close-to-open consistency: Data written by one mount client becomes visible to other mount clients after the writing client calls close() or fsync() and the reading client opens the file. Opening the file is what delivers this, so a reader that opens after the writer’s close() has returned always gets the new data — including with O_DIRECT, and no matter how soon after the write it opens. A file a reader already had open when the peer wrote is not covered: those readers catch up on their own, normally within a few milliseconds, and mmap mappings refresh on the same terms. In-progress writes that have not been flushed may not be visible to other clients at all. Checking on open costs one round trip to the metadata server per open(), which workloads that open very large numbers of files may notice. See Caching Architecture.

• Attribute and directory-entry caching: File attributes are cached by the kernel for --attrValid (default: 28800 seconds / 8 hours), and directory-entry (name) lookups — including negative/not-found results — for --entryValid (default: 1 second). The metadata server pushes invalidation notifications when remote clients modify files, so cached attributes are refreshed promptly (typically within milliseconds), and a disrupted notification channel triggers a full kernel attribute/data flush on reconnect; the --attrValid TTL is only a fallback for a missed notification. Directory-entry caches are invalidated on a best-effort basis: a create, rename, or delete on one mount usually reaches a name another mount has already cached within milliseconds, but no client can know every name the kernel holds, so cross-mount name coherence is still bounded by --entryValid in the worst case — and unlike attributes, names are not covered by the reconnect flush. (The mount client additionally keeps an internal dentry cache to cut LOOKUP RPCs; that cache is also the record it uses to decide which names are worth asking the kernel to drop — see Caching Architecture.) Both kernel caches are disabled when --acl is enforced by the mount client rather than the kernel — on kernels older than 4.9, or with --rootSquash — because every lookup must then reach the mount client so the ACL check always runs. On current kernels --acl caches exactly as a normal mount does; see POSIX ACL mode.

• mmap support: Both read-only (mmap with PROT_READ) and writable shared (MAP_SHARED with PROT_WRITE) memory mappings are supported through the kernel page cache. Pages dirtied through a shared mapping are written back to object storage on msync(), on munmap(), and by normal page-cache writeback; the resulting data is durable and becomes visible to other mounts on the usual close-to-open terms. Because flexFS does not enable the kernel’s FUSE writeback cache, mmap-dirtied pages are flushed as ordinary block writes rather than being coalesced, so a heavy random mmap-write workload is less efficient than the equivalent write() calls — but it is correct.

• O_DIRECT alignment: FlexFS imposes no alignment requirement on O_DIRECT I/O. A read or write with a misaligned offset or a length that is not a multiple of the block size succeeds rather than failing with EINVAL. This is permitted — O_DIRECT is not part of POSIX, and open(2) states that its alignment restrictions vary by filesystem and may be absent entirely. The restriction block-backed filesystems enforce comes from handing the user buffer to the block layer for DMA, so the granularity they demand is a property of the underlying device rather than of the filesystem: on a device with 512-byte logical sectors, both xfs and ext4 accept 512-granular offsets and lengths and reject anything finer with EINVAL, and that threshold moves with the device geometry and the kernel version. FlexFS has no such path — the kernel copies the data through the FUSE channel and flexFS then translates the request into block operations against object storage — so no alignment requirement exists to impose. tmpfs, which likewise has no block device beneath it, behaves the same way. Alignment is therefore a performance consideration rather than a correctness one: a sub-block write becomes a read-modify-write of the containing block. Aligning O_DIRECT I/O to the st_blksize reported by stat() — which is the volume block size — avoids that cost.

• SEEK_SET/SEEK_CUR/SEEK_END handling: Standard seek operations (SEEK_SET, SEEK_CUR, SEEK_END) are handled by the kernel’s FUSE layer. FlexFS implements SEEK_DATA and SEEK_HOLE via the Lseek operation, which queries the metadata server for sparse file information.

• Rename atomicity: Rename operations are atomic within the metadata server (the old and new entries are updated in a single transaction), but the associated block data is not moved — blocks remain in object storage under their original keys. This means rename is metadata-only and very fast, but the block key namespace reflects the original inode, not the file name.