Caching Architecture

FlexFS employs a three-tier caching architecture to minimize latency and reduce the number of requests to object storage. Each tier serves a different access pattern and can be independently configured.

Cache tiers at a glance

Tier	Location	Eviction	Writeback	Edition
L1	mount client memory	LRU	No (read cache)	Both
L2	mount client disk	LRU (clean blocks)	Optional	Both
L3	proxy server disk	LRU (clean blocks)	Optional	Enterprise

L1: In-memory LRU cache

The L1 cache is an in-memory LRU (Least Recently Used) cache that sits at the top of the store pipeline, closest to the application. It caches processed (decompressed, decrypted) blocks, so cache hits avoid all processing overhead.

Key characteristics

• Capacity: Configurable via --memCapacity. Accepts a percentage of system RAM (e.g. 2%), a human-readable size (e.g. 4G, 512M), or a plain number of blocks (e.g. 2000). When unset — or set to 0, which selects the auto-size rather than disabling the cache — it is auto-sized to 1.75% of total system RAM divided by the volume’s block size. A minimum of 32 blocks applies in every case, so the cache cannot be turned off.
• Eviction: LRU. When the cache is full, the least recently used block is evicted and its buffer is returned to the pool.
• Coalesced fetches: Concurrent requests for the same block are coalesced into a single downstream fetch using a singleflight mechanism. This prevents cache stampedes when many threads read the same file region simultaneously.
• Prefetch integration: Prefetched blocks are inserted into the L1 cache. If a block is already cached, the prefetch is skipped.
• Write-through: When a block is written (PutBlock), it is added to the L1 cache and simultaneously passed downstream.

Auto-sizing example

On a host with 32 GiB of RAM and a 4 MiB block size:

• 0.02 * 32 GiB / 4 MiB = 163 blocks (approximately 652 MiB of cached data)

L2: On-disk cache

The L2 cache persists blocks to local disk, surviving process restarts and providing much larger capacity than memory. It caches blocks in their processed form (after compression and encryption), meaning cache hits still avoid network I/O but do require decompression and decryption.

Note

The on-disk cache is available in both Enterprise and Community editions. It is not enabled by default; you must set --diskQuota to a non-zero value to activate it.

Configuration flags

Flag	Default	Description
--diskFolder	~/.flexfs/mount/cache/<pid>	Directory for cached block files
--diskMaxBlockSize	262144 (256 KiB)	Maximum processed block size to cache on disk. Blocks larger than this after processing bypass the disk cache. Set to 0 for no limit.
--diskQuota	(empty — disabled)	Maximum disk usage for the cache (e.g., 5%, 64M, 10G)
--diskWriteback	false	Enable writeback mode (see below)

LRU eviction

The disk cache maintains an LRU list for clean blocks (blocks that have been persisted to object storage). When the cache needs space for new blocks:

1. The oldest clean block is evicted from the LRU list.
2. Its disk file is deleted.
3. If no clean blocks remain (the cache is full of dirty blocks in writeback mode), the new block bypasses the cache entirely.

Writeback mode

When --diskWriteback is enabled, the disk cache operates as a writeback cache for writes:

1. The mount client writes the processed block to the disk cache.
2. The write is acknowledged to the application immediately (low latency).
3. A pool of background worker goroutines reads dirty blocks from disk and persists them to object storage asynchronously.
4. After a dirty block is successfully persisted, it transitions to a clean state and becomes eligible for LRU eviction.

Writeback mode is especially useful for:

• Latency-sensitive workloads: Write latency is bounded by local disk speed rather than object storage round-trip time.
• Bursty write patterns: The disk cache absorbs write bursts while background workers drain to storage at a sustainable rate.
• On-premises deployments: When object storage is in a remote cloud region, writeback caching masks the network latency.

The number of writeback worker goroutines equals the maxBops setting (auto-sized based on CPU count if not set). Workers retry indefinitely on transient errors, with randomized backoff capped at 10 seconds.

Caution

In writeback mode, data that has been acknowledged to the application but not yet flushed to object storage can be lost if the mount client host crashes. Call fsync() or close() to ensure data durability. The Flush() method waits for all pending writeback operations to complete before returning.

L3: Proxy group cache (Enterprise)

Proxy groups provide a shared caching layer between mount clients and object storage. They function like a content delivery network (CDN) for block data.

How proxy groups work

1. Group selection: When a mount client starts, it probes all proxy groups configured for its volume by sending a health-check request to the first address in each group. It selects the group with the lowest round-trip time (RTT), using a 125 ms timeout for the probe.

2. Block routing: Within the selected group, blocks are distributed across proxy servers using rendezvous hashing (highest random weight hashing) with xxHash. This ensures that:

   • The same block always routes to the same proxy server (cache consistency).
   • Adding or removing a proxy server only redistributes a fraction of blocks (minimal cache disruption).

3. Proxy-side caching: Each proxy server maintains its own disk cache. On a GET, if the block is cached locally, it is returned immediately. On a cache miss, the proxy fetches the block from object storage, caches it, and returns it to the client.

4. Proxy-side writeback: Proxy servers can operate in writeback mode, where PUT requests are acknowledged as soon as the block is persisted to the proxy’s local disk. The proxy then asynchronously flushes the block to object storage.

5. Graceful fallback: If the selected proxy group becomes unreachable (e.g., network failure), mount clients automatically bypass the proxy and communicate directly with object storage for 5 minutes before retrying the proxy.

Proxy group configuration

Proxy groups are created and associated with volumes via configure.flexfs:

• proxy-group: Defines a group with a provider, region, and comma-separated list of proxy server addresses.
• volume-proxy-group: Associates a proxy group with a volume. A volume can have multiple proxy groups; the mount client selects the best one based on RTT.

Max proxied blocks

Volumes have a maxProxied setting that limits how many blocks per file (by block index) are routed through proxies. Blocks beyond this index bypass the proxy and go directly to object storage. This is useful for large files where only the first portion benefits from proxy caching. When set to 0, all blocks are eligible for proxying.

Dirty block cache

In addition to the three read-cache tiers, mount.flexfs maintains an in-memory dirty block cache for write operations. This cache holds modified blocks that have not yet been synced to storage.

Setting	Default	Description
--dirtyCapacity	Auto-sized (memCapacity/4)	Maximum number of dirty blocks in memory
--dirtyActive	Auto-sized (half of dirtyCapacity)	Maximum number of dirty blocks actively syncing

Dirty blocks are flushed when:

• The application calls fsync(), fdatasync(), or close()
• The dirty cache reaches capacity (back-pressure flush)
• The FUSE Flush operation is triggered

Prefetching

Mount.flexfs includes a block prefetcher that proactively loads blocks into the L1 cache before they are requested.

Setting	Default	Description
--prefetchActive	Auto-sized (memCapacity/8)	Maximum number of concurrent prefetch operations
--noPrefetch	false	Disable prefetching entirely

Prefetched blocks share the singleflight mechanism with regular reads, so a prefetch and a concurrent read for the same block result in a single downstream fetch.

Buffer pool

All block buffers are managed by a reusable buffer pool to minimize memory allocation overhead and garbage collection pressure. The pool capacity is auto-sized based on the combined active prefetch and dirty-sync concurrency.

Setting	Default	Description
--poolCapacity	Auto-sized (prefetchActive + dirtyActive)	Number of reusable block buffers in the pool

--poolCapacity sets how many buffers are kept ready for reuse; it does not cap how many can be in use at once. The mount’s actual buffer usage is driven by demand, and mostly by --memCapacity, so that is the setting to adjust if the mount is using more memory than you expect. flexfs_mount_buffer_pool_borrowed and flexfs_mount_mem_cache_size report both.

FUSE kernel caching

In addition to the userspace caches described above, mount.flexfs configures the Linux FUSE kernel module’s caching behavior:

Setting	Default	Description
--attrValid	28800 (8 hours)	Seconds the kernel caches file attributes
--entryValid	1	Seconds the kernel caches directory entries

These kernel-level caches reduce the number of FUSE round trips for repeated stat() and directory lookups. They are separate from and additive to the block caching tiers.

Attribute coherence (event-driven)

FlexFS keeps the kernel’s attribute cache coherent with an invalidation event bus. When another mount client modifies a file, the metadata server pushes a notification over a WebSocket to all other connected clients (never back to the mount that made the change), and the receiving client immediately invalidates the affected kernel attribute and data-page cache entries — typically within milliseconds. The --attrValid TTL is therefore an upper bound on how long a missed notification could leave attributes stale, not the expected coherence window.

To make this possible each mount tracks which inodes the kernel currently holds a reference to — an nlookup ledger, named for the reference count the FUSE protocol maintains. Because the ledger records every held inode, and not just those still in the mount’s own bounded caches, the client can reach into the kernel and invalidate an inode’s attributes and data pages even after it has aged out of those caches. It also makes invalidation storm-resistant: an event for an inode this mount never referenced is a no-op, with no server-side per-client interest lists. A background check keeps the ledger honest, so flexfs_mount_ledger_size tracks the working set rather than growing without bound.

If the notification connection is disrupted and events may have been missed, the client re-invalidates every inode the ledger records as held, so no stale attribute or data page survives the gap. This recovery path is what makes the long --attrValid TTL safe. It does not extend to kernel dentries — recovery flushes inodes, not names — so names remain bounded by the much shorter --entryValid across a disruption, exactly as they are the rest of the time (see below). Health metrics for all of this are listed under metrics, with ready-made alert rules under alerting.

Close-to-open freshness

Invalidation notifications are fast, but they are not instant. On their own they would leave a brief window in which a reader that opens a file at the moment a peer’s close() returns could still be served the previous contents.

Opening the file closes that window. Whenever a mount opens a file it checks the file’s current state with the metadata server, and if another mount has changed the data since it last looked, it discards what it had cached before the open() returns. An application therefore never reads stale data from a file it has just opened, however soon after the peer’s write it opens it, and this holds for O_DIRECT readers as well.

Files that have not changed — overwhelmingly the common case — keep their cached copies, so repeat reads are still served from the kernel page cache at full speed.

The check costs one round trip to the metadata server per open(). Opening an already-cached file would otherwise need no round trips at all, so workloads that open very large numbers of files — compiler include scans, interpreter module search, large find traversals — are the ones likely to notice. Opens that cannot be serving stale data skip the check automatically: files this mount already has open, files it has just created, and files being truncated as they are opened.

flexfs_mount_cache_events_total{cache="openrefresh"} reports how many opens paid for the check and how often one found a change, so the cost is measurable on a live mount. The check has no supported off switch: the flag that disables it exists for the test suite only, because a mount running without it knowingly returns stale data.

Directories are not covered by this. Cross-mount name coherence is bounded by --entryValid instead, as described next.

Directory-entry (name) coherence

Directory-entry (dentry) caching works differently. The kernel caches name → inode lookups — including negative (not-found) results — for --entryValid seconds, and cross-mount name coherence is bounded by that TTL: in the worst case, a create, rename, or delete on one mount becomes visible to a cached name on another mount only after that mount’s entry TTL expires. (A name a mount has not looked up within the window is unaffected — it is resolved fresh on first access.) A mount’s own mutations keep its own kernel dentry cache correct as part of each syscall reply; the TTL bound applies only to names cached from a peer’s changes.

In practice most peer changes are reflected much sooner than the TTL. When a notification affects a name the kernel may have cached, the mount asks it to drop its cached answer straight away — so a file created, removed, or renamed elsewhere usually appears or disappears within milliseconds, including for names it only saw as part of a directory listing. This is a best-effort improvement rather than a guarantee: there is no way for a client to know every name the kernel has cached, so --entryValid remains the coherence bound to design against and should be kept short.

It is also the first thing to narrow after a disruption. Deciding which names are worth a notification depends on what the mount had cached, and that record is discarded on reconnect along with the rest of the userspace metadata caches — so for a short period afterwards more peer changes fall back to expiring on --entryValid rather than being reflected immediately. Names are the one thing the recovery path above cannot flush, which is the other reason to keep that timeout short.

Internal dentry cache. Beneath the kernel’s dentry cache, each mount also keeps a bounded, in-process cache of its own name → inode resolutions (including negative lookups) — a sibling of the userspace attribute cache whose only purpose is to answer a LOOKUP that reaches the client without a metadata round trip. Across mounts it is kept honest by the same event bus: a peer’s EntryNotify/DeleteNotify refreshes or drops a name this mount already holds (it never seeds a name the mount has not itself looked up), so a stale peer change is corrected within the event-driven window rather than lingering until eviction. Because it records which names this mount has resolved, it is also what decides when a peer change is worth asking the kernel to drop its own cached name — the mechanism described above. It has no configuration flag, and like the other userspace metadata caches it is purged on notification-channel disruption (see recovery above). Capacity eviction is silent and never touches the kernel.

If your workload needs tight cross-mount name freshness — for example, one client must observe another’s newly created or deleted files within seconds — lower --entryValid accordingly. The trade-off is more LOOKUP calls into the mount client (those that miss the internal dentry cache become metadata round trips). Attribute and data coherence are unaffected: those remain event-driven regardless of the entry TTL.

Negative lookups

Directory-entry caching also covers negative lookups: when a name does not exist, the kernel caches that fact for --entryValid seconds, so repeated access to a missing path (common with build tools and shell PATH scans) is answered from the kernel without a metadata round trip. Negative entries are treated exactly like positive ones (above): a create by this mount clears its own cached negative at once, and a peer’s create over a name this mount had cached as missing usually invalidates the kernel’s negative entry within milliseconds. Because that notification is best-effort, the --entryValid TTL remains the worst case — set it shorter if peers must observe newly created names quickly.

POSIX ACL mode

Extended ACLs (--acl) can be enforced on either side of the FUSE boundary, and which side is in use decides whether the kernel caches at all. mount.flexfs chooses at mount time and logs the result as ACL enforcement: kernel or ACL enforcement: client.

Kernel-enforced (Linux 4.9 and later — the normal case). The kernel evaluates ACLs itself, so a cached attribute or name is still checked before it is used. Caching therefore behaves exactly as it does without --acl: the --attrValid and --entryValid defaults above apply unchanged, and coherence works the same way, because the invalidation the mount client already sends for a peer’s change also drops the kernel’s cached ACL.

Client-enforced (the fallback). The mount client checks every operation itself, using each name lookup as the point where directory-traversal permission is verified. A cached name would let the kernel resolve a path without that check, so the kernel’s attribute and directory-entry caches are turned off (the effective --attrValid / --entryValid are 0) and every lookup, including negative ones, reaches the mount client. Cached file data is unaffected. Expect noticeably more metadata round trips — repeated stat() calls on the same file reach the mount client every time instead of being answered by the kernel.

Two situations select the client-enforced path:

• Kernels older than 4.9, which cannot evaluate ACLs for a FUSE filesystem at all. In practice this means RHEL/CentOS 7.
• --rootSquash, on any kernel. Root squashing cannot be delegated: the kernel grants root every access, so only the mount client can re-evaluate the request as the squash user.

Caution

On a kernel 4.9 or newer whose FUSE module was built without POSIX ACL support, an --acl mount fails to start rather than falling back, and the error names the running kernel. Falling back silently is not safe at that point: the kernel has already taken over basic permission checking, and it would refuse any access an ACL grants beyond the standard owner/group/other bits — so the mount would appear to work while quietly denying legitimate access.

The mount client’s own caches are unaffected by any of this and keep working in both modes.