Performance Tuning

This guide covers the key tuning parameters for optimizing flexFS performance across different workloads.

Block size

The block size determines how file data is split into chunks for storage. It is configured at volume creation time and cannot be changed afterward.

Block size	Best for	Trade-offs
256Ki (256 KiB)	Small files, random access	More metadata overhead, more requests per large file
512Ki (512 KiB)	Mixed workloads	Balanced
1Mi (1 MiB)	General purpose	Good default for most workloads
2Mi (2 MiB)	Large sequential files	Less metadata, fewer requests
4Mi (4 MiB)	Large files, HPC, genomics	Default. Optimal for files > 100 MB
8Mi (8 MiB)	Very large sequential files, streaming	Highest throughput for large files, wastes space on small files

Tip

For workloads dominated by large files (genomics BAM/CRAM, AI/ML training data, video), use 4Mi or 8Mi. For workloads with many small files (source code, configuration), use 256Ki or 512Ki.

Memory cache sizing

The in-memory block cache reduces latency by keeping recently accessed blocks in RAM.

Flag	Default	Description
--dirtyCapacity	Auto-tuned	Number of dirty (unwritten) blocks to buffer in memory before flushing.
--memCapacity	Auto-tuned	In-memory LRU cache capacity. Accepts a percentage of system RAM (e.g. 2%), a human-readable size (e.g. 4G, 512M), or a number of blocks (e.g. 2000).
--poolCapacity	Auto-tuned	Size of the block buffer pool.

The defaults are automatically calculated based on available system RAM. Override them for specific workloads:

mount.flexfs start my-volume /mnt/data --memCapacity 4G --dirtyCapacity 512

Higher --memCapacity improves read performance for working sets that fit in memory. Higher --dirtyCapacity improves write throughput by allowing more blocks to be buffered before being flushed to storage.

Disk cache

The on-disk cache provides a second tier of caching that can be much larger than the memory cache. It is per-process and does not survive a restart: the mount client deletes and recreates the cache folder at startup, so --diskFolder must point at a directory dedicated to the flexFS cache. The default path is <pid>-scoped for that reason — keep the <pid> component if you override it, or two mounts on the same host will destroy each other’s cache.

Flag	Default	Description
--diskFolder	~/.flexfs/mount/cache/<pid>	Path to the on-disk cache folder
--diskMaxBlockSize	262144	Maximum processed block size (bytes) that will be cached to disk
--diskQuota	(disabled)	Maximum disk space to use. Accepts absolute values (e.g., 50G) or percentages (e.g., 80%). Disk caching is disabled unless this is set.

mount.flexfs start my-volume /mnt/data \
  --diskFolder /var/cache/flexfs/<pid> \
  --diskQuota 100G

Disk writeback

Enable disk-level writeback caching to mask write latency:

mount.flexfs start my-volume /mnt/data \
  --diskFolder /var/cache/flexfs/<pid> \
  --diskWriteback

With --diskWriteback enabled, writes are acknowledged as soon as the block is written to the local disk cache. The block is then asynchronously uploaded to object storage (or the proxy). This significantly reduces write latency for workloads that can tolerate a short window where data exists only on local disk.

Caution

With disk writeback, data written but not yet uploaded to object storage will be lost if the local disk fails. Use a reliable local storage medium.

Write tuning

Flag	Default	Description
--dirtyActive	Auto-tuned	Maximum number of dirty blocks that can be actively syncing to storage simultaneously. Higher values increase write throughput at the cost of more concurrent network connections.
--maxBops	Auto-tuned	Maximum number of block operations (reads and writes) that can run in parallel.

For write-heavy workloads:

mount.flexfs start my-volume /mnt/data --dirtyActive 64 --maxBops 128

Prefetch tuning

Block prefetching detects sequential read patterns and preloads upcoming blocks before they are requested.

Flag	Default	Description
--noPrefetch	false	Disable prefetching entirely.
--prefetchActive	Auto-tuned	Maximum number of prefetch operations running in parallel.

Prefetching is what carries a single sequential reader. Its budget is how much data one stream can have in flight at once, and kernel readahead does not substitute for it, because one stream cannot use a readahead window larger than its own. Starve the prefetch budget and a single stream falls back to whatever its readahead window alone can keep in flight — on an object store, that is a large loss.

The default is derived from the memory cache size and is at or near optimal on the configurations we have measured, so prefer not to set --prefetchActive at all. Raising it above the default buys little, and lowering it costs a great deal on sequential reads.

--noPrefetch is for mounts serving genuinely random reads only, where the prefetches are wasted work. Be aware of what it costs anything sequential sharing that mount:

mount.flexfs start my-volume /mnt/data --noPrefetch

For a single sequential reader this is a large reduction in throughput, and widening the readahead window does not recover it. It also applies to O_DIRECT readers, which bypass the page cache and so have no readahead at all — prefetching is the only thing giving them depth.

FUSE tuning

These flags control the Linux FUSE interface behavior:

Flag	Default	Description
--attrValid	28800	Time in seconds for which file attributes are cached in the kernel. Higher values reduce metadata server load. The invalidation event bus keeps cached attributes coherent across mounts, so the TTL is only a fallback bound.
--entryValid	1	Time in seconds for which directory entry (name) lookups — including negative/not-found results — are cached in the kernel. When a peer changes a name the kernel may have cached, the client asks it to drop its cached answer straight away, so most peer changes are reflected within milliseconds. That is best-effort — no client can know every name the kernel has cached — so cross-mount name coherence is still bounded by this TTL; lower it if peers must observe created/renamed/deleted names quickly.
--noMaxPages	false	Do not set the FUSE max_pages option to its maximum value. By default, flexFS maximizes FUSE page size for best throughput.
--readAhead	auto	Kernel readahead window (e.g. 4M). Auto-sized; see Kernel readahead.

FUSE async reads are always enabled, so the kernel issues readahead requests concurrently rather than one at a time. There is no flag for this — a deep readahead window without async reads is slower than no readahead at all, because the reading process blocks until the whole window has been fetched serially.

Nothing needs to be enabled for maximum single-client throughput: the 8-hour --attrValid default already maximizes attribute caching, --entryValid is deliberately short (1 second) for cross-mount name coherence, and both the readahead window and the prefetch budget are auto-sized.

O_DIRECT is worth a note here. It bypasses the page cache, so kernel readahead does not apply to it at all and block prefetching is its only source of depth. Sequential readers gain nothing from opening files with O_DIRECT on a flexFS mount, and lose the readahead half of the two mechanisms described under Kernel readahead.

The defaults suit multi-client workloads for attributes: the invalidation event bus keeps cached attributes coherent across mounts (typically within milliseconds), and a disrupted notification channel triggers an attribute/data flush on reconnect — so attribute freshness does not depend on a short --attrValid. Directory-entry (name) coherence is weaker. A peer’s create, rename, or delete does prompt an immediate invalidation of the kernel’s cached name, so it is usually visible within milliseconds — but no client can know every name the kernel holds, so that is best-effort, and --entryValid remains the worst case for an already-cached name. It is also the one bound the reconnect flush does not cover: recovery flushes inodes, not names. The mount client’s internal dentry cache absorbs the extra LOOKUPs a lower --entryValid produces, so most are answered locally instead of becoming metadata-server round trips (see Caching Architecture). Lower --attrValid for a tighter bound on a missed attribute invalidation, and lower --entryValid if clients must observe each other’s name changes quickly:

mount.flexfs start my-volume /mnt/data \
  --attrValid 60 \
  --entryValid 60

Kernel readahead

Kernel readahead and block prefetching each carry a different case, and neither replaces the other. Prefetching gives a single stream its depth (see Prefetch tuning). The readahead window gives concurrent streams theirs, because the prefetch budget is shared across the whole mount while the window applies per stream: split across many readers the budget leaves each one less than its own window provides. With several streams running, the window is the setting that matters, and neither the prefetch budget nor the block-operation budget stands in for it.

The mount client sets the window itself at mount time, so no manual tuning is normally needed. It is auto-sized as the smallest of:

• 8 MiB, a deliberate ceiling. Above it the per-stream page commitment — the kernel allocates and locks a whole window’s worth of pages before issuing the requests — turns into reclaim pressure, and throughput falls again at high stream counts. It also bounds the mmap path, where read_ahead_kb is the read-around size and no sequential-access test applies, so faults on a mapped file fetch far more than was asked for.
• A quarter of the FUSE background request queue, so one stream cannot monopolize it.
• The client’s in-flight read budget, derived from --memCapacity, so readahead cannot evict the very blocks it is filling.

Override it with --readAhead if you have measured a reason to:

mount.flexfs start my-volume /mnt/data --readAhead 4M

Raising it well past the default is counterproductive, in two ways that pull in the same direction. Sequential throughput stops improving once the window covers a full-size FUSE request and then declines at high stream counts as the pinned pages become reclaim pressure. Meanwhile the cost on the mmap path grows in proportion to the window, since read-around fetches the whole of it for a single page fault. A window a little larger than the default is unlikely to help and certain to cost more on mapped files.

Applications that fault randomly over a large mapping should tell the kernel so with madvise(MADV_RANDOM), which disables read-around for that mapping entirely and makes the window irrelevant to them.

To inspect the effective value, resolve the mount’s backing device from its device number — for a FUSE mount this is an anonymous device allocated at mount time, so it changes on every mount and cannot be derived from /dev/fuse:

# major:minor of the mount's superblock, e.g. "0:52"
BDI=$(awk '$5=="/mnt/data" {print $3}' /proc/self/mountinfo)
cat /sys/class/bdi/$BDI/read_ahead_kb

Writing the window requires root, so an unprivileged mount keeps the kernel default and logs a warning naming the path it could not write. In a container, a read-only /sys has the same effect.

Directory tuning

Flag	Default	Description
--dirPageSize	5000	Number of directory entries per page in the directory stream. Larger values improve performance for large directories.
--dirTTL	10	Time in seconds for which directory stream pages are cached.

Proxy group placement

For Enterprise deployments using proxy groups:

• Place proxy servers in the same region as the mount clients they serve.
• Mount clients automatically select the lowest-latency proxy group via RTT probing.
• Use multiple proxy servers per group for load distribution (blocks are distributed via rendezvous hashing).

Workload-specific recommendations

Genomics / HPC

mount.flexfs start genomics-vol /mnt/data \
  --diskFolder /nvme/flexfs-cache/<pid> \
  --diskQuota 500G

The prefetch budget and readahead window are both auto-tuned and are deliberately left alone here; see Prefetch tuning.

AI/ML training data

mount.flexfs start training-vol /mnt/data \
  --diskFolder /nvme/flexfs-cache/<pid> \
  --diskQuota 1T

General interactive workloads

mount.flexfs start shared-vol /mnt/data \
  --diskFolder /var/cache/flexfs/<pid> \
  --diskQuota 80%

Next steps

• Capacity planning
• Logging and diagnostics — use RTT logging to identify bottlenecks
• Caching architecture — understand the three-tier cache model