Metrics Reference

The metadata server (meta.flexfs), the proxy server (proxy.flexfs), and the mount client (mount.flexfs) expose Prometheus metrics via an HTTP /metrics endpoint. On the servers the endpoint is always enabled and shares the main bind address; on the mount client it is opt-in (enabled with --metrics) and served on its own port.

Metadata server metrics

Namespace: flexfs_meta

Metric names in the tables below omit this prefix — volume_blocks is flexfs_meta_volume_blocks in full.

RPC metrics

Metric	Type / Labels	Description
rpc_duration_seconds	Histogram · volume_id, method	RPC operation latency in seconds.
rpc_ops_total	Counter · volume_id, method, status	Total RPC operations by method and status.

Histogram buckets (seconds): 0.00001, 0.000025, 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

Block-key and CAS metrics

The op label carries the block-key RPC method: SetBlockKeys, SetBlockKeysAndSize, ListBlockKeys, and the compare-and-swap variants SetBlockKeysCAS / SetBlockKeysAndSizeCAS. The compare-and-swap variants install each block index only if the client’s base key still matches the stored key; a mismatch is reported as a conflict for the client to re-merge and retry. The CAS metrics are recorded only for the two *CAS methods.

Metric	Type / Labels	Description
block_keys_per_call	Histogram · volume_id, op	Number of block keys touched in a single block-key RPC.
cas_conflict_retired_blocks_total	Counter · volume_id	Block objects retired because their compare-and-swap conflicted (uploaded but not installed).
cas_conflicts_per_call	Histogram · volume_id, op	Number of block indices that conflicted in a single compare-and-swap call.
cas_conflicts_total	Counter · volume_id, op	Total block indices that lost a compare-and-swap (base key no longer matched).
cas_installed_total	Counter · volume_id, op	Total block indices installed by a compare-and-swap (base key matched).

block_keys_per_call histogram buckets: 1, 4, 16, 64, 256, 1024, 4096

cas_conflicts_per_call histogram buckets: 0, 1, 2, 4, 8, 16, 64, 256

Volume info metrics

Metric	Type / Labels	Description
volume_info	Gauge · volume_id, volume_name	Volume metadata. Always 1 per volume; use labels to map IDs to names.

Volume gauge metrics

Metric	Type / Labels	Description
volume_blocks	Gauge · volume_id	Number of active blocks.
volume_blocks_retired	Gauge · volume_id	Number of retired blocks.
volume_dentries	Gauge · volume_id	Number of directory entries.
volume_inodes	Gauge · volume_id	Number of inodes.
volume_size_bytes	Gauge · volume_id	Total volume size in bytes.

Volume I/O counters

Metric	Type / Labels	Description
volume_blocks_deleted_total	Counter · volume_id	Total block objects deleted from the block store by the retirement/GC loops.
volume_blocks_read_total	Counter · volume_id	Total blocks fetched from the block store by this volume’s mounts; cache hits are not counted.
volume_blocks_written_total	Counter · volume_id	Total blocks uploaded to the block store by this volume’s mounts.
volume_bytes_read_total	Counter · volume_id	Total bytes read at FUSE boundary.
volume_bytes_written_total	Counter · volume_id	Total bytes written at FUSE boundary.

These four aggregate what every mount of the volume reports, so they carry the same meanings as the mount’s own I/O throughput counters — blocks_* counts remote block operations only, bytes_* counts traffic at the FUSE boundary. Unlike the mount-side counters they advance in steps, on each mount’s 60-second reporting interval, so query them with rate() over a window of a few minutes. They are held in memory, so they restart from zero when the metadata server does; rate() and increase() account for that, but a raw value is only cumulative since the server last started.

Session metrics

Metric	Type / Labels	Description
sessions	Gauge · volume_id	Active client sessions per volume.

Lock metrics

Metric	Type / Labels	Description
append_grants_count	Gauge · volume_id	Append grants currently held (serialized append authority).
lock_sessions_in_memory_count	Gauge · volume_id	Total lock sessions held in memory.
locks_in_memory_count	Gauge · volume_id	Total advisory locks held in memory.

Database folder metrics

Metric	Type / Labels	Description
db_folder_disk_available_bytes	Gauge	Available space on the filesystem containing the database folder. This is the value compared against --minDiskAvail.
db_folder_disk_capacity_bytes	Gauge	Total capacity of the filesystem containing the database folder.

Database internals

Metric	Type / Labels	Description
db_block_cache_hits_total	Counter	Block cache hits (shared across volumes).
db_block_cache_misses_total	Counter	Block cache misses (shared across volumes).
db_block_cache_size_bytes	Gauge	Block cache size in bytes (shared across volumes).
db_compaction_count	Counter · volume_id	Total compactions.
db_compaction_estimated_debt_bytes	Gauge · volume_id	Estimated compaction debt in bytes.
db_disk_usage_bytes	Gauge · volume_id	Total disk space used.
db_flush_count	Counter · volume_id	Total flushes.
db_memtable_count	Gauge · volume_id	Number of memtables.
db_memtable_size_bytes	Gauge · volume_id	Memtable size in bytes.
db_read_amplification	Gauge · volume_id	Read amplification.
db_table_iters	Gauge · volume_id	Open sstable iterators.
db_wal_bytes_in_total	Counter · volume_id	Logical bytes written to WAL.
db_wal_bytes_written_total	Counter · volume_id	Physical bytes written to WAL.
db_wal_files	Gauge · volume_id	Number of live WAL files.
db_wal_size_bytes	Gauge · volume_id	Size of live WAL data in bytes.

---

Proxy server metrics

Namespace: flexfs_proxy

Metric names in the tables below omit this prefix — cache_clean_blocks is flexfs_proxy_cache_clean_blocks in full.

REST metrics

Metric	Type / Labels	Description
rest_duration_seconds	Histogram · method	REST operation latency in seconds.
rest_ops_total	Counter · method, status	Total REST operations by method and status.

Histogram buckets (seconds): 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

Cache metrics

Metric	Type / Labels	Description
cache_clean_blocks	Gauge	Number of clean cached blocks.
cache_clean_bytes	Gauge	Total size of clean cached blocks in bytes.
cache_dirty_blocks	Gauge	Number of dirty blocks pending writeback.
cache_dirty_bytes	Gauge	Total size of dirty blocks in bytes.
cache_disk_capacity_bytes	Gauge	Total capacity of the filesystem containing the cache folder.
cache_disk_quota_bytes	Gauge	Configured disk quota in bytes.

Database folder metrics

Metric	Type / Labels	Description
db_folder_disk_available_bytes	Gauge	Available space on the filesystem containing the database folder. This is the value compared against --minDiskAvail.
db_folder_disk_capacity_bytes	Gauge	Total capacity of the filesystem containing the database folder.

Database internals

Metric	Type / Labels	Description
db_block_cache_hits_total	Counter	Block cache hits.
db_block_cache_misses_total	Counter	Block cache misses.
db_block_cache_size_bytes	Gauge	Block cache size in bytes.
db_compaction_count	Counter	Total compactions.
db_compaction_estimated_debt_bytes	Gauge	Estimated compaction debt in bytes.
db_disk_usage_bytes	Gauge	Total disk space used by index.
db_flush_count	Counter	Total flushes.
db_memtable_count	Gauge	Number of memtables.
db_memtable_size_bytes	Gauge	Memtable size in bytes.
db_read_amplification	Gauge	Read amplification.
db_table_iters	Gauge	Open sstable iterators.
db_wal_bytes_in_total	Counter	Logical bytes written to WAL.
db_wal_bytes_written_total	Counter	Physical bytes written to WAL.
db_wal_files	Gauge	Number of live WAL files.
db_wal_size_bytes	Gauge	Size of live WAL data in bytes.

---

Mount client metrics

Namespace: flexfs_mount

Metric names in the tables below omit this prefix — dirty_blocks is flexfs_mount_dirty_blocks in full.

Unlike the servers, the mount client only exposes metrics when started with --metrics, on the port set by --metricsPort (default 6073). When disabled, no endpoint is served and instrumentation is skipped. Values that are already tracked as live state (cache sizes, queue depths, byte counters) are read only at scrape time; the rest are recorded on the relevant code paths.

FUSE operation metrics

Metric	Type / Labels	Description
fuse_errors_total	Counter · op, status	Total non-OK FUSE replies to the kernel, by op and status (e.g. EIO, ENOENT, EPERM).
fuse_op_duration_seconds	Histogram · op	FUSE operation handling latency in seconds, by op.
fuse_ops_total	Counter · op	Total FUSE operations dispatched, by op.
open_file_handles	Gauge	Currently open file handles.

fuse_op_duration_seconds histogram buckets (seconds): 0.00001, 0.000025, 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

The NotifyReply op

Both op-labelled series above include NotifyReply, which is not driven by the workload. Every residency probe the nlookup ledger sends (the Retrieve op below) makes the kernel queue a reply that the mount must read back, so a healthy idle mount shows a steady low rate of NotifyReply — up to 64/s, and never more. The replies carry no data and need no response, so they cost a dispatch and nothing else. Exclude them when reading flexfs_mount_fuse_ops_total as a measure of workload.

Kernel notification metrics

The mount pushes invalidation/notification ops to the kernel to keep its caches coherent. The op label is one of InvalInode (drop an inode’s cached attributes and data pages), InvalEntry (drop a cached directory-entry name), Retrieve (the nlookup ledger’s residency probe — see caching), or Store.

Retrieve normally dominates the op counter and is not a sign of anything: the ledger probes inodes it has not seen referenced recently, so the rate tracks how fast the mount’s working set goes quiet, and it is capped regardless of how many inodes the mount holds. The probes are non-destructive and cost the kernel nothing it was using.

Metric	Type / Labels	Description
fuse_notify_errors_total	Counter · op	Notification ops that genuinely failed, by op. A target the kernel no longer holds is not counted, for any op: that is the expected answer for a name or inode whose kernel-side copy has already gone, and the mount deliberately keeps records that run ahead of what the kernel holds, so it is a routine outcome rather than a fault. What remains is worth attention — a kernel that declined an operation it understood, or a notification that never reached it.
fuse_notify_ops_total	Counter · op	Total kernel invalidation/notification ops issued, by op.
kernel_dentry_invalidations_dropped_total	Counter	Directory-entry notifications shed because the invalidation queue was full. Benign and needs no action: the affected name refreshes on the --entryValid timeout, which bounds name coherence in any case. Unlike attribute and data invalidations these never wait for queue room, so they are the first thing to give way when the queue fills — the right trade, since a lost attribute or data invalidation goes stale for far longer.
kernel_inval_queue_depth	Gauge	Kernel cache invalidations queued for delivery, across all drain lanes. Sustained depth means the mount is issuing them faster than the kernel is accepting them.
kernel_invalidations_coalesced_total	Counter	Kernel cache invalidations folded into one already queued for the same inode. Work avoided, not lost; a mount under peer churn shows many.
kernel_invalidations_dropped_total	Counter	Kernel cache invalidations dropped because the queue was full. Expected to stay at zero: each one leaves the kernel holding attributes or data pages this mount knows are stale, until it forgets the inode or the attribute timeout expires.
open_attr_invalidations_deferred_total	Counter	Opens where the close-to-open freshness check could not refresh the kernel’s cached attributes before answering, and had to hand the refresh to the invalidation queue instead. The refresh still happens, so this is not lost coherence — but it is no longer ordered ahead of the open, which leaves a brief window in which a peer’s size change can reach the application as a short read. Expected to stay at zero; a non-zero value means the mount is short of notification capacity, so read it alongside flexfs_mount_kernel_inval_queue_depth. See Caching Architecture.

Cache metrics

Metric	Type / Labels	Description
attr_cache_size	Gauge	Number of entries in the attribute cache.
blockkey_cache_size	Gauge	Number of inodes with cached block keys.
cache_events_total	Counter · cache, event	Cache lookups by cache (mem, disk, attr, blockkey, dentry) and outcome (hit, miss). The openrefresh cache is the close-to-open freshness check rather than a lookup: it counts only opens that checked with the metadata server, where hit means the file was unchanged and miss means a peer had changed it and the cached copy was dropped. Its total is the extra metadata load close-to-open consistency costs this mount; see Caching Architecture.
dentry_cache_size	Gauge	Number of entries in the internal (per-mount) dentry cache.
ledger_accounting_faults_total	Counter	Open files the mount was no longer tracking a kernel reference for, counted once per file when the fault is first detected. Expected to stay at zero. A non-zero value means cross-mount invalidation is suppressed for that file for as long as it stays open, so a peer’s changes to it can go unseen — capture the mount log and report it. Nothing repairs the count deliberately — it clears only when a fresh lookup of the name re-establishes the reference, or the file is closed — so read this as a count of affected files, not as a rate that should decay.
ledger_flushes_dropped_total	Counter	Cache flushes for a file with a known-bad lookup count that could not be attempted because the kernel invalidation queue was full, leaving that file’s cached copy unflushed. Indicates sustained invalidation backpressure. A subset of flexfs_mount_kernel_invalidations_dropped_total — do not sum the two, and do not read a zero here as proof that no flush was lost: a flush that folded into an invalidation already queued for the same file, which was then itself dropped, is counted only in that broader metric. It is the one to alert on; this one attributes.
ledger_leaks_reclaimed_total	Counter	Ledger entries reclaimed by the background consistency check. Routine housekeeping — a low rate is normal and needs no action. Useful only for explaining movement in flexfs_mount_ledger_size, which it is directly comparable with.
ledger_sweep_effective_confirm_ttl_seconds	Gauge	How long the background consistency check currently trusts a residency confirmation before re-asking. Floored at one hour and stretched in proportion to flexfs_mount_ledger_size so the check’s probing stays paced rather than saturating on large mounts. This is the number that explains reclamation latency: a leaked ledger entry is reclaimed within roughly this long of the kernel dropping the inode, and a multi-day value on a many-million-entry ledger is by design.
ledger_sweep_entries_examined_total	Counter	Ledger entries walked by the background consistency check, including those it skipped without probing. Diagnostic only. Bounded by a per-pass examine cap (~8k/s): a rate pinned at the cap means eligible entries are scarce and the check is pacing itself — the expected steady state on a large quiet ledger, at a fixed cost regardless of ledger size.
ledger_sweep_probes_total	Counter	Residency probes attempted by the background consistency check, including any that got no answer. Diagnostic only. Steady state sits near the paced re-confirmation target (half the 64/s cap or less) rather than at the cap; a rate pinned at 64/s means a burst — a large listing retiring many inodes into the quiet state at once — or a backlog draining, and should subside.
ledger_sweep_yields_total	Counter	Passes skipped because the kernel invalidation queue was busy. The check always defers to invalidation work and has no deadline, so occasional yields are by design; a sustained rate alongside a growing flexfs_mount_ledger_size means it is being starved.
ledger_size	Gauge	Number of inodes the kernel currently holds a lookup reference to (nlookup ledger).

I/O throughput metrics

Metric	Type / Labels	Description
block_bytes_read_total	Counter · tier	Total block bytes read, by tier (on-store buffer size).
block_bytes_written_total	Counter · tier	Total block bytes written, by tier (on-store buffer size).
blocks_read_total	Counter	Total blocks fetched from the remote block store; cache hits are not counted.
blocks_written_total	Counter	Total blocks uploaded to the remote block store.
fuse_bytes_read_total	Counter	Total bytes read at the FUSE boundary.
fuse_bytes_written_total	Counter	Total bytes written at the FUSE boundary.

All six counters update in real time, as each operation completes. The blocks_* counters count remote block operations: a read served from the mem or disk cache never reaches the block store and so is not counted, which is what makes them worth reading against the mount’s cache_events_total hit rates. They count the same operations as block_bytes_*, so dividing the bytes by the blocks gives the average on-store block size. The block_bytes_* counters record the on-store buffer size (post-compression/encryption) actually transferred, split by the tier label: proxy for traffic served through a proxy group, direct for traffic to the object store backend (including proxy-bypass fallbacks). Both tiers are always present, at zero until traffic flows. Sum over tier for the total, or keep it to attribute amplification to one tier; comparing against fuse_bytes_* quantifies read/write amplification. The tier label uses the same values as block_op_duration_seconds, so throughput and latency can be sliced or joined by tier in a single query.

Block storage metrics

Latency of remote block operations, split by tier (proxy when served through a proxy group, direct when read/written directly against the backend) and op (get, put). The tier label carries the same two values as the throughput counters, so latency and throughput can be sliced or joined by tier in a single query. Only successful operations are timed; failures are counted separately in store_errors_total.

Metric	Type / Labels	Description
block_op_duration_seconds	Histogram · tier, op	Remote block operation latency in seconds.
store_errors_total	Counter · layer, op	Block store errors, by layer (blockstore for synchronous failures, disk for asynchronous disk-cache write-back failures) and op (get, put).

block_op_duration_seconds histogram buckets (seconds): 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30

Block processing metrics

In-process transform latency. The op label is one of encrypt, decrypt, compress, decompress. Encrypt/decrypt are recorded only when volume encryption is enabled.

Metric	Type / Labels	Description
process_duration_seconds	Histogram · op	In-process block transform latency in seconds.

process_duration_seconds histogram buckets (seconds): 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5

Write-back and CAS metrics

Metric	Type / Labels	Description
cas_conflicts_total	Counter · op	Block-key compare-and-swap conflicts, by op.
cas_retries_total	Counter · op	Block-key compare-and-swap retries, by op.
dirty_blocks	Gauge	Dirty blocks awaiting write-back.
sync_slots_capacity	Gauge	Configured write-back sync slot capacity (--dirtyActive).
sync_slots_in_use	Gauge	Write-back sync slots currently in use.

Prefetch metrics

Metric	Type / Labels	Description
prefetch_active	Gauge	Currently active prefetch streams.
prefetch_issued_total	Counter	Read-ahead prefetches dispatched to the block store.
prefetch_skipped_total	Counter · reason	Scheduled prefetches skipped before fetching, by reason (dirty, empty, pending, key_error).
prefetch_slots	Gauge	Configured maximum concurrent prefetch operations (--prefetchActive).

Metadata connection metrics

Metric	Type / Labels	Description
meta_connected	Gauge	Whether the meta-server connection is up (1) or down (0).
meta_disconnects_total	Counter	Meta-server connection-loss events.
meta_inflight_requests	Gauge	In-flight requests to the meta server.
meta_rpc_duration_seconds	Histogram · method	Meta-server RPC round-trip latency in seconds, by method.
meta_rpc_errors_total	Counter · method, status	Meta-server RPC responses with a genuine failure status, by method and status. Expected outcomes (ErrNotFound, ErrConflict, etc.) are not counted as errors.
meta_send_queue_depth	Gauge	Depth of the meta-server send queue.

meta_rpc_duration_seconds histogram buckets (seconds): 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

Buffer pool metrics

The pool label identifies which pool the buffers belong to:

pool	Buffers used for
Block	File data moving between the mount and the block store. The busiest of the three, and the one sized by --memCapacity and --dirtyCapacity.
Fuse	Requests and replies exchanged with the kernel. Sized automatically. Appears once the mount has completed its handshake with the kernel, so it is briefly absent at startup.
Name	File and directory names.

Metric	Type / Labels	Description
buffer_pool_borrowed	Gauge · pool	Buffers currently in use.
buffer_pool_borrowed_total	Counter · pool	Total buffers taken from the pool over the life of the mount.
buffer_pool_created_total	Counter · pool	Total buffers allocated by the pool. Stays far below borrowed_total when the pool is reusing buffers effectively.
buffer_pool_returned_total	Counter · pool	Total buffers returned to the pool.

buffer_pool_borrowed is a current count, not a running total — use buffer_pool_borrowed_total if you want a rate.

A high steady buffer_pool_borrowed{pool="Block"} is normal and is not a leak. The caches sized by --memCapacity and --dirtyCapacity account for most of it, so it scales with those settings. Judge it by whether it grows without bound over hours, not by its absolute value.

Memory cache metrics

Always present on mount.flexfs. The in-memory (L1) block cache cannot be turned off: passing --memCapacity 0 selects the automatic size (1.75% of system RAM) rather than disabling the cache, and a floor of 32 blocks applies in every case.

Both metrics are counted in blocks. Multiply by the volume’s block size for an approximate memory footprint.

Metric	Type / Labels	Description
mem_cache_size	Gauge	Blocks currently held in the in-memory block cache.
mem_cache_capacity	Gauge	Configured maximum blocks, as resolved from --memCapacity. Constant for the life of the mount.

This cache is normally the largest single consumer of the mount’s memory, so these two metrics are what let you attribute that memory to a setting you control. To judge whether --memCapacity needs raising, read occupancy against capacity alongside the mem hit rate from cache_events_total — neither answers the question on its own:

Occupancy	mem hit rate	Reading
Below capacity	Any	Not under pressure; the working set fits.
At capacity	High	Correctly sized.
At capacity	Poor	The working set does not fit, and every miss costs a fetch from the block store. This is when raising --memCapacity pays.

# L1 utilization.
flexfs_mount_mem_cache_size / flexfs_mount_mem_cache_capacity

# Approximate resident bytes held by the L1 cache (4 MiB default block size).
flexfs_mount_mem_cache_size * 4 * 1024 * 1024

Disk cache metrics

Present only when a local disk cache is configured (--diskQuota). When none is, these series are absent rather than zero, and their absence is not a fault — it is how a mount running without an L2 cache looks. The same is true of cache_events_total for the disk cache, which reports zero hits and zero misses. Guard any expression that divides by disk_cache_quota_bytes accordingly.

Metric	Type / Labels	Description
disk_cache_clean_bytes	Gauge	Bytes of clean (uploaded) blocks held in the disk cache.
disk_cache_dirty_bytes	Gauge	Bytes of dirty (not-yet-uploaded) blocks held in the disk cache.
disk_cache_quota_bytes	Gauge	Configured disk cache capacity in bytes.

---

Standard Go and process metrics

All three components also expose the default Prometheus Go runtime and process collectors:

• go_* metrics (goroutines, GC stats, memory allocation)
• process_* metrics (CPU time, resident memory, file descriptors, start time)

Do not use process_network_* for per-mount bandwidth

process_network_receive_bytes_total and process_network_transmit_bytes_total measure the whole network namespace rather than just this process, and can count the same traffic more than once. Expect them to read well above what the mount is actually moving.

For a mount’s traffic to and from the block store, use flexfs_mount_block_bytes_read_total and flexfs_mount_block_bytes_written_total instead. Those are measured by the mount itself. Keep the process_network_* counters for checking whether a host’s network is saturated overall.

To see how much the block store is doing per byte the applications actually read or write, compare the block counters against the FUSE counters:

# Bytes fetched from the block store per byte delivered to applications.
# The block counters carry a `tier` label the FUSE counters do not, so sum it
# away first — otherwise the division matches nothing and returns no result.
sum without (tier) (rate(flexfs_mount_block_bytes_read_total[5m]))
  / rate(flexfs_mount_fuse_bytes_read_total[5m])

# The same for writes.
sum without (tier) (rate(flexfs_mount_block_bytes_written_total[5m]))
  / rate(flexfs_mount_fuse_bytes_written_total[5m])

A ratio above 1 is expected, because a cache miss fetches a whole block and a partial write rewrites one. A persistently large ratio alongside small average request sizes suggests the volume’s block size does not suit the workload.

Next steps

• Prometheus setup — scrape configuration
• Grafana dashboards — visualization
• Alerting — alert rules based on these metrics
• Logging and diagnostics — logs, RTT logging, and pprof profiling