Skip to content

Metrics Reference

The metadata server (meta.flexfs), the proxy server (proxy.flexfs), and the mount client (mount.flexfs) expose Prometheus metrics via an HTTP /metrics endpoint. On the servers the endpoint is always enabled and shares the main bind address; on the mount client it is opt-in (enabled with --metrics) and served on its own port.

Namespace: flexfs_meta

MetricTypeLabelsDescription
flexfs_meta_rpc_duration_secondsHistogramvolume_id, methodRPC operation latency in seconds.
flexfs_meta_rpc_ops_totalCountervolume_id, method, statusTotal RPC operations by method and status.

Histogram buckets (seconds): 0.00001, 0.000025, 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

The op label carries the block-key RPC method: SetBlockKeys, SetBlockKeysAndSize, ListBlockKeys, and the compare-and-swap variants SetBlockKeysCAS / SetBlockKeysAndSizeCAS. The compare-and-swap variants install each block index only if the client’s base key still matches the stored key; a mismatch is reported as a conflict for the client to re-merge and retry. The CAS metrics are recorded only for the two *CAS methods.

MetricTypeLabelsDescription
flexfs_meta_block_keys_per_callHistogramvolume_id, opNumber of block keys touched in a single block-key RPC.
flexfs_meta_cas_conflict_retired_blocks_totalCountervolume_idBlock objects retired because their compare-and-swap conflicted (uploaded but not installed).
flexfs_meta_cas_conflicts_per_callHistogramvolume_id, opNumber of block indices that conflicted in a single compare-and-swap call.
flexfs_meta_cas_conflicts_totalCountervolume_id, opTotal block indices that lost a compare-and-swap (base key no longer matched).
flexfs_meta_cas_installed_totalCountervolume_id, opTotal block indices installed by a compare-and-swap (base key matched).

block_keys_per_call histogram buckets: 1, 4, 16, 64, 256, 1024, 4096

cas_conflicts_per_call histogram buckets: 0, 1, 2, 4, 8, 16, 64, 256

MetricTypeLabelsDescription
flexfs_meta_volume_infoGaugevolume_id, volume_nameVolume metadata. Always 1 per volume; use labels to map IDs to names.
MetricTypeLabelsDescription
flexfs_meta_volume_blocksGaugevolume_idNumber of active blocks.
flexfs_meta_volume_blocks_retiredGaugevolume_idNumber of retired blocks.
flexfs_meta_volume_dentriesGaugevolume_idNumber of directory entries.
flexfs_meta_volume_inodesGaugevolume_idNumber of inodes.
flexfs_meta_volume_size_bytesGaugevolume_idTotal volume size in bytes.
MetricTypeLabelsDescription
flexfs_meta_volume_blocks_deleted_totalCountervolume_idTotal block objects deleted from the block store by the retirement/GC loops.
flexfs_meta_volume_blocks_read_totalCountervolume_idTotal blocks read.
flexfs_meta_volume_blocks_written_totalCountervolume_idTotal blocks written.
flexfs_meta_volume_bytes_read_totalCountervolume_idTotal bytes read at FUSE boundary.
flexfs_meta_volume_bytes_written_totalCountervolume_idTotal bytes written at FUSE boundary.
MetricTypeLabelsDescription
flexfs_meta_sessionsGaugevolume_idActive client sessions per volume.
MetricTypeLabelsDescription
flexfs_meta_lock_sessions_in_memory_countGaugevolume_idTotal lock sessions held in memory.
flexfs_meta_locks_in_memory_countGaugevolume_idTotal advisory locks held in memory.
MetricTypeLabelsDescription
flexfs_meta_db_folder_disk_available_bytesGauge(none)Available space on the filesystem containing the database folder. This is the value compared against --minDiskAvail.
flexfs_meta_db_folder_disk_capacity_bytesGauge(none)Total capacity of the filesystem containing the database folder.
MetricTypeLabelsDescription
flexfs_meta_db_block_cache_hits_totalCounter(none)Block cache hits (shared across volumes).
flexfs_meta_db_block_cache_misses_totalCounter(none)Block cache misses (shared across volumes).
flexfs_meta_db_block_cache_size_bytesGauge(none)Block cache size in bytes (shared across volumes).
flexfs_meta_db_compaction_countCountervolume_idTotal compactions.
flexfs_meta_db_compaction_estimated_debt_bytesGaugevolume_idEstimated compaction debt in bytes.
flexfs_meta_db_disk_usage_bytesGaugevolume_idTotal disk space used.
flexfs_meta_db_flush_countCountervolume_idTotal flushes.
flexfs_meta_db_memtable_countGaugevolume_idNumber of memtables.
flexfs_meta_db_memtable_size_bytesGaugevolume_idMemtable size in bytes.
flexfs_meta_db_read_amplificationGaugevolume_idRead amplification.
flexfs_meta_db_table_itersGaugevolume_idOpen sstable iterators.
flexfs_meta_db_wal_bytes_in_totalCountervolume_idLogical bytes written to WAL.
flexfs_meta_db_wal_bytes_written_totalCountervolume_idPhysical bytes written to WAL.
flexfs_meta_db_wal_filesGaugevolume_idNumber of live WAL files.
flexfs_meta_db_wal_size_bytesGaugevolume_idSize of live WAL data in bytes.

Namespace: flexfs_proxy

MetricTypeLabelsDescription
flexfs_proxy_rest_duration_secondsHistogrammethodREST operation latency in seconds.
flexfs_proxy_rest_ops_totalCountermethod, statusTotal REST operations by method and status.

Histogram buckets (seconds): 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

MetricTypeLabelsDescription
flexfs_proxy_cache_clean_blocksGauge(none)Number of clean cached blocks.
flexfs_proxy_cache_clean_bytesGauge(none)Total size of clean cached blocks in bytes.
flexfs_proxy_cache_dirty_blocksGauge(none)Number of dirty blocks pending writeback.
flexfs_proxy_cache_dirty_bytesGauge(none)Total size of dirty blocks in bytes.
flexfs_proxy_cache_disk_capacity_bytesGauge(none)Total capacity of the filesystem containing the cache folder.
flexfs_proxy_cache_disk_quota_bytesGauge(none)Configured disk quota in bytes.
MetricTypeLabelsDescription
flexfs_proxy_db_folder_disk_available_bytesGauge(none)Available space on the filesystem containing the database folder. This is the value compared against --minDiskAvail.
flexfs_proxy_db_folder_disk_capacity_bytesGauge(none)Total capacity of the filesystem containing the database folder.
MetricTypeLabelsDescription
flexfs_proxy_db_block_cache_hits_totalCounter(none)Block cache hits.
flexfs_proxy_db_block_cache_misses_totalCounter(none)Block cache misses.
flexfs_proxy_db_block_cache_size_bytesGauge(none)Block cache size in bytes.
flexfs_proxy_db_compaction_countCounter(none)Total compactions.
flexfs_proxy_db_compaction_estimated_debt_bytesGauge(none)Estimated compaction debt in bytes.
flexfs_proxy_db_disk_usage_bytesGauge(none)Total disk space used by index.
flexfs_proxy_db_flush_countCounter(none)Total flushes.
flexfs_proxy_db_memtable_countGauge(none)Number of memtables.
flexfs_proxy_db_memtable_size_bytesGauge(none)Memtable size in bytes.
flexfs_proxy_db_read_amplificationGauge(none)Read amplification.
flexfs_proxy_db_table_itersGauge(none)Open sstable iterators.
flexfs_proxy_db_wal_bytes_in_totalCounter(none)Logical bytes written to WAL.
flexfs_proxy_db_wal_bytes_written_totalCounter(none)Physical bytes written to WAL.
flexfs_proxy_db_wal_filesGauge(none)Number of live WAL files.
flexfs_proxy_db_wal_size_bytesGauge(none)Size of live WAL data in bytes.

Namespace: flexfs_mount

Unlike the servers, the mount client only exposes metrics when started with --metrics, on the port set by --metricsPort (default 6073). When disabled, no endpoint is served and instrumentation is skipped. Values that are already tracked as live state (cache sizes, queue depths, byte counters) are read only at scrape time; the rest are recorded on the relevant code paths.

MetricTypeLabelsDescription
flexfs_mount_fuse_errors_totalCounterop, statusTotal non-OK FUSE replies to the kernel, by op and status (e.g. EIO, ENOENT, EPERM).
flexfs_mount_fuse_op_duration_secondsHistogramopFUSE operation handling latency in seconds, by op.
flexfs_mount_fuse_ops_totalCounteropTotal FUSE operations dispatched, by op.
flexfs_mount_open_file_handlesGauge(none)Currently open file handles.

fuse_op_duration_seconds histogram buckets (seconds): 0.00001, 0.000025, 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

The mount pushes invalidation/notification ops to the kernel to keep its caches coherent. The op label is one of InvalInode (drop an inode’s cached attributes and data pages), InvalEntry (drop a cached directory-entry name), Retrieve (the nlookup ledger’s residency probe — see caching), or Store.

Retrieve normally dominates the op counter and is not a sign of anything: the ledger probes inodes it has not seen referenced recently, so the rate tracks how fast the mount’s working set goes quiet, and it is capped regardless of how many inodes the mount holds. The probes are non-destructive and cost the kernel nothing it was using.

MetricTypeLabelsDescription
flexfs_mount_fuse_notify_errors_totalCounteropNotification ops that genuinely failed, by op. A target the kernel no longer holds is not counted, for any op: that is the expected answer for a name or inode whose kernel-side copy has already gone, and the mount deliberately keeps records that run ahead of what the kernel holds, so it is a routine outcome rather than a fault. What remains is worth attention — a kernel that declined an operation it understood, or a notification that never reached it.
flexfs_mount_fuse_notify_ops_totalCounteropTotal kernel invalidation/notification ops issued, by op.
flexfs_mount_kernel_dentry_invalidations_dropped_totalCounter(none)Directory-entry notifications shed because the invalidation queue was full. Benign and needs no action: the affected name refreshes on the --entryValid timeout, which bounds name coherence in any case. Unlike attribute and data invalidations these never wait for queue room, so they are the first thing to give way when the queue fills — the right trade, since a lost attribute or data invalidation goes stale for far longer.
flexfs_mount_kernel_inval_queue_depthGauge(none)Kernel cache invalidations queued for delivery, across all drain lanes. Sustained depth means the mount is issuing them faster than the kernel is accepting them.
flexfs_mount_kernel_invalidations_coalesced_totalCounter(none)Kernel cache invalidations folded into one already queued for the same inode. Work avoided, not lost; a mount under peer churn shows many.
flexfs_mount_kernel_invalidations_dropped_totalCounter(none)Kernel cache invalidations dropped because the queue was full. Expected to stay at zero: each one leaves the kernel holding attributes or data pages this mount knows are stale, until it forgets the inode or the attribute timeout expires.
flexfs_mount_open_attr_invalidations_deferred_totalCounter(none)Opens where the close-to-open freshness check could not refresh the kernel’s cached attributes before answering, and had to hand the refresh to the invalidation queue instead. The refresh still happens, so this is not lost coherence — but it is no longer ordered ahead of the open, which leaves a brief window in which a peer’s size change can reach the application as a short read. Expected to stay at zero; a non-zero value means the mount is short of notification capacity, so read it alongside flexfs_mount_kernel_inval_queue_depth. See Caching Architecture.
MetricTypeLabelsDescription
flexfs_mount_attr_cache_sizeGauge(none)Number of entries in the attribute cache.
flexfs_mount_blockkey_cache_sizeGauge(none)Number of inodes with cached block keys.
flexfs_mount_cache_events_totalCountercache, eventCache lookups by cache (mem, disk, attr, blockkey, dentry) and outcome (hit, miss). The openrefresh cache is the close-to-open freshness check rather than a lookup: it counts only opens that checked with the metadata server, where hit means the file was unchanged and miss means a peer had changed it and the cached copy was dropped. Its total is the extra metadata load close-to-open consistency costs this mount; see Caching Architecture.
flexfs_mount_dentry_cache_sizeGauge(none)Number of entries in the internal (per-mount) dentry cache.
flexfs_mount_ledger_accounting_faults_totalCounter(none)Open files the mount was no longer tracking a kernel reference for, counted once per file when the fault is first detected. Expected to stay at zero. A non-zero value means cross-mount invalidation is suppressed for that file for as long as it stays open, so a peer’s changes to it can go unseen — capture the mount log and report it. Nothing repairs the count deliberately — it clears only when a fresh lookup of the name re-establishes the reference, or the file is closed — so read this as a count of affected files, not as a rate that should decay.
flexfs_mount_ledger_flushes_dropped_totalCounter(none)Cache flushes for a file with a known-bad lookup count that could not be attempted because the kernel invalidation queue was full, leaving that file’s cached copy unflushed. Indicates sustained invalidation backpressure. A subset of flexfs_mount_kernel_invalidations_dropped_total — do not sum the two, and do not read a zero here as proof that no flush was lost: a flush that folded into an invalidation already queued for the same file, which was then itself dropped, is counted only in that broader metric. It is the one to alert on; this one attributes.
flexfs_mount_ledger_leaks_reclaimed_totalCounter(none)Ledger entries reclaimed by the background consistency check. Routine housekeeping — a low rate is normal and needs no action. Useful only for explaining movement in flexfs_mount_ledger_size, which it is directly comparable with.
flexfs_mount_ledger_sweep_entries_examined_totalCounter(none)Ledger entries walked by the background consistency check, including those it skipped without probing. Diagnostic only. Compare its rate against flexfs_mount_ledger_size: roughly one ledger per second means every pass is scanning everything to find little, which is the signal that the check’s scan is costing more than it returns.
flexfs_mount_ledger_sweep_probes_totalCounter(none)Residency probes attempted by the background consistency check, including any that got no answer. Diagnostic only. A rate sitting at the 64/s cap means the check is running flat out and entries are re-examined more slowly the larger the mount’s working set — expected on large mounts, and harmless, since a late reclaim costs only memory.
flexfs_mount_ledger_sweep_yields_totalCounter(none)Passes skipped because the kernel invalidation queue was busy. The check always defers to invalidation work and has no deadline, so occasional yields are by design; a sustained rate alongside a growing flexfs_mount_ledger_size means it is being starved.
flexfs_mount_ledger_sizeGauge(none)Number of inodes the kernel currently holds a lookup reference to (nlookup ledger).
MetricTypeLabelsDescription
flexfs_mount_block_bytes_read_totalCountertierTotal block bytes read, by tier (on-store buffer size).
flexfs_mount_block_bytes_written_totalCountertierTotal block bytes written, by tier (on-store buffer size).
flexfs_mount_blocks_read_totalCounter(none)Total blocks read from the block store.
flexfs_mount_blocks_written_totalCounter(none)Total blocks written to the block store.
flexfs_mount_fuse_bytes_read_totalCounter(none)Total bytes read at the FUSE boundary.
flexfs_mount_fuse_bytes_written_totalCounter(none)Total bytes written at the FUSE boundary.

The fuse_bytes_* counters update in real time; the blocks_* counters advance on the 60-second reporting interval, so query them with rate() over a window of a few minutes. The block_bytes_* counters record the on-store buffer size (post-compression/encryption) actually transferred, split by the tier label: proxy for traffic served through a proxy group, direct for traffic to the object store backend (including proxy-bypass fallbacks). Both tiers are always present, at zero until traffic flows. Sum over tier for the total, or keep it to attribute amplification to one tier; comparing against fuse_bytes_* quantifies read/write amplification. The tier label uses the same values as block_op_duration_seconds, so throughput and latency can be sliced or joined by tier in a single query.

Latency of remote block operations, split by tier (proxy when served through a proxy group, direct when read/written directly against the backend) and op (get, put). The tier label carries the same two values as the throughput counters, so latency and throughput can be sliced or joined by tier in a single query. Only successful operations are timed; failures are counted separately in store_errors_total.

MetricTypeLabelsDescription
flexfs_mount_block_op_duration_secondsHistogramtier, opRemote block operation latency in seconds.
flexfs_mount_store_errors_totalCounterlayer, opBlock store errors, by layer (blockstore for synchronous failures, diskcache for asynchronous write-back failures) and op (get, put).

block_op_duration_seconds histogram buckets (seconds): 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30

In-process transform latency. The op label is one of encrypt, decrypt, compress, decompress. Encrypt/decrypt are recorded only when volume encryption is enabled.

MetricTypeLabelsDescription
flexfs_mount_process_duration_secondsHistogramopIn-process block transform latency in seconds.

process_duration_seconds histogram buckets (seconds): 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5

MetricTypeLabelsDescription
flexfs_mount_cas_conflicts_totalCounteropBlock-key compare-and-swap conflicts, by op.
flexfs_mount_cas_retries_totalCounteropBlock-key compare-and-swap retries, by op.
flexfs_mount_dirty_blocksGauge(none)Dirty blocks awaiting write-back.
flexfs_mount_sync_slots_capacityGauge(none)Configured write-back sync slot capacity (--dirtyActive).
flexfs_mount_sync_slots_in_useGauge(none)Write-back sync slots currently in use.
MetricTypeLabelsDescription
flexfs_mount_prefetch_activeGauge(none)Currently active prefetch streams.
flexfs_mount_prefetch_issued_totalCounter(none)Read-ahead prefetches dispatched to the block store.
flexfs_mount_prefetch_skipped_totalCounterreasonScheduled prefetches skipped before fetching, by reason (dirty, empty, pending, key_error).
flexfs_mount_prefetch_slotsGauge(none)Configured maximum concurrent prefetch operations (--prefetchActive).
MetricTypeLabelsDescription
flexfs_mount_meta_connectedGauge(none)Whether the meta-server connection is up (1) or down (0).
flexfs_mount_meta_disconnects_totalCounter(none)Meta-server connection-loss events.
flexfs_mount_meta_inflight_requestsGauge(none)In-flight requests to the meta server.
flexfs_mount_meta_rpc_duration_secondsHistogrammethodMeta-server RPC round-trip latency in seconds, by method.
flexfs_mount_meta_rpc_errors_totalCountermethod, statusMeta-server RPC responses with a genuine failure status, by method and status. Expected outcomes (ErrNotFound, ErrConflict, etc.) are not counted as errors.
flexfs_mount_meta_send_queue_depthGauge(none)Depth of the meta-server send queue.

meta_rpc_duration_seconds histogram buckets (seconds): 0.00005, 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

The pool label identifies which pool the buffers belong to:

poolBuffers used for
BlockFile data moving between the mount and the block store. The busiest of the three, and the one sized by --memCapacity and --dirtyCapacity.
FuseRequests and replies exchanged with the kernel. Sized automatically. Appears once the mount has completed its handshake with the kernel, so it is briefly absent at startup.
NameFile and directory names.
MetricTypeLabelsDescription
flexfs_mount_buffer_pool_borrowedGaugepoolBuffers currently in use.
flexfs_mount_buffer_pool_borrowed_totalCounterpoolTotal buffers taken from the pool over the life of the mount.
flexfs_mount_buffer_pool_created_totalCounterpoolTotal buffers allocated by the pool. Stays far below borrowed_total when the pool is reusing buffers effectively.
flexfs_mount_buffer_pool_returned_totalCounterpoolTotal buffers returned to the pool.

buffer_pool_borrowed is a current count, not a running total — use buffer_pool_borrowed_total if you want a rate.

A high steady buffer_pool_borrowed{pool="Block"} is normal and is not a leak. The caches sized by --memCapacity and --dirtyCapacity account for most of it, so it scales with those settings. Judge it by whether it grows without bound over hours, not by its absolute value.

Always present on mount.flexfs. The in-memory (L1) block cache cannot be turned off: passing --memCapacity 0 selects the automatic size (1.75% of system RAM) rather than disabling the cache, and a floor of 32 blocks applies in every case.

Both metrics are counted in blocks. Multiply by the volume’s block size for an approximate memory footprint.

MetricTypeLabelsDescription
flexfs_mount_mem_cache_sizeGauge(none)Blocks currently held in the in-memory block cache.
flexfs_mount_mem_cache_capacityGauge(none)Configured maximum blocks, as resolved from --memCapacity. Constant for the life of the mount.

This cache is normally the largest single consumer of the mount’s memory, so these two metrics are what let you attribute that memory to a setting you control. To judge whether --memCapacity needs raising, read occupancy against capacity alongside the mem hit rate from cache_events_total — neither answers the question on its own:

Occupancymem hit rateReading
Below capacityAnyNot under pressure; the working set fits.
At capacityHighCorrectly sized.
At capacityPoorThe working set does not fit, and every miss costs a fetch from the block store. This is when raising --memCapacity pays.
# L1 utilization.
flexfs_mount_mem_cache_size / flexfs_mount_mem_cache_capacity
# Approximate resident bytes held by the L1 cache (4 MiB default block size).
flexfs_mount_mem_cache_size * 4 * 1024 * 1024

Present only when a local disk cache is configured (--diskQuota). When none is, these series are absent rather than zero, and their absence is not a fault — it is how a mount running without an L2 cache looks. The same is true of cache_events_total for the disk cache, which reports zero hits and zero misses. Guard any expression that divides by disk_cache_quota_bytes accordingly.

MetricTypeLabelsDescription
flexfs_mount_disk_cache_clean_bytesGauge(none)Bytes of clean (uploaded) blocks held in the disk cache.
flexfs_mount_disk_cache_dirty_bytesGauge(none)Bytes of dirty (not-yet-uploaded) blocks held in the disk cache.
flexfs_mount_disk_cache_quota_bytesGauge(none)Configured disk cache capacity in bytes.

All three components also expose the default Prometheus Go runtime and process collectors:

  • go_* metrics (goroutines, GC stats, memory allocation)
  • process_* metrics (CPU time, resident memory, file descriptors, start time)

To see how much the block store is doing per byte the applications actually read or write, compare the block counters against the FUSE counters:

# Bytes fetched from the block store per byte delivered to applications.
rate(flexfs_mount_block_bytes_read_total[5m]) / rate(flexfs_mount_fuse_bytes_read_total[5m])
# The same for writes.
rate(flexfs_mount_block_bytes_written_total[5m]) / rate(flexfs_mount_fuse_bytes_written_total[5m])

A ratio above 1 is expected, because a cache miss fetches a whole block and a partial write rewrites one. A persistently large ratio alongside small average request sizes suggests the volume’s block size does not suit the workload.