Grafana Dashboards

This page covers how to set up Grafana to visualize flexFS metrics collected by Prometheus.

Prerequisites

• A running Prometheus instance scraping flexFS metrics
• Grafana 9.0 or later

Adding the data source

1. Navigate to Configuration > Data Sources in Grafana.
2. Click Add data source and select Prometheus.
3. Set the URL to your Prometheus server (e.g., http://prometheus:9090).
4. Click Save & Test to verify the connection.

Key panels

The following panel definitions cover the most important flexFS metrics. You can combine them into a single dashboard or create separate dashboards for the metadata server, proxy server, and mount client.

Metadata server panels

RPC operations per second

rate(flexfs_meta_rpc_ops_total[5m])

Display as a time series, grouped by method. This shows the throughput of metadata operations (Lookup, GetAttr, SetAttr, Create, Mkdir, etc.).

RPC latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_meta_rpc_duration_seconds_bucket[5m])) by (le, method))

Display as a time series, grouped by method. Alerts should trigger if p99 exceeds your SLA threshold.

Active sessions per volume

flexfs_meta_sessions

Display as a stat panel or time series grouped by volume_id. Shows how many mount clients are connected to each volume.

Volume size

flexfs_meta_volume_size_bytes

Display as a time series or bar gauge grouped by volume_id. Use the flexfs_meta_volume_info metric to map volume IDs to human-readable names:

flexfs_meta_volume_size_bytes * on(volume_id) group_left(volume_name) flexfs_meta_volume_info

Inode and block counts

flexfs_meta_volume_inodes
flexfs_meta_volume_blocks
flexfs_meta_volume_dentries

Display as stat panels or time series grouped by volume_id.

Volume I/O throughput

rate(flexfs_meta_volume_bytes_read_total[5m])
rate(flexfs_meta_volume_bytes_written_total[5m])

Display as a time series with read and write as separate series.

Database disk usage

flexfs_meta_db_disk_usage_bytes

Display as a time series grouped by volume_id. Compare against flexfs_meta_db_folder_disk_capacity_bytes to calculate utilization:

sum without (volume_id) (flexfs_meta_db_disk_usage_bytes) / flexfs_meta_db_folder_disk_capacity_bytes

Database folder space available

flexfs_meta_db_folder_disk_available_bytes

Display as a stat panel in bytes (IEC), with a threshold at the server’s low-disk floor (1 GiB by default) — the point at which the server shuts itself down.

Proxy server panels

REST operations per second

rate(flexfs_proxy_rest_ops_total[5m])

Display as a time series, grouped by method.

REST latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_proxy_rest_duration_seconds_bucket[5m])) by (le, method))

Cache hit rate

Approximate cache effectiveness by comparing read operations to cache block counts over time. A stable or growing flexfs_proxy_cache_clean_blocks alongside read traffic indicates good cache utilization.

Cache utilization

flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes

Compare against the disk quota:

(flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes) / flexfs_proxy_cache_disk_quota_bytes

Dirty block writeback queue

flexfs_proxy_cache_dirty_blocks
flexfs_proxy_cache_dirty_bytes

Display as a time series. A growing dirty queue may indicate writeback is not keeping up with write load.

Disk capacity

flexfs_proxy_cache_disk_capacity_bytes
flexfs_proxy_cache_disk_quota_bytes

Display as a bar gauge showing quota usage against total disk capacity.

Mount client panels

Mount metrics are only present when the client is started with --metrics. Add an instance (or host) variable to filter panels to a specific mount.

FUSE operation throughput

sum(rate(flexfs_mount_fuse_op_duration_seconds_count[5m])) by (op)

Time series grouped by op. Per-operation FUSE request rate (Read, Write, Lookup, GetAttr, etc.).

FUSE operation latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_fuse_op_duration_seconds_bucket[5m])) by (le, op))

Grouped by op. This is the user-visible responsiveness of the filesystem.

FUSE error rate

sum(rate(flexfs_mount_fuse_errors_total[5m])) by (op, status)

Non-OK replies returned to the kernel (e.g. EIO, ENOENT), by op and status.

Cache hit rate

sum(rate(flexfs_mount_cache_events_total{event="hit"}[5m])) by (cache)
/
sum(rate(flexfs_mount_cache_events_total[5m])) by (cache)

Per-cache hit ratio (mem, disk, attr, blockkey, dentry). A low ratio on mem or attr points to cache-sizing or workload issues.

The disk series is flat at zero when no disk cache is configured, and the disk_cache_* gauges are absent entirely; that is a mount running without an L2 cache, not a fault.

Memory cache utilization

flexfs_mount_mem_cache_size / flexfs_mount_mem_cache_capacity

Pair this with the mem series from the hit-rate panel above — neither answers the sizing question alone. At capacity with a high hit rate, the cache is correctly sized. At capacity with a falling hit rate, the working set no longer fits and every miss costs a fetch from the block store; that combination is the signal to raise the in-memory cache capacity. Below capacity, sizing is not the constraint.

This cache is normally the largest single consumer of the mount’s memory. Multiply mem_cache_size by the volume’s block size to attribute process_resident_memory_bytes to it.

Metadata RPC latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_meta_rpc_duration_seconds_bucket[5m])) by (le, method))

Grouped by method. Metadata-path latency, often the dominant driver of interactive responsiveness.

Object-store and proxy latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_block_op_duration_seconds_bucket[5m])) by (le, tier, op))

Grouped by tier and op. Backend GET/PUT latency, split by proxy versus direct object store.

I/O throughput

rate(flexfs_mount_fuse_bytes_read_total[5m])
rate(flexfs_mount_fuse_bytes_written_total[5m])

Read and write bytes per second at the FUSE boundary.

Read/write amplification

sum without (tier) (rate(flexfs_mount_block_bytes_read_total[5m])) / rate(flexfs_mount_fuse_bytes_read_total[5m])
sum without (tier) (rate(flexfs_mount_block_bytes_written_total[5m])) / rate(flexfs_mount_fuse_bytes_written_total[5m])

Bytes actually transferred to/from the stores (on-store, post-compression/encryption) per byte crossing the FUSE boundary. Add a {tier="direct"} or {tier="proxy"} selector to attribute amplification to a single tier.

sum without (tier) rather than a plain sum: the two tiers must be summed, but instance and job have to survive so the division matches the corresponding fuse_bytes_* series. A bare sum(...) discards every label and the division then matches nothing, yielding an empty result.

Bytes per second against latency, by tier

sum(rate(flexfs_mount_block_bytes_read_total[5m])) by (tier)
histogram_quantile(0.99, sum(rate(flexfs_mount_block_op_duration_seconds_bucket{op="get"}[5m])) by (le, tier))

Throughput and latency share the tier label, so both queries produce one series per tier and line up on the same legend — putting them on a shared panel shows whether a latency rise is the proxy tier or the object store.

Write-back pressure

flexfs_mount_dirty_blocks
flexfs_mount_sync_slots_in_use

Dirty blocks awaiting upload and sync slots in use. Sustained sync_slots_in_use near flexfs_mount_sync_slots_capacity means write-back is saturated.

Metadata connection health

flexfs_mount_meta_connected
rate(flexfs_mount_meta_disconnects_total[5m])
rate(flexfs_mount_meta_connects_total[5m])
flexfs_mount_meta_inflight_requests

Connection up/down, reconnect churn, and in-flight request depth. Plot the two rates together: they track each other through a flapping connection, and a disconnect rate that stops being matched by connects is a client that has not got back in.

Disk cache utilization

(flexfs_mount_disk_cache_clean_bytes + flexfs_mount_disk_cache_dirty_bytes) / flexfs_mount_disk_cache_quota_bytes

Local disk-cache fill versus --diskQuota. Only present when a disk cache is configured.

Kernel cache coherence health

flexfs_mount_ledger_size
rate(flexfs_mount_ledger_leaks_reclaimed_total[5m])
increase(flexfs_mount_ledger_accounting_faults_total[1h])
flexfs_mount_kernel_inval_queue_depth
flexfs_mount_kernel_inval_queue_max_lane_depth
rate(flexfs_mount_kernel_invalidations_coalesced_total[5m])
rate(flexfs_mount_kernel_invalidations_deferred_total[5m])
increase(flexfs_mount_kernel_invalidations_dropped_total[1h])

How many inodes the kernel is holding, and the health of the bookkeeping behind cross-mount invalidation (see caching). Ledger size should follow the workload’s working set; the reclaim rate is routine housekeeping and needs no attention. Neither is worth alerting on.

Plot both queue depths on the same panel. Work is spread across many delivery queues by file, so the total says how much is outstanding overall while the max shows whether any one queue has stopped moving — and one stuck queue barely registers in the total. Both rising together is a busy mount, which a cache flush or a burst of activity on other mounts will do. The max staying up on its own is the one to chase: updates for a share of the mount’s files are held up behind a single file, usually because reads from the object store are slow. The deferral rate is the early warning — rising while the depths stay flat is normal contention being handled; rising along with them means the mount is falling behind.

Three further counters — flexfs_mount_ledger_sweep_entries_examined_total, ..._probes_total and ..._yields_total — plus the gauge ..._effective_confirm_ttl_seconds are diagnostic rather than operational: none is worth a dashboard panel by default, so reach for them only when tuning a very large mount. The counters describe the cost of the background check itself and the gauge describes its pacing. Both per-pass costs are capped, so the readings to know are: the examined rate pinned at its ~8k/s cap means eligible entries are scarce and the check is pacing itself (the expected steady state on a large quiet ledger, at a fixed cost), and the probe rate sitting near the paced re-confirmation target rather than the 64/s budget is likewise normal — pinned at 64/s means a burst or backlog is draining and should subside. The TTL gauge stretches in proportion to ledger size and is the number that explains how long a leaked entry can take to be reclaimed; multi-day values on many-million-entry ledgers are by design and cost only memory held a little longer.

The accounting-fault counter is the one to alert on rather than watch. It should stay flat at zero; anything else means cross-mount invalidation is suppressed for a file that is currently open, and it will stay that way until the file is closed. It counts each affected file once, when the fault is first detected, so a flat non-zero total is one unresolved incident rather than a resolved one — the graph will not fall back to zero on its own.

The last three cover the queue that carries invalidations to the kernel. Depth should sit near zero: a notification write is synchronous and can wait on an in-flight block read for the same inode, so sustained depth points at slow reads rather than at the invalidation path itself. A healthy mount under peer churn coalesces heavily, since repeat changes to one inode collapse into one queued op. Drops should never happen and are alerted on — each one leaves the kernel serving state the mount knows is stale.

Sample dashboard JSON

Tip

To import a dashboard, go to Dashboards > Import in Grafana, paste the JSON, and select your Prometheus data source.

Below is a minimal dashboard definition with core panels. Customize panel sizes and positions to suit your layout.

{
  "dashboard": {
    "title": "flexFS Overview",
    "tags": ["flexfs"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Meta RPC ops/s",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
        "targets": [
          {
            "expr": "sum(rate(flexfs_meta_rpc_ops_total[5m])) by (method)",
            "legendFormat": "{{ method }}"
          }
        ]
      },
      {
        "title": "Meta RPC Latency p99",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(flexfs_meta_rpc_duration_seconds_bucket[5m])) by (le, method))",
            "legendFormat": "{{ method }}"
          }
        ]
      },
      {
        "title": "Active Sessions",
        "type": "stat",
        "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 },
        "targets": [
          {
            "expr": "sum(flexfs_meta_sessions)",
            "legendFormat": "Total"
          }
        ]
      },
      {
        "title": "Volume Size",
        "type": "bargauge",
        "gridPos": { "h": 8, "w": 6, "x": 6, "y": 8 },
        "targets": [
          {
            "expr": "flexfs_meta_volume_size_bytes * on(volume_id) group_left(volume_name) flexfs_meta_volume_info",
            "legendFormat": "{{ volume_name }}"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "bytes" } }
      },
      {
        "title": "Proxy REST ops/s",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
        "targets": [
          {
            "expr": "sum(rate(flexfs_proxy_rest_ops_total[5m])) by (method)",
            "legendFormat": "{{ method }}"
          }
        ]
      },
      {
        "title": "Proxy Cache Utilization",
        "type": "gauge",
        "gridPos": { "h": 4, "w": 6, "x": 0, "y": 16 },
        "targets": [
          {
            "expr": "(flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes) / flexfs_proxy_cache_disk_quota_bytes",
            "legendFormat": "utilization"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 } }
      },
      {
        "title": "Proxy Dirty Blocks",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 6, "x": 6, "y": 16 },
        "targets": [
          {
            "expr": "flexfs_proxy_cache_dirty_blocks",
            "legendFormat": "dirty blocks"
          }
        ]
      },
      {
        "title": "Mount FUSE Latency p99",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(flexfs_mount_fuse_op_duration_seconds_bucket[5m])) by (le, op))",
            "legendFormat": "{{ op }}"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "s" } }
      },
      {
        "title": "Mount Cache Hit Rate",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
        "targets": [
          {
            "expr": "sum(rate(flexfs_mount_cache_events_total{event=\"hit\"}[5m])) by (cache) / sum(rate(flexfs_mount_cache_events_total[5m])) by (cache)",
            "legendFormat": "{{ cache }}"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 } }
      }
    ],
    "schemaVersion": 39,
    "version": 1
  }
}

Customization tips

• Variable for volume: Add a Grafana variable with the query label_values(flexfs_meta_volume_info, volume_name) to filter panels by volume.
• Variable for proxy instance: Use label_values(flexfs_proxy_rest_ops_total, instance) to select specific proxy servers.
• Variable for mount host: Use label_values(flexfs_mount_fuse_ops_total, instance) to focus the mount panels on a single client.
• Annotations: Add annotations from flexfs_meta_rpc_ops_total to mark deployment events or configuration changes.
• Thresholds: Set color thresholds on latency panels (green < 10ms, yellow < 100ms, red > 100ms) to quickly spot performance issues.

Next steps

• Metrics reference — full catalog of all metrics
• Alerting — Prometheus alert rules
• Logging and diagnostics — mount client logs and profiling