Grafana Dashboards

This page covers how to set up Grafana to visualize flexFS metrics collected by Prometheus.

Prerequisites

• A running Prometheus instance scraping flexFS metrics
• Grafana 9.0 or later

Adding the data source

1. In current Grafana releases, go to Connections > Add new connection, search for Prometheus, and click Add new data source. In older releases, go to Configuration > Data sources, click Add data source, and select Prometheus.
2. Set the server URL to your Prometheus server (e.g., http://prometheus:9090).
3. Click Save & test to verify the connection.

Key panels

The following panel definitions cover the most important flexFS metrics. You can combine them into a single dashboard or create separate dashboards for the metadata server, proxy server, and mount client.

Metadata server panels

RPC operations per second

rate(flexfs_meta_rpc_ops_total[5m])

Display as a time series, grouped by method. This shows the throughput of metadata operations (Lookup, GetAttr, SetAttr, MkNod, MkDir, etc.).

RPC latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_meta_rpc_duration_seconds_bucket[5m])) by (le, method))

Display as a time series, grouped by method. Alerts should trigger if p99 exceeds your SLA threshold.

Active sessions per volume

flexfs_meta_sessions

Display as a stat panel or time series grouped by volume_id. Shows how many mount clients are connected to each volume.

Volume size

flexfs_meta_volume_size_bytes

Display as a time series or bar gauge grouped by volume_id. Use the flexfs_meta_volume_info metric to map volume IDs to human-readable names:

flexfs_meta_volume_size_bytes * on(volume_id) group_left(volume_name) flexfs_meta_volume_info

Inode and block counts

flexfs_meta_volume_inodes
flexfs_meta_volume_blocks
flexfs_meta_volume_dentries

Display as stat panels or time series grouped by volume_id.

Volume I/O throughput

rate(flexfs_meta_volume_bytes_read_total[5m])
rate(flexfs_meta_volume_bytes_written_total[5m])

Display as a time series with read and write as separate series.

Database disk usage

flexfs_meta_db_disk_usage_bytes

Display as a time series grouped by volume_id. Compare against flexfs_meta_db_folder_disk_capacity_bytes to calculate utilization:

sum without (volume_id) (flexfs_meta_db_disk_usage_bytes) / flexfs_meta_db_folder_disk_capacity_bytes

Database folder space available

flexfs_meta_db_folder_disk_available_bytes

Display as a stat panel in bytes (IEC), with a threshold at the server’s low-disk floor (1 GiB by default) — the point at which the server shuts itself down.

Proxy server panels

REST operations per second

rate(flexfs_proxy_rest_ops_total[5m])

Display as a time series, grouped by method.

REST latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_proxy_rest_duration_seconds_bucket[5m])) by (le, method))

Cache hit rate

Approximate cache effectiveness by comparing read operations to cache block counts over time. A stable or growing flexfs_proxy_cache_clean_blocks alongside read traffic indicates good cache utilization.

Cache utilization

flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes

Compare against the disk quota:

(flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes) / flexfs_proxy_cache_disk_quota_bytes

Dirty block writeback queue

flexfs_proxy_cache_dirty_blocks
flexfs_proxy_cache_dirty_bytes

Display as a time series. A growing dirty queue may indicate writeback is not keeping up with write load.

Disk capacity

flexfs_proxy_cache_disk_capacity_bytes
flexfs_proxy_cache_disk_quota_bytes

Display as a bar gauge showing quota usage against total disk capacity.

Mount client panels

Mount metrics are only present when the client is started with --metrics. Add an instance (or host) variable to filter panels to a specific mount.

FUSE operation throughput

sum(rate(flexfs_mount_fuse_op_duration_seconds_count{op!="NotifyReply"}[5m])) by (op)

Time series grouped by op. Per-operation FUSE request rate (Read, Write, Lookup, GetAttr, etc.). The query leaves out NotifyReply, which reflects the mount’s own cache maintenance rather than the workload (see The NotifyReply op).

FUSE operation latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_fuse_op_duration_seconds_bucket[5m])) by (le, op))

Grouped by op. This is the user-visible responsiveness of the filesystem.

FUSE error rate

sum(rate(flexfs_mount_fuse_errors_total[5m])) by (op, status)

Non-OK replies returned to the kernel (e.g. EIO, ENOENT), by op and status.

Cache hit rate

sum(rate(flexfs_mount_cache_events_total{event="hit"}[5m])) by (cache)
/
sum(rate(flexfs_mount_cache_events_total[5m])) by (cache)

Per-cache hit ratio (mem, disk, attr, blockkey, dentry, openrefresh). A low ratio on mem or attr points to cache-sizing or workload issues.

The disk series has no data when no disk cache is configured (it has no hits or misses to divide), and the disk_cache_* gauges are absent entirely; that is a mount running without an L2 cache, not a fault.

Memory cache utilization

flexfs_mount_mem_cache_size / flexfs_mount_mem_cache_capacity

Pair this with the mem series from the hit-rate panel above — neither answers the sizing question alone. At capacity with a high hit rate, the cache is correctly sized. At capacity with a falling hit rate, the working set no longer fits and every miss costs a fetch from the block store; that combination is the signal to raise the in-memory cache capacity. Below capacity, sizing is not the constraint.

This cache is normally the largest single consumer of the mount’s memory. Multiply mem_cache_size by the volume’s block size to attribute process_resident_memory_bytes to it.

Metadata RPC latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_meta_rpc_duration_seconds_bucket[5m])) by (le, method))

Grouped by method. Metadata-path latency, often the dominant driver of interactive responsiveness.

Object-store and proxy latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_block_op_duration_seconds_bucket[5m])) by (le, tier, op))

Grouped by tier and op. Backend GET/PUT latency, split by proxy versus direct object store.

I/O throughput

rate(flexfs_mount_fuse_bytes_read_total[5m])
rate(flexfs_mount_fuse_bytes_written_total[5m])

Read and write bytes per second at the FUSE boundary.

Read/write amplification

sum without (tier) (rate(flexfs_mount_block_bytes_read_total[5m])) / rate(flexfs_mount_fuse_bytes_read_total[5m])
sum without (tier) (rate(flexfs_mount_block_bytes_written_total[5m])) / rate(flexfs_mount_fuse_bytes_written_total[5m])

Bytes actually transferred to/from the stores (on-store, post-compression/encryption) per byte crossing the FUSE boundary. Add a {tier="direct"} or {tier="proxy"} selector to attribute amplification to a single tier.

sum without (tier) rather than a plain sum: the two tiers must be summed, but instance and job have to survive so the division matches the corresponding fuse_bytes_* series. A bare sum(...) discards every label and the division then matches nothing, yielding an empty result.

Bytes per second against latency, by tier

sum(rate(flexfs_mount_block_bytes_read_total[5m])) by (tier)
histogram_quantile(0.99, sum(rate(flexfs_mount_block_op_duration_seconds_bucket{op="get"}[5m])) by (le, tier))

Throughput and latency share the tier label, so both queries produce one series per tier and line up on the same legend — putting them on a shared panel shows whether a latency rise is the proxy tier or the object store.

Write-back pressure

flexfs_mount_dirty_blocks
flexfs_mount_sync_slots_in_use

Dirty blocks awaiting upload and sync slots in use. Sustained sync_slots_in_use near flexfs_mount_sync_slots_capacity means write-back is saturated.

Metadata connection health

flexfs_mount_meta_connected
rate(flexfs_mount_meta_disconnects_total[5m])
rate(flexfs_mount_meta_connects_total[5m])
flexfs_mount_meta_inflight_requests

Connection up/down, reconnect churn, and in-flight request depth. Plot the two rates together: they track each other through a flapping connection, and a disconnect rate that stops being matched by connects is a client that has not got back in.

Disk cache utilization

(flexfs_mount_disk_cache_clean_bytes + flexfs_mount_disk_cache_dirty_bytes) / flexfs_mount_disk_cache_quota_bytes

Local disk-cache fill versus --diskQuota. Only present when a disk cache is configured.

Kernel cache coherence health

flexfs_mount_ledger_size
rate(flexfs_mount_ledger_leaks_reclaimed_total[5m])
increase(flexfs_mount_ledger_accounting_faults_total[1h])
flexfs_mount_kernel_inval_queue_depth
flexfs_mount_kernel_inval_queue_max_lane_depth
rate(flexfs_mount_kernel_invalidations_coalesced_total[5m])
rate(flexfs_mount_kernel_invalidations_deferred_total[5m])
increase(flexfs_mount_kernel_invalidations_dropped_total[1h])

How many files and directories the kernel has cached from this mount, and whether the mount is keeping those caches up to date with changes made on other mounts (see caching). Ledger size should follow the workload’s working set; the reclaim rate is routine housekeeping and needs no attention. Neither is worth alerting on.

Plot both queue depths on the same panel. Work is spread across many delivery queues by file, so the total says how much is outstanding overall while the max shows whether any one queue has stopped moving — and one stuck queue barely registers in the total. Both rising together is a busy mount, which a cache flush or a burst of activity on other mounts will do. The max staying up on its own is the one to chase: updates for a share of the mount’s files are held up behind a single file, usually because reads from the object store are slow. The deferral rate is the early warning — rising while the depths stay flat is normal contention being handled; rising along with them means the mount is falling behind.

The flexfs_mount_ledger_sweep_* metrics describe the mount’s background housekeeping and are diagnostic only; none needs a dashboard panel. See the metrics reference for what their readings mean.

The accounting-fault counter is the one to alert on rather than watch. It should stay flat at zero; anything else means changes made on other mounts to a file that is open here may not be seen until the file is closed. It counts each affected file once, when the fault is first detected, so a flat non-zero total is one unresolved incident rather than a resolved one — the graph will not fall back to zero on its own.

A high coalescing rate is normal on a mount whose files change often on other mounts: repeat changes to one file are merged into one update. Drops should stay at zero and are alerted on — each one leaves the kernel serving state the mount knows is stale.

Sample dashboard JSON

Tip

To import a dashboard, go to Dashboards > New > Import in Grafana, paste the JSON into Import via dashboard JSON model, click Load, and then click Import. The panels below do not name a data source, so they use Grafana’s default data source; make your Prometheus data source the default, or select it on each panel after import.

Below is a minimal dashboard definition with core panels. Customize panel sizes and positions to suit your layout.

{
  "title": "flexFS Overview",
  "tags": ["flexfs"],
  "timezone": "browser",
  "panels": [
    {
      "title": "Meta RPC ops/s",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
      "targets": [
        {
          "expr": "sum(rate(flexfs_meta_rpc_ops_total[5m])) by (method)",
          "legendFormat": "{{ method }}"
        }
      ]
    },
    {
      "title": "Meta RPC Latency p99",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(flexfs_meta_rpc_duration_seconds_bucket[5m])) by (le, method))",
          "legendFormat": "{{ method }}"
        }
      ]
    },
    {
      "title": "Active Sessions",
      "type": "stat",
      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 },
      "targets": [
        {
          "expr": "sum(flexfs_meta_sessions) or vector(0)",
          "legendFormat": "Total"
        }
      ]
    },
    {
      "title": "Volume Size",
      "type": "bargauge",
      "gridPos": { "h": 8, "w": 6, "x": 6, "y": 8 },
      "targets": [
        {
          "expr": "flexfs_meta_volume_size_bytes * on(volume_id) group_left(volume_name) flexfs_meta_volume_info",
          "legendFormat": "{{ volume_name }}"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "bytes" } }
    },
    {
      "title": "Proxy REST ops/s",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
      "targets": [
        {
          "expr": "sum(rate(flexfs_proxy_rest_ops_total[5m])) by (method)",
          "legendFormat": "{{ method }}"
        }
      ]
    },
    {
      "title": "Proxy Cache Utilization",
      "type": "gauge",
      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 16 },
      "targets": [
        {
          "expr": "(flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes) / flexfs_proxy_cache_disk_quota_bytes",
          "legendFormat": "utilization"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 } }
    },
    {
      "title": "Proxy Dirty Blocks",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 6, "x": 6, "y": 16 },
      "targets": [
        {
          "expr": "flexfs_proxy_cache_dirty_blocks",
          "legendFormat": "dirty blocks"
        }
      ]
    },
    {
      "title": "Mount FUSE Latency p99",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(flexfs_mount_fuse_op_duration_seconds_bucket[5m])) by (le, op))",
          "legendFormat": "{{ op }}"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "s" } }
    },
    {
      "title": "Mount Cache Hit Rate",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
      "targets": [
        {
          "expr": "sum(rate(flexfs_mount_cache_events_total{event=\"hit\"}[5m])) by (cache) / sum(rate(flexfs_mount_cache_events_total[5m])) by (cache)",
          "legendFormat": "{{ cache }}"
        }
      ],
      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 } }
    }
  ],
  "schemaVersion": 39,
  "version": 1
}

Customization tips

• Variable for volume: Metadata server series are labeled by volume_id. Add a Grafana variable with the query label_values(flexfs_meta_volume_info, volume_id) and filter panels with volume_id=~"$volume". flexfs_meta_volume_info maps each volume_id to its volume_name.
• Variable for proxy instance: Use label_values(flexfs_proxy_rest_ops_total, instance) to select specific proxy servers.
• Variable for mount host: Use label_values(flexfs_mount_fuse_ops_total, instance) to focus the mount panels on a single client.
• Thresholds: Set color thresholds on latency panels (green < 10ms, yellow < 100ms, red > 100ms) to quickly spot performance issues.

Next steps

• Metrics reference — full catalog of all metrics
• Alerting — Prometheus alert rules
• Logging and diagnostics — mount client logs and verbose mode