Grafana Dashboards

This page covers how to set up Grafana to visualize flexFS metrics collected by Prometheus.

Prerequisites

• A running Prometheus instance scraping flexFS metrics
• Grafana 9.0 or later

Adding the data source

1. Navigate to Configuration > Data Sources in Grafana.
2. Click Add data source and select Prometheus.
3. Set the URL to your Prometheus server (e.g., http://prometheus:9090).
4. Click Save & Test to verify the connection.

Key panels

The following panel definitions cover the most important flexFS metrics. You can combine them into a single dashboard or create separate dashboards for the metadata server, proxy server, and mount client.

Metadata server panels

RPC operations per second

rate(flexfs_meta_rpc_ops_total[5m])

Display as a time series, grouped by method. This shows the throughput of metadata operations (Lookup, GetAttr, SetAttr, Create, Mkdir, etc.).

RPC latency (p99)

histogram_quantile(0.99, rate(flexfs_meta_rpc_duration_seconds_bucket[5m]))

Display as a time series, grouped by method. Alerts should trigger if p99 exceeds your SLA threshold.

Active sessions per volume

flexfs_meta_sessions

Display as a stat panel or time series grouped by volume_id. Shows how many mount clients are connected to each volume.

Volume size

flexfs_meta_volume_size_bytes

Display as a time series or bar gauge grouped by volume_id. Use the flexfs_meta_volume_info metric to map volume IDs to human-readable names:

flexfs_meta_volume_size_bytes * on(volume_id) group_left(volume_name) flexfs_meta_volume_info

Inode and block counts

flexfs_meta_volume_inodes
flexfs_meta_volume_blocks
flexfs_meta_volume_dentries

Display as stat panels or time series grouped by volume_id.

Volume I/O throughput

rate(flexfs_meta_volume_bytes_read_total[5m])
rate(flexfs_meta_volume_bytes_written_total[5m])

Display as a time series with read and write as separate series.

Database disk usage

flexfs_meta_db_disk_usage_bytes

Display as a time series grouped by volume_id. Compare against flexfs_meta_db_folder_disk_capacity_bytes to calculate utilization:

sum(flexfs_meta_db_disk_usage_bytes) / flexfs_meta_db_folder_disk_capacity_bytes

Database folder space available

flexfs_meta_db_folder_disk_available_bytes

Display as a stat panel in bytes (IEC), with a threshold at the server’s --minDiskAvail floor (1 GiB by default) — the point at which the server shuts itself down.

Proxy server panels

REST operations per second

rate(flexfs_proxy_rest_ops_total[5m])

Display as a time series, grouped by method.

REST latency (p99)

histogram_quantile(0.99, rate(flexfs_proxy_rest_duration_seconds_bucket[5m]))

Cache hit rate

Approximate cache effectiveness by comparing read operations to cache block counts over time. A stable or growing flexfs_proxy_cache_clean_blocks alongside read traffic indicates good cache utilization.

Cache utilization

flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes

Compare against the disk quota:

(flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes) / flexfs_proxy_cache_disk_quota_bytes

Dirty block writeback queue

flexfs_proxy_cache_dirty_blocks
flexfs_proxy_cache_dirty_bytes

Display as a time series. A growing dirty queue may indicate writeback is not keeping up with write load.

Disk capacity

flexfs_proxy_cache_disk_capacity_bytes
flexfs_proxy_cache_disk_quota_bytes

Display as a bar gauge showing quota usage against total disk capacity.

Mount client panels

Mount metrics are only present when the client is started with --metrics. Add an instance (or host) variable to filter panels to a specific mount.

FUSE operation throughput

sum(rate(flexfs_mount_fuse_op_duration_seconds_count[5m])) by (op)

Time series grouped by op. Per-operation FUSE request rate (Read, Write, Lookup, GetAttr, etc.).

FUSE operation latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_fuse_op_duration_seconds_bucket[5m])) by (le, op))

Grouped by op. This is the user-visible responsiveness of the filesystem.

FUSE error rate

sum(rate(flexfs_mount_fuse_errors_total[5m])) by (op, status)

Non-OK replies returned to the kernel (e.g. EIO, ENOENT), by op and status.

Cache hit rate

sum(rate(flexfs_mount_cache_events_total{event="hit"}[5m])) by (cache)
/
sum(rate(flexfs_mount_cache_events_total[5m])) by (cache)

Per-cache hit ratio (mem, disk, attr, blockkey, dentry). A low ratio on mem or attr points to cache-sizing or workload issues.

The disk series is flat at zero when no disk cache is configured, and the disk_cache_* gauges are absent entirely; that is a mount running without an L2 cache, not a fault.

Memory cache utilization

flexfs_mount_mem_cache_size / flexfs_mount_mem_cache_capacity

Pair this with the mem series from the hit-rate panel above — neither answers the sizing question alone. At capacity with a high hit rate, the cache is correctly sized. At capacity with a falling hit rate, the working set no longer fits and every miss costs a fetch from the block store; that combination is the signal to raise --memCapacity. Below capacity, sizing is not the constraint.

This cache is normally the largest single consumer of the mount’s memory. Multiply mem_cache_size by the volume’s block size to attribute process_resident_memory_bytes to it.

Metadata RPC latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_meta_rpc_duration_seconds_bucket[5m])) by (le, method))

Grouped by method. Metadata-path latency, often the dominant driver of interactive responsiveness.

Object-store and proxy latency (p99)

histogram_quantile(0.99, sum(rate(flexfs_mount_block_op_duration_seconds_bucket[5m])) by (le, tier, op))

Grouped by tier and op. Backend GET/PUT latency, split by proxy versus direct object store.

I/O throughput

rate(flexfs_mount_fuse_bytes_read_total[5m])
rate(flexfs_mount_fuse_bytes_written_total[5m])

Read and write bytes per second at the FUSE boundary.

Read/write amplification

sum without (tier) (rate(flexfs_mount_block_bytes_read_total[5m])) / rate(flexfs_mount_fuse_bytes_read_total[5m])
sum without (tier) (rate(flexfs_mount_block_bytes_written_total[5m])) / rate(flexfs_mount_fuse_bytes_written_total[5m])

Bytes actually transferred to/from the stores (on-store, post-compression/encryption) per byte crossing the FUSE boundary. Add a {tier="direct"} or {tier="proxy"} selector to attribute amplification to a single tier.

sum without (tier) rather than a plain sum: the two tiers must be summed, but instance and job have to survive so the division matches the corresponding fuse_bytes_* series. A bare sum(...) discards every label and the division then matches nothing, yielding an empty result.

Bytes per second against latency, by tier

sum(rate(flexfs_mount_block_bytes_read_total[5m])) by (tier)
histogram_quantile(0.99, sum(rate(flexfs_mount_block_op_duration_seconds_bucket{op="get"}[5m])) by (le, tier))

Throughput and latency share the tier label, so both queries produce one series per tier and line up on the same legend — putting them on a shared panel shows whether a latency rise is the proxy tier or the object store.

Write-back pressure

flexfs_mount_dirty_blocks
flexfs_mount_sync_slots_in_use

Dirty blocks awaiting upload and sync slots in use. Sustained sync_slots_in_use near flexfs_mount_sync_slots_capacity means write-back is saturated.

Metadata connection health

flexfs_mount_meta_connected
rate(flexfs_mount_meta_disconnects_total[5m])
flexfs_mount_meta_inflight_requests

Connection up/down, reconnect churn, and in-flight request depth.

Disk cache utilization

(flexfs_mount_disk_cache_clean_bytes + flexfs_mount_disk_cache_dirty_bytes) / flexfs_mount_disk_cache_quota_bytes

Local disk-cache fill versus --diskQuota. Only present when a disk cache is configured.

Kernel cache coherence health

flexfs_mount_ledger_size
rate(flexfs_mount_ledger_leaks_reclaimed_total[5m])
increase(flexfs_mount_ledger_accounting_faults_total[1h])
flexfs_mount_kernel_inval_queue_depth
rate(flexfs_mount_kernel_invalidations_coalesced_total[5m])
increase(flexfs_mount_kernel_invalidations_dropped_total[1h])

How many inodes the kernel is holding, and the health of the bookkeeping behind cross-mount invalidation (see caching). Ledger size should follow the workload’s working set; the reclaim rate is routine housekeeping and needs no attention. Neither is worth alerting on.

Three further counters — flexfs_mount_ledger_sweep_entries_examined_total, ..._probes_total and ..._yields_total — describe the cost of the background check itself. They are diagnostic rather than operational, and none is worth a dashboard panel by default; add them when tuning a very large mount. Two comparisons are worth knowing: the examined rate against flexfs_mount_ledger_size (roughly one ledger per second means each pass scans everything to find little), and the probe rate against its 64/s cap (pinned at the cap means the check is running flat out, which is expected on a large mount and costs only slower reclamation of memory).

The accounting-fault counter is the one to alert on rather than watch. It should stay flat at zero; anything else means cross-mount invalidation is suppressed for a file that is currently open, and it will stay that way until the file is closed. It counts each affected file once, when the fault is first detected, so a flat non-zero total is one unresolved incident rather than a resolved one — the graph will not fall back to zero on its own.

The last three cover the queue that carries invalidations to the kernel. Depth should sit near zero: a notification write is synchronous and can wait on an in-flight block read for the same inode, so sustained depth points at slow reads rather than at the invalidation path itself. A healthy mount under peer churn coalesces heavily, since repeat changes to one inode collapse into one queued op. Drops should never happen and are alerted on — each one leaves the kernel serving state the mount knows is stale.

Sample dashboard JSON

Tip

To import a dashboard, go to Dashboards > Import in Grafana, paste the JSON, and select your Prometheus data source.

Below is a minimal dashboard definition with core panels. Customize panel sizes and positions to suit your layout.

{
  "dashboard": {
    "title": "flexFS Overview",
    "tags": ["flexfs"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Meta RPC ops/s",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
        "targets": [
          {
            "expr": "sum(rate(flexfs_meta_rpc_ops_total[5m])) by (method)",
            "legendFormat": "{{ method }}"
          }
        ]
      },
      {
        "title": "Meta RPC Latency p99",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(flexfs_meta_rpc_duration_seconds_bucket[5m])) by (le, method))",
            "legendFormat": "{{ method }}"
          }
        ]
      },
      {
        "title": "Active Sessions",
        "type": "stat",
        "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 },
        "targets": [
          {
            "expr": "sum(flexfs_meta_sessions)",
            "legendFormat": "Total"
          }
        ]
      },
      {
        "title": "Volume Size",
        "type": "bargauge",
        "gridPos": { "h": 8, "w": 6, "x": 6, "y": 8 },
        "targets": [
          {
            "expr": "flexfs_meta_volume_size_bytes * on(volume_id) group_left(volume_name) flexfs_meta_volume_info",
            "legendFormat": "{{ volume_name }}"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "bytes" } }
      },
      {
        "title": "Proxy REST ops/s",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
        "targets": [
          {
            "expr": "sum(rate(flexfs_proxy_rest_ops_total[5m])) by (method)",
            "legendFormat": "{{ method }}"
          }
        ]
      },
      {
        "title": "Proxy Cache Utilization",
        "type": "gauge",
        "gridPos": { "h": 4, "w": 6, "x": 0, "y": 16 },
        "targets": [
          {
            "expr": "(flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes) / flexfs_proxy_cache_disk_quota_bytes",
            "legendFormat": "utilization"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 } }
      },
      {
        "title": "Proxy Dirty Blocks",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 6, "x": 6, "y": 16 },
        "targets": [
          {
            "expr": "flexfs_proxy_cache_dirty_blocks",
            "legendFormat": "dirty blocks"
          }
        ]
      },
      {
        "title": "Mount FUSE Latency p99",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(flexfs_mount_fuse_op_duration_seconds_bucket[5m])) by (le, op))",
            "legendFormat": "{{ op }}"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "s" } }
      },
      {
        "title": "Mount Cache Hit Rate",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
        "targets": [
          {
            "expr": "sum(rate(flexfs_mount_cache_events_total{event=\"hit\"}[5m])) by (cache) / sum(rate(flexfs_mount_cache_events_total[5m])) by (cache)",
            "legendFormat": "{{ cache }}"
          }
        ],
        "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1 } }
      }
    ],
    "schemaVersion": 39,
    "version": 1
  }
}

Customization tips

• Variable for volume: Add a Grafana variable with the query label_values(flexfs_meta_volume_info, volume_name) to filter panels by volume.
• Variable for proxy instance: Use label_values(flexfs_proxy_rest_ops_total, instance) to select specific proxy servers.
• Variable for mount host: Use label_values(flexfs_mount_fuse_ops_total, instance) to focus the mount panels on a single client.
• Annotations: Add annotations from flexfs_meta_rpc_ops_total to mark deployment events or configuration changes.
• Thresholds: Set color thresholds on latency panels (green < 10ms, yellow < 100ms, red > 100ms) to quickly spot performance issues.

Next steps

• Metrics reference — full catalog of all metrics
• Alerting — Prometheus alert rules
• Logging and diagnostics — mount client logs and profiling