Alerting

This page provides sample Prometheus alerting rules for monitoring flexFS infrastructure. Adapt the thresholds to your environment and SLAs.

Alert rules file

Add the following to a Prometheus rules file (e.g., flexfs-alerts.yml) and reference it in your prometheus.yml:

prometheus.yml

rule_files:
  - "flexfs-alerts.yml"

Metadata server alerts

groups:
  - name: flexfs-meta
    rules:

      - alert: FlexFSMetaServerDown
        expr: up{job="flexfs-meta"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "flexFS metadata server is down"
          description: "{{ $labels.instance }} has been unreachable for more than 2 minutes."

      - alert: FlexFSMetaHighRPCLatency
        expr: |
          histogram_quantile(0.99, sum(rate(flexfs_meta_rpc_duration_seconds_bucket[5m])) by (le, instance))
          > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS metadata server p99 RPC latency is high"
          description: "{{ $labels.instance }} p99 RPC latency has been above 100ms for 5 minutes."

      - alert: FlexFSMetaNoActiveSessions
        expr: sum(flexfs_meta_sessions) == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "No active flexFS mount sessions"
          description: "No mount clients have been connected for 10 minutes. This may indicate a network or configuration issue."

      - alert: FlexFSMetaDiskUsageHigh
        expr: |
          sum without (volume_id) (flexfs_meta_db_disk_usage_bytes)
          / flexfs_meta_db_folder_disk_capacity_bytes
          > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS metadata database disk usage above 85%"
          description: "Database disk usage on {{ $labels.instance }} is {{ $value | humanizePercentage }}."

      - alert: FlexFSMetaDiskUsageCritical
        expr: |
          sum without (volume_id) (flexfs_meta_db_disk_usage_bytes)
          / flexfs_meta_db_folder_disk_capacity_bytes
          > 0.95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "flexFS metadata database disk usage above 95%"
          description: "Database disk usage on {{ $labels.instance }} is {{ $value | humanizePercentage }}. Immediate action required."

      - alert: FlexFSMetaDiskAvailableLow
        expr: flexfs_meta_db_folder_disk_available_bytes < 4 * 1024 * 1024 * 1024
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "flexFS metadata server is approaching its disk floor"
          description: "Only {{ $value | humanize1024 }}B remain available on the database folder filesystem of {{ $labels.instance }}. The server shuts itself down gracefully once available space falls below its configured floor (1 GiB by default)."

      - alert: FlexFSMetaHighCompactionDebt
        expr: flexfs_meta_db_compaction_estimated_debt_bytes > 1073741824
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "flexFS metadata database compaction debt is high"
          description: "Volume {{ $labels.volume_id }} has {{ $value | humanize1024 }} of compaction debt."

      - alert: FlexFSMetaWriteStalls
        expr: increase(flexfs_meta_db_write_stall_seconds_total[10m]) > 30
        labels:
          severity: warning
        annotations:
          summary: "flexFS metadata database is stalling writes"
          description: "Volume {{ $labels.volume_id }} spent {{ $value | humanizeDuration }} refusing writes in the last 10 minutes. Mutating RPCs block for the duration of each stall; stalls longer than 45 seconds disconnect busy mounts."

Proxy server alerts

  - name: flexfs-proxy
    rules:

      - alert: FlexFSProxyServerDown
        expr: up{job="flexfs-proxy"} == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy server is down"
          description: "{{ $labels.instance }} has been unreachable for more than 2 minutes. Mount clients will fall back to direct storage access."

      - alert: FlexFSProxyHighRESTLatency
        expr: |
          histogram_quantile(0.99, sum(rate(flexfs_proxy_rest_duration_seconds_bucket[5m])) by (le, instance))
          > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy server p99 REST latency is high"
          description: "{{ $labels.instance }} p99 REST latency has been above 500ms for 5 minutes."

      - alert: FlexFSProxyCacheNearQuota
        expr: |
          (flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes)
          / flexfs_proxy_cache_disk_quota_bytes
          > 0.90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy cache is above 90% of quota"
          description: "{{ $labels.instance }} cache utilization is {{ $value | humanizePercentage }}."

      - alert: FlexFSProxyDirtyBlocksHigh
        expr: flexfs_proxy_cache_dirty_blocks > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy has a large dirty block queue"
          description: "{{ $labels.instance }} has {{ $value }} dirty blocks pending writeback. Writeback may not be keeping up with write load."

      - alert: FlexFSProxyDiskCapacityLow
        expr: |
          (flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes)
          / flexfs_proxy_cache_disk_capacity_bytes
          > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy disk usage is high"
          description: "{{ $labels.instance }} is using {{ $value | humanizePercentage }} of disk capacity."

      - alert: FlexFSProxyDiskAvailableLow
        expr: flexfs_proxy_db_folder_disk_available_bytes < 4 * 1024 * 1024 * 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy is approaching its disk floor"
          description: "Only {{ $value | humanize1024 }}B remain available on the database folder filesystem of {{ $labels.instance }}. The server shuts itself down gracefully once available space falls below its configured floor (1 GiB by default)."

Mount client alerts

Mount metrics require the client to be started with --metrics. These rules assume an instance label identifying each mount host, set in your scrape config. Because mounts are per-host and self-healing (they reconnect and fall back to direct storage), these are warning-level rather than critical.

  - name: flexfs-mount
    rules:

      - alert: FlexFSMountMetaDisconnected
        expr: flexfs_mount_meta_connected == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client is disconnected from the metadata server"
          description: "{{ $labels.instance }} has had no metadata server connection for 5 minutes. Metadata operations will stall."

      - alert: FlexFSMountHighMetaLatency
        expr: |
          histogram_quantile(0.99, sum(rate(flexfs_mount_meta_rpc_duration_seconds_bucket[5m])) by (le, instance))
          > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client p99 metadata RPC latency is high"
          description: "{{ $labels.instance }} p99 metadata RPC latency has been above 100ms for 5 minutes."

      - alert: FlexFSMountIOErrors
        expr: sum(rate(flexfs_mount_fuse_errors_total{status="EIO"}[5m])) by (instance) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client is returning I/O errors"
          description: "{{ $labels.instance }} has returned EIO to applications for 5 minutes. Check store and metadata connectivity."

      - alert: FlexFSMountStoreErrors
        expr: sum(rate(flexfs_mount_store_errors_total[5m])) by (instance, layer, op) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client block store errors"
          description: "{{ $labels.instance }} is seeing block store {{ $labels.op }} failures at the {{ $labels.layer }} layer."

      - alert: FlexFSMountWriteBackSaturated
        expr: flexfs_mount_sync_slots_in_use >= flexfs_mount_sync_slots_capacity
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client write-back is saturated"
          description: "{{ $labels.instance }} has had all write-back sync slots busy for 10 minutes; uploads are not keeping up with writes."

      - alert: FlexFSMountKernelInvalidationsDropped
        expr: increase(flexfs_mount_kernel_invalidations_dropped_total[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount dropped a kernel cache invalidation"
          description: "{{ $labels.instance }} could not queue a cross-mount cache update, so this mount may keep serving file data or attributes that another mount has already changed. Check flexfs_mount_kernel_inval_queue_max_lane_depth first: a single backed-up queue is the usual cause, and the overall depth can look healthy while that is happening. Then check object store latency, since slow reads are what holds the queue up."

      - alert: FlexFSMountInvalidationLaneStalled
        expr: min_over_time(flexfs_mount_kernel_inval_queue_max_lane_depth[10m]) >= 16
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount has a stalled kernel invalidation lane"
          description: "{{ $labels.instance }} has had one cache-update queue backed up for 10 minutes. Updates for a share of this mount's files are being held behind a single file, so changes made on other mounts may not be visible here. The usual cause is slow reads from the object store; check object store latency and the mount's read metrics. A brief spike during a cache flush is normal, which is why this only fires when the backlog never clears."

      - alert: FlexFSMountLedgerAccountingFault
        expr: increase(flexfs_mount_ledger_accounting_faults_total[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount lost track of a kernel reference"
          description: "{{ $labels.instance }} found a file it has open that it is no longer tracking a kernel reference for. An open file always holds one, so a reference was lost. Attribute and data invalidations for that file are suppressed while the reference is missing, so a peer's changes to it can go unseen; it clears only when a fresh lookup of the file's name re-establishes the reference, or when the file is closed. Note the counter increments once per affected file, when the fault is first detected, so this alert resolves an hour later even though the condition has not: treat it as a report-and-investigate signal, not a live indicator. Capture the mount log — the matching entry reads 'has an open file handle but no nlookup ledger entry', and any nearby 'nlookup underflow' lines are related — and report it."

      - alert: FlexFSMountLedgerFlushesDropped
        expr: increase(flexfs_mount_ledger_flushes_dropped_total[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount could not flush a stale kernel cache entry"
          description: "{{ $labels.instance }} detected a wrong kernel lookup count for a file but had no room on the invalidation queue to flush what the kernel had cached for it, so a stale copy may be served until the kernel forgets the inode or the attribute-cache validity window expires. Indicates sustained invalidation backpressure; check for a saturated notification path."

      - alert: FlexFSMountDiskCacheNearQuota
        expr: |
          (flexfs_mount_disk_cache_clean_bytes + flexfs_mount_disk_cache_dirty_bytes)
          / flexfs_mount_disk_cache_quota_bytes
          > 0.90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount disk cache is above 90% of quota"
          description: "{{ $labels.instance }} local disk cache utilization is {{ $value | humanizePercentage }}. Eviction keeps it within quota; benign unless hit rates suffer."

Alertmanager integration

Configure Prometheus to send alerts to Alertmanager:

prometheus.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager:9093"

Tip

The proxy server down alert uses severity: warning rather than severity: critical because mount clients automatically fall back to direct storage access when proxy servers are unavailable. The metadata server down alert is critical because it directly affects mount client operations.

Operational runbook references

Alert	Immediate action
FlexFSMetaDiskAvailableLow	Free or add space now: the server stops itself once available space falls below its configured floor, and it will refuse to start again until space is recovered.
FlexFSMetaDiskUsageHigh	Add disk space or move the database folder to a larger volume. Check for excessive retention.
FlexFSMetaHighRPCLatency	Check disk I/O, database compaction debt, network latency. Consider increasing database memory.
FlexFSMetaServerDown	Check systemd status, server logs, network connectivity. Restart if needed.
FlexFSMetaWriteStalls	Check db_write_stalls_total by reason. L0 stalls that line up with POST /compact calls or the weekly compaction mean a manual compaction is starving automatic ones: stop compacting on a schedule and run it on demand after large deletions. Memtable stalls mean flushes cannot keep up with the write rate: check disk I/O.
FlexFSMountHighMetaLatency	Check metadata server load/latency and the network path from this host.
FlexFSMountIOErrors	Check store/proxy connectivity and metadata health; inspect mount logs for the failing operations.
FlexFSMountMetaDisconnected	Check the network path to the metadata server, meta server health, and the mount client logs.
FlexFSMountStoreErrors	Check object store / proxy reachability and credentials; a disk layer error indicates local disk-cache problems.
FlexFSMountWriteBackSaturated	Check storage backend throughput; consider increasing write-back concurrency or reducing write pressure.
FlexFSProxyCacheNearQuota	Increase --diskQuota or add disk space. Eviction will keep the cache within quota.
FlexFSProxyDirtyBlocksHigh	Check storage backend health and latency. Consider increasing the proxy’s write-back concurrency.
FlexFSProxyDiskAvailableLow	Free or add space on the database folder filesystem: the server stops itself once available space falls below its configured floor.
FlexFSProxyServerDown	Check systemd status, server logs. Mount clients will use direct storage in the meantime.

Next steps

• Metrics reference — full metrics catalog
• Prometheus setup — scrape configuration
• Grafana dashboards — visualization