Alerting

This page provides sample Prometheus alerting rules for monitoring flexFS infrastructure. Adapt the thresholds to your environment and SLAs.

Alert rules file

Add the following to a Prometheus rules file (e.g., flexfs-alerts.yml) and reference it in your prometheus.yml:

prometheus.yml

rule_files:
  - "flexfs-alerts.yml"

Metadata server alerts

groups:
  - name: flexfs-meta
    rules:

      - alert: FlexFSMetaServerDown
        expr: up{job="flexfs-meta"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "flexFS metadata server is down"
          description: "{{ $labels.instance }} has been unreachable for more than 2 minutes."

      - alert: FlexFSMetaHighRPCLatency
        expr: |
          histogram_quantile(0.99, sum(rate(flexfs_meta_rpc_duration_seconds_bucket[5m])) by (le, instance))
          > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS metadata server p99 RPC latency is high"
          description: "{{ $labels.instance }} p99 RPC latency has been above 100ms for 5 minutes."

      - alert: FlexFSMetaNoActiveSessions
        expr: sum(flexfs_meta_sessions) == 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "No active flexFS mount sessions"
          description: "No mount clients have been connected for 10 minutes. This may indicate a network or configuration issue."

      - alert: FlexFSMetaDiskUsageHigh
        expr: |
          sum(flexfs_meta_db_disk_usage_bytes)
          / flexfs_meta_db_folder_disk_capacity_bytes
          > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS metadata database disk usage above 85%"
          description: "Database disk usage on {{ $labels.instance }} is {{ $value | humanizePercentage }}."

      - alert: FlexFSMetaDiskUsageCritical
        expr: |
          sum(flexfs_meta_db_disk_usage_bytes)
          / flexfs_meta_db_folder_disk_capacity_bytes
          > 0.95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "flexFS metadata database disk usage above 95%"
          description: "Database disk usage on {{ $labels.instance }} is {{ $value | humanizePercentage }}. Immediate action required."

      - alert: FlexFSMetaDiskAvailableLow
        expr: flexfs_meta_db_folder_disk_available_bytes < 4 * 1024 * 1024 * 1024
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "flexFS metadata server is approaching its disk floor"
          description: "Only {{ $value | humanize1024 }}B remain available on the database folder filesystem of {{ $labels.instance }}. The server shuts itself down gracefully below --minDiskAvail (1 GiB by default)."

      - alert: FlexFSMetaHighCompactionDebt
        expr: flexfs_meta_db_compaction_estimated_debt_bytes > 1073741824
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "flexFS metadata database compaction debt is high"
          description: "Volume {{ $labels.volume_id }} has {{ $value | humanize1024 }} of compaction debt."

Proxy server alerts

  - name: flexfs-proxy
    rules:

      - alert: FlexFSProxyServerDown
        expr: up{job="flexfs-proxy"} == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy server is down"
          description: "{{ $labels.instance }} has been unreachable for more than 2 minutes. Mount clients will fall back to direct storage access."

      - alert: FlexFSProxyHighRESTLatency
        expr: |
          histogram_quantile(0.99, sum(rate(flexfs_proxy_rest_duration_seconds_bucket[5m])) by (le, instance))
          > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy server p99 REST latency is high"
          description: "{{ $labels.instance }} p99 REST latency has been above 500ms for 5 minutes."

      - alert: FlexFSProxyCacheNearQuota
        expr: |
          (flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes)
          / flexfs_proxy_cache_disk_quota_bytes
          > 0.90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy cache is above 90% of quota"
          description: "{{ $labels.instance }} cache utilization is {{ $value | humanizePercentage }}."

      - alert: FlexFSProxyDirtyBlocksHigh
        expr: flexfs_proxy_cache_dirty_blocks > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy has a large dirty block queue"
          description: "{{ $labels.instance }} has {{ $value }} dirty blocks pending writeback. Writeback may not be keeping up with write load."

      - alert: FlexFSProxyDiskCapacityLow
        expr: |
          (flexfs_proxy_cache_clean_bytes + flexfs_proxy_cache_dirty_bytes)
          / flexfs_proxy_cache_disk_capacity_bytes
          > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy disk usage is high"
          description: "{{ $labels.instance }} is using {{ $value | humanizePercentage }} of disk capacity."

      - alert: FlexFSProxyDiskAvailableLow
        expr: flexfs_proxy_db_folder_disk_available_bytes < 4 * 1024 * 1024 * 1024
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS proxy is approaching its disk floor"
          description: "Only {{ $value | humanize1024 }}B remain available on the database folder filesystem of {{ $labels.instance }}. The server shuts itself down gracefully below --minDiskAvail (1 GiB by default)."

Mount client alerts

Mount metrics require the client to be started with --metrics. These rules assume an instance label identifying each mount host, set in your scrape config. Because mounts are per-host and self-healing (they reconnect and fall back to direct storage), these are warning-level rather than critical.

  - name: flexfs-mount
    rules:

      - alert: FlexFSMountMetaDisconnected
        expr: flexfs_mount_meta_connected == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client is disconnected from the metadata server"
          description: "{{ $labels.instance }} has had no metadata server connection for 5 minutes. Metadata operations will stall."

      - alert: FlexFSMountHighMetaLatency
        expr: |
          histogram_quantile(0.99, sum(rate(flexfs_mount_meta_rpc_duration_seconds_bucket[5m])) by (le, instance))
          > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client p99 metadata RPC latency is high"
          description: "{{ $labels.instance }} p99 metadata RPC latency has been above 100ms for 5 minutes."

      - alert: FlexFSMountIOErrors
        expr: sum(rate(flexfs_mount_fuse_errors_total{status="EIO"}[5m])) by (instance) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client is returning I/O errors"
          description: "{{ $labels.instance }} has returned EIO to applications for 5 minutes. Check store and metadata connectivity."

      - alert: FlexFSMountStoreErrors
        expr: sum(rate(flexfs_mount_store_errors_total[5m])) by (instance, layer, op) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client block store errors"
          description: "{{ $labels.instance }} is seeing block store {{ $labels.op }} failures at the {{ $labels.layer }} layer."

      - alert: FlexFSMountWriteBackSaturated
        expr: flexfs_mount_sync_slots_in_use >= flexfs_mount_sync_slots_capacity
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount client write-back is saturated"
          description: "{{ $labels.instance }} has had all write-back sync slots busy for 10 minutes; uploads are not keeping up with writes."

      - alert: FlexFSMountKernelInvalidationsDropped
        expr: increase(flexfs_mount_kernel_invalidations_dropped_total[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount dropped a kernel cache invalidation"
          description: "{{ $labels.instance }} could not queue a kernel cache invalidation, so the kernel may still be serving attributes or data pages this mount knows are stale. Indicates the invalidation queue is saturated — check flexfs_mount_kernel_inval_queue_depth and whether block reads are stalling, since a notification write waits on in-flight reads."

      - alert: FlexFSMountLedgerAccountingFault
        expr: increase(flexfs_mount_ledger_accounting_faults_total[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount lost track of a kernel reference"
          description: "{{ $labels.instance }} found a file it has open that it is no longer tracking a kernel reference for. An open file always holds one, so a reference was lost. Attribute and data invalidations for that file are suppressed while the reference is missing, so a peer's changes to it can go unseen; it clears only when a fresh lookup of the file's name re-establishes the reference, or when the file is closed. Note the counter increments once per affected file, when the fault is first detected, so this alert resolves an hour later even though the condition has not: treat it as a report-and-investigate signal, not a live indicator. Capture the mount log — the matching entry reads 'has an open file handle but no nlookup ledger entry', and any nearby 'nlookup underflow' lines are related — and report it."

      - alert: FlexFSMountLedgerFlushesDropped
        expr: increase(flexfs_mount_ledger_flushes_dropped_total[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount could not flush a stale kernel cache entry"
          description: "{{ $labels.instance }} detected a wrong kernel lookup count for a file but had no room on the invalidation queue to flush what the kernel had cached for it, so a stale copy may be served until the kernel forgets the inode or `--attrValid` expires. Indicates sustained invalidation backpressure; check for a saturated notification path."

      - alert: FlexFSMountDiskCacheNearQuota
        expr: |
          (flexfs_mount_disk_cache_clean_bytes + flexfs_mount_disk_cache_dirty_bytes)
          / flexfs_mount_disk_cache_quota_bytes
          > 0.90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "flexFS mount disk cache is above 90% of quota"
          description: "{{ $labels.instance }} local disk cache utilization is {{ $value | humanizePercentage }}. Eviction keeps it within quota; benign unless hit rates suffer."

Alertmanager integration

Configure Prometheus to send alerts to Alertmanager:

prometheus.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager:9093"

Tip

The proxy server down alert uses severity: warning rather than severity: critical because mount clients automatically fall back to direct storage access when proxy servers are unavailable. The metadata server down alert is critical because it directly affects mount client operations.

Operational runbook references

Alert	Immediate action
FlexFSMetaServerDown	Check systemd status, server logs, network connectivity. Restart if needed.
FlexFSMetaHighRPCLatency	Check disk I/O, database compaction debt, network latency. Consider increasing database memory.
FlexFSMetaDiskUsageHigh	Add disk space or move the database folder to a larger volume. Check for excessive retention.
FlexFSMetaDiskAvailableLow	Free or add space now: the server stops itself once available space falls below --minDiskAvail, and it will refuse to start again until space is recovered.
FlexFSProxyServerDown	Check systemd status, server logs. Mount clients will use direct storage in the meantime.
FlexFSProxyCacheNearQuota	Increase --diskQuota or add disk space. Eviction will keep the cache within quota.
FlexFSProxyDiskAvailableLow	Free or add space on the database folder filesystem: the server stops itself once available space falls below --minDiskAvail.
FlexFSProxyDirtyBlocksHigh	Check storage backend health and latency. Consider increasing --writebackActive.
FlexFSMountMetaDisconnected	Check the network path to the metadata server, meta server health, and the mount client logs.
FlexFSMountHighMetaLatency	Check metadata server load/latency and the network path from this host.
FlexFSMountIOErrors	Check store/proxy connectivity and metadata health; inspect mount logs for the failing operations.
FlexFSMountStoreErrors	Check object store / proxy reachability and credentials; a diskcache layer error indicates local disk problems.
FlexFSMountWriteBackSaturated	Check storage backend throughput; consider increasing --dirtyActive or reducing write pressure.

Next steps

• Metrics reference — full metrics catalog
• Prometheus setup — scrape configuration
• Grafana dashboards — visualization