Troubleshooting

/dev/fuse not available

Symptom: Node pods crash with errors about FUSE device access.

Cause: The worker node does not have the FUSE kernel module loaded or /dev/fuse is not accessible.

Solution:

1. Verify the FUSE device exists on the host:

   ls -la /dev/fuse

2. If missing, load the kernel module:

   sudo modprobe fuse

3. To make it persistent across reboots, add fuse to /etc/modules-load.d/fuse.conf.

Most Linux distributions (Ubuntu, Amazon Linux 2, RHEL 8+) include FUSE3 by default.

Privileged mode denied

Symptom: Node DaemonSet pods fail to schedule with “privileged” security policy errors.

Cause: The cluster’s Pod Security Policy, Pod Security Standards, or OPA/Gatekeeper rules prevent privileged pods.

Solution: The flexFS CSI node pods require privileged mode because they need to:

• Access /dev/fuse for FUSE mounts
• Perform bind mounts into the kubelet directory tree with bidirectional mount propagation

You must allow privileged pods in the kube-system namespace for the flexfs-csi-node DaemonSet. Example for Pod Security Standards:

apiVersion: v1
kind: Namespace
metadata:
  name: kube-system
  labels:
    pod-security.kubernetes.io/enforce: privileged

Pod stuck in ContainerCreating

Symptom: A pod using a flexFS PVC stays in ContainerCreating state.

Diagnosis:

kubectl describe pod <pod-name>
kubectl logs -n kube-system <flexfs-csi-node-pod> -c flexfs-csi-driver

Common causes:

Cause	Log message	Solution
Admin server unreachable	request creation failed or connection timeouts	Verify the adminAddr in your Secret is correct and reachable from the cluster.
Invalid account token	statusCode: 401 or statusCode: 403	Verify the token in your Secret is valid.
Volume does not exist	statusCode: 404	For static provisioning, verify the volumeHandle matches an existing volume name.
Encryption secret missing	Error: encryption secret must be at least 8 characters	Add the secret field to your Kubernetes Secret.
Stale FUSE mount	detected a broken base mount	The driver will attempt to unmount it automatically. If it persists, manually unmount from the node.

Secret misconfiguration

Symptom: Volumes fail to mount with “missing admin addr” or “missing account token”.

Solution: Verify your Secret contains the correct field names (adminAddr, token) and that the Secret is in the correct namespace:

kubectl get secret flexfs-secret -n default -o yaml

The Secret must be referenced correctly in both the StorageClass (for dynamic provisioning) and the PersistentVolume (for static provisioning).

PVC stays Pending

Symptom: A claim using a flexFS StorageClass never binds. Its events show only ExternalProvisioning -- waiting for a volume to be created, or a ProvisioningFailed warning.

Check the provisioner sidecar first:

kubectl -n kube-system logs -l app=flexfs-csi-controller -c csi-provisioner --tail=50

Two causes account for nearly all of these:

• leases ... is forbidden — the controller’s ServiceAccount cannot use the Lease it needs for leader election, so the sidecar never starts provisioning and the claim gets no error of its own. The Helm chart grants this; if you manage RBAC yourself, the ClusterRole needs coordination.k8s.io/leases with get, list, watch, create, update, patch and delete.
• failed to create volume ... statusCode: 400 — the StorageClass does not tell the admin server where to put the volume. Supply provider + region, or metaStore + blockStore. providerCode and regionCode are not the parameter names the driver reads and are silently ignored.

PVC resize does nothing

Symptom: You edit a claim’s resources.requests.storage, and it stays at the old size with the event waiting for an external controller to expand this PVC.

Cause: The csi-resizer sidecar is not running, or lacks RBAC. Expansion is performed by that sidecar calling the driver — the driver advertising the capability is not enough on its own.

Solution: Confirm the controller Deployment has three containers (driver, csi-provisioner, csi-resizer), that the StorageClass sets allowVolumeExpansion: true and a csi.storage.k8s.io/controller-expand-secret-name, and that the ClusterRole covers persistentvolumeclaims/status and pods:

kubectl -n kube-system logs -l app=flexfs-csi-controller -c csi-resizer --tail=30

If the resize is applied but a running pod still reports ENOSPC, wait a minute: the raised quota reaches an existing mount on its next volume-settings refresh, and df updates at the same time. Neither a remount nor a pod restart is required.

PVC resize fails with “nothing to expand”

Symptom: The claim carries a VolumeResizeFailed event reading volume "..." has no storage quota, so there is nothing to expand.

Cause: That volume has no storage limit, so it already permits more than the claim is asking for. Expansion raises an existing quota; there is nothing here to raise, and recording the unlimited sentinel as the claim’s capacity would tell Kubernetes the claim is exabytes in size.

Solution: Leave it alone if the volume is meant to be unlimited — revert the claim’s request to its previous value. If you want a limit on that volume, set one deliberately:

configure.flexfs update volume <volume-name> --maxBlocks <blocks>

Note this applies to any volume whose max_blocks is 0, which includes volumes bound through a static PV.

Image pull errors

Symptom: CSI pods show ImagePullBackOff or ErrImagePull.

Solution:

1. Verify the image name and tag in your values or manifests. The default image is flexfs/csi-driver-1.9:latest.
2. If using a private registry, create an imagePullSecret and reference it in the pod spec.
3. Verify network connectivity from the nodes to the container registry.

Kubelet directory mismatch (MicroK8s and others)

Symptom: Pods start and the driver reports success, but the volume’s mount point inside the pod is empty. The base FUSE mount is visible on the host (e.g. under <kubeletDir>/flexfs/<volume-name>), yet the pod sees nothing — and there are no errors in the driver logs.

Cause: The kubelet on your nodes uses a non-standard --root-dir (not /var/lib/kubelet). MicroK8s is the common case: it uses /var/snap/microk8s/common/var/lib/kubelet.

The driver builds its FUSE base mounts under <kubeletDir>/flexfs and then bind-mounts them onto the per-pod targetPath that kubelet supplies. That targetPath lives under kubelet’s real root directory. If kubeletDir does not match the real root, the bind mount is created at a path outside the bidirectionally-propagated kubelet-dir volume, so it never propagates back to the pod. Every step still “succeeds,” which is why nothing is logged.

Solution: Set the kubeletDir value to the node’s real kubelet root directory. This single value drives the DaemonSet host paths, the registrar’s --kubelet-registration-path, the container mountPath, and the --kubeletDir argument on the driver — they must all agree.

# MicroK8s (run from the directory where you extracted flexfs-csi-driver.tgz)
helm upgrade --install flexfs-csi-driver ./flexfs-csi-driver \
  --namespace kube-system \
  --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet

# Then restart the node pods (OnDelete update strategy)
kubectl delete pods -n kube-system -l app=flexfs-csi-node

To determine the kubelet root directory on a node, inspect the running kubelet for a --root-dir flag (ps aux | grep kubelet); if absent, the default /var/lib/kubelet applies.

Transport endpoint not connected

Symptom: Pods that were running become unresponsive when accessing the mounted volume, with “transport endpoint is not connected” errors.

Cause: The underlying FUSE process (mount.flexfs) was terminated or crashed.

Cause, most often: the node plugin pod was deleted, upgraded, or evicted. Each volume’s mount.flexfs daemon is a child of the node plugin container, so removing that pod takes down every flexFS mount on the node even though the mount points themselves remain. This is why the DaemonSet ships with updateStrategy: OnDelete — upgrades do not roll node plugins automatically, and you should drain a node (or accept the interruption) before deleting its plugin pod.

Solution: The CSI driver detects a dead mount and rebuilds it on the next NodePublishVolume call. To force recovery:

1. Delete the affected pod (Kubernetes will recreate it, which triggers a fresh publish)
2. If the issue persists, check the mount client log at <kubeletDir>/flexfs/<volume-name>/log-rw inside the node pod (where <kubeletDir> is your configured kubelet root, /var/lib/kubelet by default)

A running container cannot be rescued in place. Each pod’s mount is a bind of the base mount made when that pod was published, and the driver re-binds a pod’s target only during NodePublishVolume — which the kubelet issues when a pod starts. Rebuilding the base mount therefore fixes the next pod to be published, not the ones already running: they keep a bind of the dead session and continue to report transport endpoint is not connected until they are restarted.

mountPropagation: HostToContainer does not change this, which is worth stating because it is a common suggestion for FUSE CSI drivers. It governs whether host-side mounts under a container’s mount point become visible inside it; it does nothing about a pod’s own target path being re-bound, and that is what recovery needs. A running pod with that setting and one without it behave identically here: both keep the dead mount, and both recover on restart.

So the recovery step is simply to restart the affected pods.

Debug logging

Enable verbose logging on the CSI driver by adding the --verbose flag to the csi.flexfs start command in both the Deployment and DaemonSet specs:

containers:
  - name: flexfs-csi-driver
    image: flexfs/csi-driver-1.9:latest
    args: ["start", "--verbose"]

To enable verbose logging on the mount client, pass verbose as a mount option on the PV:

spec:
  mountOptions:
    - verbose

Mount client logs are written to <kubeletDir>/flexfs/<volume-name>/log-rw (or log-ro for read-only mounts) inside the node pod, where <kubeletDir> is your configured kubelet root (/var/lib/kubelet by default).

Checking CSI driver logs

# Controller logs
kubectl logs -n kube-system deployment/flexfs-csi-controller -c flexfs-csi-driver

# Node logs (pick a specific node pod)
kubectl logs -n kube-system <flexfs-csi-node-pod> -c flexfs-csi-driver

# Provisioner sidecar logs
kubectl logs -n kube-system deployment/flexfs-csi-controller -c csi-provisioner

Next steps

• CSI Driver Overview
• Configuration reference
• General troubleshooting