Troubleshooting

/dev/fuse not available

Symptom: Node pods crash with errors about FUSE device access.

Cause: The worker node does not have the FUSE kernel module loaded or /dev/fuse is not accessible.

Solution:

1. Verify the FUSE device exists on the host:

   ls -la /dev/fuse

2. If missing, load the kernel module:

   sudo modprobe fuse

3. To make it persistent across reboots, add fuse to /etc/modules-load.d/fuse.conf.

Most Linux distributions (Ubuntu, Amazon Linux 2, RHEL 8+) include FUSE3 by default.

Privileged mode denied

Symptom: Node DaemonSet pods fail to schedule with “privileged” security policy errors.

Cause: The cluster’s Pod Security Policy, Pod Security Standards, or OPA/Gatekeeper rules prevent privileged pods.

Solution: The flexFS CSI node pods require privileged mode because they need to:

• Access /dev/fuse for FUSE mounts
• Perform bind mounts into the kubelet directory tree with bidirectional mount propagation

You must allow privileged pods in the kube-system namespace for the flexfs-csi-node DaemonSet. Example for Pod Security Standards:

apiVersion: v1
kind: Namespace
metadata:
  name: kube-system
  labels:
    pod-security.kubernetes.io/enforce: privileged

Pod stuck in ContainerCreating

Symptom: A pod using a flexFS PVC stays in ContainerCreating state.

Diagnosis:

kubectl describe pod <pod-name>
kubectl logs -n kube-system <flexfs-csi-node-pod> -c flexfs-csi-driver

Common causes:

Cause	Log message	Solution
Admin server unreachable	request creation failed or connection timeouts	Verify the adminAddr in your Secret is correct and reachable from the cluster.
Invalid account token	statusCode: 401 or statusCode: 403	Verify the token in your Secret is valid.
Volume does not exist	statusCode: 404	For static provisioning, verify the volumeHandle matches an existing volume name.
Encryption secret missing	Error: encryption secret must be at least 8 characters	Add the secret field to your Kubernetes Secret.
Stale FUSE mount	detected a broken base mount	The driver will attempt to unmount it automatically. If it persists, manually unmount from the node.

Secret misconfiguration

Symptom: Volumes fail to mount with “missing admin addr” or “missing account token”.

Solution: Verify your Secret contains the correct field names (adminAddr, token) and that the Secret is in the correct namespace:

kubectl get secret flexfs-secret -n default -o yaml

The Secret must be referenced correctly in both the StorageClass (for dynamic provisioning) and the PersistentVolume (for static provisioning).

Image pull errors

Symptom: CSI pods show ImagePullBackOff or ErrImagePull.

Solution:

1. Verify the image name and tag in your values or manifests. The default image is flexfs/csi-driver-1.9:latest.
2. If using a private registry, create an imagePullSecret and reference it in the pod spec.
3. Verify network connectivity from the nodes to the container registry.

Kubelet directory mismatch (MicroK8s and others)

Symptom: Pods start and the driver reports success, but the volume’s mount point inside the pod is empty. The base FUSE mount is visible on the host (e.g. under <kubeletDir>/flexfs/<volume-name>), yet the pod sees nothing — and there are no errors in the driver logs.

Cause: The kubelet on your nodes uses a non-standard --root-dir (not /var/lib/kubelet). MicroK8s is the common case: it uses /var/snap/microk8s/common/var/lib/kubelet.

The driver builds its FUSE base mounts under <kubeletDir>/flexfs and then bind-mounts them onto the per-pod targetPath that kubelet supplies. That targetPath lives under kubelet’s real root directory. If kubeletDir does not match the real root, the bind mount is created at a path outside the bidirectionally-propagated kubelet-dir volume, so it never propagates back to the pod. Every step still “succeeds,” which is why nothing is logged.

Solution: Set the kubeletDir value to the node’s real kubelet root directory. This single value drives the DaemonSet host paths, the registrar’s --kubelet-registration-path, the container mountPath, and the --kubeletDir argument on the driver — they must all agree.

# MicroK8s (run from the directory where you extracted flexfs-csi-driver.tgz)
helm upgrade --install flexfs-csi-driver ./flexfs-csi-driver \
  --namespace kube-system \
  --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet

# Then restart the node pods (OnDelete update strategy)
kubectl delete pods -n kube-system -l app=flexfs-csi-node

To determine the kubelet root directory on a node, inspect the running kubelet for a --root-dir flag (ps aux | grep kubelet); if absent, the default /var/lib/kubelet applies.

Transport endpoint not connected

Symptom: Pods that were running become unresponsive when accessing the mounted volume, with “transport endpoint is not connected” errors.

Cause: The underlying FUSE process (mount.flexfs) was terminated or crashed.

Solution: The CSI driver detects this condition and attempts to re-establish the mount automatically during the next NodePublishVolume call. To force recovery:

1. Delete the affected pod (Kubernetes will recreate it)
2. If the issue persists, check the mount client log at <kubeletDir>/flexfs/<volume-name>/log-rw inside the node pod (where <kubeletDir> is your configured kubelet root, /var/lib/kubelet by default)

Debug logging

Enable verbose logging on the CSI driver by adding the --verbose flag to the csi.flexfs start command in both the Deployment and DaemonSet specs:

containers:
  - name: flexfs-csi-driver
    image: flexfs/csi-driver-1.9:latest
    args: ["start", "--verbose"]

To enable verbose logging on the mount client, pass verbose as a mount option on the PV:

spec:
  mountOptions:
    - verbose

Mount client logs are written to <kubeletDir>/flexfs/<volume-name>/log-rw (or log-ro for read-only mounts) inside the node pod, where <kubeletDir> is your configured kubelet root (/var/lib/kubelet by default).

Checking CSI driver logs

# Controller logs
kubectl logs -n kube-system deployment/flexfs-csi-controller -c flexfs-csi-driver

# Node logs (pick a specific node pod)
kubectl logs -n kube-system <flexfs-csi-node-pod> -c flexfs-csi-driver

# Provisioner sidecar logs
kubectl logs -n kube-system deployment/flexfs-csi-controller -c csi-provisioner

Next steps

• CSI Driver Overview
• Configuration reference
• General troubleshooting