Skip to content
flexFS

Logging and Diagnostics

FlexFS provides multiple logging and diagnostic tools for troubleshooting performance issues and understanding system behavior.

Mount client log file

Section titled “Mount client log file”

In daemon mode (the default), the mount client writes logs to a file. Specify the log file path with:

Terminal window
mount.flexfs start <volume-name> /mnt/data --logFile /var/log/flexfs/mount.log

If no --logFile is specified, logs are written to ~/.flexfs/mount/logs/<name>-<pid>.log by default, where <name> is the volume name and <pid> is the mount process ID.

In foreground mode (--foreground), and when started by systemd, logs are written to stdout and fatal errors to stderr.

Log file cleanup

Section titled “Log file cleanup”

When a mount in daemon mode ends normally, for example after umount or a stop signal, the mount client deletes its log file. The log file is kept when:

  • The mount client exits because of an error, crashes, or is killed, so the log is available to find the cause.
  • An auto-update hands the mount to a new mount client process and the log file path has no <pid> placeholder. The new process continues writing to the same file. With a path containing <pid>, the new process writes to a new file and the old process deletes its own log.
  • The mount was started with --keepLog.

Kept logs do not accumulate indefinitely when their path contains <pid>, as the default path does. Every running mount client whose log file path contains <pid> deletes log files that match its own --logFile path, with any process ID in place of <pid> and any volume name in place of <name>, once their process is no longer running and they have not been modified for at least an hour. It checks when it starts and every 10 minutes. Log files whose path has no <pid> placeholder are never deleted this way.

Verbose logging

Section titled “Verbose logging”

Enable detailed logging with the --verbose flag:

Terminal window
mount.flexfs start <volume-name> /mnt/data --verbose

Verbose mode produces significantly more output and should be used for debugging only.

For servers:

Terminal window
meta.flexfs start --verbose
proxy.flexfs start --verbose

Metrics endpoint

Section titled “Metrics endpoint”

The mount client can expose a Prometheus /metrics endpoint with --metrics, served over plain HTTP on --metricsPort (default 6075). This is the recommended way to monitor a running mount. See the Metrics Reference for the full catalog and Prometheus setup for scrape configuration.

Server logging

Section titled “Server logging”

The metadata server and proxy server log to stdout by default. Use --verbose for detailed operation logging.

When running under systemd, logs are captured by the journal:

Terminal window
# Metadata server logs
journalctl -u flexfs-meta -f


# Proxy server logs
journalctl -u flexfs-proxy -f

API access rules

Section titled “API access rules”

Where an access file is in use, lines like these describe it. All of them are logged whether or not --verbose is set: a refused request is either an attack or a rule that is wrong, and both need to be visible in a default deployment.

Access rules loaded from "/root/.flexfs/meta/access": global 10.0.0.0/8, 1 endpoint rule(s)
  access | DELETE /locks                                  10.1.2.3
  access | global rule gates: /backup, /compact, /locks, /metrics, /status, /verify, ...
  access | not gated by the global rule: / (add an [endpoint."..."] section to gate one)
Denied "DELETE" "/locks" from "203.0.113.9" (access rules)
Error (access): cannot reload "/root/.flexfs/meta/access", keeping the rules already in force: ...
Access rules removed ("/root/.flexfs/meta/access") - all endpoints unrestricted

The global rule gates: line is the one to read after an edit — it names every endpoint the save just put behind the whitelist, which on the admin and free-tier servers includes the endpoints mount clients and the metadata server call.

A denial is logged once per source, method and endpoint every ten seconds. Requests to paths the server does not serve share one entry per source and method, so a scanner walking URLs cannot flood the log; whatever the throttle drops is counted on the next line it writes (... 49 further denial(s) not logged). A reload failure repeats no more than once a minute for as long as the file stays broken.

Worth alerting on: any Error (access) line, and a /status response whose access.state is stale — both mean the file on disk is not what the server is enforcing. See API Access Control.

CSI driver logging

Section titled “CSI driver logging”

The CSI driver logs to stdout within the container. To enable verbose logging, add --verbose to the driver container’s args (see Debug logging). To view the logs:

Terminal window
# View CSI driver logs
kubectl logs -n kube-system <flexfs-csi-node-pod> -c flexfs-csi-driver

Mount client logs within the CSI driver are written to <kubeletDir>/flexfs/<volume-name>/log-rw (or log-ro for read-only mounts), where <kubeletDir> is the driver’s --kubeletDir (/var/lib/kubelet by default).

Next steps

Section titled “Next steps”