Maintenance

Database folder

The metadata server stores all filesystem metadata in a database folder. The default location is:

~/.flexfs/meta/data

This can be changed with the --dbFolder flag. The folder is created automatically on first start.

Folder contents

The database folder contains the metadata database files. These files should not be modified manually. The metadata server manages compaction, garbage collection, and space reclamation automatically.

Storage requirements

Metadata database size depends on:

• Number of files and directories — Each inode (file, directory, symlink) consumes metadata entries.
• Extended attributes and ACLs — Volumes with extensive xattrs or ACLs require more space.
• Retention window — When block retention is enabled (for time-travel), historical metadata versions are retained, increasing database size.

As a rough guideline, expect 1-2 KiB of metadata per file/directory. A volume with 10 million files requires approximately 10-20 GiB of metadata storage.

Tip

Place the database folder on a fast local disk (SSD or NVMe). Metadata operations are I/O-sensitive, and database performance directly impacts filesystem latency.

Caution

Never let the database folder’s filesystem fill up. The server checks the available space on startup and every minute thereafter: below a minimum free-space threshold (1 GiB by default) it refuses to start, or shuts itself down gracefully while there is still room to close the database cleanly. Alert on flexfs_meta_db_folder_disk_available_bytes so you act first — see Low disk space protection.

Backup strategies

Online backup (no downtime)

The recommended way to back up the metadata database is an online checkpoint, taken while the server keeps running. The server writes a consistent, point-in-time copy of each volume’s database into a staging directory using the POST /v2/backup endpoint. Because the staging copy is a self-contained snapshot that nothing writes to, it is safe to rsync offsite — unlike the live database folder, which is mutated under any copy.

Trigger a checkpoint, then ship the staging directory to your backup target. A once-per-minute cron job is a reasonable schedule:

set -euo pipefail

# 1. Ask the running metadata server for a fresh consistent checkpoint.
curl -fsk -X POST \
  -H "Authorization: Bearer <meta-token>" \
  "https://localhost:8443/v2/backup?blocking=true"

# 2. Ship the staging directory (a static, consistent snapshot) offsite.
rsync -a --delete --exclude '.incoming/' \
  /root/.flexfs/meta/data/.backup/ backup-host:/backups/meta/

• <meta-token> is the metadata server’s token — the UUID configured with meta.flexfs init creds.
• blocking=true makes the request wait for the checkpoint and return its outcome; without it you get an immediate 202 (fire-and-forget) and a failure would be invisible to the cron job.
• curl -f makes the cron job fail loudly if the checkpoint returns an error, so a bad backup is never silent.
• Staging directory: with no dest parameter, checkpoints are written to <dbFolder>/.backup (e.g. ~/.flexfs/meta/data/.backup in the home folder of the user running the server; the example uses root’s). Pass ?dest=<dir> to choose another location. For the checkpoint to be near-instant it must be on the same filesystem as --dbFolder (the database’s immutable files are hard-linked rather than copied); across filesystems every file is copied instead.
• Disk overhead is bounded: each run replaces the previous checkpoint, so the staging directory holds roughly one checkpoint’s worth of data. A checkpoint does temporarily pin the database files it references until it is replaced, so avoid retaining stale checkpoints in place.

Tip

The staging directory must not be, or contain, the database folder itself — the server rejects such a dest. The default <dbFolder>/.backup is safe because it sits beneath the db folder and is ignored by volume loading.

To restore from an online backup, place the per-volume <volumeID>/ directories from your backup into the metadata server’s --dbFolder and start the server. Each directory is already a complete, valid database captured at a single point in time.

Offline backup (cold copy)

If you prefer a whole-folder copy — for example, to archive a server that is already being taken down — stop the metadata server first, then copy the database folder. This example is for a server run as the root systemd service:

set -e
sudo systemctl stop flexfs-meta
sudo cp -a /root/.flexfs/meta/data /backup/meta-data-$(date +%Y%m%d)
sudo systemctl start flexfs-meta

Caution

Stopping the server is only required for this cold-copy method. Do not copy the live database folder without stopping the server — the files may be mid-write and the copy inconsistent. Prefer the online checkpoint method above, which needs no downtime.

Block storage durability

The block data itself is stored in cloud object storage (S3, GCS, Azure Blob, OCI), which provides its own durability guarantees (typically 99.999999999% for standard storage classes). Block data does not need to be backed up separately.

Point-in-time recovery

For volumes with block retention configured, the time-travel feature provides point-in-time access to historical filesystem state. This can serve as a complement to traditional backups for data recovery scenarios.

Memory tuning

Database cache

The metadata database keeps an in-memory cache of index and data blocks, and this cache is the most impactful factor in metadata server performance. By default it is sized to 50% of system RAM.

Guidelines:

• For dedicated metadata servers, the default of 50% of system RAM is a good starting point.
• On servers that run alongside other services, ensure enough RAM remains for those workloads and for the operating system.
• If the metadata database is larger than the available cache, frequently accessed metadata will still be served from cache, but random access patterns may incur disk I/O.
• Monitor the server’s memory usage and provision RAM accordingly if the system is under memory pressure.

Note

The metadata server periodically releases freed memory back to the operating system to prevent excessive RSS growth from database cache fragmentation.

Sync mode

By default the metadata server commits writes without fsync, favoring write throughput at the cost of a small crash-durability window. The --sync flag opts into fsyncing every write operation, so each metadata change is durable before it is acknowledged.

• async (default): Higher write throughput. A crash may lose the last few seconds of metadata operations, but the database remains consistent (no corruption).
• --sync: Every metadata write is durable before acknowledgment. A hard crash or power loss cannot lose acknowledged metadata. Choose this when durability of the most recent operations matters more than metadata write latency.

Metadata read errors

If the metadata server cannot read part of a volume’s database, for example because of a disk read error or a damaged database file, it stops accepting changes for that volume instead of acting on data it could not read. Other volumes on the same server are not affected.

While a volume is in this state:

• Its mounts pause. Applications see their file operations wait rather than fail, and the mounts reconnect on their own.
• A recount of that volume with blocking=true returns 503 Service Unavailable with Retry-After: 60. Without blocking=true it is accepted with 202 and the failure is only logged. A recount of all volumes skips it. Find, find-old, analyze, duplicates, size bins and missing objects requests for that volume return 503 Service Unavailable with Retry-After: 60, and find.flexfs, analyze.flexfs and dedup.flexfs report that its metadata cannot be read. A query already running when the error occurs stops; a find or find-old query that has already started returning results is cut off instead of receiving the 503. A verify request still runs and returns the 503 if it hits a read error. Size bins requests for all volumes leave that volume out, a missing objects request for all volumes reports an error entry for it, and a verify request for all volumes reports an error for it if it hits a read error.
• Background maintenance, such as block cleanup and reconciliation, skips the volume. A reconcile request skips it too, and with blocking=true still returns 200. Reconciliation also skips other volumes that share its block store location.
• The metadata server logs volume <id> stopped accepting changes after a metadata read error and reports the error.

Every five minutes the metadata server reads the volume’s entire database again. As soon as a full read succeeds, it logs volume <id> is accepting changes again and the paused mounts continue where they left off.

If the full read keeps failing, check the health of the disk that holds the database folder. If the database itself is damaged, restore it from a backup (see Backup strategies).

Scaling considerations

The metadata server is a single-instance service. Scaling is vertical:

• CPU: The metadata server benefits from multiple cores for concurrent RPC session handling and background maintenance tasks.
• Memory: More memory allows a larger database cache, reducing disk I/O.
• Storage: Fast local storage (NVMe) reduces metadata operation latency.
• Network: The metadata server handles RPC connections from all mount clients and REST requests from utilities. Ensure sufficient network bandwidth for your expected session count.

For very large deployments (thousands of concurrent mounts), consider:

• Splitting volumes across multiple metadata servers (each metadata server handles a subset of volumes).
• Increasing the database cache size.
• Using the fastest available local storage.