Maintenance

Database folder

The metadata server stores all filesystem metadata in a database folder. The default location is:

~/.flexfs/meta/data

This can be changed with the --dbFolder flag. The folder is created automatically on first start.

Folder contents

The database folder contains the metadata database files. These files should not be modified manually. The metadata server manages compaction, garbage collection, and space reclamation automatically.

Storage requirements

Metadata database size depends on:

• Number of files and directories — Each inode (file, directory, symlink) consumes metadata entries.
• Extended attributes and ACLs — Volumes with extensive xattrs or ACLs require more space.
• Retention window — When block retention is enabled (for time-travel), historical metadata versions are retained, increasing database size.

As a rough guideline, expect 1-2 KiB of metadata per file/directory. A volume with 10 million files requires approximately 10-20 GiB of metadata storage.

Tip

Place the database folder on a fast local disk (SSD or NVMe). Metadata operations are I/O-sensitive, and database performance directly impacts filesystem latency.

Caution

Never let the database folder’s filesystem fill up. The server checks the available space on startup and every minute thereafter: below --minDiskAvail (1 GiB by default) it refuses to start, or shuts itself down gracefully while there is still room to close the database cleanly. Alert on flexfs_meta_db_folder_disk_available_bytes so you act first — see Low disk space protection.

Backup strategies

Online backup (no downtime)

The recommended way to back up the metadata database is an online checkpoint, taken while the server keeps running. The server writes a consistent, point-in-time copy of each volume’s database into a staging directory using the POST /backup endpoint. Because the staging copy is a self-contained snapshot that nothing writes to, it is safe to rsync offsite — unlike the live database folder, which is mutated under any copy.

Trigger a checkpoint, then ship the staging directory to your backup target. A once-per-minute cron job is a reasonable schedule:

# 1. Ask the running metadata server for a fresh consistent checkpoint.
curl -fsk -X POST \
  -H "Authorization: Bearer $META_TOKEN" \
  "https://localhost:443/backup?blocking=true"

# 2. Ship the staging directory (a static, consistent snapshot) offsite.
rsync -a --delete ~/.flexfs/meta/data/.backup/ backup-host:/backups/meta/

• $META_TOKEN is the metadata server’s token — the UUID configured with meta.flexfs init creds.
• blocking=true makes the request wait for the checkpoint and return its outcome; without it you get an immediate 202 (fire-and-forget) and a failure would be invisible to the cron job.
• curl -f makes the cron job fail loudly if the checkpoint returns an error, so a bad backup is never silent.
• Staging directory: with no dest parameter, checkpoints are written to <dbFolder>/.backup (e.g. ~/.flexfs/meta/data/.backup). Pass ?dest=<dir> to choose another location. For the checkpoint to be near-instant it must be on the same filesystem as --dbFolder (the database’s immutable files are hard-linked rather than copied); across filesystems every file is copied instead.
• Disk overhead is bounded: each run replaces the previous checkpoint, so the staging directory holds roughly one checkpoint’s worth of data. A checkpoint does temporarily pin the database files it references until it is replaced, so avoid retaining stale checkpoints in place.

Tip

The staging directory must not be, or contain, the database folder itself — the server rejects such a dest. The default <dbFolder>/.backup is safe because it sits beneath the data directory and is ignored by volume loading.

To restore from an online backup, place the per-volume <volumeID>/ directories from your backup into the metadata server’s --dbFolder and start the server. Each directory is already a complete, valid database captured at a single point in time.

Offline backup (cold copy)

If you prefer a whole-folder copy — for example, to archive a server that is already being taken down — stop the metadata server first, then copy the database folder:

sudo systemctl stop flexfs-meta
cp -a ~/.flexfs/meta/data /backup/meta-data-$(date +%Y%m%d)
sudo systemctl start flexfs-meta

Caution

Stopping the server is only required for this cold-copy method. Do not copy the live database folder without stopping the server — the files may be mid-write and the copy inconsistent. Prefer the online checkpoint method above, which needs no downtime.

Block storage durability

The block data itself is stored in cloud object storage (S3, GCS, Azure Blob, OCI), which provides its own durability guarantees (typically 99.999999999% for standard storage classes). Block data does not need to be backed up separately.

Point-in-time recovery

For volumes with block retention configured, the time-travel feature provides point-in-time access to historical filesystem state. This can serve as a complement to traditional backups for data recovery scenarios.

Memory tuning

Database cache

The --dbMemCapacity flag controls how much memory the metadata database allocates for its in-memory block cache (index and data blocks). This is the most impactful tuning parameter for metadata server performance.

Value	Description
40% (default)	40% of system RAM
4G	4 GiB absolute
512M	512 MiB absolute

Guidelines:

• For dedicated metadata servers, the default of 40% is a good starting point.
• For servers running alongside other services, reduce to 10-20% or use an absolute value.
• If the metadata database is larger than available cache, frequently accessed metadata will still be served from cache, but random access patterns may incur disk I/O.
• Monitor the server’s memory usage and adjust if the system is under memory pressure.

Note

The metadata server periodically releases freed memory back to the operating system to prevent excessive RSS growth from database cache fragmentation.

Sync mode

By default the metadata server commits writes without fsync, favoring write throughput at the cost of a small crash-durability window. The --sync flag opts into fsyncing every write operation, so each metadata change is durable before it is acknowledged.

• async (default): Higher write throughput. A crash may lose the last few seconds of metadata operations — bounded by the periodic flush interval (~5 seconds) — but the database remains consistent (no corruption).
• --sync: Every metadata write is durable before acknowledgment. A hard crash or power loss cannot lose acknowledged metadata. Choose this when durability of the most recent operations matters more than metadata write latency.

Scaling considerations

The metadata server is a single-instance service. Scaling is vertical:

• CPU: The metadata server benefits from multiple cores for concurrent RPC session handling and background maintenance tasks.
• Memory: More memory allows a larger database cache, reducing disk I/O.
• Storage: Fast local storage (NVMe) reduces metadata operation latency.
• Network: The metadata server handles RPC connections from all mount clients and REST requests from utilities. Ensure sufficient network bandwidth for your expected session count.

For very large deployments (thousands of concurrent mounts), consider:

• Splitting volumes across multiple metadata servers (each metadata server handles a subset of volumes).
• Increasing the database cache size.
• Using the fastest available local storage.