Proxy Groups

Note

Proxy groups are an Enterprise feature. Community edition users can use local on-disk writeback caching (--diskWriteback with --diskQuota) on the mount client as an alternative latency mitigation strategy.

Proxy groups are managed through the admin server using configure.flexfs. A proxy group defines a set of proxy server addresses that collectively cache blocks for associated volumes.

Creating a proxy group

Use configure.flexfs to create a proxy group:

configure.flexfs create proxy-group \
  --providerCode aws \
  --regionCode <region> \
  --addresses proxy1.example.com:443,proxy2.example.com:443

The provider and region must already exist (see Providers and Regions); otherwise the request fails with 400: referenced record not found (foreign key constraint).

A proxy group has the following properties:

Property	Description
ID	Auto-assigned unique identifier (integer)
Provider Code	Provider where the proxy servers are deployed
Region Code	Region where the proxy servers are deployed
Addresses	Comma-separated list of proxy server host:port addresses

Associating proxy groups with volumes

A volume can be associated with multiple proxy groups. This is done by creating volume-proxy-group associations:

configure.flexfs create volume-proxy-group \
  --volumeID <volume-name> \
  --proxyGroupID 1

You can associate multiple proxy groups with the same volume:

configure.flexfs create volume-proxy-group \
  --volumeID <volume-name> \
  --proxyGroupID 1

configure.flexfs create volume-proxy-group \
  --volumeID <volume-name> \
  --proxyGroupID 2

When a mount client connects, it receives all proxy groups associated with its volume and selects the one with the lowest RTT.

Dynamic membership

Proxy group membership can be modified while mounts are active:

Adding a proxy server

configure.flexfs update proxy-group 1 \
  --addresses proxy1.example.com:443,proxy2.example.com:443,proxy3.example.com:443

Mount clients pick the new proxy server up without a remount and begin routing blocks to it. Only blocks that hash to the new server are affected — existing routing assignments remain stable.

Removing a proxy server

A proxy server acknowledges writes once they are on its local disk and uploads them to object storage afterward, so it may hold the only copy of recently written blocks. Drain it before taking it out of service:

1. Update the addresses list without it:

   configure.flexfs update proxy-group 1 \
     --addresses proxy1.example.com:443,proxy2.example.com:443

2. Keep the removed server running until its flexfs_proxy_cache_dirty_blocks metric reaches 0 and stays there. A proxy server waits only 30 seconds for pending uploads when it stops, and uploads what is left the next time it starts with the same cache folder.

3. Stop and decommission the server.

Decommissioning a server, or deleting its cache folder, before its dirty blocks reach 0 loses writes that clients were told had succeeded.

Following these steps loses nothing, but while the removed server still holds dirty blocks, reads of those blocks go to another group member that cannot find them in object storage yet. The mount retries for up to 15 seconds, then sends all its block traffic directly to object storage for 5 minutes, and the direct read waits until the upload finishes. Readers of recently written data therefore stall, and mounts bypass the group for a while. To keep this short, remove servers during low write activity.

Blocks previously routed to the removed server are redistributed among the remaining servers. Clients using the updated routing table get cache misses for those blocks, which are fetched from object storage and cached on their new target.

Scaling considerations

• Adding servers: Causes a partial cache warm-up as some blocks are re-routed. The more servers in the group, the smaller the fraction of blocks affected.
• Removing servers: Causes the removed server’s block assignments to shift to other servers, triggering cache misses.
• Replacing servers: Equivalent to adding the new server and removing the old one.

Removing a proxy group

A proxy group cannot be deleted while a volume uses it, including a retired volume whose data has not been deleted yet: the delete is refused with 409 (in use by a volume). Unlink each volume from the group first, then delete the group:

configure.flexfs delete volume-proxy-group <volume-name> 1
configure.flexfs delete proxy-group 1

As when removing a single server, keep the group’s servers running until their flexfs_proxy_cache_dirty_blocks metric reaches 0 before decommissioning them.

How mount clients discover proxy groups

1. During startup, the mount client fetches volume settings from the admin server. The settings include the list of proxy groups (and their addresses) associated with the volume.
2. When it starts, whenever the volume’s proxy groups change, and 5 minutes after a proxy request fails, the client probes every server of every group in parallel.
3. Among the groups whose servers all answer within 125 ms, the one whose slowest server answers fastest is selected. After a failure, any other group that answers is preferred over the one that failed.
4. Membership changes made later reach the client while it stays mounted, and it re-selects its group when they do.

If no proxy group answers, the mount client falls back to direct object storage access and probes again after 5 minutes. If requests to the selected group fail, the client goes direct for 5 minutes and then probes again, returning to the failed group only if no other group answers. After switching, it stays on the new group until that group fails, the volume’s proxy groups change, or the volume is remounted.

SSL certificates for proxy groups

Mount clients connect to proxy servers over HTTPS by default, and each proxy server auto-generates a self-signed certificate on first start. To use your own certificate, start the proxy server with --sslCert and --sslKey (see TLS Certificates).