Proxy Server Overview
Proxy servers (proxy.flexfs) act as a CDN-like block caching layer between mount clients and object storage. They receive block read and write requests over HTTPS, cache blocks on local disk, and read from or upload to the underlying object store. Proxy servers are organized into proxy groups — sets of one or more proxy servers that collectively cache blocks for one or more volumes.
Why use proxy servers?
Section titled “Why use proxy servers?”- Multi-client read amplification — When many mount clients read the same data, proxy servers serve cached blocks instead of each client hitting object storage independently.
- Egress cost reduction — Blocks served from the proxy cache avoid object storage egress charges.
- Regional and edge caching — Place proxy groups near compute workloads in different regions. Mount clients automatically select the lowest-latency group whose servers all answer within 125 ms.
- Hybrid cloud / on-prem — Proxy groups with writeback caching bridge on-premises compute with cloud object storage, masking the higher latency of cross-network writes. See Hybrid Deployments.
Architecture
Section titled “Architecture”Proxy groups
Section titled “Proxy groups”A proxy group is a collection of one or more proxy server addresses. Each volume can be associated with multiple proxy groups (e.g., one per region). The admin server stores the mapping between volumes and proxy groups.
RTT-based group selection
Section titled “RTT-based group selection”When a mount client starts, and whenever its volume’s proxy groups change, it probes every server of every proxy group configured for its volume in parallel, and selects the group with the lowest round-trip time (RTT), taking a group’s RTT from its slowest server. A group counts as unreachable unless all of its servers answer within 125 ms (the Internal mount flag --proxyProbeTO). If no proxy group answers, the mount client bypasses proxies and communicates directly with object storage, probing again after 5 minutes. After a proxy request still fails after 15 seconds of retries, it goes direct for 5 minutes and then probes all groups again in the background, preferring any other group that answers over the one that failed. It then stays on the new group until that group fails, the volume’s proxy groups change, or the volume is remounted. When the volume sets maxProxied, each file’s blocks beyond that count also go direct, and block deletion always does. This fallback ensures mounts never fail due to proxy unavailability, but it means mount clients always need a network path to object storage.
Block routing
Section titled “Block routing”Within the selected proxy group, mount clients spread blocks across the proxy servers by hashing each block’s key. This ensures:
- Deterministic routing — All clients route the same block to the same proxy server, maximizing cache hit rates.
- Minimal disruption — When a proxy server joins or leaves the group, only the blocks assigned to that server are redistributed. Other assignments remain stable.
Dynamic membership
Section titled “Dynamic membership”Proxy server addresses within a group can be added or removed via the admin server while mounts are active. The change reaches connected mount clients without a remount, and each one re-selects its group and rebuilds its routing table when it arrives.
Data flow
Section titled “Data flow”Read path
Section titled “Read path”- Mount client determines the target proxy server from the block’s key.
- Proxy server checks its local disk cache.
- Cache hit: Block is returned immediately from disk.
- Cache miss: Proxy server fetches the block from object storage, caches it locally, and returns it to the client.
Write path
Section titled “Write path”Proxy servers always cache writes (writeback); there is no write-through mode.
- Mount client sends the block to the target proxy server.
- Proxy server writes the block to its local disk cache and acknowledges the client immediately.
- The proxy server uploads the block to object storage in the background.
Until that upload finishes, the proxy server’s disk holds the only copy of the block. See Removing a proxy server before taking one out of service.
When to use proxies
Section titled “When to use proxies”| Scenario | Benefit |
|---|---|
| Multiple clients reading the same dataset | Shared cache reduces object storage load and egress |
| Cross-region compute accessing a central bucket | Regional proxy groups reduce read latency |
| On-premises compute with cloud storage | Writeback caching masks WAN latency for writes |
| Large-scale training or analytics jobs | Proxy cache absorbs repeated data access patterns |
| Egress cost sensitivity | Cached reads avoid per-GB egress charges |