Hybrid Deployments

A common use case for proxy groups is bridging on-premises compute infrastructure with cloud-based object storage. Proxy servers deployed on-prem serve as writeback caches, masking the higher latency of cross-network writes and providing a local read cache for frequently accessed data.

Architecture

In a hybrid deployment, the typical topology is:

Cloud: Object storage bucket (S3, GCS, Azure Blob, or OCI) + admin server + metadata server.
On-premises: One or more proxy servers + mount clients on compute nodes.

The proxy group running on-prem is associated with the volume via the admin server. Mount clients on-prem discover the local proxy group, measure its RTT (which will be low since it is on the same LAN), and route block traffic through it. Every server in a group must answer the probe within 125 ms for the group to be selected; otherwise the group counts as unreachable, and without another group mounts go directly to object storage.

Write path

Mount client writes a block to the on-prem proxy server over the local network.
Proxy server writes the block to its local disk cache and acknowledges the client immediately (writeback).
The proxy server asynchronously flushes the block to cloud object storage over the WAN link.

This decouples write latency from WAN latency. The mount client experiences local-network write speeds, while the proxy server handles the slower WAN transfer in the background.

Read path

Mount client requests a block from the on-prem proxy server.
If the block is in the local cache, it is served immediately.
If not, the proxy server fetches it from cloud object storage, caches it locally, and returns it.

Subsequent reads of the same block (by the same client or any other client in the proxy group) are served from the local cache.

Deployment steps

1. Deploy proxy servers on-prem

Install proxy.flexfs on one or more on-prem machines with fast local storage (SSD or NVMe). This example runs it as a root systemd service; it can also be run directly as any user (see Proxy Server Setup):

# Initialize credentials with cloud storage keys for root, which the systemd service runs as
sudo proxy.flexfs init creds \
  --blockUser <username> \
  --blockPass <password>

# Create systemd service
sudo proxy.flexfs init systemd --now \
  --startFlags "--diskFolder /data/proxy-cache --diskQuota 1T"

2. Create a proxy group

On a machine with access to the admin server, create a provider and region for the data center if they do not exist yet, then create a proxy group containing the on-prem proxy server addresses:

configure.flexfs create provider --code onprem --name "On-premises"
configure.flexfs create region --providerCode onprem --code dc1 --name "Data center 1"

configure.flexfs create proxy-group \
  --providerCode onprem --regionCode dc1 \
  --addresses proxy1.internal:443,proxy2.internal:443

3. Associate the proxy group with your volume

configure.flexfs create volume-proxy-group \
  --volumeID <volume-name> \
  --proxyGroupID 1

4. Mount on compute nodes

On-prem compute nodes mount normally. The mount client will discover the on-prem proxy group and route traffic through it:

mount.flexfs start <volume-name> /mnt/data

Multi-region hybrid

For organizations with multiple data centers or regions, deploy proxy groups in each location:

# Regions for each data center (the onprem provider already exists)
configure.flexfs create region --providerCode onprem --code us-dc --name "US data center"
configure.flexfs create region --providerCode onprem --code eu-dc --name "EU data center"

# Proxy group in US data center
configure.flexfs create proxy-group \
  --providerCode onprem --regionCode us-dc \
  --addresses us-proxy1.internal:443,us-proxy2.internal:443

# Proxy group in EU data center
configure.flexfs create proxy-group \
  --providerCode onprem --regionCode eu-dc \
  --addresses eu-proxy1.internal:443

# Associate both with the volume
configure.flexfs create volume-proxy-group \
  --volumeID shared-data \
  --proxyGroupID 1

configure.flexfs create volume-proxy-group \
  --volumeID shared-data \
  --proxyGroupID 2

Mount clients in each location automatically select the proxy group with the lowest RTT, provided every server in it answers within 125 ms.

Tuning for hybrid workloads

Cache sizing

Size the on-prem proxy cache to hold the active working set. If your workload accesses 500 GiB of data regularly, configure at least that much cache:

proxy.flexfs start --diskFolder /data/cache --diskQuota 600G

Writeback tuning

For high-throughput write workloads over WAN links:

Writeback concurrency is sized automatically from the proxy host’s CPU count and needs no tuning in most deployments. It is not derived from the WAN link, so on a link shared with other traffic, shape it at the network rather than expecting the proxy to back off on its own.
Enable --sync if you need guaranteed crash durability for writes before they reach object storage.

Network considerations

Bandwidth: Writeback throughput is limited by the WAN link. Size your link based on the expected write rate.
Latency: Read latency for cached data is determined by the local network. Read latency for cache misses includes WAN round-trip time.
Firewall: The proxy server needs outbound HTTPS access to the cloud object storage endpoint. Mount clients need HTTPS access to the proxy servers, and to the object storage endpoint for the times they bypass the proxy (see RTT-based group selection). Mount clients also need access to the admin server and the metadata server, wherever those run.
DNS: Ensure proxy server hostnames resolve correctly from mount client machines.

Egress cost optimization

Proxy groups can significantly reduce cloud egress costs in hybrid deployments:

Read caching: Data read once from the cloud is cached on-prem. Subsequent reads are served locally without egress charges.
Block routing: Each block is routed to one proxy server in the group, so it is cached once rather than duplicated across servers, maximizing effective cache capacity.