Hybrid Deployments

Note

Proxy groups are an Enterprise feature. Community edition users can use local on-disk writeback caching (--diskWriteback) on the mount client to mitigate latency for individual clients.

One of the most powerful use cases for proxy groups is bridging on-premises compute infrastructure with cloud-based object storage. Proxy servers deployed on-prem serve as writeback caches, masking the higher latency of cross-network writes and providing a local read cache for frequently accessed data.

Architecture

In a hybrid deployment, the typical topology is:

1. Cloud: Object storage bucket (S3, GCS, Azure Blob, or OCI) + admin server + metadata server.
2. On-premises: One or more proxy servers + mount clients on compute nodes.

The proxy group running on-prem is associated with the volume via the admin server. Mount clients on-prem discover the local proxy group, measure its RTT (which will be low since it is on the same LAN), and route all block traffic through it.

Write path

1. Mount client writes a block to the on-prem proxy server over the local network.
2. Proxy server writes the block to its local disk cache and acknowledges the client immediately (writeback).
3. The proxy server asynchronously flushes the block to cloud object storage over the WAN link.

This decouples write latency from WAN latency. The mount client experiences local-network write speeds, while the proxy server handles the slower WAN transfer in the background.

Read path

1. Mount client requests a block from the on-prem proxy server.
2. If the block is in the local cache, it is served immediately.
3. If not, the proxy server fetches it from cloud object storage, caches it locally, and returns it.

Subsequent reads of the same block (by the same client or any other client in the proxy group) are served from the local cache.

Deployment steps

1. Deploy proxy servers on-prem

Install proxy.flexfs on one or more on-prem machines with fast local storage (SSD or NVMe):

# Initialize credentials with cloud storage keys
proxy.flexfs init creds \
  --blockUser $USERNAME \
  --blockPass $PASSWORD

# Create systemd service
sudo proxy.flexfs init systemd --now \
  --startFlags "--diskFolder /data/proxy-cache --diskQuota 1T"

2. Create a proxy group

On a machine with access to the admin server, create a proxy group containing the on-prem proxy server addresses:

configure.flexfs create proxy-group \
  --providerCode onprem --regionCode dc1 \
  --addresses proxy1.internal:443,proxy2.internal:443

3. Associate the proxy group with your volume

configure.flexfs create volume-proxy-group \
  --volumeID my-volume \
  --proxyGroupID 1

4. Mount on compute nodes

On-prem compute nodes mount normally. The mount client will discover the on-prem proxy group and route traffic through it:

mount.flexfs start my-volume /mnt/data

Multi-region hybrid

For organizations with multiple data centers or regions, deploy proxy groups in each location:

# Proxy group in US data center
configure.flexfs create proxy-group \
  --providerCode onprem --regionCode us-dc \
  --addresses us-proxy1.internal:443,us-proxy2.internal:443

# Proxy group in EU data center
configure.flexfs create proxy-group \
  --providerCode onprem --regionCode eu-dc \
  --addresses eu-proxy1.internal:443

# Associate both with the volume
configure.flexfs create volume-proxy-group \
  --volumeID shared-data \
  --proxyGroupID 1

configure.flexfs create volume-proxy-group \
  --volumeID shared-data \
  --proxyGroupID 2

Mount clients in each location automatically select the nearest proxy group based on RTT.

Tuning for hybrid workloads

Cache sizing

Size the on-prem proxy cache to hold the active working set. If your workload accesses 500 GiB of data regularly, configure at least that much cache:

proxy.flexfs start --diskFolder /data/cache --diskQuota 600G

Writeback tuning

For high-throughput write workloads over WAN links:

• Increase --writebackActive if you have sufficient WAN bandwidth and want faster flush times.
• Use --writebackDelay to throttle writeback if the WAN link is shared with other traffic.
• Enable --sync if you need guaranteed crash durability for writes before they reach object storage.

Network considerations

• Bandwidth: Writeback throughput is limited by the WAN link. Size your link based on the expected write rate.
• Latency: Read latency for cached data is determined by the local network. Read latency for cache misses includes WAN round-trip time.
• Firewall: The proxy server needs outbound HTTPS access to the cloud object storage endpoint. Mount clients need HTTPS access to the proxy servers.
• DNS: Ensure proxy server hostnames resolve correctly from mount client machines.

Egress cost optimization

Proxy groups can significantly reduce cloud egress costs in hybrid deployments:

• Read caching: Data read once from the cloud is cached on-prem. Subsequent reads are served locally without egress charges.
• Writeback batching: Writes are batched and flushed asynchronously, allowing the proxy server to optimize network utilization.
• Rendezvous hashing: Ensures each block is cached exactly once in the proxy group (not duplicated across servers), maximizing effective cache capacity.