Advantages

What primarily sets flexFS apart from traditional parallel file systems is its ability to directly target object storage as a native block storage. Object storage systems typically have thousands of always-on servers, high aggregate throughput, low usage costs, and high durability. Since flexFS mount clients can load / store file data blocks directly to an object storage service, without passing all I/O operations through a storage service, it is possible for flexFS to have as much aggregate throughput as the underlying object storage service itself.

A potential drawback of this direct throughput optimized mode, however, is that object storage services tend to have relatively high latency. To help mitigate that latency when reading and writing small files, flexFS has an optional ability to route only the first n blocks (where n is configurable) through intermediate writeback caching clusters called proxy groups. The end result is a file system with high throughput and durability that requires minimal infrastructure. And with minimal infrastructure comes lower TCO and operational overhead. Meanwhile, there is also a great deal of flexibility in tuning the latency versus throughput characteristics of flexFS.

Traditional parallel file system

Traditional parallel file systems were originally designed and built prior to the popularization of public clouds, and their mount clients are not capable of communicating directly with object storage. In general, all I/O operations are routed to a storage service (usually a cluster of instances with ephemeral NVMe drives). Such a service is costly to run / maintain, and can easily result in a throughput-limiting bottleneck.

flexFS file system

The flexFS file system, however, was built from the ground-up to be able to directly leverage bulk storage services such as object storage. Any unfavorable latency characteristics of those bulk storage services can be mitigated by making use of proxy groups that function as writeback caches for only a configurable fraction of the file data blocks.

Typical object storage workflow

Workflows that make use of files in object storage, but need to interact with those file using POSIX utilities tend to follow an inefficient pattern of downloading the full object to local storage (typically a block storage device), performing some processing, and then uploading results back to object storage. With this pattern comes a variety of risks that increase engineering complexity and can also increase computing time and associated costs.

flexFS workflow

Compared with the typical object storage, the flexFS workflow is much easier to engineer, and often results in lower total I/O wait times due to efficient flexFS proficiency in block streaming and compression. This allows tasks the complete faster and can yield dramatic computing cost savings.