There’s a quiet contradiction sitting at the bottom of almost every modern AI and analytics stack, and most teams have just learned to live with it. Your applications want files. Your wallet wants object storage. Those two desires don’t get along, and the gap between them has been costing companies real money and real GPU hours for years now.
At the 68th IT Press Tour in Boston this week, Paradigm4 walked us through how they’ve been closing that gap with a product called flexFS. The briefing featured Marilyn Matz, Co-founder and CEO; Gary Planthaber, CTO and the engineer who actually built the thing; and David Freund on technical sales. What follows is what stood out, why it matters, and where the rough edges still are.
The Problem Nobody Wants to Keep Paying For
Here’s the thing most IT pros already know in their bones: the workloads doing the actual work speak POSIX. AI training, HPC pipelines, analytics engines, even the newer AI agents—they all expect files, directories, and low-latency I/O. That expectation isn’t going anywhere.
Object storage, meanwhile, is the obvious economic answer at scale. It’s elastic, durable, and cheap. But it comes with high-latency metadata and file I/O, plus its own API that most software was never written to speak. As David Freund put it during the briefing, the typical AI stack “has cracks in its foundation,” and those cracks show up as GPU idle time, slow pipelines, and data scientists doing plumbing instead of science.
So companies have tried a few well-worn workarounds, and each one has a catch:
- Lift-and-shift parallel file systems like Lustre, DDN, or WEKA into the cloud—which ties your throughput directly to deployed capacity and ignores the years hyperscalers spent building good object storage.
- The copy-in, copy-out dance—pull files from object storage to local NVMe, compute, push results back. Simple, but your expensive CPUs and GPUs sit idle waiting on transfers.
- Cloud file services like EFS or FSx for Lustre—great interfaces and low latency, but you hit a cost ceiling fast, and shrinking a volume is, in Freund’s words, a “world of hurt.”
Paradigm4 didn’t set out to build a storage company. They build data and analytics tooling for life sciences, and they ran into this wall while serving customers chewing through population-scale genomics data—think UK Biobank, where licensees paid a fortune for time-limited exclusive access and needed to mine petabytes fast. Gary Planthaber’s team evaluated JuiceFS, ObjectiveFS, s3backer, EFS, FSx for Lustre, and a pile of S3-backed FUSE projects. Nothing hit all four needs—POSIX correctness, high throughput, low operational overhead, and a price their regulated customers could stomach. So they built flexFS.
What flexFS Actually Is
At its core, flexFS is an object-native parallel file system. Rather than bolting object storage on as a cold tier, it treats the object store as dumb block storage and adds the file-system smarts on top. The architecture has three moving parts worth understanding.
Chunks, Not One-to-One Files
When you write a file, flexFS breaks it into chunks, gives each chunk an object ID, and stores those objects in the back end—letting the hyperscaler handle parallel access. This is a one-to-many mapping, not one file per object, and that detail matters. It increases parallelism on reads and makes byte-range I/O far more efficient, because you’re not plowing through giant monolithic objects to grab a small slice.
The Metadata Server
High-latency metadata was one of the two big object-storage weaknesses. flexFS moves metadata into its own persistent, low-latency metadata service. There’s no sharding of a single volume today; instead, each metadata server owns one or more volumes (or runs in an HA replica configuration for durability). You can run as many metadata servers as you need, each handling a subset of namespaces. Under the hood, Planthaber’s team built the metadata engine on PebbleDB—the same key-value store behind CockroachDB—chosen specifically for MVCC and performance. The rest is written in Go.
The Proxy Group (The Optional Cache)
This is the piece that’s easy to misread. The proxy group is a write-back cache, and Paradigm4 deliberately designed it to behave like a CDN—mount clients automatically pick the best-performing proxy for a given volume. A few things make it genuinely clever:
- It’s optional and fractional. Big sequential files where latency doesn’t matter? Go direct to the object store. Lots of small random-I/O files on the same volume? Tell the proxy to cache, say, the first 100 blocks of each file and go direct beyond that. The mount clients know the rules without chattering across the network to figure out where data lives.
- Caching is multi-level. Both the compute client and the proxy servers use RAM plus directly attached NVMe.
- Most customers don’t even use it. Planthaber was candid: most deployments—including Paradigm4’s own—run fine without proxies. They get pulled in for small-file-heavy workloads where they can cut latency by 5x to 10x.
One nuance to keep straight: the proxies act like a CDN but aren’t one. You could put an actual CDN like CloudFront in front of flexFS if you wanted, but that only helps reads.
Let’s Talk About the Money, Because That’s the Headline
You can describe architecture all day, but the briefing’s most convincing material was a single customer story with receipts. A top-5 global biopharma company moved its Research Data Commons—a global repository for clinical and research data—onto flexFS backed by S3, replacing a combination of AWS S3, EFS, EBS, and FSx for Lustre.
The numbers, covering September 2022 through March 2026 (43 months) at 1.14 PB and over 160 million files:
- $2.53M on flexFS + S3 versus $5.65M for the provisioned AWS alternative—roughly $3.13M in cumulative savings, about 55% of the AWS cost avoided.
- $1.44M saved in 2025 alone, or 59% of the competing AWS spend.
- By March 2026, the entire flexFS + S3 bill ran $110K/month versus $274K/month for the AWS equivalent.
And here’s the line that genuinely landed in the room: at current scale, the whole flexFS + S3 bill ($110K/month) is less than the competing EFS storage component alone ($141K/month).
There’s a counterintuitive twist worth pausing on. You’d expect storage costs to grow linearly with data. With flexFS they get cheaper per terabyte as you scale, because the pricing rides the slope of object storage rather than the steeper slopes of managed file services. Their effective rate fell from about $90/TB-month at 25 TB in 2022 to roughly $66/TB-month at 1.14 PB in 2026. EFS Standard, by contrast, stays flat around $307/TB-month and FSx around $174/TB-month. As Freund put it, the more you store, the more you save—a sentence that shouldn’t make sense for storage, and yet here we are.
A big chunk of the savings comes from elasticity. FSx for Lustre provisions capacity in 2.4 TiB increments and historically couldn’t shrink—that inelasticity wasted an estimated $332K over the 43 months. flexFS grows and shrinks automatically, and you pay for the bytes you actually store. Literally the bytes: no rounding up to the next block, no charging for metadata overhead.
The Features That Aren’t About Price
Cost gets the attention, but several capabilities matter just as much for day-to-day operations and risk.
Built-In Time Travel
Because flexFS never overwrites an object—it allocates a new one and remembers the old chunk and its metadata for a settable retention period—you get continuous, point-in-time recovery at no added cost. Storage veterans will recognize this as a redirect-on-write scheme, essentially continuous data protection. You can mount a read-only copy of a volume at any past moment, examine or recover data, then dismiss it, all without stopping production. In an era where a rogue AI agent can wipe a database and then cheerfully admit it ignored instructions, the ability to simply unwind what happened is a genuinely useful insurance policy.
Non-Disruptive Updates
Mount clients update in place using a FUSE session handoff—no unmount, no interruption, no data loss. Server-side updates pause I/O for under a second. For shops that have lost 24 to 48 hours of scientist time reprovisioning a Lustre volume, that’s a meaningful difference.
The Rest of the Toolbox
flexFS handles a lot of housekeeping itself rather than leaning on the back end:
- Deduplication runs as a manual (schedulable) utility that verifies duplicates by checksum and byte-for-byte comparison before replacing them with hard links—and it’ll tell you how much you’d reclaim before you commit.
- An optimized find utility queries the metadata server directly instead of traversing the mounted volume, returning results orders of magnitude faster on volumes with hundreds of millions of files.
- Client-side compression (LZ4 by default, plus Zstandard and Snappy, or off entirely) happens on the compute nodes, so it scales out and shrinks the packets crossing the network—sometimes improving throughput.
- End-to-end encryption with keys held on the compute node, opaque to Paradigm4, the hyperscaler, and anything in the network path. Plus a native Kubernetes CSI driver with a Helm chart, Linux advisory locking, and extended POSIX ACLs that work uniformly across HPC, AWS Batch, and Databricks.
Worth noting on durability and redundancy: flexFS leans on the object store’s own erasure coding and bucket policies rather than reinventing them. It can target any S3-compatible object store or key-value store, on-prem or across the four major clouds, in single-region, multi-region, multi-cloud, hybrid, or converged configurations. That converged setup—storage services co-resident on compute nodes—was validated jointly with Oracle on OCI and reached near-local-NVMe performance over networked object storage.
Where flexFS Is Headed: Lakehouses, Databases, and Agents
Paradigm4 sees flexFS reaching well beyond life sciences, and they brought benchmarks to make the case.
Data Lakehouse Acceleration
Spark and Presto-style engines querying Parquet on object storage hit a “metadata tax” and small-file congestion. flexFS sits between the engine and the object store, dropping query planning from 30–60 seconds to under a second. On a TPC-H run at scale factor 100, Spark with Gluten on flexFS (cached) finished all 22 queries in about 176 seconds versus 1,191 seconds for plain Spark on S3—close to a 7x gain.
There’s an honest wrinkle here, and the team didn’t hide it. Spark with Comet ran worse on non-cached flexFS than on S3, because Comet does a lot of serialization and many of these engines optimized their code paths for S3 first, treating POSIX as an afterthought. With caching enabled, flexFS more than recovered. The takeaway Freund offered: the proxy tier isn’t just a cache, it’s a compatibility and optimization layer that can soften an engine’s less favorable I/O behavior.
Coupled-Architecture DBMS Modernization
MPP data warehouses, graph, and vector databases that depend on direct-attached disks and strict POSIX can move to elastic object storage via flexFS—decoupling compute from storage growth, cutting TCO by as much as 60% with no code changes, and gaining zero-copy snapshots through that time-travel metadata mapping.
AI/ML Training and Agentic Workspaces
For training, flexFS reports a 2x speedup over S3 direct even without a cache, optimized byte-range reads for GPU efficiency, and instant checkpointing that lets GPU clusters resume in seconds rather than minutes. For agents, it offers a POSIX scratchpad with persistence, byte-range access so agents read only the relevant slice of a 500MB PDF instead of the whole thing, and the point-in-time recovery that matters when an autonomous agent goes off the rails.
A Question They’re Asking the Industry: The “File Lakehouse”
Paradigm4 used part of the briefing to float a category they’re considering: the File Lakehouse, sitting alongside the familiar data lakehouse and coupled-architecture DBMS spaces, with flexFS as the common substrate beneath all three. The idea covers unstructured data workloads—AI training and inference, agent workspaces, images, PDFs, video—where POSIX and byte-range I/O matter more than SQL over Parquet.
It’s an open question, and the team genuinely wanted feedback. One sharp suggestion from the room: if you’re serious about agentic AI, the metadata server probably needs graph and vector capabilities so it can reason about the meaning of data, not just file attributes. Freund took the point—the picture looks incomplete without it. Honestly, that’s a fair critique, and it’s a more useful one than most vendor Q&A produces.
The Honest Bottom Line
A few realities keep this grounded. flexFS is a small product from a roughly 12-person floating team inside Paradigm4, born out of their REVEAL life-sciences stack. There are around a dozen production deployments today, some running over three years, including some very large named accounts. They have no pure on-prem customers yet (a couple POCs are scheduled), and frankly flexFS offers little advantage on-prem over the mature open-source file systems already built for that world—its real edge is cloud economics and the cloud’s inelastic file-service pricing.
The product is mature where it counts: installation typically under an hour, most customers need only a single server, it’s a drop-in replacement for EFS, FSx for Lustre, OCI File Storage, GC Filestore, and Azure Files, and there’s a free Community Edition giving you the full system (minus proxy caching) against your own S3 bucket up to 5 TB.
For data center teams staring at a cloud storage bill that climbs faster than their actual data, the pitch is refreshingly concrete: keep the POSIX semantics your software already speaks, keep the object-storage economics your CFO already likes, and stop paying the tax in between. The biopharma numbers suggest that’s not just marketing—it’s a line item that went down. And in this market, a storage story where the cost curve bends the right way is worth a serious look.
flexFS is available to try at docs.flexfs.io. Paradigm4 is actively building out channel and SI partnerships and exploring OEM relationships across the data lakehouse and DBMS spaces.
##
