Object storage was supposed to be the easy part. You toss your data into a bucket, forget about it, and trust Amazon’s many nines of durability to keep it safe forever. For years that worked fine. Then AI training jobs showed up with petabytes of files, and suddenly the limits of that “giant digital attic” started to matter again.
At the 68th IT Press Tour in Boston last week, Joe Zhou, Developer Advocate at Juicedata, walked through why his company built JuiceFS, a POSIX-compatible distributed file system that sits on top of object storage. His talk doubled as a comparison against Amazon’s own answer to the same problem, S3 Files, which launched in April. The short version? Both tools solve a real pain. Only one of them, in Zhou’s telling, solves all of it.
Let me walk through what he covered, because there’s a lot here for anyone running storage behind AI or data-heavy applications.
Why Object Storage Became the New Backend
Zhou opened with a confession that probably sounds familiar to a lot of engineers. Before joining Juicedata, he was a backend developer who treated object storage as a black box. Put metadata in a database, dump the actual files into S3, and never think about it again. S3 was one of the first AWS services ever shipped, so it’s been a cornerstone of cloud architecture from the start.
What changed is that object storage stopped being just a dumping ground and started becoming the foundation other systems are built on. Vector databases, SQL-compatible databases, message queues, they’re all moving onto buckets. Zhou rattled off real names that should ring bells for VMblog readers:
- LanceDB and Turbopuffer, vector databases that store their indexes directly on object storage. Notion uses Turbopuffer, and Zhou pointed out it’s part of why Notion could drop the price tag on its AI assistant for new users.
- Neon, the Postgres-compatible database that Databricks acquired, which keeps its storage on object stores.
- WarpStream, a Kafka-compatible streaming service that does the same.
The appeal makes sense once you list what object storage gives you. The API is simple but powerful, the namespace is a flat and effectively infinite map, durability is extreme, and the cost runs around two cents per gigabyte per month in popular regions. Add multi-region replication, immutability, and versioning, and you can see why people keep building on it.
The CAS Trick, and Where It Breaks
Here’s the part I found genuinely clever. Zhou explained how engineers build coordination systems on object storage using a single primitive called compare-and-swap, or CAS. Imagine a queue stored as one JSON object. A worker wants to append an item, so it reads the object and writes it back with a condition: only commit if the version I read is still current. If two workers race, one wins and the other re-reads and retries. The workers stay stateless. The object store handles the atomicity.
It works, but it’s a workaround, and Zhou was honest about that. You can’t let the queue object grow too large, because you’re repeatedly reading and rewriting the whole thing. Google Cloud Storage caps CAS at one operation per second, so high concurrency is out. This is the recurring theme of his talk: object storage primitives are strong, but you’re constantly designing around their limits.
What Object Storage Can’t Do
So why bolt a file system on top of a bucket? Because object storage has a list of stubborn shortcomings that don’t go away no matter how clever your CAS logic gets. Zhou laid them out plainly.
- No in-place updates. Objects are immutable. Want to change a few bytes? You rewrite the entire object as a new version.
- No real hierarchy. Prefixes look like folders but they’re just long indexed keys. There’s no actual directory tree.
- Costly metadata operations and higher latency. Batch work is expensive, and you’re always reaching across a network rather than a local disk.
There’s also no direct execution. If you have a SQLite file sitting in a bucket and want to run a query against it in place, you can’t. A file system layer is what gives you POSIX, in-place edits with something as ordinary as sed, a rename with mv, low latency through caching, and the ability to operate on structured data directly. That’s the whole reason these layers exist.
Amazon’s Answer: S3 Files
Amazon clearly saw the same problem, probably pushed along by the rise of AI agents that expect a file system to work with. In April it released S3 Files, a POSIX-compatible distributed file system built on two existing, mature services: EFS as a high-performance caching tier and S3 as the source of truth backing storage.
The idea is tidy. You mount a bucket as a volume. EFS handles caching and metadata, S3 holds the durable data, and the system keeps a strict one-to-one mapping between files and objects. EFS is roughly ten times more expensive than S3, so the design offloads as much as it can down to the cheaper tier. You control the behavior with a few parameters: a prefix for what to sync, a trigger for when to sync, and a sizeLessThan value that defaults to 128 KiB.
That default tells you something. S3 Files is tuned for small files. Zhou’s read was that this fits classic machine learning datasets well, think the tiny handwritten-digit images everyone trained on early in their ML journey. For other workloads, the tradeoffs get sharper.
The Catch: Write Amplification and Costly Renames
Zhou gave a vivid example, and it’s worth sitting with because it explains a lot. Say you have a 2 GB video sitting in S3 and you want to add a few bytes to the end, like a signature. Ideally you’d just append. Instead, because of the one-to-one file-to-object mapping, your application server has to pull the entire 2 GB object, append your bytes, and write the whole thing back. Two gigabytes of transfer to add two kilobytes. And the operation isn’t atomic, so anything can happen in between.
Renames are the other sore spot. In a file system, mv /foo /bar is an instant metadata change. On object storage there’s no real directory, so renaming a prefix means rewriting every object underneath it. Move a folder with a million files and you’ve triggered a million background object rewrites.
Then there’s pricing, which Zhou described as genuinely hard to reason about. A write lands in the EFS tier first, so you’re charged for the write and for EFS storage while it sits there. After about sixty seconds it gets aggregated and synced to S3, which means a charge to read it back out of EFS, plus S3 storage and access costs. As he put it, price-oriented development on these services can be a headache, even when the engineering is solid.
So does S3 Files solve the problem? Zhou’s scorecard was fair. It nails POSIX support and file hierarchy, and the caching tier does speed things up. But in-place updates, batch metadata operations, application execution, and direct work on structured data all run into the file-size and object-rewrite limits. A decent approach, he called it. Decent, not complete.
JuiceFS: A Different Architecture
This is where Zhou pivoted to his own product, and the design difference is the whole story. He used a Netflix system-design analogy that lands well. When you stream a video, a request first hits a metadata engine to locate the file, then goes to backing storage to fetch the bytes. JuiceFS works the same way, with metadata and data fully separated rather than stacked on object storage like S3 Files does.
JuiceFS has three pieces:
- A metadata engine. In the Community Edition you bring your own database, Redis, PostgreSQL, MySQL, even SQLite. The Enterprise Edition ships a purpose-built engine tuned only for file system tree structures, with a Raft-style consensus protocol for high availability and disaster recovery.
- Object storage of your choice, any mainstream cloud or on-prem store, MinIO, Ceph, and the like.
- A client that runs on your application servers and exposes the volume, typically through FUSE, with support for HDFS and even an S3 gateway.
One point Zhou hammered, because he gets asked constantly: JuiceFS does not relay your data traffic. The client talks to your object store directly, so you pay data transfer costs only to whichever object store you picked. There’s no extra transfer charge and no per-client license fee. The Enterprise pricing is based on capacity, layered on top of what you already pay your object store provider.
Chunking Is the Secret Sauce
Unlike S3 Files, JuiceFS does not keep a one-to-one file-to-object mapping. It splits files into chunks, with a configurable maximum block size (4 MB by default, fixed per volume). That single design choice fixes the problems Zhou spent the first half of the talk describing.
Go back to the 2 GB video and its appended signature. Because the file is already split into many chunks, an append only touches the last block. Change something in the middle and you rewrite one or a few blocks, not the whole file. The chunks themselves are still immutable, since the object store demands it, but you’re never rewriting gigabytes to change kilobytes.
Renames work the same clean way. Because metadata is separated from data, mv /foo /bar updates the metadata engine instantly, and the chunk pointers still point where they always did. No background rewrite of a million objects. Just a pointer change.
Cost Tiering and Scale
A newer Community Edition feature lets you map JuiceFS storage tiers to different object storage classes. Hot data on Standard, warmer data on Intelligent-Tiering, archives on Glacier Instant Retrieval, all configurable down to individual directories or files. Zhou noted this one came from community feedback, got built first in the open-source edition, and is now being promoted up into Enterprise, which is a nice inversion of the usual flow.
On scale, the numbers got attention. Community Edition handles hundreds of millions of files comfortably. Enterprise pushed its soft limit from 100 billion files a year or so ago to 500 billion files in a single volume after a round of optimization, with room to grow as customers ask for more.
The Mirror File System and Who’s Using It
The feature Zhou seemed most fond of is the Mirror File System, and it’s the one Amazon can’t easily match because AWS is a single cloud. JuiceFS can mirror a file system across regions or even across clouds, keeping one source region as the source of truth.
There are two flavors. In the cache-only mirror, the mirror region holds a distributed cache and pulls what it needs, so you don’t replicate the entire object store. In the full mirror, you replicate the objects too, giving the mirror region a complete local copy. Which one you pick depends on whether your bigger worry is storage cost or traffic cost.
The real-world example was MiniMax, the AI company whose models show up alongside the big names in the rankings. MiniMax runs a hybrid-cloud setup, keeping GPU-heavy critical workloads in its own data center for cost reasons and pushing other components to the cloud. It uses cache-only mirrors today, with one source region feeding around ten mirror regions placed close to expensive GPU resources, and it’s already weighing a move to full mirrors.
How Customers Actually Use It
The adopter stories were the most fun part of the briefing, partly because they show creativity that goes beyond the marketing slides. JuiceFS competes directly with parallel file systems like Lustre and BeeGFS, and Zhou was candid that a proof of concept usually decides the winner.
A few examples stuck with me. A company called Zilliz-adjacent shop built lightweight containers for running AI agents, each needing a persistent volume that survives a container restart. They embedded a SQLite-backed JuiceFS into each container and paired it with Litestream, an open-source project that streams the SQLite file out to object storage. Restart the container, restore the SQLite, and the agent’s file system is right back. A fintech insurer ran ClickHouse on JuiceFS to make a single instance stateless. Xiaomi and MiniMax use the Community Edition to unify their data foundations.
Zhou also addressed the migration question directly, which matters for anyone with a full bucket already. If you point JuiceFS at an existing object store and do nothing, it can’t read those files, because they aren’t in JuiceFS’s chunked format. But a single import command loads the metadata and gives you a read-only, in-place view of the original data. Read from it freely; a write creates a new copy in JuiceFS format rather than touching the original. That’s a meaningful difference from S3 Files, and a fair thing to know before you plan a rollout.
The Business Behind the Software
A detail that says a lot about Juicedata’s posture: the company is profitable, runs around 30 to 35 people with most of the team in China, and according to Zhou, doesn’t need to raise money. CEO Davis Liu is playing a long game, building stable storage software meant to keep working for years rather than chasing aggressive growth. Zhou himself is the company’s first marketing hire in North America, so the go-to-market motion is still taking shape, with early OEM and reseller conversations happening in places like Sri Lanka and India.
Go-to-market runs two ways. There’s a managed cloud service built on the Enterprise solution, which gets you started about as easily as S3 Files. And there’s on-prem Enterprise deployment for companies like MiniMax that want everything in their own data center, with metrics collection scaled to whatever the customer’s security posture allows.
Why “For AI” Isn’t Just a Label
Zhou caught himself near the end, realizing he hadn’t fully explained the “for AI” part of the tagline. The origin story helps. JuiceFS was conceived to replace things like HDFS and build a unified foundation for storage and training. Then every company started slapping AI on everything, and for JuiceFS that label actually fit.
The reason is practical. A toy MNIST model trains fine on a laptop. Real training runs against petabyte-scale data, and researchers shouldn’t have to rewrite their PyTorch or framework code to talk to object storage APIs when their tools already expect POSIX. JuiceFS lets those AI workloads run in distributed mode against a file system that behaves like a file system, while the bytes live on cheap, durable object storage underneath.
The Takeaway for VMblog Readers
Here’s the honest assessment, the same one Zhou gave about his own product. There’s no silver bullet, and JuiceFS is a distributed system with the operational weight that implies. If you npm install against a default JuiceFS volume, it’ll feel slow until you tune it, because that’s a different workload than streaming large training files. As one journalist in the room noted, plenty of legacy commercial applications still demand a file protocol and can’t be changed, and that’s exactly the gap a file system on object storage fills.
What JuiceFS gets right is the architecture. By separating metadata from data and chunking files instead of mapping them one-to-one, it sidesteps the write amplification, costly renames, and rewrite penalties that limit a one-to-one approach like S3 Files. Add cross-cloud mirroring, a self-funded company with a long-horizon mindset, and a measured set of feature claims, and you have a storage layer worth a serious proof of concept if you’re running AI or data-heavy workloads on object storage.
If you want to look closer, the project lives at juicefs.com and the open-source code is at github.com/juicedata/juicefs. For a team trying to get a real file system out of a bucket without paying the object-rewrite tax, it’s a name worth knowing.
##
