TL;DR — CubeFS is a CNCF Incubating distributed storage system that serves both POSIX and S3 APIs from the same cluster. It separates metadata, data, and object subsystems, supports erasure coding and multi-AZ replication, and is designed for large-scale AI/ML workloads — billions of files, petabytes of data, high-throughput checkpoint and training I/O.
What it is
CubeFS (formerly ChubaoFS) is an open-source, cloud-native distributed file system and object store. It provides concurrent access via POSIX, HDFS, and S3-compatible interfaces from a single storage cluster. It's a CNCF Incubating project, production-proven at JD.com, OPPO, and NetEase at multi-petabyte scale. In the AI Native landscape it's in AI Native Infra › Storage.
Why it exists
AI workloads need both: POSIX for training (DataLoader, random reads) and S3 for checkpoints, model artifacts, and data lakes. Running two separate systems doubles ops burden. CubeFS serves both interfaces from one cluster — write a checkpoint via S3, read training data via POSIX mount, same underlying storage.
Fig 1 — One CubeFS cluster serves POSIX, S3, and HDFS interfaces simultaneously.
How it works
CubeFS has three core subsystems: a metadata subsystem (distributed, Raft-replicated metadata partitions), a data subsystem (data nodes holding replicas or erasure-coded chunks), and an object subsystem (S3-compatible gateway). The FUSE client and S3 gateway both hit the same metadata and data nodes, so data is shared across interfaces. Volumes can use replication (3-copy) or erasure coding (lower storage overhead).
Key features
- Multi-protocol — POSIX (FUSE), S3, HDFS from one cluster; no data duplication.
- Erasure coding — storage-efficient data protection with configurable EC policies.
- Multi-AZ — cross-zone replication for disaster recovery.
- Multi-tenancy — volume-level isolation with QoS (IOPS/bandwidth limits).
- Kubernetes CSI — mount CubeFS volumes as PVs in pods.
- Scale — production-proven at billions of files, petabytes of data.
Quick start
Deploy a cluster with Docker or Kubernetes, then mount:
# start a minimal cluster (docker-compose for dev)
docker-compose -f docker/docker-compose.yml up -d
# mount via FUSE
cfs-client -c /etc/cubefs/fuse.json
# now accessible at /mnt/cubefsOn Kubernetes, use the CSI driver to create PVs backed by CubeFS volumes.
When to use, when to skip
Use it when you need a single storage system that serves both file and object access — training jobs reading via POSIX and checkpointing via S3, multi-tenant GPU clusters, or when you want one system instead of separate NFS + S3. Also when you need erasure coding and multi-AZ at the storage layer.
Skip it if you only need object storage (use MinIO) or only need a cached POSIX layer on top of existing S3 (use JuiceFS). CubeFS is a full storage system — more to operate than a caching layer.
vs / alongside
| Tool | Approach | Note |
|---|---|---|
| CubeFS | Full distributed FS + S3 | One cluster, multi-protocol |
| JuiceFS | POSIX FS over object store | Lighter, needs external metadata + object store |
| MinIO | S3 object store only | No POSIX, simpler |
| Alluxio | Caching layer | Overlay, not a storage system |
References
- CubeFS documentation — official docs.
- cubefs/cubefs — source (CNCF Incubating).
- CNCF project page — maturity and landscape.
Extra reads
- CubeFS blog — architecture deep dives.
- CubeFS on Kubernetes — CSI and Helm deployment.
Verified against CubeFS docs (cubefs.io), May 2026.