CubeFS

CubeFS is a distributed storage system for unstructured data. It presents the same storage platform through file and object interfaces, including POSIX, HDFS, and S3-compatible access.

The central mental model is one logical volume, split metadata and data services, several access paths.

applications
   |-- POSIX through a CubeFS client
   |-- HDFS-compatible access
   `-- S3-compatible access through ObjectNode
                    |
                 volume
          +---------+---------+
          |                   |
   metadata partitions   data partitions or blobs
       on MetaNodes       on DataNodes or BlobNodes
          \                   /
           `---- Masters ----'
              coordinate

CubeFS exists for teams that need shared storage across many machines and applications. It separates file metadata from file data, partitions both, and distributes those partitions across a cluster. You can expand the relevant subsystem as demand grows.

This is infrastructure, not a hosted storage account. Your team deploys the components, supplies disks and network capacity, monitors them, and plans failure recovery.

What a volume means

A volume is the unit users work with. Through a file client, it looks like a file system that can be mounted by multiple clients. Through the object gateway, the same volume corresponds to a bucket.

That shared backing does not make POSIX and object semantics identical. The object gateway converts bucket and key operations into volume and path operations. A key such as a/b.txt maps to a file path. Some object names that are legal in a pure key-value namespace can conflict after this conversion. Treat cross-protocol use as a design decision and test the exact naming and update patterns you need.

The control and storage components

Master

Master nodes manage cluster resources and volume information. They track DataNode and MetaNode health, allocate partitions, and coordinate background work. Multiple Masters use Raft for consistent control metadata and persist that metadata in RocksDB.

The Master is the control point, not the normal path for every file byte. Clients cache topology information and communicate with the relevant metadata and data services.

MetaNode

MetaNodes store file-system metadata. CubeFS divides a volume's metadata into meta partitions. Each partition covers a range of inode identifiers and contains inode and directory-entry indexes.

Metadata partitions use Multi-Raft replication. They can split as the namespace grows. This design avoids placing the entire namespace behind one metadata server.

DataNode and the replica subsystem

DataNodes store file data in data partitions. Replicas of a data partition form a replica group. CubeFS uses this subsystem for replicated storage and can also use it as a cache in front of erasure-coded data.

Replication keeps complete copies. It is straightforward to read and repair, but every additional copy consumes more raw capacity.

BlobStore and the erasure-code subsystem

BlobStore is CubeFS's erasure-coded storage subsystem. Its main components include Access, BlobNode, ClusterManager, Proxy, and Scheduler.

Erasure coding splits data into data shards and calculates parity shards. It can use less raw capacity than keeping several complete replicas. The tradeoff is a larger component set and extra work during encoding, repair, and some degraded reads.

CubeFS can run the replica and erasure-code subsystems together or independently. Choose from workload behavior, recovery needs, latency, capacity cost, and your team's ability to operate each design.

ObjectNode

ObjectNode is a stateless S3-compatible gateway. It obtains a volume view from the Master, then works directly with metadata and data services. You can scale gateway capacity separately from storage capacity.

ObjectNode maps a volume to a bucket and a path to an object key. It gives S3 clients an access path into CubeFS; it does not turn every CubeFS behavior into Amazon S3 behavior.

Client

The CubeFS client commonly uses FUSE to mount a volume. It caches topology and file metadata to reduce control-plane requests. That cache affects the consistency model: official client design guidance permits several clients to read the same file, but warns against several clients writing the same file at the same time.

Where CubeFS fits

CubeFS can serve several storage patterns:

• shared persistent file storage for containers;
• data used by big-data tools through HDFS-compatible access;
• training data and model artifacts for machine-learning systems;
• S3-compatible object access to unstructured data;
• storage and compute separation for selected databases and middleware;
• mixed hot and colder data when replica, erasure-code, and cache choices fit the workload.

Kubernetes integration does not remove the storage cluster. CubeFS components still need stable resources, disks, networking, and failure-domain planning. A CSI driver can connect Kubernetes workloads to CubeFS volumes, while the storage system remains a separate operational responsibility.

How to evaluate it

Start with the access contract. Decide whether applications need mounted files, HDFS compatibility, S3 compatibility, or controlled access through more than one protocol.

Then examine the workload:

1. Estimate namespace size, file-size distribution, read and write patterns, and concurrency.
2. Choose replicated or erasure-coded storage from latency, capacity, and recovery goals.
3. Map Masters, MetaNodes, DataNodes, and optional BlobStore components across failure domains.
4. Test client caching and concurrent access with representative applications.
5. Measure normal operation and degraded operation. A healthy empty cluster proves little about repair load or tail latency.
6. Plan monitoring, upgrades, capacity expansion, disk replacement, authentication, and backup before production use.

Limits and tradeoffs

CubeFS has more moving parts than a local file system or a managed object-storage service. Its split architecture creates independent scaling options, but it also creates more component boundaries to monitor.

Protocol compatibility has boundaries. The documentation says CubeFS relaxes some POSIX consistency requirements for performance. Its S3-compatible gateway converts object names into paths, which introduces name conflicts that pure object stores do not have. The client metadata cache also constrains safe concurrent writes to one file.

Erasure coding saves capacity only by adding encoding, repair, and operational complexity. Replication is not free either; it consumes full copies. Neither engine is the automatic answer for every workload.

Use CubeFS when its shared, scalable, multi-protocol storage model solves a concrete problem and your team can own the cluster lifecycle. Prefer a simpler file system, a managed service, or a single-protocol store when those options satisfy the requirement with less operational risk.