Distributed Storage

Distributed storage keeps data across multiple machines while presenting a useful storage service to clients. You may see a file system, an object API, a key-value interface, or a database. Behind that interface, the system must place data, coordinate updates, detect failures, and restore protection.

The goal is not to make failure disappear. The goal is to define which failures the service can tolerate and what clients observe while those failures happen.

Why one machine stops being enough

A single machine has finite capacity, throughput, and fault tolerance. Adding machines can increase capacity and parallelism. It also creates network delays, partial failures, and copies that can temporarily disagree.

Distributed storage is useful when you need one or more of these outcomes:

• More data than one machine can hold
• More read or write traffic than one machine can serve
• Continued service after a disk, host, rack, or site fails
• Data access from many clients or locations
• Independent scaling of storage and compute

These outcomes are not automatic. They depend on the data model, placement policy, protection scheme, consistency model, and operating discipline.

The five jobs every system must do

First, the system divides data into manageable units. A file system may use blocks or chunks. An object store uses objects. A key-value store uses records or key ranges.

Second, it places those units on storage nodes. Placement may use a central metadata service, a distributed map, consistent hashing, or a calculated rule such as CRUSH.

Third, it protects data. Replication stores complete copies. Erasure coding stores data chunks plus coding chunks that can reconstruct missing data.

Fourth, it coordinates access. The system defines when a write is acknowledged, which version a read may return, and how concurrent writes are ordered or reconciled.

Fifth, it repairs protection after change. When a node fails or capacity is added, the system rebuilds or moves data. This background work competes with client traffic.

Data plane and control plane

The data plane carries client data to and from storage nodes. The control plane tracks membership, topology, placement, health, and configuration.

Some designs send client data through a central server. Others let clients contact storage nodes directly after consulting metadata. Ceph clients, for example, calculate object placement with CRUSH and send data to object storage daemons. HDFS clients ask the NameNode for block locations, then transfer data with DataNodes.

Separating these planes helps you ask a useful question: does a component coordinate data, carry data, or do both? A metadata service can be small in bytes yet critical to availability.

Partitioning and placement

Partitioning divides a data set so different nodes can own different portions. Hash partitioning can spread keyed data. Range partitioning keeps adjacent keys together. Large files can be split into fixed-size chunks.

A placement policy maps each partition or chunk to storage nodes. Good placement considers failure domains. Three copies on one host do not protect against that host failing. Copies or coding chunks must cross the host, rack, zone, or site boundaries named in the service objective.

Placement also affects balance. A poor key or uneven workload can create a hotspot, where one partition receives much more traffic or data than others.

Replication and erasure coding

Replication is direct: write multiple complete copies. It is often a good fit for frequently updated data and fast recovery, but it consumes raw capacity in proportion to the copy count.

Erasure coding divides data into data chunks and calculates coding chunks. The system can reconstruct missing chunks when enough surviving chunks remain. It usually reduces capacity overhead for large data sets. It adds computation, network work, and recovery complexity.

Neither method is a backup. Replication can copy an authorized deletion, application bug, or corrupt update. Recovery from those events requires protected history, snapshots, versioning, or backups with an independent retention policy.

Quorums and consistency

A quorum is the number of participants required for an operation or decision. Majority quorums are common in consensus systems. A five-member Raft group can continue while a majority can communicate, but it stops making progress when it loses that majority.

Storage systems expose different consistency guarantees. Strong consistency makes completed updates appear in one agreed order under the stated contract. Eventual consistency allows replicas to differ temporarily, with the expectation that they converge when updates stop and communication recovers.

The CAP result concerns behavior during a network partition. A system cannot both guarantee linearizable responses and guarantee a response from every partitioned side. This does not mean you choose only two properties for all time. You examine the exact operation, failure, and client-visible guarantee.

A write is a policy decision

Trace one write before trusting a design:

1. A client sends data to a coordinator or storage node.
2. The system determines the target replicas or chunks.
3. Participants store or journal the update.
4. The system acknowledges after its configured durability condition is met.
5. Lagging copies catch up later if the design permits it.

An acknowledgment can mean memory, a local journal, several durable copies, or a quorum. Read the service contract. The word “replicated” alone does not define when a write is safe.

Failure, recovery, and degraded states

Distributed storage assumes components will fail. Operators should distinguish three states:

• Available: the service can answer an operation under its contract.
• Durable: acknowledged data survives the failures covered by its contract.
• Fully protected: every item has the intended number of copies or chunks across the intended failure domains.

A system can remain available while under-replicated. It can also preserve data while refusing writes because it lacks a quorum. “Healthy” must therefore be translated into specific conditions and risks.

Recovery is not free. Rebuilding a failed disk reads surviving data, sends it across the network, and writes replacement data. Throttling recovery protects foreground latency but extends the period of reduced protection.

How to evaluate a system

Start with the workload and failure objectives, not a product name.

Define the interface your application needs. Measure object sizes, request rates, read and write ratios, concurrency, latency targets, and growth. State the required consistency and durability. Name the failures you must tolerate, including correlated failures such as a rack or site outage.

Then test normal load, a failed node, loss of a failure domain, network delay, network partition, capacity expansion, and rebuild. Observe both client behavior and time to restore protection. A benchmark without failures measures only the easy half of distributed storage.

Limits and good judgment

Distribution adds coordination, network dependence, operational state, and more ways to be partially healthy. It can be the wrong choice for a small workload that fits comfortably on one reliable system with tested backups.

More replicas do not fix a poor failure-domain layout. More nodes do not fix a hot partition. Stronger consistency can add coordination latency. Weaker consistency can move conflict handling into the application. Erasure coding can save capacity while increasing repair work.

Choose explicit guarantees. Test them under failure. Keep backups outside the live failure path. Monitor capacity, latency, error rate, data protection, and repair progress as separate signals.