Ceph Fundamentals

Ceph turns a group of servers and storage devices into one distributed storage system. It can present block devices, a shared file system, and object APIs from the same underlying cluster.

That range is useful, but it can hide the central idea. Ceph stores data as objects in RADOS, its distributed object store. The block, file, and object services translate client requests into RADOS operations.

Use this mental model:

applications
    -> RBD block devices | CephFS files | RGW object APIs
        -> RADOS pools and placement groups
            -> OSD daemons and storage devices

Ceph is not a storage appliance with one controller. Clients obtain a cluster map from monitors. They use that map and the CRUSH algorithm to calculate where data belongs. They then communicate with the responsible OSD daemons.

Why Ceph exists

A single storage server has finite capacity and creates a failure boundary. Traditional shared-storage designs often place controllers in the data path. Those controllers must handle every request and remain available.

Ceph distributes storage and placement work across the cluster. You add capacity by adding OSDs and hosts. CRUSH changes the placement calculation when the topology changes, so the cluster can move affected placement groups instead of consulting a central object-location table.

This design targets large, fault-tolerant storage systems. It also creates operational work. You must plan failure domains, networks, media, capacity, protection, upgrades, monitoring, and recovery.

The RADOS foundation

RADOS is Ceph's reliable, autonomic distributed object store. All client interfaces ultimately store data as RADOS objects in pools.

An object has an identifier, binary data, and metadata. OSD daemons store those objects and handle reads, writes, replication, recovery, and scrubbing.

A pool is a logical partition of the RADOS object space. A pool defines placement and protection settings. It also forms an authorization boundary for many client capabilities.

Ceph maps every object to a placement group. A placement group collects many objects into a unit that Ceph can place, peer, recover, and rebalance. CRUSH then maps the placement group to an ordered set of OSDs.

object -> pool -> placement group -> acting set of OSDs

This extra layer matters. Managing placement for every object independently would create too much cluster metadata. Placement groups let Ceph manage object collections at a practical granularity.

The core daemons

Monitor

A Ceph Monitor, or MON, maintains the authoritative cluster maps. Clients and daemons contact monitors to learn cluster membership, topology, and state.

Monitors form a quorum. A majority must agree on cluster state. A three-monitor deployment can tolerate one monitor failure while retaining a majority. Production designs spread monitors across separate failure domains.

Monitors are not proxies for normal data traffic. A client gets a current map from a monitor, calculates placement, and contacts OSDs directly.

OSD

An Object Storage Daemon, or OSD, controls data on a storage device. OSDs serve client I/O, replicate or encode objects, detect peer state, recover data, and scrub stored objects.

An OSD has two independent state labels. up means its daemon is running and reachable. in means CRUSH includes it in data placement. An OSD can therefore be up but out, or down but still in.

Manager

A Ceph Manager, or MGR, exposes monitoring data and hosts manager modules. Those modules support functions such as orchestration, dashboards, metrics, and balancing.

The manager complements the monitors. It does not replace monitor quorum or store application data as an OSD.

Metadata Server

A Ceph Metadata Server, or MDS, manages CephFS metadata. It handles the file-system namespace, including directories, ownership, and access modes.

CephFS needs MDS daemons because file metadata has different access patterns from object data. RBD and RGW do not require MDS daemons.

CRUSH and failure domains

CRUSH stands for Controlled Replication Under Scalable Hashing. It calculates data placement from the cluster map, a placement rule, and an item identifier.

The CRUSH map represents a hierarchy such as device, host, rack, row, and room. A CRUSH rule can place copies or erasure-coded chunks across a chosen failure domain.

Suppose a replicated pool stores three copies and uses host as its failure domain. CRUSH should select OSDs on three different hosts. Three copies on three drives inside one host would not protect against that host failing.

CRUSH is deterministic. Clients and OSDs with the same map calculate the same expected locations. A topology change alters only the placements affected by the new calculation, which limits unnecessary movement.

Pools, placement groups, and protection

Pools divide cluster storage for workloads and policies. A pool chooses a replicated or erasure-coded protection model, a CRUSH rule, and placement-group behavior.

A replicated pool keeps complete copies of each object. The pool's size controls the copy count. The min_size setting controls how many copies must be available before the pool accepts I/O in a degraded state.

Ceph documentation recommends three copies rather than two for replicated high availability. That choice consumes more raw capacity but leaves more room for failure and recovery.

An erasure-coded pool divides an object into data chunks and coding chunks. A profile described as K plus M creates K data chunks and M coding chunks. This can reduce capacity overhead, but it adds computation and has workload-specific constraints.

Protection settings are not backups. Replication and erasure coding preserve current data through selected failures. They can also preserve an unwanted deletion or corrupted application state. Keep independent recovery copies when the workload requires historical recovery.

Three client-facing services

RBD: block storage

RADOS Block Device, or RBD, presents thin-provisioned block images. Clients can use kernel drivers, QEMU, or libraries to access those images.

RBD fits virtual-machine disks and applications that expect a block device. Its features include snapshots, layering, and replication options. The client still owns the file system or database layout placed on the image.

CephFS: file storage

CephFS presents a POSIX-compatible shared file system. Clients can mount it with a kernel client or Filesystem in Userspace.

MDS daemons manage file metadata, while file data remains in RADOS pools. CephFS fits workloads that need paths, directories, file permissions, and concurrent access to one namespace.

RGW: object storage

The RADOS Gateway, or RGW, provides object-storage APIs compatible with Amazon S3 and OpenStack Swift. RGW translates those API requests into RADOS operations.

RGW is a gateway in the client path, unlike native RBD and CephFS data access. Deploy several RGW instances with load balancing when the service needs scale and availability.

These interfaces share a storage foundation, but they do not become interchangeable. A block image does not gain file semantics. An RGW bucket is not a CephFS directory. Choose the interface that matches the application.

Cluster state and recovery

Ceph describes cluster health as HEALTH_OK, HEALTH_WARN, or HEALTH_ERR. The summary is only a starting point. Named health checks explain the condition that produced a warning or error.

Placement groups also expose state. active+clean means a placement group is available for I/O and has the expected complete set of replicas or chunks. Other states can show peering, degradation, recovery, remapping, or inconsistency.

When an OSD fails, monitors update cluster state after receiving failure reports. Placement groups peer to agree on their authoritative state. CRUSH can map affected placement groups to replacement OSDs, and recovery restores the configured protection level.

Recovery consumes the same devices and networks that serve clients. A cluster may remain available while latency rises. Capacity headroom matters because recovery needs somewhere to place replacement data.

Authentication is not encryption

Cephx authenticates Ceph clients and daemons with shared secrets and tickets. Capabilities limit what a Ceph client identity can do against monitors, OSDs, and other services.

Use separate client identities and least-privilege capabilities for applications. Do not distribute the administrative keyring as an application credential.

Cephx does not itself provide encryption at rest. It also does not cover every connection outside the native Ceph client-to-daemon boundary. Plan network encryption, device encryption, gateway security, and key management separately.

Where Ceph fits

Ceph is a strong candidate when you need shared storage across many hosts, expect capacity growth, and can operate a distributed system. It is common beneath private clouds, virtualization platforms, container platforms, data services, and large object stores.

Ceph may be the wrong choice for a small workload that fits safely on one managed storage service. The cluster has a real minimum operating footprint. It needs quorum, redundant storage, monitoring, upgrades, and staff who can diagnose recovery behavior.

Hardware labels alone do not predict results. OSD media, CPU, memory, network bandwidth, latency, failure-domain layout, client pattern, protection scheme, and recovery load all affect performance.

Build a representative test cluster before production. Test normal I/O, a full host failure, recovery, capacity pressure, upgrades, and restoration from an independent backup.

A learning path

Start by tracing one write from a client interface into a pool, placement group, and OSD acting set. Then learn the responsibility of each daemon.

Next, study CRUSH failure domains and pool protection. These settings decide whether replicas or chunks survive the failures you expect.

After that, choose one client interface and learn it in depth. Add cephadm operations, health interpretation, capacity planning, security, upgrades, and disaster recovery.

The durable mental model is this: monitors publish the map, CRUSH calculates placement, and OSDs store the objects. RBD, CephFS, and RGW turn that object foundation into application-facing storage.