Don't Panic: Ceph Fundamentals

Ceph is what happens when one storage server stops being enough, and the replacement is a cluster that has opinions about maps, quorum, and where every byte ought to live. It provides block devices, a shared file system, and object APIs, but those are different doors into the same building. The building is RADOS, which stores data as objects in pools.

The important trick is CRUSH, the placement calculation. A client gets a cluster map from monitors, works out where an object belongs, then speaks to the OSDs that hold it. The monitors do not carry ordinary data traffic. They publish the map. Efficient, certainly, until the map must describe the real building, racks, hosts, and all the ways reality can become unavailable.

An OSD stores objects and does the strenuous chores: I/O, replication, recovery, and scrubbing. Objects collect into placement groups, which give Ceph something manageable to place and repair. The surprise is that three copies are not automatically three-host safety. Three drives in one host are three copies with one shared bad day. CRUSH rules and failure domains decide what the copies actually survive.

Ceph offers three client contracts. RBD supplies a block image for a VM or block-oriented application. CephFS supplies paths, directories, and file semantics, with metadata servers for the namespace. RGW supplies S3-compatible and Swift-compatible object APIs. One cluster can serve all three, which is handy. It does not make a bucket into a directory, any more than putting a hat on a disk turns it into a filing cabinet.

Health is where the distributed-system bill arrives. A placement group that is active plus clean accepts I/O and has its expected protection set. After an OSD failure, placement groups peer, CRUSH remaps work, and OSDs recover data. The cluster can remain available while recovery competes with client traffic. Replication and erasure coding protect current state; neither rescues an unwanted deletion or bad application write. Keep an independent recovery plan.

Start with the Intro for the complete architecture. Use Slides for the map of daemons, placement, and interfaces. Keep the Cheatsheet nearby while comparing pool protection, health signals, and failure domains. Then use the practice reference and exercise on a disposable lab, where a stopped OSD is a lesson rather than an incident report.

Don't Panic: Ceph Fundamentals

Relevant careers