Don't Panic: Database Failover and Disaster Recovery Drills

Database failover moves the writable role from one database to another. A disaster recovery drill proves whether that move, the surrounding application path, and the return to protection work under a defined scenario. The database accepting one query is not the same as the service being recovered, although it may look very pleased with itself.

The short sequence is fence, promote, redirect, validate, and re-protect. Fencing prevents the old primary from accepting writes. It comes before promotion because a hidden old writer can create split brain, where two databases accept different histories. Network silence is not fencing. It is only silence, and silence has caused enough meetings already.

Promotion makes the standby or restored copy authoritative. A planned switchover happens while both sides are reachable. A forced failover handles a source that is unavailable or unsafe, and may accept data loss from unreplayed transactions. That is why the drill needs a stated RPO, the allowed data gap, and RTO, the allowed service recovery time.

Measure the real path. Endpoints, DNS, connection pools, secrets, identity, jobs, monitoring, and downstream consumers can all fail after the database role changes. Use canary transactions and high-water marks to prove what data arrived. Validate critical reads, one safe write, integrity checks, scheduled jobs, change streams, audit logs, and normal security controls.

Promotion is not the finish line. The new primary may lack a ready standby. The former primary must be rewound, re-seeded, or restored before it rejoins. Failback is another role transition, not a victory lap.

Use the Practice Reference to set pass conditions and timestamps. Do the Exercise to run a tabletop drill. The Cheatsheet keeps objectives, transitions, split-brain controls, and validation in view. A drill proves one scenario at one time, so rotate scenarios before reality chooses one for you.

Don't Panic: Database Failover and Disaster Recovery Drills

Relevant careers