Database High Availability

Database high availability keeps a database service usable when part of its supporting system fails. It combines redundant data copies, failure detection, controlled role changes, and client routing.

No design removes every outage. Your job is to define the failures the service must survive, then prove the design meets that target.

This course gives you a practical map. You will learn how replication, quorum, failover, routing, durability, and recovery objectives fit together.

Start with the service objective

Availability belongs to the whole service, not one database process. A healthy standby does not help if applications cannot find it after failover.

Define the outcome before choosing a topology:

Recovery time objective (RTO): the maximum acceptable time a system resource can remain unavailable.
Recovery point objective (RPO): the point in time to which data must be recovered after an outage. An RTO shapes detection, election, promotion, routing, and application recovery. An RPO shapes replication and commit choices.

Ask which failures count. A design that survives one host failure may still fail when a rack, network zone, region, identity system, or router fails.

Build one service from several roles

A common topology has one read-write primary and one or more standby servers. The primary records changes. Replication carries those changes to each standby.

A warm standby waits for promotion before serving traffic. A hot standby can also serve read-only queries.

Redundancy helps only when copies do not share every failure boundary. Place enough database members, routing components, and control services across the failures you intend to tolerate.

Capacity matters after failure. The remaining system must carry the required workload while a member or location is unavailable.

Choose the replication guarantee

Asynchronous replication lets the primary acknowledge a commit before a standby confirms it. This usually reduces write latency. A failover can lose commits that had not reached the promoted standby.

Synchronous replication waits for a defined standby acknowledgment before reporting commit success. This improves durability across a primary failure, but adds response time and can reduce write availability.

The exact acknowledgment matters. A standby may confirm receipt, durable storage, or application of a change. Read the database manual before translating “synchronous” into an RPO claim.

Replication lag measures how far a standby trails the primary. Monitor bytes or log positions, replay delay, and time. One number cannot describe every failure consequence.

Use quorum to protect one writable history

Distributed members can lose contact while they remain healthy. This is a network partition.

A quorum is the minimum voting set allowed to make a decision. Majority-based systems require more than half of the configured votes.

Quorum prevents both sides of a partition from independently choosing a primary. The minority side stops accepting writes because two writable histories would be harder to reconcile than a temporary outage.

An odd number of voting members can provide a useful majority without an unused tie vote. Member placement matters more than the number alone. A single location must not hold every vote needed for progress.

Do not force quorum or promote a node just to clear an alert. First establish which members contain the authoritative history and ensure the old primary cannot accept writes.

Treat failover as a controlled state change

Failover moves service from a failed or unreachable primary to a suitable standby. A sound sequence has several stages:

Detect that the primary is not serving correctly.
Confirm enough voters can make a safe decision.
Prevent the old primary from accepting writes.
Select a standby with an acceptable data state.
Promote or elect the new primary.
Redirect new client connections.
Verify writes, reads, replication, and application behavior.

Preventing the old primary from serving writes is called fencing. It can use power control, storage exclusion, network isolation, or a database-native mechanism.

Failover is automatic when software performs the sequence. Switchover is a planned role change while the current primary is healthy.

Automation reduces reaction time, but a poor detector can turn a brief network delay into an unnecessary failover. Tune detection against measured conditions and test the complete sequence.

Keep clients inside the design

Applications need a stable connection path that follows the writable primary. A proxy, database router, virtual endpoint, service discovery system, or driver can provide that path.

Existing connections usually break when their server fails. Applications must reconnect, use bounded timeouts, and retry only operations that are safe to repeat.

Connection pools can preserve stale connections. Transactions may have an unknown outcome when a connection drops during commit. Application logic must handle that uncertainty without duplicating business actions.

Read routing creates another choice. A standby can reduce primary read load, but asynchronous replicas can return stale data. Route consistency-sensitive reads according to the database guarantee you require.

Monitor the ability to fail over

A green primary is incomplete evidence. Monitor the entire failover path:

member health, role, and voting state;
replication transport, durable position, replay position, and lag;
quorum and failure-domain coverage;
router health and connection success;
remaining capacity after a member failure;
failed elections, unexpected role changes, and recovery time;
backup status and restore-test results.

Alert on loss of redundancy before the primary fails. A service running on its last healthy copy is available now but unable to tolerate the next planned failure.

Keep backup and disaster recovery separate

Replication maintains another current copy. It can also carry accidental deletion, bad updates, or corruption to that copy.

Backups preserve recovery points. They support recovery to an earlier state and remain necessary even when failover is fast.

High availability usually addresses component failures within a designed topology. Disaster recovery addresses wider loss and may use another location, restored backups, or a separate replicated cluster.

The boundaries can overlap. Name the protected failure, target RTO, target RPO, and activation process instead of relying on labels.

Test the claims

Run planned switchovers to exercise routine role changes. Run failover tests that remove the primary, isolate a network path, or make the router unavailable.

Measure detection, election, promotion, reconnection, and application recovery separately. Confirm the data point after failover and compare it with the RPO.

Test degraded capacity. Test a failed standby rebuild. Test restoration from backup because quorum cannot recover deleted data or a completely lost cluster.

Record evidence and update the runbook. A topology diagram describes intent. Repeated tests show what the service can actually survive.

Limits

High availability cannot guarantee uninterrupted service during every failure. Quorum loss, correlated faults, software defects, operator mistakes, and dependency outages can stop the database path.

More replicas do not automatically improve safety. Poor placement can preserve shared failure. Incorrect failover can create conflicting writers.

Synchronous replication does not replace backups. Automatic failover does not replace application retries. Read scaling does not prove write availability.