openskills.info

Disaster Recovery

itPlatform engineering and SRE

Disaster Recovery

Disaster recovery is the work of restoring technology services after a severe disruption. The disruption might destroy infrastructure, corrupt data, disable a region, or make a primary site unusable.

Your goal is not to predict one dramatic event. Your goal is to preserve the business outcomes that depend on technology when normal operating assumptions fail.

That makes disaster recovery broader than keeping a backup. A usable recovery capability combines priorities, people, procedures, data, infrastructure, communications, and tests.

Start with business impact

A business impact analysis identifies the processes a system supports, their dependencies, and the harm caused by downtime or data loss. It gives you an order of recovery instead of a flat list of "critical" systems.

Use the analysis to answer four questions:

  1. Which business process must return first?
  2. Which applications, data, identities, networks, facilities, and suppliers support it?
  3. How much service interruption can the process tolerate?
  4. How much recent data can the process lose?

The answers produce recovery requirements. They should come from business owners and technical owners together. A technically impressive recovery design is still wrong if it restores the wrong service first.

Use two time objectives

The recovery time objective, or RTO, is the maximum acceptable time between disruption and restoration. It describes the allowed downtime.

The recovery point objective, or RPO, is the point in time to which data must be recovered. It describes the acceptable data-loss window.

Suppose a service has a four-hour RTO and a fifteen-minute RPO. The recovery capability must restore the service within four hours. It must also recover data to a point no more than fifteen minutes before the disruption.

RTO and RPO are requirements, not measurements of an actual recovery. A test produces the observed recovery time and observed recovery point. Compare those results with the objectives.

Near-zero objectives usually require more replication, automation, capacity, and operational care. Set them from impact and risk, then balance them against cost and complexity.

Separate related capabilities

High availability handles expected component failures while the service continues operating. It might replace a failed instance or route around a failed zone.

Disaster recovery handles a distinct event with a larger or longer effect. It activates an alternate recovery path and may accept some downtime or data loss.

Business continuity covers the wider organization. It includes people, facilities, suppliers, communications, and manual processes as well as technology recovery.

Incident response contains and investigates an event. Disaster recovery restores the required service. A ransomware incident may need both: responders determine a safe recovery point, while the recovery team rebuilds and validates the service.

These capabilities overlap, but none replaces the others.

Choose a recovery posture

Recovery strategies form a cost-and-speed spectrum:

  • Backup and restore recreates infrastructure and restores protected data after the event. It has low standby cost but usually takes the longest.
  • Pilot light keeps the most critical core components ready while the rest is provisioned during recovery.
  • Warm standby runs a reduced-capacity copy that can scale and receive traffic.
  • Active-active runs service from multiple locations and shifts traffic when one location fails. It is fast, but it adds cost and consistency challenges.

Do not select a posture from its label alone. Confirm that the complete recovery path meets the objectives. That path includes identity, network, configuration, secrets, application code, data, capacity, external services, and traffic changes.

Replication is not automatically a safe backup. It can copy corruption or destructive changes to the secondary location. Recovery design often needs separate, protected recovery points as well as replicated service state.

Turn strategy into an executable plan

A recovery plan should tell an authorized team what to do under pressure. NIST organizes execution into three phases:

  1. Activation and notification assesses the outage, applies activation criteria, names the decision authority, and contacts the recovery team.
  2. Recovery restores components in dependency order and verifies the required functions.
  3. Reconstitution validates the recovered system, returns to a normal operating state, cleans up temporary resources, and records lessons.

The plan needs named roles, current contacts, prerequisites, decision points, escalation paths, communication procedures, and ordered recovery steps. Store a protected copy where the team can reach it even when the primary environment is unavailable.

Define success in business terms. "The database is online" is a component check. "An authorized customer can place an order and the order is recorded correctly" is a service check.

Test the recovery capability

A document review can find missing names or steps. A tabletop exercise can expose unclear decisions and coordination gaps. A simulation can exercise procedures. A full recovery test can validate infrastructure, data, dependencies, and reconstitution.

Use a test scope that matches the risk. Protect production while still exercising the parts that could fail during a real recovery.

Record the observed recovery time, recovered data point, failed steps, manual work, and unexpected dependencies. Correct the plan and the system. Then test again.

A successful backup job proves that data was copied. A successful restore test proves that the copy can participate in recovery. A complete disaster recovery test goes further and proves that the restored service performs its required business functions.

Know the limits

Disaster recovery cannot preserve a business process that depends on unavailable staff, facilities, suppliers, or legal authority. Business continuity planning must cover those constraints.

A cloud provider supplies recovery features, not your finished recovery capability. You still define the objectives, select locations, protect credentials, configure replication or backups, document decisions, and test the workload.

A plan also ages. Architecture, contacts, vendors, data volumes, and recovery objectives change. Update the plan after system changes, tests, and real incidents.

A practical learning path

Begin with a business impact analysis and a dependency map. Define RTO and RPO for each important service flow. Compare recovery postures against those objectives. Write one executable plan with clear activation criteria. Test a restore, then a service recovery, then a coordinated exercise. Use every result to improve both the plan and the system.