Disaster recovery plans that get tested, not just written

A disaster recovery plan that's never been tested is a document, not a plan — and the gap between the two usually only becomes visible during an actual incident.

Why untested DR plans fail in predictable ways

We regularly review client DR documentation that looks thorough on paper — defined RTO/RPO targets, documented failover procedures, backup schedules — and then find, during a first test, that the actual failover takes three times longer than the documented target, or that a critical dependency (a third-party API key, a DNS change that needs manual approval from someone on vacation) was never accounted for. These gaps are invisible until you actually try to execute the plan, which is precisely why testing matters more than the document itself.

The testing cadence that actually catches problems

We recommend and build toward quarterly DR tests at minimum, with two tiers: a tabletop exercise (walking through the plan verbally, identifying gaps in logic or missing steps) roughly monthly, and a full technical failover test — actually failing over to backup infrastructure and validating the application works — quarterly. The technical test is the one that catches the real gaps, because tabletop exercises can't surface issues like "the backup database restore script hasn't been updated since the schema changed six months ago."

A concrete example of what testing catches

For a financial services client with a documented RTO (recovery time objective) of 4 hours, our first full failover test took just under 11 hours — nearly three times the documented target. The gap traced to several specific issues: the runbook referenced an outdated network configuration, a required SSL certificate for the failover environment had expired without anyone noticing because it wasn't in active use, and two team members listed as required for the failover process were no longer with the company. None of these would have been visible from reading the document — they only surfaced by actually executing it.

After remediation and a second test three months later, failover time was 3 hours 40 minutes — within target, and validated by actually doing it rather than assumed from documentation.

RTO and RPO — the numbers that should drive every decision

Recovery Time Objective (how long can you be down) and Recovery Point Objective (how much data loss is acceptable, measured in time) aren't abstract compliance checkboxes — they should directly determine your architecture. A 15-minute RPO requirement means near-continuous replication; a 24-hour RPO tolerance means nightly backups are sufficient and much cheaper. We push clients to set these numbers based on actual business impact analysis, not a generic industry default, because over-engineering DR for a system that could tolerate a day of downtime wastes budget that would matter more elsewhere.

How Ndakum approaches it

DR planning and testing is part of our Cloud Engineering work — we build the plan and then schedule the actual test, because an unproven plan gives false confidence.