Disaster Recovery You Can Rehearse: Building and Rehearsing the System
A DR environment is useful only if it can be operated under pressure. For our Kubernetes platform, that meant more than a second cluster in another region. Infrastructure, identities, secrets, applications, and traffic had to come together in the right order. Wrong order meant engineers debugging pods that would not start. We wanted recovery repeatable and safe, not dependent on a few people remembering commands during an outage.
That gave us three practical engineering problems to solve. We needed:
- Safeguards against build scripts touching production
- Secrets synchronised without values in Git
- An affordable standby that scales quickly for recovery
The final test was a repeatable drill that brought all three together.
Build in the Right Order
A DR environment is a chain of dependencies: each layer must be ready before the next.
We started with a dedicated DR project and VPC, then a Kubernetes cluster with private nodes and workload identity.
Next came application identities, then empty secret placeholders with production names. Only then could secret synchronisation and the platform installer be set up.
We added a temporary VPN and DNS link, scaling and traffic scripts, and a drill checklist. Applications restored too early would fail to start.
We Made the Installer Hard to Misuse
The platform installer removed a whole class of failure. It runs 14 idempotent steps. A second run does not create different state or break what is already there. Steps can run individually, and a dry run shows what would happen before changes.
More importantly: It checks the connected Kubernetes cluster first. If the context is not the expected DR cluster, it stops, and it refuses to run against anything resembling production.
The reason is simple: People make mistakes when tired or under pressure.
The goal: make the most dangerous mistake difficult.
Keeping Secrets in Sync Without Putting Secrets in Git
Restored applications expect their production secret names to exist in DR. Creating those names was straightforward. The harder part was keeping their values synchronised without putting sensitive data into our infrastructure code or Git.
We used Terraform to create secret placeholders in DR. They have the same names as their production counterparts but contain no values.
A scheduled sync job compares the current production value with the DR version and copies only new or changed versions. Before every drill, we force a sync.
Values never enter Git or Terraform state. The team has a clear RPO: secrets are current as of the last successful sync. Sync is no longer a memory step; it is fixed in every drill.
Warm When Idle, Hot When We Need It
A full production-sized platform in a non-serving region is expensive; too little capacity makes recovery slow. So DR has two modes. Warm is the default: minimum size, ready infrastructure. Hot is for drills or failover.
We keep no spare application machines idle in DR. When restored pods need capacity, GKE creates machines automatically. After a drill, the platform scales back down. This keeps standby affordable without making capacity a recovery bottleneck.
The Drill: 12 Steps, Every Time
A recovery process is useful only if it works the same when calm and when the primary region is down. Practice runs and real failovers follow the same steps. The 12 steps fall into five parts: start recovery, prepare the environment, restore and validate applications, move traffic, and measure.
Start the Recovery
1. Declare the drill or incident: We assign one owner, one communication channel and start the clock.
2. Define the scope: Is the whole region being recovered, or only part of it? Traffic should not move on a hunch.
Prepare the Environment
3. Sync secrets: We force a sync and check the critical ones.
4. Scale up: The DR platform moves from warm to hot.
5. Check the platform: Nodes, workload identity and the service mesh are verified before applications are restored.
Restore and Validate
6. Restore applications: Admission policies automatically fix the environment-specific settings inherited from production. The drill has restored 219 namespaces in minutes.
7. Fix images: Anything still pointing to the wrong registry is corrected.
8. Check readiness: Health checks, gateways and dependent services, including AWS, are verified.
Move Traffic
9. Open a DNS pull request: A script reads the live gateway addresses and proposes the change.
10. Cut over: The change is reviewed, merged and applied, and the team watches the traffic move.
Using a pull request instead of changing DNS directly in a console makes the change reviewed, logged and easy to undo.
Measure and Close
11. Record the recovery: We stop the clock and write down RTO and RPO.
12. Plan failback separately: The return to primary gets its own plan rather than being improvised while tired.
What Changed When We Started Rehearsing
The biggest change was not just faster recovery. It was knowing recovery could actually run.
Before, recovery was an untested document; there was no second region, applications were restored manually, team by team, and knowledge lived with a few people.
Now there is a second region at minimum size, a practised 12-step failover, and documented owners, steps, and runbooks.
We started with “We don't know how long recovery would take.” Now we can run, measure, and improve the process repeatedly. The real achievement is predictable recovery.
What We Left Out on Purpose
We did not attempt full disaster recovery at once. We started with compute: Kubernetes clusters, platform components, and application recovery.
The data layer is still ahead. Cloud SQL, Postgres, and Memorystore need their own DR setup.
AWS is not independent yet. DR applications reach S3, SQS, and SNS through the same network path as production, so a failure of that path could still affect recovery.
We avoided static keys: they might make a drill pass but would turn a temporary workaround into a permanent security risk. For now, we know exactly where the recovery boundary ends.
Closing Thoughts
The goal was never perfect DR in one shot. It was to turn recovery from something written down into something we could run. The remaining gaps are known, the process is repeatable, and every drill gives us another chance to improve it.
More from
Engineering
category
