Disaster Recovery AI Agent

The problem

DR runbooks are written once, tested rarely, and executed under stress. When the incident hits, the person who wrote the runbook may not be on-call, and the runbook has drifted from reality — updated database versions, migrated backup locations, changed credential paths. Half the outage window is spent figuring out which step to run next and whether the current step is still even valid. The cost of this is not the runbook itself; it's the SLA breach, the customer-facing incident, and the exhausted engineer who has to reconstruct what was supposed to happen while the pager keeps going off. Every organization eventually has a "the DR runbook didn't work when we needed it" story. The pattern is universal.

Key challenges

Automated DR sounds appealing until you consider what happens when the automation gets it wrong. Restoring the wrong backup can be worse than the original incident. Restarting a service that shouldn't be restarted can compound cascading failures. Any DR automation has to be trustworthy at the moments the humans running it are most stressed and least able to sanity-check its outputs — which means the design has to prioritize explainability, safe defaults, and human overrides at every layer.

What we built

AR Data built an AI agent that operates the DR runbook — detects the failure pattern, selects the appropriate recovery path, orchestrates backup restoration and verification, and escalates to humans at defined decision points. The agent explains what it is doing at each step so on-call engineers can override with context. The architecture separates detection (deterministic monitors for known failure signatures), decision (agent selects recovery path from a defined set), execution (deterministic runbook steps, one at a time, with verification between each), and escalation (defined criteria for when to hand off to a human). This structure keeps the agent's decision space narrow enough to be safe and wide enough to be useful.

Our approach

1. 1

   Detection is deterministic, decision is agentic

   Failure detection uses deterministic monitors (thresholds, health check patterns). The agent doesn't guess whether there's an incident; it responds to structured incident signals. Agent judgment is applied to path selection, not to detection.

2. 2

   Path selection from a defined set

   The agent chooses among a finite set of recovery paths, each with its own preconditions and verification steps. This is much safer than open-ended recovery planning and much easier to audit.

3. 3

   Per-step verification before advance

   Every runbook step verifies its own success before the next step runs. If verification fails, the agent stops and escalates rather than plowing forward.

4. 4

   Escalation is a first-class outcome

   The agent has defined escalation triggers — unfamiliar failure pattern, verification failure, cross-region blast radius, request outside the automation scope. Escalation preserves full context for the human on-call.

Key architectural decisions

Results

• Automated failure detection via structured monitors
• Guided recovery execution with per-step verification
• Human-in-the-loop overrides at every decision point
• Escalation on unfamiliar patterns rather than best-effort execution
• Post-incident reports auto-generated with full agent reasoning
• Reduced mean-time-to-recovery on covered failure patterns
• Runbook drift eliminated — the agent IS the runbook, versioned in code

Impact

The DR agent turns runbooks from something-that-drifts into something-that-runs. The value isn't that a machine handles the incident — it's that the machine handles the parts of the incident that are known and hands the rest to humans with full context. That reframing (agent as executor of well-defined procedures, human as decider on ambiguity) is the pattern we bring to other operational-AI engagements: the automation earns trust by having a small, well-defined scope and honestly escalating outside it.

Tech stack