# Disaster Recovery Runbook (KAR-520)

> Closes KAR-520. Companion: ADR 003 (V1 dedicated environments), ADR 012 (dedicated DB for sensitive tenants).
> Date: 2026-05-23.
> Owner: operator (DR drills) + Architecture Lead (procedure).

This runbook covers the recovery procedure for the BMW-pilot Supabase project. The skeleton ships intentionally compact: one DB, one Supabase project, one Vercel deploy. Each section says what to do, what to expect, and what to verify before declaring "recovered".

## Recovery Time / Point Objectives

| Metric | Target | Notes |
|---|---|---|
| **RTO** (time to restore service) | 4 hours | Bound by Supabase point-in-time restore + Vercel rebuild |
| **RPO** (acceptable data loss window) | 1 hour | Supabase PITR has ~minute-level granularity but operator alert latency dominates |
| **MTTR target** (typical incident) | 30 minutes | Most incidents are deploy-rollback, not full DR |

Numbers are nominal pilot-scope. They tighten when contractual SLAs land.

## Pre-incident checklist (do these NOW, not during the incident)

- [ ] Confirm Supabase **PITR (Point-In-Time Recovery) is enabled** on the production project. Dashboard → Database → Backups. PITR retention default is 7 days for Pro tier.
- [ ] Operator owns at least one **out-of-band credential** to Supabase (not stored in this repo, not in Vercel — e.g. password manager). Documented in `service-role-key-rotation.md`.
- [ ] Operator owns at least one **out-of-band credential** to Vercel (same comment).
- [ ] **Phone tree / contact list** updated in operator-private store. Not in repo.
- [ ] DNS provider login documented in operator-private store (DR includes the case where Vercel itself is down; we may need to point DNS elsewhere).

## Incident categories

### Cat-1 — Deploy regression

A recent merge broke production. Data is intact, application is broken.

1. Open Vercel → Deployments → identify the last known-good deployment by timestamp.
2. Promote it: `…` menu → "Promote to Production".
3. Verify in <1 min: smoke-test the home page and one authenticated route.
4. Open an incident ticket with the failing PR linked.
5. Document the root-cause in `docs/operations/incidents/<date>-<slug>.md`.

**Estimated downtime: 5 minutes.**

### Cat-2 — Data corruption (logical)

A query, migration, or buggy code wrote bad data. Schema is intact, rows are wrong.

1. **Stop writes** if the failing path is identifiable — promote a Vercel deployment that disables the route, or flip a feature flag.
2. Open Supabase → Database → Backups → Point in Time Recovery.
3. Restore to the **moment just before the corruption event**. PITR creates a **new project** with the restored state.
4. Compare the restored DB against the live DB to identify the divergence. Tools: `pg_dump` of both, diff the relevant tables.
5. **Surgical recovery** (preferred): copy only the affected rows from restored → live via `pg_dump --table` + `psql`.
6. **Full restore** (if surgical is infeasible): point the app at the new project ref by rotating `NEXT_PUBLIC_SUPABASE_URL` + `SUPABASE_SERVICE_ROLE_KEY` in Vercel, redeploy.
7. Document the divergence and the corruption root-cause.

**Estimated downtime: 1-3 hours.**

### Cat-3 — Data loss (deletion)

A user or admin deleted rows that should not have been deleted.

Same procedure as Cat-2 but the surgical recovery is INSERTs from the PITR project rather than UPDATEs.

### Cat-4 — Tenant project unavailable

The Supabase project is unreachable (Supabase region outage, project-level lock).

1. Status check: https://status.supabase.com.
2. If region-level, **wait** — usually under 1 hour. Communicate to users.
3. If project-level (extremely rare), open Supabase support ticket + escalate via paid-tier support phone.
4. Worst-case fallback: PITR-restore to a **different region**, repoint app. ETA 2-4 hours.

### Cat-5 — Vercel / DNS layer down

App is unreachable but DB is fine.

1. Status: https://www.vercel-status.com.
2. If Vercel-wide, wait. Communicate.
3. If DNS — verify the domain at https://dnschecker.org. Re-point if necessary via the DNS provider.

### Cat-6 — Total infrastructure compromise

Service-role key leaked, credentials exposed, or unauthorized access detected.

This branches to two procedures:

- **`service-role-key-rotation.md`** — rotate Supabase service-role key, Vercel envs, all sessions invalidated.
- **Full PITR restore to a fresh project** — same as Cat-2 Step 6, but the trigger is compromise not corruption.

## Backup-Restore drill (quarterly)

To validate the runbook is current, run a drill **once per quarter**:

1. Pick a random Tuesday. Operator notifies stakeholders 24 h ahead.
2. Choose a Cat-2-shaped scenario from above (e.g. "user reports the projects table is wiped").
3. **In staging only**: trigger PITR to recover. Capture timing of each step.
4. Compare to the RTO target. Findings go in `docs/operations/dr-drill-<date>.md`.

The drill is the only way to keep this runbook honest. Without one, the procedure rots quietly.

## What this runbook does NOT cover

- **Production deploy gate**: that is the CI workflow + branch protection settings. Drift between this runbook and reality is itself a finding.
- **Tenant-specific compliance reporting** (e.g. BMW audit trail requests). Tracked separately.
- **Multi-tenant DR fan-out** — V1 is single-tenant per Supabase project; multi-tenant DR plan ships with the multi-tenant migration ADR.

## References

- ADR 003 — V1 dedicated environments.
- ADR 012 — Dedicated DB for sensitive tenants.
- `docs/security/service-role-key-rotation.md`.
- Supabase docs: https://supabase.com/docs/guides/platform/backups
- Vercel docs: https://vercel.com/docs/deployments/managing-deployments