# Runbook — Incident Response

> Audience: on-call engineer, vendor support, BMW IT during go-live.
> Goal: short, scannable steps when something has gone wrong in
> production. Detailed background lives in
> `docs/foundation/` and `SECURITY.md`.

---

## Severity classification

| Level | Examples | Time-to-acknowledge | Time-to-mitigate |
| --- | --- | --- | --- |
| SEV-1 | Production down, mass data loss, secret exposure | 15 min | 1 h |
| SEV-2 | Tenant outage, RLS bypass, auth failure for >1 user | 1 h | 4 h |
| SEV-3 | Single user blocked, slow response, intermittent error | 4 h | 1 working day |
| SEV-4 | Cosmetic, single non-critical workflow | 2 working days | next sprint |

## First 15 minutes

1. **Acknowledge.** Reply in the operations channel with the time + your name.
2. **Capture state.** Run:
   ```bash
   curl -sI https://kadi-v2.vercel.app/login
   curl -sI https://kadi-v2.vercel.app/api/csp-report -X POST
   vercel inspect <url>            # latest deployment
   gh pr list --state open         # active PRs that may have introduced
   git log --oneline -10           # recent commits
   ```
3. **Identify if pre-existing or recent.** Compare current commit to last known-good
   deployment (Vercel dashboard → Deployments).
4. **Decide rollback or fix-forward.**
   - Rollback: in Vercel, click the previous successful deployment → "Promote to Production".
   - Fix-forward: open hotfix branch, push, merge. Use only if rollback would
     lose successful new behaviour.
5. **Post status.** Even one line ("Investigating supplier-search timeout, no rollback yet").

## Categories

### A — Database outage or RLS bypass

1. Run `supabase/bootstrap/supabase-audit-phase1-2.sql` against the active project. Compare
   to the latest snapshot in `docs/foundation/rls-audit-2026-04-21.md`.
2. If RLS bypass: rotate the service-role key immediately (see
   `secrets-rotation.md`).
3. If outage: check Supabase status page; check connection-pool exhaustion
   (`SELECT count(*) FROM pg_stat_activity` on the project SQL editor).

### B — Auth failure (login broken)

1. Check `https://kadi-v2.vercel.app/login` — does it render?
2. Test sign-in with the masteradmin account.
3. Inspect Supabase logs: Dashboard → Logs → Auth.
4. Common causes:
   - `NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY` mismatched after rotation
   - Site URL or Redirect URL drift in Supabase Auth settings
   - Magic-link expired (if onboarding flow)

### C — Secret exposure

If a secret has been committed to git, posted in chat, or otherwise leaked:

1. Treat as SEV-1.
2. Rotate the secret immediately (`secrets-rotation.md`).
3. Search git history for the leaked string:
   ```bash
   git log -S '<secret>' --all
   git rev-list --all | xargs git grep '<secret>'
   ```
4. If committed: file a `git filter-repo` or `bfg-repo-cleaner` ticket
   (history rewrite needs maintainer-approval and force-push).
5. Notify the operator and document in `docs/runbooks/incidents/<date>.md`.

### D — Cost spike on Vercel / Supabase

1. Check Vercel Function invocations on the dashboard.
2. Check Supabase usage on the dashboard.
3. If brute-force suspected: identify the IP (Vercel logs → filter by 4xx/5xx),
   add to Vercel firewall rules.
4. Flag pre-Azure: deploy Azure-Cache-for-Redis-backed rate limiter
   (`docs/foundation/rate-limiting-readiness.md`).

## After mitigation

- [ ] Write a 1-page incident write-up under
      `docs/runbooks/incidents/<YYYY-MM-DD>-<short-name>.md`.
- [ ] Add a regression test if the incident was code-caused.
- [ ] If a new failure mode: extend this runbook.
- [ ] Notify operator and stakeholders that mitigation is complete.