# Observability Runbook (KAR-519)

> Closes KAR-519. Companion: ADR 016 (logging facade) + KAR-532 (@vercel/otel).
> Date: 2026-05-23.

## What is wired

| Layer | Tool | Where |
|---|---|---|
| Errors / breadcrumbs / traces | Sentry (`@sentry/nextjs`) | `sentry.server.config.ts`, `sentry.edge.config.ts` |
| Structured logs | `lib/logger.ts` (ADR 016) | JSON-in-prod, pretty-in-dev, scrubs `SENSITIVE_KEYS` |
| Auto-instrumentation | `@vercel/otel` (KAR-532) | `instrumentation.ts` |
| Custom spans / metrics | `lib/observability/spans.ts` (KAR-532) | opt-in per call site |
| Field Web Vitals | `@vercel/speed-insights` (KAR-522 QW3) | `app/layout.tsx` |
| CI Web Vitals | Lighthouse-CI (KAR-524) | `.github/workflows/lighthouse-ci.yml` |

## Env vars

| Var | Required | Default | Notes |
|---|---|---|---|
| `SENTRY_DSN` | optional | (none) | When unset, Sentry init is a no-op. Set for staging + prod. |
| `SENTRY_TRACES_SAMPLE_RATE` | optional | `0.1` | Float 0..1. Lower in high-traffic prod, higher in staging. |
| `LOG_LEVEL` | optional | `info` (prod) / `debug` (dev) | Lowest level the logger emits. |
| `LOG_SERVICE` | optional | `kadi-backend` | Identifier in log records and OTel service name. |

## Alert rules to configure in Sentry UI

These cannot be checked into code (Sentry's API for alert-rules is fragile and project-scoped). Operator sets them up post-DSN-rotation:

1. **Error rate spike** — fires if `event.type:error` count > 50 / 5 min on production environment.
2. **New issue** — fires on every previously-unseen `issue.title` in production. Slack channel + email.
3. **Performance regression** — fires if p95 trace duration of `transaction:GET /api/*` > 1s for 10 min.
4. **High 5xx ratio** — fires if 5xx rate > 1% for 5 min.

Each alert targets an on-call rotation (TBD per operator setup).

## On-incident checklist

1. Open the Sentry issue from the alert.
2. Identify the failing transaction (Trace tab → top of the trace tree).
3. Cross-reference with Vercel logs (Functions tab) for the same request_id.
4. If a release is suspected, check `release` field on the issue. Rollback via Vercel Promotions.
5. Capture the trace_id in the incident channel before resolving — it ties Sentry, logs, and Vercel logs together.

## Local dev

Without a Sentry DSN set:

- Sentry init becomes a no-op.
- `lib/logger.ts` writes pretty-printed lines to stdout.
- `@vercel/otel` exports nothing (no exporter wired).

With a staging DSN set in `.env.local`:

- Errors during local dev surface in the staging Sentry project.
- Traces use the staging service-name. Don't mix prod data.

## What this runbook does NOT cover

- **PII in logs sweep** — tracked in KAR-535. The logger's `SENSITIVE_KEYS` set scrubs structured fields, but message strings need a manual audit.
- **JWT-key rotation** — tracked in KAR-537.
- **OWASP-Top-10-CI** — tracked in KAR-518. Adds dependency + SAST gates.
- **Rate-limiting** — tracked in KAR-517. Sentry shows the symptom; Upstash limits the cause.

## References

- ADR 016 — Logging and Observability Facade.
- KAR-532 — OpenTelemetry helpers.
- KAR-522 audit Source 8 (OpenTelemetry) + Source 1 (12-Factor XI Logs as event streams).