# Operational Runbooks

## Tenant Suspension

**When to use**: Non-payment, policy violation, legal hold.

**Steps**:
1. Owner Portal → Tenants → select tenant → detail page
2. Click **Suspend** and enter reason
3. Or via API: `POST /api/owner/tenants/{tenantId}/suspend` with `{ "reason": "..." }`

**Effect**:
- Tenant status → `SUSPENDED`
- Tenant app serves requests but `requireActiveTenant()` returns 403
- Tenant data is preserved — no deletion
- Entitlement checks still run but `requireActiveTenant()` blocks before they're reached

**Reversal**:
```sql
UPDATE cp_tenants
SET status = 'ACTIVE', suspended_at = NULL, suspended_reason = NULL, updated_at = now()
WHERE id = '{tenantId}';
```
Then immediately create an audit event manually or via owner API PATCH.

---

## Tenant Archival

**When to use**: Contract ended, tenant requested deletion.

**Prerequisites**: Tenant must be `SUSPENDED` first.

**Steps**:
1. Suspend the tenant (see above)
2. Owner Portal → Tenant → click **Archive**
3. Or: `POST /api/owner/tenants/{tenantId}/archive`

**Effect**:
- Tenant status → `ARCHIVED`
- Tenant cannot be reactivated via UI (PLATFORM_OWNER-only database operation)
- Data preserved for compliance retention period

**Irreversibility**: Archival is not reversible via the portal. Direct DB access required.

---

## Emergency Feature Disable (Kill Switch)

**When to use**: Bug in production affecting a module, security issue, rollout problem.

**Steps** (manual SQL until UI is built):
```sql
INSERT INTO cp_release_rollouts (feature_id, type, is_active, reason, tenant_ids, set_by, expires_at)
SELECT
  id,
  'KILL_SWITCH',
  true,
  'Bug in MODULE_OEE trend chart causing data corruption — disabling globally',
  NULL,          -- NULL = all tenants
  '{ownerUserId}',
  now() + interval '24 hours'
FROM cp_features
WHERE code = 'MODULE_OEE';
```

**Effect**: Entitlement engine returns `kill_switch` denial for all tenants immediately.
Entitlement cache invalidates within 5 minutes (or immediately if `invalidateEntitlementCache()` is called).

**Reversal**:
```sql
UPDATE cp_release_rollouts
SET is_active = false, updated_at = now()
WHERE type = 'KILL_SWITCH' AND is_active = true
  AND feature_id = (SELECT id FROM cp_features WHERE code = 'MODULE_OEE');
```

---

## Support Access Review

**When to use**: Auditing who accessed tenant data, reviewing active sessions.

**Query active sessions**:
```sql
SELECT
  s.id,
  s.reason,
  s.expires_at,
  s.created_at,
  u.email AS owner_email,
  t.company_name AS tenant_name
FROM cp_support_access_sessions s
JOIN cp_owner_users u ON u.user_id = s.owner_user_id
JOIN cp_tenants t ON t.id = s.tenant_id
WHERE s.terminated_at IS NULL
  AND s.expires_at > now()
ORDER BY s.created_at DESC;
```

**Terminate a session**:
```
DELETE /api/owner/tenants/{tenantId}/support-access?sessionId={sessionId}
```

**Audit history**:
```sql
SELECT * FROM cp_audit_events
WHERE action IN ('support_access.created', 'support_access.terminated')
ORDER BY created_at DESC;
```

---

## Entitlement Cache Invalidation

If a tenant reports that a feature toggle isn't taking effect:

**Via API** (owner applies module change — cache auto-invalidates)

**Manual invalidation**: Currently in-memory only. Cache TTL is 5 minutes.
On multi-instance deployments, all instances must be restarted or Redis-backed cache used.

**Check current entitlement for a tenant**:
```typescript
import { checkEntitlement } from '@/lib/entitlement/engine'
const result = await checkEntitlement({
  tenantId: '{tenantId}',
  feature: 'MODULE_OEE',
})
// result.granted, result.reason
```

---

## Provisioning Monitoring

**Check for stuck jobs** (RUNNING for >30 minutes):
```sql
SELECT j.id, j.status, j.started_at, t.company_name
FROM cp_provisioning_jobs j
JOIN cp_tenants t ON t.id = j.tenant_id
WHERE j.status = 'RUNNING'
  AND j.started_at < now() - interval '30 minutes';
```

**Check recent failure rate**:
```sql
SELECT
  status,
  COUNT(*) AS count,
  MAX(created_at) AS latest
FROM cp_provisioning_jobs
WHERE created_at > now() - interval '24 hours'
GROUP BY status;
```

**Stuck job recovery** (manually reset to retry):
```sql
UPDATE cp_provisioning_jobs SET status = 'FAILED', error_message = 'Manually reset — was stuck RUNNING'
WHERE id = '{jobId}';
```
Then retry from Owner Portal.