# Runbook: Failed Provisioning

## Symptoms
- Tenant status = `PROVISIONING_FAILED`
- Provisioning job status = `FAILED`
- One step shows status `FAILED` with `error_detail`

## Diagnosis

1. **Owner Portal** → Provisioning → find the job → inspect step log
2. Check which step failed and its `error_detail`
3. Check `output_data` on completed steps for context

### Common Failure Modes

| Step | Common Cause | Fix |
|------|-------------|-----|
| `create_supabase_project` | Missing `SUPABASE_MANAGEMENT_API_KEY` | Set env var, retry |
| `apply_schema_migration` | No tenant DB credentials | Configure environment first |
| `set_secrets` | Missing `VERCEL_API_TOKEN` | Set env var, retry |
| `seed_branding` | DB write conflict | Safe to retry — idempotent |
| `seed_modules` | No plan assigned | Assign plan to tenant, retry |
| `create_tenant_admin` | Tenant DB not accessible | Configure credentials first |
| `activate_tenant` | Tenant already in wrong state | Check tenant status, retry |

## V1 Stub Steps

The following steps are **stubs** in V1 — they complete immediately with a note:
- `create_supabase_project` (requires Supabase Management API key)
- `apply_schema_migration` (requires tenant DB credentials)
- `configure_storage` (runs at project creation)
- `set_secrets` (requires Vercel API token)
- `create_tenant_admin` (requires tenant DB service_role)

The following steps are **real** and execute for real:
- `seed_branding` — creates `cp_branding_profiles` row
- `seed_modules` — validates plan assignment
- `activate_tenant` — sets tenant status to ACTIVE

## Recovery

### Retry via Owner Portal
1. Go to `/owner/provisioning/[jobId]`
2. Click **Retry Job**
3. Completed steps will be automatically skipped
4. Failed step and subsequent steps will re-run

### Retry via API
```bash
curl -X POST https://{domain}/api/owner/provisioning/{jobId}/retry \
  -H "Cookie: {session_cookie}"
```

### Manual Recovery (if retry fails repeatedly)
1. Identify the failing step
2. Complete that step manually (e.g. configure Supabase project via dashboard)
3. Update `cp_provisioning_job_steps` to COMPLETED manually
4. Retry the job — it will skip the manually-completed step

```sql
-- Mark a step as manually completed (use with caution)
UPDATE cp_provisioning_job_steps
SET status = 'COMPLETED', 
    completed_at = now(),
    output_data = '{"manually_completed": true, "note": "Completed by operator"}'
WHERE job_id = '{jobId}'
  AND step_name = '{stepName}';
```

## Post-Recovery Verification
- [ ] Tenant status = `ACTIVE`
- [ ] All provisioning steps = `COMPLETED` or `SKIPPED`
- [ ] Branding profile exists in `cp_branding_profiles`
- [ ] Tenant can log in
- [ ] Entitlement check returns correct features