# API Error Runbook

Updated: 2026-03-29
Article: https://spec-coding.dev/blog/spec-first-error-handling-patterns-for-apis

## Purpose

Use this runbook when an API error alert fires or a consumer reports failures. It provides a lookup table, diagnostic steps, and escalation paths so the on-call engineer can resolve issues without guessing.

## Error Code Lookup Table

| HTTP Status | Error Code | Meaning | Likely Cause | First Response |
|---|---|---|---|---|
| 400 | `VALIDATION_FAILED` | Request body or query param failed schema validation | Client sending malformed payload | Check request body against OpenAPI spec; respond to consumer with specific field errors |
| 401 | `AUTH_TOKEN_EXPIRED` | Bearer token past expiry | Client not refreshing tokens | Confirm token TTL; advise consumer to implement refresh flow |
| 401 | `AUTH_TOKEN_INVALID` | Token signature does not verify | Wrong signing key, token from different environment | Check which IdP issued the token; verify `iss` and `aud` claims |
| 403 | `INSUFFICIENT_SCOPE` | Token valid but missing required scope | Consumer provisioned with wrong role | Verify RBAC config; escalate to IAM team if scope should be granted |
| 404 | `RESOURCE_NOT_FOUND` | Requested entity does not exist | Stale ID, deleted resource, wrong environment | Confirm resource exists in DB; check if consumer is hitting correct environment |
| 409 | `CONFLICT` | State conflict (e.g., duplicate create, stale update) | Missing idempotency key or concurrent write | Check `Idempotency-Key` header; review optimistic locking version |
| 422 | `BUSINESS_RULE_VIOLATION` | Request is syntactically valid but violates a domain rule | e.g., cancelling an already-shipped order | Read `detail` field in error body; no infra action needed -- consumer logic issue |
| 429 | `RATE_LIMIT_EXCEEDED` | Consumer exceeded request quota | Burst traffic or missing backoff | Check rate limit dashboard; if legitimate traffic, raise quota in config |
| 500 | `INTERNAL_ERROR` | Unhandled exception in service | Bug in code, nil pointer, missing env var | Check application logs for stack trace; roll back last deploy if correlated |
| 502 | `UPSTREAM_FAILURE` | Gateway received invalid response from upstream service | Upstream service crashed or returned malformed response | Check upstream service health; verify network connectivity between services |
| 503 | `SERVICE_UNAVAILABLE` | Service is up but cannot serve requests | Database connection pool exhausted, dependency circuit open | Check DB connection metrics and circuit breaker state |
| 504 | `GATEWAY_TIMEOUT` | Upstream did not respond in time | Slow query, deadlock, resource starvation | Check slow query log and upstream p99 latency; consider increasing timeout or optimizing query |

## Diagnostic Steps for Any Error Spike

1. **Confirm scope**: Is the error rate elevated globally, or isolated to one consumer / one endpoint?
2. **Correlate with deploys**: Check if a deployment happened in the last 60 minutes (`git log --since='1 hour ago'` on the service repo).
3. **Check dependencies**: Dashboard for database latency, cache hit rate, and downstream service health.
4. **Read structured logs**: Filter by `trace_id` from the consumer's error response to get the full request path.
5. **Verify contract**: Confirm the live schema matches the deployed OpenAPI spec (drift can happen with manual deploys).

## Escalation Paths

| Situation | Escalate To | Channel | SLA |
|---|---|---|---|
| Error rate > 5% for 5 minutes | On-call engineer (L1) | PagerDuty | Acknowledge in 5 min |
| Error rate > 20% or full outage | Engineering lead (L2) | PagerDuty + Slack #incidents | Acknowledge in 2 min |
| Consumer-reported issue, no alert fired | API support team | Slack #api-support | Respond in 1 hour |
| Security-related error (401/403 spike from unknown IPs) | Security on-call | PagerDuty security service | Acknowledge in 5 min |
| Data integrity issue (409s from internal services) | Data platform team | Slack #data-oncall | Respond in 30 min |

## Common Fix Procedures

**Rolled-back deploy caused schema mismatch:**
1. Verify the running container image tag matches the expected release.
2. If mismatched, redeploy the correct version.
3. Run contract tests against the live endpoint to confirm alignment.

**Database connection pool exhaustion (503s):**
1. Check `active_connections` metric vs. pool max.
2. Identify long-running transactions: `SELECT * FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC`.
3. Kill stuck transactions if safe, then investigate root cause.

**Rate limit misconfiguration (unexpected 429s):**
1. Check the rate limit config for the affected consumer key.
2. Compare with the documented quota in the API agreement.
3. Adjust limit and deploy config change; no code deploy needed.

## Monitoring Alert Response

| Alert | Dashboard Link | Runbook Step |
|---|---|---|
| `api_error_rate_5xx > 5%` | Grafana: API Health | Run Diagnostic Steps 1-5, escalate if not resolved in 15 min |
| `api_latency_p99 > 2s` | Grafana: API Latency | Check slow query log and upstream health |
| `api_error_rate_429 > 10%` | Grafana: Rate Limits | Review rate limit config; check for bot traffic |
| `api_auth_failure_spike` | Grafana: Auth | Check IdP status; review token issuance logs |
| `contract_drift_detected` | CI: Contract Tests | Compare deployed spec with repo spec; redeploy if mismatched |

## Notes

- Update this runbook after every postmortem. If the team encountered a new failure mode, add it to the lookup table.
- Keep revision history with date, owner, and reason for change.
