# Reconciliation Exception Playbook

Updated: 2026-03-29
Article: https://spec-coding.dev/blog/postmortem-preventing-billing-incident-with-spec-first

Use this playbook when automated reconciliation between two financial systems (billing, payments, ledger) flags a mismatch. Every exception must be categorized, investigated, resolved, and reported within the SLA for its category.

## Exception Categories

| Category | Description | Example | Resolution SLA |
|----------|-------------|---------|---------------|
| E1 - Missing record | Transaction exists in source but not in target system | Payment captured in gateway but absent from ledger | 4 hours |
| E2 - Amount mismatch | Records exist in both systems but monetary values differ | Invoice shows $120.00, ledger entry shows $12.00 (decimal shift) | 4 hours |
| E3 - Status mismatch | Records exist in both systems but lifecycle state disagrees | Gateway says refunded, billing still shows active subscription | 8 hours |
| E4 - Duplicate record | Same transaction appears more than once in one system | Retry storm created two payment capture records | 8 hours |
| E5 - Timing variance | Records will eventually match but fall outside the reconciliation window | Batch settlement arrives after nightly recon cutoff | 24 hours |

## Investigation Steps

For every exception, follow this sequence before attempting resolution:

1. **Identify the source of truth.** Determine which system is the authoritative record holder for this transaction type. Document your reasoning.
2. **Pull raw data.** Export the full transaction record from both systems, including timestamps (UTC), correlation IDs, and status history.
3. **Check for timing.** Compare transaction timestamps. If the delta is less than the recon window buffer (default: 2 hours), reclassify as E5 and monitor the next cycle.
4. **Trace the mutation chain.** Walk the event log or audit trail to find where the records diverged. Note the exact event that introduced the mismatch.
5. **Quantify impact.** Calculate the total monetary value affected. If aggregate impact exceeds $1,000 or affects more than 10 customers, escalate to finance immediately.

## Resolution Procedures

| Category | Resolution Action |
|----------|-------------------|
| E1 - Missing record | Replay the event from source system; if replay fails, create a manual adjustment entry with dual approval |
| E2 - Amount mismatch | Correct the erroneous system to match source of truth; file a correcting journal entry; root-cause the data transformation bug |
| E3 - Status mismatch | Trigger a state sync from the authoritative system; verify downstream effects (e.g., dunning paused, access revoked) |
| E4 - Duplicate record | Void the duplicate; verify idempotency key implementation; add regression test to contract test suite |
| E5 - Timing variance | No action if resolved in next cycle; if persists for 2 cycles, reclassify as E1 |

## Exception Report Format

File one report per reconciliation run. Use this structure:

```
Recon Run ID:       [auto-generated]
Run Date (UTC):     [YYYY-MM-DD HH:MM]
Systems Compared:   [System A] <-> [System B]
Total Records:      [count]
Matched:            [count] ([percentage]%)
Exceptions:         [count] ([percentage]%)

Exception Breakdown:
  E1 Missing:       [count] | Total Value: $[amount]
  E2 Amount:        [count] | Total Value: $[amount]
  E3 Status:        [count]
  E4 Duplicate:     [count] | Total Value: $[amount]
  E5 Timing:        [count]

Open from Prior Run: [count]
Resolved This Run:   [count]
Net Open:            [count]
```

## Escalation Thresholds

- Exception rate exceeds 0.5% of total records: notify engineering lead and finance
- Any single E2 exception above $5,000: immediate finance escalation
- E1 exceptions unresolved past SLA: page on-call SRE
- Pattern of recurring exceptions on the same transaction type: open a root-cause investigation ticket with 48-hour deadline

## Notes

- Run reconciliation at minimum daily. High-volume systems should run every 4 hours.
- Keep resolved exceptions in the archive for 90 days to support pattern analysis.
- After every P0 billing incident, update this playbook with lessons learned.
