Contract Testing Plan: From OpenAPI to CI

According to Postman's 2025 State of APIs report, 60% of API-first teams ship at least one breaking change per quarter that their spec did not account for. The root cause is almost always the same: the OpenAPI document exists, but nothing enforces it. This guide covers how to turn your OpenAPI spec into a living contract test suite, which tools fit which scenarios, how to wire them into CI, and what to do when tests fail in consumer versus provider pipelines.

Published on 2026-03-12 · ✓ Updated 2026-03-20 · 7 min read · Author: Daniel Marsh · Review policy: Editorial Policy

The gap between spec and implementation

Spec drift — where the implementation diverges silently from the OpenAPI document — is one of the most common failure modes in API-first teams. It happens because the spec is written upfront, implementation happens over several sprints, and there's no automated check that the two stay in sync. By the time a consumer integrates against the spec, the actual server behavior has already drifted.

A contract testing plan closes this gap. It converts the spec from a static document into an executable test suite that runs on every commit. Three decisions to make upfront: which tool, which direction (consumer-driven or provider-driven), and how failures are reported and blocked.

Three tools, three philosophies

The main tools for OpenAPI-based contract testing are Dredd, Schemathesis, and Pact. They solve different problems:

Dredd reads your OpenAPI file and sends example requests from the spec to a running server, then validates the responses against the documented schemas. It is good for simple provider-side validation, though it only tests what examples you've written into the spec.

Schemathesis is a property-based testing engine that generates many request variations from the schema and looks for crashes, 5xx responses, and schema violations. It excels at finding edge cases the spec examples missed, but does not test consumer behavior.

Pact takes a consumer-driven approach. Each consumer publishes a pact file describing the interactions it expects, and the provider CI runs all consumer pacts against the real server. It works well for multi-consumer APIs where each consumer has different expectations, though it requires consumer teams to maintain pact files.

Starting with Dredd for quick provider validation

Dredd is the easiest entry point if you have an OpenAPI 3.x spec and a running test server. Add it to CI with a minimal configuration:

# dredd.yml
dry-run: false
hookfiles: ./dredd-hooks.js
language: nodejs
sandbox: false
server: npm run start:test
server-wait: 3
endpoint: http://localhost:3000
blueprint: ./openapi/api.yaml
reporter: [cli, junit]
output: [./reports/dredd.xml]

In CI, the step becomes:

# .github/workflows/contract-test.yml
- name: Run Dredd contract tests
  run: |
    npx dredd openapi/api.yaml http://localhost:3000 \
      --reporter junit \
      --output reports/dredd.xml

A Dredd failure means the live server returned a response that does not match the OpenAPI spec. That is a contract violation, not a unit test failure — treat it as a blocking signal.

Adding Schemathesis for property-based edge case coverage

After Dredd catches obvious spec mismatches, Schemathesis finds the paths Dredd's hand-written examples missed. It generates hundreds of request variations — null values, empty strings, boundary integers, extra fields — and looks for responses that violate the spec or return 5xx status codes.

# Run Schemathesis against a running test server
schemathesis run openapi/api.yaml \
  --base-url http://localhost:3000 \
  --checks all \
  --stateful=links \
  --junit-xml reports/schemathesis.xml

Schemathesis will find cases like: the spec says a field is nullable but the server panics when it receives null. Or the spec defines a maximum string length of 255 but the server throws a 500 on a 256-character input. These are real bugs, and they are the kind that slip through example-based tests.

Pact for consumer-driven contracts across teams

When multiple teams consume your API — a web frontend, a mobile app, a third-party integration — Pact shifts the testing model. Instead of the provider deciding what to test, each consumer publishes a pact file describing what interactions they depend on. The provider runs all consumer pacts on every deploy.

// consumer pact definition (JavaScript)
const { Pact } = require('@pact-foundation/pact');

provider
  .addInteraction({
    state: 'an order with id 42 exists',
    uponReceiving: 'a GET request for order 42',
    withRequest: { method: 'GET', path: '/v1/orders/42' },
    willRespondWith: {
      status: 200,
      body: { id: 42, status: 'pending', total_cents: 1500 }
    }
  })

The provider runs this pact in its own CI pipeline using pact-verifier. If the provider changes the response shape — removes total_cents, renames status — the pact verification fails before the change can be deployed. The consumer team gets notified before their code breaks in production.

What a contract test failure actually means

Not all contract failures are equivalent. The response to a failure depends on which side broke the contract:

A provider failure from Dredd or Schemathesis means the implementation does not match the spec. The provider team must fix the implementation or update the spec through a formal change review. The PR cannot merge.

A consumer pact failure from Pact means the provider changed behavior that a consumer depends on. The provider team must either not ship the change, negotiate a versioned migration with the consumer team, or both parties update their pacts simultaneously.

If spec drift is discovered during CI, it means the spec was updated without running tests. Add a pre-commit hook that runs a quick schema lint so this is caught locally before it reaches CI.

Handling spec drift in long-running projects

Spec drift accumulates when engineers fix bugs directly in the implementation without updating the spec. The fix is to make the spec the merge gate, not an afterthought. Two practices help:

First, add an OpenAPI linting step to CI that runs before contract tests. Tools like Spectral catch structural problems and custom rules can enforce team conventions:

# .spectral.yml
extends: spectral:oas
rules:
  operation-success-response: error
  no-$ref-siblings: error
  oas3-api-servers: warn

Second, require spec updates in the PR template. Make "Does this change affect the OpenAPI spec?" a checklist item that the author must answer before requesting review.

Wiring the full pipeline in CI

A complete contract testing pipeline runs in this order: lint the spec, start the test server, run Dredd for schema compliance, run Schemathesis for edge cases, run Pact verifier for consumer contracts. Each step gates the next.

# .github/workflows/contract-tests.yml
jobs:
  contract-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Lint OpenAPI spec
        run: npx spectral lint openapi/api.yaml
      - name: Start test server
        run: npm run start:test &
      - name: Wait for server
        run: npx wait-on http://localhost:3000/health
      - name: Dredd schema compliance
        run: npx dredd openapi/api.yaml http://localhost:3000
      - name: Schemathesis property tests
        run: schemathesis run openapi/api.yaml --base-url http://localhost:3000 --checks all
      - name: Pact provider verification
        run: npm run pact:verify
        env:
          PACT_BROKER_URL: ${{ secrets.PACT_BROKER_URL }}
          PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }}

Choosing the right tool for your stage

Not every team needs all three tools from day one. Start with Dredd if you have a spec and want fast feedback that your server matches it. Add Schemathesis when you want deeper edge case coverage before a major release. Add Pact when you have multiple consumer teams who need independent deployment confidence.

The contract testing plan should be in the spec itself — not a separate document. State which tools are required, which pipelines they run in, and what constitutes a blocking failure versus a warning. When the testing strategy is written in the spec before implementation starts, everyone who works on the feature knows the bar they need to meet.

When contract tests find bugs the spec already permits

Sometimes a contract test fails because the spec itself is wrong — it permits behavior that should not be allowed, or it omits a response variant that the implementation correctly handles. This is a spec update, not an implementation fix. The process is: update the spec, get it reviewed, then update the tests.

Never skip the review step just because the test is "obviously wrong." Contract tests that can be silenced without review aren't contract tests — they're assertions with an off switch. Keep the review gate in place.