Quality Gates for AI-Assisted Development Specs

The quality gates that stop AI-assisted code from shipping unreviewed: pre-prompt spec check, post-generation diff review, test-evidence verification, and the human sign-off rule. Skip all four and a one-line "harmless" change can cost a weekend of rollbacks, because it was not harmless at all.

In practice: the gate that stops a pretty but unsafe diff

The common failure is not ugly generated code. It is polished code with no rollback story. A quality gate should ask for evidence that survives beyond the local test run.

Quality gate row:
Risk class: medium
Required before merge: unit test, integration test, rollback note
Required before release: log query and alert threshold
Release owner: on-call lead
Stop signal: error_rate_refund_worker > 1% for 10 minutes

Four Gates, In Order, Non-Negotiable

AI-assisted work should pass four gates, enforced in order. Gate 1 fires before the model sees a prompt: the spec has to be complete. Gate 2 fires right after generation: the diff has to match what the prompt said it would touch. Gate 3 fires after the code is wired up: tests have to exist and they have to actually fail when the behavior breaks. Gate 4 fires before merge: a named human has to have read the thing, not just clicked approve.

Any team that skips one of these is betting that the other three catch it. That is a worse bet than it looks, because the gaps tend to line up: the vague spec that slips past Gate 1 is the same one that produces the weak tests that slip past Gate 3.

Gate 1: Pre-Prompt Spec Check

Before the model generates a single token, an automated check runs over the spec file. It looks for five sections: problem statement, scope, non-goals, acceptance criteria, risks. If any of them is empty, missing, or reduced to a single sentence like "make it work," the runner refuses to call the model. No spec, no generation. The automated check is deliberately crude — it counts sections, it cannot judge them, so a human pass over the same file still matters: the spec review checklist is what that reader works through.

This sounds bureaucratic until you watch it work. Plenty of "quick tickets" dissolve the moment someone has to write down a non-goal. If a feature cannot survive five headings, it is not ready for AI, and honestly it is not ready for a human either.

The trap is letting Gate 1 become a checkbox. If every spec passes, the gate is not measuring anything. Audit a random sample of passing specs every couple of weeks and look for the ones that used filler text. Kick those back visibly, so the whole team sees what filler looks like.

Gate 2: Post-Generation Diff Review

Right after the model finishes, before any tests run, check the diff against the prompt. Three questions: Did the diff touch only the files the prompt named? Did it add any dependencies nobody authorized? Did it create files outside the declared output paths?

All three can be automated. A script reads the prompt's declared file list, diffs it against the actual changed files, and flags anything unexpected. New entries in package.json, pyproject.toml, or go.mod trigger a hard stop. Generated files in weird directories trigger a hard stop.

A typical failure here is the model quietly adding a utility dependency because "it would be cleaner." Sometimes it would. But that decision should be surfaced, not smuggled.

Gate 3: Test Evidence, Including the Mutation Check

Tests passing is not evidence. Tests passing and then failing when you intentionally break the code is evidence. Require both. The full argument for that distinction, including what the mutation check catches that coverage never does, is in AI coding with test evidence gates.

For every acceptance criterion, there has to be at least one test that maps to it. Writing acceptance criteria in Given/When/Then makes the mapping mechanical:

- Given a user with an expired session token
  When they request a protected resource
  Then the response is 401 and no row is written to the audit log

- Given a payment webhook with a replayed event ID
  When the handler receives it
  Then the handler returns 200 and does not re-charge the card

After the tests pass, run a mutation check on the critical paths. If you can break the function body and the test still passes, the test is a liar. Better to find that out now than during an incident. Gate 3 fails more often than people expect, usually because the AI wrote a test that mocks the thing it was supposed to verify.

Gate 4: The "I Read This" Button

Gate 4 is the one that teams rubber-stamp, and it is the one that saves you. A named human has to read the diff end to end and attest to it. Not "approved," not a thumbs-up emoji. A comment that references specific lines, specific choices, or specific risks. Something that proves they were there.

The reason to be strict about this is that Gates 1 through 3 are machine-checkable, which means they are also machine-foolable. A reasonable-looking spec, a diff that stays in its lane, and a test suite that goes green will sail through. None of that catches a silent behavior change where the code is technically correct but no longer does what the business wants.

A PR That Passed 1-3 And Got Caught At Gate 4

Consider a PR that refactors pricing rounding logic. The spec is complete. The diff is bounded. Tests pass and survive mutation. It looks perfect.

The reviewer on Gate 4 notices that one branch now uses banker's rounding where the old code used half-up. The tests miss it because the test inputs happen to round the same way under both rules. On real production amounts, some invoices would shift by a cent — small per invoice, but enough to become a month-end reconciliation problem, and in a regulated business possibly an audit finding.

Gates 1, 2, and 3 have no opinion about rounding policy. A reviewer who knows how these products are priced reads the diff and asks "wait, why did this change?" That is the gate.

The Rubber-Stamp Problem

Gate 4 dies quietly. Reviewers see green CI, skim the diff, click approve, and move on. After a few months the gate is a vanity stamp and nobody notices until something gets through.

Countermeasures that work: rotate reviewers so the same person is not rubber-stamping the same author every week; require the review comment to cite a specific line or decision; audit a sample of approvals monthly and ask the reviewer to walk through what they looked at. If they cannot, that is data.

Measuring gate health is worth doing. Track how many PRs get rejected at each gate, and more importantly, which rejections were valuable versus which were noise. A gate that rejects nothing is either perfect or broken, and it is never perfect.

What CI Enforces Versus What Humans Enforce

Gates 1, 2, and 3 are CI's job. Spec schema validation, diff scope checks, test-to-criterion mapping, mutation sampling - all of it runs without a human in the loop. If a human has to remember to do these, they will forget.

Gate 4 cannot be automated. The whole point is that a human with context looked at the change. What can be automated is the evidence trail: the reviewer has to leave a comment that references the diff, and a bot records it against the PR. If the audit later asks "who signed off on this," there is an answer.

Bypass Policy

Sometimes you need to override a gate. Production is on fire, the spec is a one-liner, and you need the fix out now. That is acceptable, but with rules: only two named people can bypass, every bypass writes a row to an audit table with the reason, and every bypass generates a follow-up ticket that has to be closed within a week. Without that, "emergency" becomes the default path and the gates stop existing.

The Gate That Probably Should Exist

The four gates above all stop at merge, and that leaves a gap. A post-merge regression gate — a 24-hour smoke check that compares key metrics against the previous day: error rates, latency percentiles, business metrics like conversion — would catch the silent regressions that tests cannot. A change that quietly degrades conversion or nudges error rates up would surface in hours instead of days. It is the natural fifth gate once the first four are real.

What To Actually Do Monday

Pick one gate and make it real this week. If you have nothing, start with Gate 1 because it is the cheapest to automate and the highest leverage. If you have Gate 1, harden Gate 3 with a mutation check on your top ten critical files. If you have all four on paper but they are soft, audit last month's approvals and see how many reviewers can still tell you what they looked at. The answer will tell you which gate to fix first.

AI Review Packet to Copy

Use this when the spec is ready but the AI implementation still needs a gate. The packet defines which proof has to appear before a generated diff can be trusted.

AI coding review packet: Quality Gates for AI-Assisted Development Specs

Decision to make:
- The quality gates that stop AI-assisted code from shipping unreviewed: pre-prompt spec check, post-generation diff review, test-evidence verification, and the human sign-off rule.

Owner check:
- Product owner:
- Engineering owner:
- QA or operations reviewer:

Scope boundary:
- In scope:
- Out of scope:
- Assumption that still needs approval:

Acceptance evidence:
- Test or fixture:
- Log, metric, or screenshot:
- Manual review step:

AI boundary: generated changes must stay inside the written scope and attach evidence for each acceptance criterion.

Reviewer prompt:
- What would still be ambiguous to someone who missed the planning meeting?
- What evidence would make this safe enough to ship?

Gate Rollout Checklist

A quality gate is useful only when it blocks the right change for a clear reason. Roll it out with owners, evidence, and a bypass rule before asking every team to follow it.

Download: ai-quality-gate-policy-v2.md

• Assign one owner for each gate: spec completeness, diff scope, test evidence, and human review.
• Make machine-checkable gates run in CI so reviewers are not responsible for remembering them.
• Require human review comments to cite a line, decision, or risk instead of using a blank approval.
• Log every bypass with reason, approver, expiry date, and follow-up ticket.
• Audit rejected and bypassed changes monthly to remove noisy checks and strengthen useful ones.

Evidence to Track

Track rejection reason, bypass count, review latency, post-merge defect escapes, and whether a blocked PR later produced a clearer spec. A gate that never rejects anything is not automatically healthy; it may simply be invisible.

AI change gate:
- Applies to: auth, billing, permissions, data migration, public API
- Required before merge:
  - acceptance criteria mapped to tests
  - changed files match allowed_files list
  - reviewer checks forbidden_changes list
  - production metric named for first-hour monitoring
- Bypass:
  - only incident commander can bypass
  - bypass expires in 24 hours
  - follow-up spec review is required