Agentic test pyramid

The agentic test pyramid is a verification architecture for agent-produced work that stratifies checks by determinism rather than by test scope: a wide base of deterministic gates that run identically on every artifact, a middle tier of model-based judgment for the dimensions a rule cannot express, and a deliberately scarce apex of human review reserved for what only judgment can certify.

How it works

The base is deterministic: builds, type checks, schema validation, and rule gates that run the same way on every pass, fail closed, and do not care how confident the agent sounded. The middle is model-based judgment, a judge model scoring output against a rubric or a review agent reading a diff, which reaches nuance a rule cannot express but is probabilistic and sensitive to how the rubric is phrased. The apex is human review, the scarcest and most fatigable layer, reserved for taste, architecture, intent, and the failure modes nothing below it can articulate. Each layer filters for the one above it, so the human reads a small, pre-screened surface instead of everything the agent produced. Vendor engineering guidance is consistent with the shape from two angles: containment designed at the environment layer first, and evals that prove reliable graduating into regression suites that run continuously. What makes the pyramid agentic is the volume assumption: where agent output scales like machine output, any layer that cannot run at machine rate must be placed where its scarcity is a feature rather than a bottleneck.

Why it matters

The classic pyramid assumed a human wrote the code and tests verified it; under agent authorship the question inverts to how much of the verification can be trusted to machines, and the determinism axis is the honest answer to what each layer can be trusted with. Research keeps finding real ceilings in the middle tier: judge verdicts can shift under nothing more than rephrasing, and even the best current judges miss a large share of seeded failures on evidence-based agentic evaluation benchmarks. That is the argument for the shape: a middle tier that cannot be the floor needs a deterministic floor beneath it, and an apex that cannot scale needs the layers below to keep it scarce. Teams that underbuild the base compensate with review queues that grow with agent volume, the ice-cream-cone shape classic testing already named, manual checking piled on top of too little automation, only faster. The honest limit is that the base only catches what it can articulate: a gate-passing artifact can still be wrong on a dimension no rule covers, which is why the upper layers exist at all, and the base itself runs thin where the artifact has no stable schema or executable check. Read the pyramid as a placement map drawn on top of the risk map: scope still tells you where failure has blast radius, determinism tells you which verifier earns the first pass at it, and a recurring failure class migrates down a layer once it can be expressed at acceptable error rates.

In practice

An agent ships a change: the build, the type check, and a set of rule validators run first and fail closed on anything they can name. A judge model then scores the change against a pinned rubric and flags one section as off-spec. The human reviews the flagged section and the consequential parts of the diff rather than the whole artifact, a scoping the team earned by first calibrating the judge's flags against human-reviewed samples. Most of the verification ran at machine rate; the scarce human attention landed on the one place where judgment couldn't be delegated.

Practical considerations

Place each check at the most deterministic layer that can express it, and treat every escalation upward as a cost to be justified. Calibrate the middle tier before trusting it: check a judge's agreement against a sample of human verdicts, pin and version the rubric, and expect to recalibrate when the underlying model changes. Keep the apex scarce by design, since human review absorbs growth badly; the base is where volume should land. Route recurring failures downward, because once a failure class can be expressed as a rule it stops being a judgment call and starts being a validator. Watch for the inverted shape creeping back, which presents as review queues growing with agent volume while the deterministic suite stays thin. The layers also fail differently: a well-instrumented validator fails loudly while a drifting judge fails quietly, so the middle tier deserves its own periodic baseline checks and the base deserves negative fixtures that prove it still bites.

Defined by Ready Solutions AI