A verification stack in four ideas
Five layers, where each behaviour gets proven, what becomes of test specifications and manual testing, and the loop that keeps the stack alive.
Everything we know about verifying an agent’s work fits in four ideas. The rest is technique, to be looked up when needed rather than held in the head.
- One principle. The autonomy you grant an agent equals your capacity to verify. Investing in verification is buying autonomy.
- One structure. A chain of gates from cheap-and-certain to expensive-and-rare: deterministic machine checks, then a model as triage, then a person, who answers the two questions only a person can and closes what the triage flagged, then reality. The only operating rule: push everything down to the lowest layer that can settle it.
- One allocation rule. Each behaviour is proven once, at the lowest layer able to prove it, with selective coverage above for the wiring that joins behaviours together. Manual testing takes only what cannot be specified.
- One maintenance loop. Every defect that escapes must answer “which layer should have caught this”, and that layer is patched permanently. The machine is fed by its own failures.
Mutation testing, the policy on flaky tests, test-first for bug fixes, a test specification rendered from the acceptance criteria, acceptance testing as a confirmation ritual: all of it is the execution of those four. Here is the execution.
The five layers
Deterministic gates. Type check, build, lint, unit and integration tests, contract tests, licence and security scans, and the architecture fitness functions, since a rule the machine does not enforce does not exist for an agent. The greatest value of this layer is not catching errors. It is that it makes the agent’s self-correction loop run: binary feedback, fast, no person needed, the agent fixes until green and only then does a person look. That makes the speed of this layer a strategic asset. A forty-minute pipeline kills the agent loop; tests that run in a few minutes have a different return on investment than they used to. And a new enemy rises to the same rank: the flaky test. At human cadence a flaky test is an annoyance. At agent cadence, with the loop running dozens of times, it is poison: the agent fixes faults that do not exist, or learns to distrust the signal. The policy has to be ruthless: a flaky test is quarantined at once, with an owner and a deadline.
Checking the checker. The fatal weakness of the first layer in this era: if the tests were written by the same agent, in the same context, as the code, they share its wrong assumptions, and green proves only that the code matches the agent’s reading. Four remedies, in order of value. Test-first for bug fixes, with the red test written before the fix and a hook that locks the test files during the fixing task, blocking reward hacking by structure rather than by trust. Tests anchored to the acceptance criteria rather than to the code: the input-and-output table in the criteria becomes the test cases directly, which is where the quality of the spec pays its dividend, since the checker is only as good as the specification it checks against. Property-based testing: instead of listing examples, state the invariant, “the total after splitting equals the total before”, and let the machine generate a thousand cases; it suits the pure core the design separated out. And mutation testing, to measure whether tests are real or decorative: inject faults into the code and see whether the tests go red. Ninety percent coverage with a forty percent mutation score is a suite performing a play. Never make coverage a target for an agent; it is the textbook Goodhart case, and an agent reaches any coverage number in an afternoon with tests that assert nothing.
An independent verifier. A second agent, with a fresh context, that reads only the spec, the criteria and the plan, never the reasoning of the agent that wrote the code, and plays the adversary: checking the diff against the criteria line by line, hunting uncovered cases, trying to break it. For a user interface it is a browser agent that actually clicks through the flow and compares screenshots. Independence is the whole value of this layer; a verifier that can read the maker’s thinking is the maker grading itself under a different signature. And set the expectation correctly: this layer is triage, not a gate. A model as evaluator has blind spots for a model’s own mistakes, so “the review passed” is never the merge condition. Its job is to rank severity, point human attention at the right place, and catch the semantic class of error the first layer cannot see.
A person. Two questions only a person can answer remain, does it match the intent and does it fit the architecture, plus the findings the verifier could only flag: a correctness doubt it could not settle, a security pattern, a concurrency risk, a test that asserts too little. The deterministic layers settled what they can prove; the person settles what they cannot. Allocate by risk tier, as the pipeline piece set out. The health measure of the whole stack lives here: the fraction of diffs reaching a person that the person has to change. High means the layers below are leaking. On the right path it falls week by week, while autonomy is widened in step.
Reality. There is a class of error no previous layer catches, right against the spec while the spec was wrong about the intent, which is the validation gap. So the last layer is a canary or a shadow run, monitoring within bands, and the iron rule of the flywheel: every escaped defect raises one mandatory question, which layer should have caught it, and one permanent addition to that layer: an evaluation case, a new property, a new fitness rule. The verification machine is never designed once. It is fed.
Seen against the chain this series has built, verification is not a final phase but spread along the whole of it. A verifiable criterion is verification at the spec; a contract and a type are verification frozen into the design; a node’s done condition is verification at decomposition; the plan gate verifies the translation of intent; the five layers verify the code; the canary verifies against reality. Each upstream link done well makes the downstream layer lighter, which is why the series went in this order.
Where each behaviour is proven
The question “what goes in unit tests, what in integration, what in end-to-end, what in manual” is the test strategy, and it is the piece the five layers do not answer on their own. The layers say which gates exist. The strategy says where each thing to be checked is cheapest to check. Without it a team checks the same behaviour at three layers, or nobody checks it because everyone assumed another layer would.
The classic frame still holds: a wide base of unit tests, integration in the middle, a narrow top of end-to-end tests, manual above that. The shape comes from three trade-offs: speed, stability, realism. A unit test runs in milliseconds and is never flaky but is far from reality; an end-to-end test is closest to reality and is slow, expensive to maintain and prone to flake. What changed for this era is that the cost of writing tests collapsed, since the model writes them, while the cost of running and maintaining them did not. So the pyramid matters more than before, because the new temptation is to let the model emit five hundred end-to-end tests “to be safe” and receive a fifty-minute pipeline full of flakes, poison for the agent loop. The pyramid is kept in shape now not to save writing effort but to keep the loop fast and the signal clean.
Three rules of allocation.
Prove each behaviour at the lowest layer possible. Calculation, branching, validation: unit, and this is where the pure core pays, since most behaviour in a pure core is testable without standing anything up. The correctness of queries, schema and transactions: integration against a real database, never a mocked one, since a mocked database tests your belief rather than the truth. The joints between modules or services: contract tests, which prove the two sides agree on the shape of the exchange and prove nothing about whether the right side effect happened, so the wiring of a rule into the application still gets selective integration coverage; a unit-tested rule can be correct and connected wrong. End-to-end only for a handful of critical journeys, sign-up, payment, the main business flow, one happy path and one failure path each, counted in tens rather than hundreds. The check: if this behaviour fails, which layer reports the fault fastest and most precisely? A red end-to-end test says something is broken somewhere; a red unit test says this function, this line.
Prove each behaviour once, and the wiring selectively. Proving the same rule at three layers sounds safe and costs twice: the pipeline slows, and every behaviour change has to be fixed in several places, which for an agent means more opportunities to fix the test to match the code. So the rule is exhaustive at the cheapest layer that can prove it, then a thin layer of integration and journey tests whose job is not the rule but the connections: that the rule is called, with the right data, and its effect lands. Allow one exception beyond that, deliberately: money and security flows may be double-covered as an intentional insurance premium.
Manual testing takes what cannot be specified, not what has not been automated yet. The feel of a flow, off-script combinations, “the right thing”: exploratory. But “this case is not automated yet so a tester clicks through it” is debt with a name and a date, not a strategy.
Applied to the password example from the criteria piece: the validity rule, twelve to 128 characters plus the common-password list, is a unit test, property-based if possible; “invalidate every other session within sixty seconds” is integration, since it needs the real session store; the WRONG_PASSWORD error in the right shape is a contract test; the whole journey, a user changes their password and signs in again, is one end-to-end test; whether the form is clear and the message does not frighten anyone is exploratory. The criteria table itself decides the allocation, which is why the right place for the test strategy is a column in the criteria: each line labelled with its layer when the spec is written, and the plan gate checks the labels. The strategy is therefore not a phase or a weighty document. It is one page of convention for the whole project, which kind of behaviour goes to which layer, a ceiling on end-to-end tests, the flaky policy, plus the label on each criterion. Teams that start with the inverted shape, manual thickest, a few end-to-end tests, almost no unit tests, which is the most expensive and slowest shape for agent cadence, should flip it from both ends: build the unit base for the newly written core without trying to cover old code, and retire scripted manual regression on a schedule, with the middle built contract by contract as each one is fixed.
Test specifications, manual testing, acceptance
The picture above leans toward what can be automated, and real projects still revolve around test case documents and rounds of manual testing before anyone signs anything. Neither disappears. Both change roles.
A test case document, the table of steps, data and expected result, used to exist for two reasons: as a quality contract with whoever accepts the work, and as a script for a tester to click through. The first reason stands. The second shrinks hard. What is worth seeing is that the test case document stands in the same place as the acceptance criteria: it is a behaviour specification in table form. So the right flow is now criteria with their input-and-output table, then the test case document generated from them, then the automated tests generated from the same source, three renderings of one truth instead of three hand-written documents that drift apart. The model does this rendering well because it is translation, not decision. Where a test specification with evidence is required, by a client’s acceptance process, a compliance audit or a procurement questionnaire, generate the document from the criteria, run the suite, export the results and the screenshots. One machine serves the pipeline, which needs only a verdict, and the person who has to accept the release, who needs to know which criterion each case came from. What a person keeps doing is reviewing coverage: the table is where “we are missing this case” is easiest to see, far easier than in test code, so it remains a thinking tool while it stops being an execution tool.
Manual testing splits in two, and both halves need saying. Scripted regression, clicking through every old scenario before each release, has to be retired on a schedule, because at an agent’s merge cadence human effort cannot keep up, and keeping it means every bit of speed bought upstream queues to death downstream. Exploratory testing, a person with business intuition hunting faults outside any script, is not only kept but grows, because it aims at the three blind spots of all five machine layers: the feel of a thing wrong, an awkward flow, a strange word, which the machine has no concept of; the off-script combination nobody wrote as a case; and above all validation, “it passes everything and still is not what was wanted”, which only a person using the thing can find. The machine verifies against the specification. Exploratory testing hunts what the specification forgot to say, because it is the part that needs a person who can be surprised. It runs on a cadence, daily or every few days, on a staging environment that is always current, focused by risk tier on what was just merged, not blocking individual merges, since that would make it the new bottleneck. Its findings enter the flywheel like every other defect, with the mandatory question, and any exploratory case that keeps paying graduates into an automated test. Today’s exploration is tomorrow’s regression suite.
So is a separate testing phase needed? Internally, no: everything that can be automated already lives inside the life of each task, and exploration runs on its cadence. Contractually, yes: there is still a named acceptance stage before a milestone is signed, because the person accepting the work needs a point at which to sign. What changes is its inside. Instead of a multi-week examination by a room of people, it becomes a ritual of confirmation on continuously verified ground: the suite run again in front of the acceptor or its evidence exported, plus a round of the acceptor using the thing, which is exploratory testing from the other side. If acceptance keeps finding large defects, that is not a sign that acceptance is important. It is a sign that the five layers are leaking, and each defect it finds is traced back to the layer that should have caught it.
The harder half is not technical. A regression pass staffed by a dozen people was proof that quality was being taken seriously, and the proof was visible, including to people who had never read the code. Six people in a room are visible. An export has to be read. So never retire the visible thing without naming what replaced it: which layer catches what now, how many cases cover it, what ships with each release, and who owns that layer. None of this holds if nobody outside the team ever has to accept anything. If someone does, a release manager, an auditor, a client signing off, the evidence is the only thing that moves them, and it moves them only if they can read it. Where the work is done for a fixed price, the same evidence is also what settles a warranty dispute; the arithmetic of that is its own piece.
In Rindo a test case’s steps are versioned, a run is a fixed batch of cases, a recorded result pins the revision of the steps it exercised and names who recorded it, and when a result is recorded again the audit trail keeps the earlier one; results from a pipeline arrive through an import, a release’s readiness reads the active cases, and steps written by an agent always wait as a draft for a person to approve.