Where people belong in a pipeline run by agents
Six layers, three new bottlenecks, what to measure, and how analysts, testers and product managers move from producing to judging.
If the autonomy you can give an agent is bounded by your capacity to verify, then a development process is really a design for where people stand. Put them where their hour earns the most, protect their attention from everything a machine can settle, and measure the queues rather than the output. Here is the shape we use, layer by layer, and why a person is where they are in it.
Six layers
Intent and specification. The highest-leverage hour in the whole process. The spec is becoming the code: the quality of the requirements, the context and the acceptance criteria now decides directly how well the agent works. One operating rule follows. Review intent before generating code, which means the first review gate sits at the spec, not at the pull request. An hour spent by the analyst or the product manager on the spec saves many hours of review in the wrong direction, because the cost of fixing intent before generation is nearly zero, and after it you pay for a regeneration and a second review.
Generation. Parallelise freely, but constrain the size. Let agents run several threads at once, and enforce small changes from the moment work is assigned by cutting the spec into units that merge independently. Large pull requests kill review capacity faster than anything else. A study of about 34,000 agent-authored pull requests on GitHub found two regimes: agents are excellent at narrow, well-bounded automation, which gets merged at once, and poor at iterative refinement, where they tend to go quiet when the feedback turns subjective, leaving what the authors call an attention tax on the maintainer. The design lesson is not to let every agent pull request queue alike. Predict which ones will be expensive and route them before a person looks.
The deterministic gate. Type check, build, lint, tests, contract checks, the architecture rules. A failure goes straight back to the agent to fix. A person never sees it.
Machine review. A model’s pass over every change, instantly and consistently whatever the depth of the queue: logic errors, security patterns, performance regressions, missing coverage. This is a soft filter. The lesson from the verification piece applies: a model-based evaluator has systematic blind spots for a model’s own mistakes, so machine review is triage, not a gate. It ranks what a person should look at first.
Human review. Two questions only a person can answer, does the change match the intent and does it fit the architecture, plus closing whatever triage surfaced and no deterministic gate settled: an unresolved correctness doubt, a security pattern, a concurrency risk, a test that asserts too little. The person does not re-read for syntax; the person decides what the machine could only flag. Tiered by risk: a change that touches authentication, privacy or sensitive data gets explicit criteria and early routing to a reviewer who can judge it; a change on the edge gets a glance. And one principle of accountability stays: the engineer who prompted the agent owns the output. That rule stops diffs from being thrown over the wall into a queue nobody owns.
Staged rollout. Validation by reality rather than prediction: offline evaluation on curated and adversarial cases, replay against historical cases, simulation of what-if scenarios, a shadow run beside the real process, then a monitored canary with a limited blast radius, each stage with an acceptance threshold set in advance. The person here is the product manager or the domain expert judging outcomes, and the judgement flows back upstream: a defect that escaped becomes an evaluation case, a misread intent becomes an amended spec. That loop is what makes the system learn rather than merely run.
Three new bottlenecks
Put sensors on three places from the first day. The review queue, which is the obvious one. The attention tax of the feedback loop: not the first review but the later rounds, where the agent responds to feedback and then drops the thread, and a person has to pick it up. And the one almost nobody watches, the supply of ground truth: how fast domain experts can produce reference outputs and annotate outcomes bounds how fast your evaluations improve, and weak evaluations blind every machine layer above them.
What to measure
The trap is to measure output. Individuals report large speedups while the organisation stands still, and in one controlled study experienced developers felt faster while measuring slower; the feeling of a new process is as suspect as the feeling of a new tool. Measure the queues before buying anything: time to first review, median pull request size, how concentrated review is on a few senior people, escaped defects, rework, lead time for a change. If the interventions do not move lead time, the problem is no longer review habits; it is the operating model. The last word belongs to the organisation-level measures, read together rather than one at a time: delivery (deployment frequency, lead time), stability (change failure rate, time to restore), and cost (hours and rework per change). A gain that shows in none of them is a feeling; a gain that shows as the same delivery at lower cost, or the same cost at better stability, is real, and only the full set tells those apart.
One quick health check: count the hours in the pipeline spent on work a machine could do. A person reading a diff for syntax errors is waste. A person writing acceptance criteria, deciding a risk tier, or annotating an outcome is in the right place.
From producing to judging
Every role shifts the same way. The change is not that the agent writes code instead of people. It is that people now manage a growing volume of work done by agents, which means designing constraints, validating results and measuring effect, rather than producing each artifact by hand.
The analyst becomes the context architect. This is the role that gains the most, because the analyst stands exactly at the intent gap. Upstream, the classic work, user stories and acceptance criteria, is now direct input to the machine; the quality of the wording sets the quality of the gate. Downstream, the analyst is the oracle on the user’s behalf: validating that what was built matches what was asked, with checks and acceptance tests rather than a hopeful nod, and owning the measures that say whether the change worked. And in governance, the analyst is the person who can say when a model’s recommendation should not be followed, and the domain knowledge that work depends on often lives with whoever ordered it.
The tester moves from checking behaviour to checking meaning. For decades testing meant validating software behaviour against a deterministic specification: click the button, an API call fires, the response matches the schema. That contract dissolves when the product runs on a model whose output is probabilistic and context-sensitive. Testing used to validate correctness; now it has to judge quality, intent and safety, which pushes the role from the verification side to the validation side. Concretely: move the best manual testers out of scripted regression and into exploratory testing, risk analysis and test design, because exploratory testing is pure validation, a person hunting for the expectations nobody wrote down. New shapes of the role appear, the investigator who uses generated test ideas and human curiosity to probe odd combinations, the domain translator who turns business risk into tests, and on model-driven products the reviewer of the model’s own output. The tester enters earlier and sharper, because more features and more hidden risk now arrive at once. And accountability for quality cannot be automated: in a verified loop, the tester is the person who signs the gate the agent is not allowed to sign.
The product manager owns the right thing. Validation ends in a question of value, and that is the product manager’s ground. In a modern evaluation process the product manager and the domain expert annotate whether an outcome is acceptable, testers review the failure cases, and leadership reads the benchmark to see whether quality is rising. Validation is layered by role, not dumped on one person. And since reference outputs must be approved by a domain expert rather than generated by another model, the domain expert becomes the scarcest input in the pipeline. A note on the word: in product companies “PM” means product manager. A project manager does not own the definition of the right thing, but decides whether validation gets time in the plan; teams that lower their test standards to make a deadline are making a project management decision. Where one head holds both jobs, which is common, the two decisions still have to be made separately.
The developer takes on part of both. From implementer to specifier, verifier and orchestrator of systems that do the implementing; the value moves from writing code to defining intent and designing guardrails, and owning test automation becomes a core skill rather than a specialism.
One frame holds it together. The analyst holds the input: is the intent written down clearly enough. The tester holds the middle: does the system show behaviour outside what anyone expected. The product manager and the domain expert hold the output: is the outcome worth having. The developer holds the machinery: the deterministic gates and the evaluations that run. The irony is plain. The soft skills that used to be discounted, interviewing a stakeholder, writing an unambiguous criterion, curious exploration, domain judgement, are now the bottleneck of the whole line, because they are exactly the steps where no proxy can replace the oracle. Read predictions about roles with that in mind, though: most of them come from tool vendors and training companies with an incentive to dramatise the shift. The direction is confirmed from several independent sources. The pace in any given team is not.
In Rindo these jobs have names of their own: a work role such as analyst, tester or reviewer carries an instructions page that is handed to every assistant holding the role before it works, the tasks an agent files land in a Triage queue for a person to sort, and the audit trail records who approved what and through which channel.