The Verification Bound: How Much Evaluation Does an Agent Actually Need?

A first-principles model of the economics of agent evaluation — how many test cases buy enough confidence to ship, why more is sometimes worse, and the contrarian lever that beats a bigger benchmark

Brian James Curry · Vector1 Research Fourth in a series on the layer around the model.

“You can’t test quality into a product.” — a maxim from manufacturing that the agent era is busy rediscovering the hard way.\

Abstract

The first three installments of this series each landed on the same meta-thesis: the binding constraint in applied AI is rarely the model itself but the layer around it — the causal layer beneath agentic optimization, the memory layer behind long-horizon recall, the autonomy layer that decides when to let an agent act unsupervised. This installment closes the loop on the last of those, because the autonomy piece ended on a promise: as work shifts from doing the task to checking the agent, checking becomes the job. This is a model of checking.

I treat pre-deployment evaluation as the purchase of confidence and ask the only question a budget owner actually cares about: how many test cases should you run before you ship? I derive a closed-form optimum, N* = (4/δ²)·W(2K²), where W is the Lambert function and δ is the discriminating power of a single evaluation case. The model yields a three-region phase diagram — skip the eval, batch eval, ship and monitor — and a contrarian result with a clean proof: the dominant lever is the discrimination of your eval, not its volume. Halving discrimination quadruples the cases you need for the same confidence, while a tenfold increase in stakes raises the required volume only logarithmically. The closed form matches a Monte Carlo simulation of the full decision process to within Monte Carlo error.

1. The reliability problem nobody puts in the budget

The agent field has spent two years optimizing the wrong moment. Benchmarks report how often a model gets a task right once — and that number has been climbing impressively. The number that decides whether you can deploy is different: how often it gets the task right every time you run it.

The gap between those two is brutal. On τ-bench, the Princeton/Sierra benchmark that simulates real customer-service workflows, a leading model around its release cleared roughly 61% of retail tasks on a single attempt — but the probability it solved the same task correctly across eight independent runs fell below 25%. Run a task eight times and there is only about a one-in-four chance the agent handles it perfectly every time. For a system touching millions of interactions, that is not a deployable number; it is a liability with a good demo.

This pattern is not an artifact of one benchmark. A 2026 reliability-science study of long-horizon agents, measured across more than 23,000 episodes, found that reliability degrades super-linearly with task complexity — the longer and more structured the task, the faster the failure rate compounds, and that decay is invisible to any benchmark that reports only single-shot accuracy. Meanwhile the methodology literature has been issuing a quieter warning: evaluation scores themselves are noisy. Work on putting error bars on evals (Miller, 2024) and on quantifying benchmark variance has shown that many published differences between systems are statistically indistinguishable, and that LLM-as-judge pipelines add their own measurement noise on top.

So the practitioner faces two costs at once. Under-evaluate, and you ship an agent whose true reliability you never measured — and pay for it in production, where the stakes are real. Over-evaluate, and you burn money and calendar time on test cases that no longer change the decision — and τ-bench’s own authors note that running a single trial of one model pairing across their task set cost on the order of a few hundred dollars in API calls. Evaluation is neither free nor free of diminishing returns. That makes “how much is enough?” an optimization problem, not a slogan. Let’s solve it.

2. Evaluation is the purchase of confidence

The right frame is sixty years old. Ronald Howard’s information value theory (1966) established that information is worth exactly the improvement in expected outcome it produces by changing a decision — and not a cent more. Abraham Wald’s sequential analysis (1947), built for wartime munitions inspection, asked the structurally identical question we are asking: how many samples must you draw before you can accept or reject a batch with acceptable error? An evaluation suite is an acceptance test for an agent. Every test case is a sample drawn at a price, and you keep buying samples only while the confidence they buy is worth more than they cost.

That framing has an immediate and underappreciated consequence. The value of a test case is not in its existence but in its power to change your mind. A case that a good agent and a bad agent both pass — or both fail — tells you nothing and costs the same as one that cleanly separates them. This is the quantity that will turn out to dominate everything, so it deserves a name and a number.

3. A model of the verification bound

Set up the decision cleanly. An agent has an unknown true per-task error rate. Before deploying, we must decide whether it is good enough — whether its true rate sits below some ship bar — and we make that decision by running N evaluation cases and watching how it does. There are two worlds we might be in: a good agent (error rate p_g) or a bad one (error rate p_b > p_g), with some prior probability on each before we test.

The discriminating power of a single test case is the standardized distance between how a good and a bad agent perform on it:

The per-test discrimination δ: how many standard deviations of signal one evaluation case provides. Sharp evals — clean rubrics, decisive tasks, low grading noise — have high δ; flaky, ambiguous, or judge-graded cases have low δ.

This is a signal-detection quantity — the d′ of the evaluation literature (Green & Swets, 1966), the same lens the memory installment used for retrieval. A single case carries δ units of signal; N independent cases carry δ√N, because signal accumulates as the square root of sample size. The two worlds' observed scores are therefore δ√N standard errors apart, and with a sensibly placed decision threshold the probability of a wrong call — shipping a bad agent, or rejecting a good one — is:

Decision-error probability as a function of eval volume. Φ is the standard normal CDF. The error falls, but with the punishing slowness of the square root: to halve the exponent you must quadruple N.

That square root is the whole story in miniature. It is why you cannot test your way to certainty cheaply, and why the quality of each test — δ — matters more than how many you run.

4. The cost of confidence

Now price the decision. Each test case costs c — and c should bundle everything a case actually consumes: compute, the time of whoever or whatever grades it, and, critically, the delay it imposes on shipping a good agent, since calendar time is the most underpriced line item in evaluation. Against that, a wrong decision costs S, the stake: the probability-weighted exposure of shipping a bad agent into production (errors × volume × per-error harm) plus the smaller cost of wrongly rejecting a good one. Total expected cost as a function of how much we evaluate:

Two terms in tension. Eval cost rises linearly with N. Residual risk falls — but ever more slowly. Their sum is what you actually pay.

The cost of confidence has a floor and a wall. Early test cases buy steep reductions in risk; past the optimum, you are paying linearly for risk reduction that has gone asymptotically flat. The shaded zone to the right is real, common, and expensive: over-testing is net-negative. Worked example: a good agent failing 5% of tasks versus a bad one failing 15% (δ ≈ 0.33), with stakes 330× the cost of a test case.

The shape is the point. There is a genuine interior minimum — a N* that is neither zero nor "run the whole benchmark again." To the left of it you are under-confident and exposed; to the right you are buying decimal places of certainty that no longer change whether you ship. Most teams I have seen live somewhere on the right wall, mistaking the activity of evaluation for the value of it.

5. The optimum, in closed form

Setting the marginal cost of one more test equal to its marginal risk reduction gives the first-order condition:

Buy test cases until the next one’s price equals the risk it removes. φ is the normal density.

This is transcendental, but it solves exactly through the Lambert W function (the inverse of x·eˣ), which is precisely the tool for equations where the unknown appears both inside and outside an exponential:

The verification bound. The optimal number of evaluation cases, in closed form.

Everything important is visible in the structure. The prefactor is 1/δ² — the optimal eval volume scales with the inverse square of discrimination. The stake-to-cost ratio S/c enters only inside the Lambert W, and for any non-trivial stake the asymptotics of W reduce to a logarithm:

The asymptotic law. Required evaluation grows only logarithmically in the stakes, and quadratically in the inverse of discrimination.

Read those two facts together and the strategic implication is stark. If the stakes of a wrong decision rise tenfold — a far larger deployment, a more dangerous failure mode — the eval you need grows by a constant additive amount, not tenfold. But if your evaluation is blunt — low δ — the volume you need explodes quadratically. Stakes are a logarithmic lever. Discrimination is a quadratic one. That asymmetry is the thesis, and it has a clean proof.

6. Three regimes

Because N* is a function of just two numbers — discrimination δ and the stake-to-cost ratio S/c — the entire space of evaluation decisions fits on one diagram.

The whole landscape of “how much to evaluate” on two axes. Three regimes, separated by where the optimal eval volume falls.

Faith (skip the eval). When stakes are low relative to cost, or discrimination is so poor that no feasible test changes the decision, the optimum is fewer than one test case. Running an eval here is theater — it consumes budget and produces no decision-relevant information. Ship, and spend the saved effort elsewhere. Most internal tooling lives here and should stop apologizing for it.

Batch eval (test, then ship). The broad middle, where a finite, knowable N* is worth running before deployment. This is where formal evaluation earns its keep, and where the closed form tells you exactly when to stop. Notice how wide this region is at high δ: a sharp eval keeps you in cheap, decisive batch territory even as stakes climb toward the top of the chart.

Continuous monitoring (ship, then watch). The high-stakes, low-discrimination corner, where the optimal N* exceeds any feasible pre-deployment batch. Here the lesson is not "test more" — you can't test enough up front — but to change strategy entirely: deploy behind guardrails and evaluate continuously in production, amortizing the cost of verification against live signal rather than trying to buy certainty in advance. This is the eval analogue of the memory installment's third regime, where a cheap recency window beat a bad retriever: when the up-front instrument is too weak, you stop front-loading and start streaming.

The boundaries are not vibes; they are the level sets N* = 1 and N* = N_max computed from the closed form. The worked example sits comfortably in batch territory at about forty cases.

7. The lever: sharpen the eval, don’t enlarge it

Here is the result that should change how teams budget. The separation between a good and a bad agent after N cases is δ√N. To reach a given confidence, that product must hit a target — so if you halve the discrimination of your eval, you must quadruple the number of cases to stand still:

The same-confidence law. It is exact, not asymptotic. A blunt eval is not a little worse than a sharp one; it is quadratically worse.

Two evaluation suites chasing the same confidence. The sharp eval (δ = 0.33) reaches a target residual risk in about 40 cases; the blunt eval (δ = 0.17) needs about 160 — four times as many — to land in the same place, and pays four times the cost and delay to get there.

The industry’s reflex is to make benchmarks bigger. The math says the high-leverage move is to make each case sharper: design tasks where a good agent and a bad agent visibly diverge, write rubrics that grade decisively rather than mushily, strip out flaky cases that add variance but no signal, and reduce judge noise — because every bit of grading noise drives δ down and N* up by its square. The recent move toward variance-reduced and actively-sampled evaluation (e.g., 2026 work achieving the same confidence interval with several-fold fewer queries) is exactly this lever pulled in practice: not more questions, but more informative ones.

Why the lever asymmetry matters at scale. Each curve is the optimal eval volume as stakes grow. Raising discrimination from 0.15 to 1.0 collapses the entire curve; raising the stakes a thousandfold walks you only gently up a single curve. You cannot out-spend a blunt eval — you can only sharpen it.

This also reframes a finding the agent community keeps rediscovering empirically: that the architecture around the model — how it plans, recovers from errors, and applies context — often moves reliability more than swapping the underlying model does. In this model, those architectural choices are partly a δ-and-p story: a better scaffold both lowers the true error rate and makes good and bad configurations easier to tell apart. The leverage was never only in the weights.

8. Does the math hold?

A closed form is a hypothesis until it survives a simulation that makes none of its approximations. I ran the full decision process directly — drawing a true world (good or bad agent) according to the prior, generating N noisy evaluation signals, applying the ship/reject rule, and tallying realized cost over 400,000 trials per setting — and compared the simulated cost curve and decision-error rates against the analytic predictions.

N C(N) analytic C(N) Monte Carlo β analytic β Monte Carlo 10 108.70 108.39 0.2991 0.2985 20 95.25 95.91 0.2280 0.2305 40 (≈N*) 88.15 88.03 0.1459 0.1455 50 89.37 89.43 0.1193 0.1193 80 102.45 102.58 0.0680 0.0685 150 156.80 156.82 0.0206 0.0207

The analytic optimum (N* = 39.82 from the Lambert-W form) matches the numerical minimizer to four decimals, and the false-ship probabilities track the simulation to within a few ten-thousandths across the whole curve. The model is not a metaphor; it is a calculator.

9. What this means for how we build

Strip the math away and four working rules remain.

Stop measuring evaluation by how much you run. The activity of evaluating is not the value of it. The value is the change in your ship decision, and past N* that change is zero while the bill keeps climbing. Know where your wall is.

Spend the marginal dollar on sharpness, not size. Because discrimination is a quadratic lever and volume is a square-root one, the highest-return investment in your eval program is almost never “more cases.” It is decisive tasks, clean rubrics, and lower grading noise. A small, sharp eval beats a large, mushy one — and beats it quadratically.

Match the strategy to the regime. Faith for the low-stakes tail; batch eval where a knowable N* is affordable; continuous monitoring when no batch can reach confidence. The most expensive mistake is running a batch eval in a regime that calls for monitoring, and calling the resulting false confidence "tested."

This is the verification layer, and it is now the job. The autonomy installment argued that work shifts from doing the task to checking the agent. This is the economics of that check. Across four pieces the pattern has held without exception: the causal layer makes agentic optimization trustworthy, the memory layer makes long-horizon agents affordable, the autonomy layer decides when to let them act, and the verification layer decides when to believe them. The model keeps improving. The leverage keeps living in the layer around it.

A note on what this model does not capture

Honesty about the boundaries is part of the method. The clean closed form assumes decision errors are governed by a normal approximation with a well-placed threshold; under strongly asymmetric stakes the optimal threshold shifts (a Neyman–Pearson/Wald point), and the symmetric Φ(−δ√N/2) should be read as the balanced-error case. More seriously, the model assumes test cases are independent and identically distributed draws of a stable quantity — and three real-world facts strain that. First, production distributions drift, so a pre-deployment δ decays after launch; this is itself an argument for the monitoring regime. Second, agent failures are heavy-tailed — a rare catastrophic action dominates expected loss — and when the loss distribution has fat enough tails, the central limit theorem that underwrites the normal approximation can mislead, a caveat the 2025 evaluation-methodology literature has made forcefully. Third, the model assumes the agent is not adversarial; the sandbagging literature shows systems can strategically underperform on evaluations they recognize, which corrupts δ in a way no sample size fixes. Each of these is a direction the model points toward rather than resolves, and each is a candidate for the next installment. The value of a simple model is not that it is complete — it is that it tells you exactly which complications are worth the trouble.

References

Wald, A. (1947). Sequential Analysis. Wiley.
Howard, R. A. (1966). Information Value Theory. IEEE Transactions on Systems Science and Cybernetics.
Green, D. M., & Swets, J. A. (1966). Signal Detection Theory and Psychophysics. Wiley.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Erlbaum.
Raiffa, H., & Schlaifer, R. (1961). Applied Statistical Decision Theory. Harvard.
Yao, S., et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045 (ICLR 2025).
Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.
Madaan, L., et al. (2024). Quantifying Variance in Evaluation Benchmarks. arXiv:2406.10229.
Vendrow, J., et al. (2025). Do Large Language Model Benchmarks Test Reliability? arXiv:2502.03461.
(2025). When Not to Rely on the Central Limit Theorem in LLM Evaluation. ICML 2025, Position Paper Track.
van der Weij, T., et al. (2025). AI Sandbagging: Language Models Can Strategically Underperform on Evaluations. arXiv:2406.07358.
Evaluation and Benchmarking of LLM Agents: A Survey. (2025). KDD ‘25.
Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents. (2026). arXiv:2603.29231.
Efficient Evaluation of LLM Performance with Statistical Guarantees. (2026). arXiv:2601.20251.

Code: sim4.py reproduces the closed form, the numerical optimum, and the Monte Carlo validation table. Drop it into a Gist where marked.

Brian James Curry is the founder of Vector1 Research, an independent lab working at the intersection of marketing economics, causal inference, cognitive AI systems, and the economics of intelligent automation. This is the fourth piece in a series modeling the layer around the model. Reach him at brian@vector1.ai or vector1.ai.

Tags: #AIAgents #Evaluation #MLOps #AgentReliability #AppliedAI #DecisionTheory #Vector1 #LLM #AIArchitecture #AppliedResearch