A 200-Question Test Set Cannot See a 7-Point Regression

Most GenAI teams ship on impressions. Here are the statistics that turn evaluation into a release gate, with the code on Substack.

Someone changes the system prompt, runs twenty questions, likes the answers and merges. Months later a model version is retired, the replacement behaves differently on a tenth of queries, and nobody notices until a wrong answer reaches an executive.

The fix is to make evaluation the release process itself: versioned test sets of known size, judges calibrated against people, and a statistical comparison with production before anything ships. My full deep dive works through it on Microsoft Foundry with runnable Python. This piece covers the ideas that matter most.

Version what you actually ship

A GenAI release is five things at once: the model version, the system prompt, the retrieval index snapshot, the tool definitions and the guardrail policy. Each changes on its own schedule, and on Foundry each lives in a different place. Hash every component, derive one release ID, and let the difference between candidate and production decide which test suites run. A nightly index rebuild then counts as a release, which it is. The full article includes the manifest code.

Your test set has a resolution

Every evaluation set has a minimum detectable effect: the smallest drop in pass rate it can reliably catch. Near a 90% pass rate, at the usual 5% significance and 80% power, it is roughly 2.487 × √(2p(1 − p) / n).

A 200-row set has a resolution of about 7.5 percentage points. At 1,000 rows it is 3.3 points. So “no regression on our 200 questions” really means “no regression larger than about 7 points”, which for a busy system is hundreds of wrong answers a day.

Pair your comparisons

When the candidate and production answer the same rows, only the rows where they disagree carry evidence. McNemar’s test uses exactly those. Detecting a 3-point drop with about 8% of rows in disagreement takes roughly 548 paired rows, against about 1,397 per arm for an unpaired comparison. Pairing cuts the labeling bill by more than half.

In the worked example, a candidate loses 52 rows and gains 31 out of 1,000, a net drop of 2.1 points with p = 0.028. An unpaired test on the same pass rates would miss it. The 52 lost rows then become a ready-made debugging list. The full article includes the exact-test code and the sample-size formula.

Check the judge before you trust it

LLM judges make big test sets affordable, but a judge is a model with its own error. In the example, a judge agrees with human experts 85% of the time, which sounds fine. Its Cohen’s kappa is 0.571, it is 5 points more lenient than the experts, and it catches only 60% of real failures. A gate built on it would let 2 in 5 failures through.

The full article shows how to measure this on a few hundred labeled rows and three ways to fix it, in order of cost.

Measure retrieval on its own, and watch production

Groundedness judges check answers against whatever was retrieved, so they miss retrieval failures entirely. Measure recall@k and mean reciprocal rank separately. In document-heavy industries add a third metric: how often a superseded revision of a specification or drawing makes the top results.

In production, control charts tell a real dip from noise. Judging 400 requests a day against a 90% baseline gives daily limits of 0.855 to 0.945, so a slow slide to 0.87 never alarms. Pool a week and the lower limit tightens to 0.883, which catches it.

What’s in the Full Article

• A content-addressed release manifest that versions model, prompt, index, tools and guardrails together
• Stratified golden-set design with Neyman allocation
• Judge calibration, McNemar and retrieval-metric code you can run
• Control-chart monitoring and model-routing cost math (a 61% cost cut and what it trades away)
• Where these methods break, and how to keep them vendor-proof

Written for ML engineers and AI platform leads who need GenAI releases to be as repeatable as software releases.

Read the full deep dive on Substack → https://brianjamescurry.substack.com/p/evaluation-is-the-release-process

Subscribe on Substack for the next piece in the series, on running agents in production.

About the Author

Brian Curry is a Kansas City-based machine learning and AI strategist, marketing economist, and applied researcher. He leads Vector1, an independent research studio working on cognitive AI, causal inference, marketing economics, and knowledge engineering, with a focus on simulation-driven intelligence systems.

Over more than twenty years, Brian has led data science and analytics initiatives at enterprises ranging from $4B to $125B in revenue, including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, and AT&T. His work applies economics and causal inference to connect marketing investment to structural outcomes, operational decisions to economic value, and AI adoption to durable competitive advantage.

He is also the founder of KC AI Lab, a 1,500-member AI and ML education community in the Midwest. He lives in Kansas City and is an active part of the Kansas City community.

Research and Open-Source Projects

The modeling approach in this paper draws on PyCausalSim and Papilon.

Causal Inference and Economics

• PyCausalSim: Simulation-driven causal discovery and evaluation, using counterfactual scenarios to isolate structural drivers.
• Papilon: Complex-system simulation and optimization, with applications in media mix modeling and adaptive architectures.
• Daedalus: Causal analytics for money-flow mapping and macroeconomic tracking.
• The Autonomy Economy: A macroeconomic framework for the combined effects of rapid productivity growth and structural price deflation in an AI-automated economy.

Forecasting and Applied Modeling

• Regimecast: Time-series regime detection, structural breaks, and economic forecasting.
• CarryNet: Financial modeling for commodity carry trade analysis.
• Foodmesh-ml: Machine learning, optimization, and simulation for distributed food systems.

Cognitive AI and Semantic Architecture

• Memory-Node Encapsulation (MNE): A cognitive AI architecture and data structure for artificial episodic memory (NeoCortex-M).
• MeaningFlow: Semantic content modeling, embeddings, clustering, and coverage gap analysis.

Contact: brian@vector1.ai | GitHub @Bodhi8 | Substack

The text of this article may be shared with attribution under Creative Commons Attribution 4.0 International (CC BY 4.0). Code and models referenced here are available separately to paid subscribers.