Your AI Metric Is Probably Lying to You
Four new papers, one simulation, and a test for knowing when a proxy stops meaning anything
Autonomous coding agents raise commit volume by 240 percent. They raise shipped releases by 30 percent.
Both numbers come from the same study of more than 500,000 GitHub developers (Demirer, Musolff, and Yang, NBER Working Paper 35275). Which one ends up in the board deck depends on who makes the slides. The gap between the number a system is optimized on and the outcome it was supposed to represent is one of the most underpriced risks in applied AI.
This week’s research on that gap spans four fields. The full technical write-up, with the math, the causal inference tools, and runnable code, is on my Substack. Here are the three ideas worth taking away even if you never click through.
1. Report the attenuation alongside the lift
In the coding study, the effect of autonomous agents shrinks at every stage of the pipeline: +240 percent at commits, +80 percent at projects, +30 percent at releases. Roughly seven eighths of the headline gain is gone before code ships.
The authors also estimate an elasticity of substitution between AI and human effort of 0.23. In production-function terms that is close to perfect complements. Output is capped by the scarcer input, and the scarce input is now human review and release capacity. More generated code stops helping once verification is the bottleneck.
The practical rule: whenever you report a first-stage gain, report the ratio at each downstream link as well. The link where the ratio collapses is where your proxy stops carrying value.
2. Harder optimization can make outcomes worse
It is tempting to assume a noisy proxy only makes results less precise. That holds when the noise is well behaved. It fails when the proxy’s errors have heavier tails than the signal.
A simple Monte Carlo shows the difference. Each candidate has a true value, the optimizer sees only the true value plus error, and it picks the best of n candidates. With Gaussian error, harder search keeps helping. With heavy-tailed error, the true value of the pick peaks at around 16 candidates and falls to roughly zero at 16,384, while the proxy score keeps climbing past 150.
OpenAI researchers found the same shape when optimizing language models against learned reward models (Gao, Schulman, and Hilton): the proxy reward rises steadily while the true reward rises, peaks, and falls. Two defenses showed up in the simulation. Ranking on the median of three independent proxies delayed the collapse without stopping it. A small gold-standard audit of 30 samples, used to decide when to stop searching, landed at or near the peak in every run. Cheap metrics need an expensive referee.
3. A proxy is a surrogate, and surrogates have testable conditions
Econometrics and biostatistics already have a framework for this, and most ML teams never use it. A short-run measure is a valid surrogate for a long-run outcome only if the treatment affects the outcome entirely through that measure (the Prentice criterion). The surrogate index of Athey, Chetty, Imbens, and Kang builds on that condition; in a California job training experiment it recovered long-run employment effects from six quarters of data with 35 percent smaller standard errors.
Coding agents break the condition. They plausibly affect releases through review load and integration conflicts as well as commit volume, and they change the commit-to-release relationship itself. Medicine has a harsher version called the surrogate paradox: in the Cardiac Arrhythmia Suppression Trial, drugs that suppressed arrhythmias increased mortality.
If you have a subsample where the real outcome is observed, you can test this directly. Regress the outcome on treatment and the proxy, and check whether treatment still matters once the proxy is controlled for. If it does, the proxy is missing a channel.
Three questions for anyone who owns a KPI
The full piece also covers banks managing ratings around IFRS 9 provisioning thresholds, LLM and Q-learning agents destabilizing a simulated fund market through different mechanisms, and intrusion detectors whose benchmark gains come from preprocessing choices rather than model architecture. The common thread fits in three questions:
- What is the cheapest way to raise this number without helping the business?
- Has anyone actually tried it?
- What do the downstream outcomes show?
The complete article, with the formal models, the strategic classification literature, a four-step protocol for estimating attenuation, the full simulation code, and 17 sources, is on Substack: The Proxy Trap: Why AI Systems Keep Optimizing the Wrong Number
About the Author
Brian Curry is a Kansas City-based machine learning and AI strategist, marketing economist, and applied researcher. He leads Vector1, an independent research studio working on cognitive AI, causal inference, marketing economics, and knowledge engineering, with a focus on simulation-driven intelligence systems.
Over more than twenty years, Brian has led data science and analytics initiatives at enterprises ranging from $4B to $125B in revenue, including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, and AT&T. His work applies economics and causal inference to connect marketing investment to structural outcomes, operational decisions to economic value, and AI adoption to durable competitive advantage.
He is also the founder of KC AI Lab, a 1,500-member AI and ML education community in the Midwest. He lives in Kansas City and is an active part of the Kansas City community.
Research and Open-Source Projects
The modeling approach in this paper draws on PyCausalSim and Papilon.
Causal Inference and Economics
- PyCausalSim: Simulation-driven causal discovery and evaluation, using counterfactual scenarios to isolate structural drivers.
- Papilon: Complex-system simulation and optimization, with applications in media mix modeling and adaptive architectures.
- Daedalus: Causal analytics for money-flow mapping and macroeconomic tracking.
- The Autonomy Economy: A macroeconomic framework for the combined effects of rapid productivity growth and structural price deflation in an AI-automated economy.
Forecasting and Applied Modeling
- Regimecast: Time-series regime detection, structural breaks, and economic forecasting.
- CarryNet: Financial modeling for commodity carry trade analysis.
- Foodmesh-ml: Machine learning, optimization, and simulation for distributed food systems.
Cognitive AI and Semantic Architecture
- Memory-Node Encapsulation (MNE): A cognitive AI architecture and data structure for artificial episodic memory (NeoCortex-M).
- MeaningFlow: Semantic content modeling, embeddings, clustering, and coverage gap analysis.
Contact: brian@vector1.ai | GitHub @Bodhi8 | Substack
The text of this article may be shared with attribution under Creative Commons Attribution 4.0 International (CC BY 4.0). Code and models referenced here are available separately to paid subscribers.
