Your MMM Is Confidently Wrong About Paid Search

A synthetic test put search ROAS at 10.51 against a true 2.96. Here is what Meridian 2.0 and PyMC-Marketing 1.2 changed, and what they still cannot fix.

Marketing mix modeling had a big few months. Google shipped Meridian 2.0.0 with JAX as the default backend and built-in calibration from experiments. PyMC Labs shipped PyMC-Marketing 1.0.0, 1.1.0 and 1.2.0 in eight weeks. Vendors from Haus to Sellforte now put agents on top of their models.

My read is less exciting than the release notes. Fitting an MMM is close to solved. Identifying one is not. The full deep dive on Substack has the dated changelogs, a runnable Python model, templates for both libraries and the references. This is the short version.

The bias no backend can fix

Strip a model down to one channel and one thing you did not measure. If spend responds to a demand shock you cannot see, the estimated return converges to the truth plus a bias term: beta_hat = beta + gamma * Cov(x, d) / Var(x).

Put realistic numbers in. A true ROAS of 4.2, a demand shock that adds $150k of sales and $20k of search spend, and some independent spend noise give a bias of 6.0. The model reports about 10.2. A preprint from a Zalando researcher found the same order of error in a richer simulation: 10.66x estimated against 4.20x true.

Adstock curves, saturation curves, seasonality terms and geo hierarchies change the shape of the response. They do not change where the variation in spend came from, so they leave this bias alone. Only a better control, an experiment, or an outside estimate that never saw the demand shock can shrink it. The full article walks through each of the three with the math.

Calibration helps, but not as much as you think

The standard fix is to turn a lift test into a prior. Treat that as a precision-weighted average, (m/sd_m^2 + l/sd_l^2) / (1/sd_m^2 + 1/sd_l^2), and the problem is visible. Combine the 10.66x model with an unbiased test of 4.20 and standard deviation 1.0, and you get 5.26. The test carries 84% of the weight and the answer is still 1.06 too high.

The full article shows how PyMC-Marketing puts lift tests in the likelihood, how Meridian 2.0 turns them into priors, and why fitting the experiment’s full time series, as the Zalando paper does, is the direction both should move.

An optimizer amplifies the error

I ran a small reference MMM on synthetic data with confounded search spend. Without calibration it put search ROAS at 10.51 against a true 2.96. One go-dark test moved it to 7.21. A second test at a higher spend level moved it to 4.53, still too high.

The budget result is the one to show a CMO. Every plan spent the same $124.6k a week. Leaving the allocation alone lost $11.8k a week against the true optimum. The optimizer on the uncalibrated model lost $88.5k a week, about $4.6 million a year. Across six random seeds, none of the optimized plans beat the status quo.

The full article includes the complete script, its output, both charts and the six-seed check, so you can break it yourself.

Agents speed up the plumbing, not the identification

Meridian now ships five agent skills with human checkpoints. PyMC Labs published the most candid test I have seen: a written skill raised a coding agent’s pass rate from 60% to 93% on PyMC tasks, but automated skill improvement on real pymc-marketing tasks stalled when the optimizer wrote version-wrong API code from memory. Meridian 2.0 changed backends, precision and several argument names in one release, and PyMC-Marketing changed what its optimizer returns, so this failure mode is not hypothetical. Agents that move budget faster on a biased model lose money faster.

What’s in the Full Article

• Dated release notes for Meridian 1.7.1 through 2.1.0 and PyMC-Marketing 1.0.0 through 1.2.0
• A runnable NumPy and SciPy MMM with lift-test calibration and budget optimization
• Minimal Meridian 2.x and PyMC-Marketing 1.2 templates with calibration and diagnostics
• A five-way comparison: identification, calibration, speed, optimizer and agents
• When not to use each tool, and what to do this quarter

Written for marketing leaders who fund MMM and the data scientists who build it.

Read the full deep dive on Substack → https://brianjamescurry.substack.com/p/the-experiment-is-the-model-marketing

Subscribe on Substack for the next deep dive on applied ML in marketing.

About the Author

Brian Curry is a Kansas City-based machine learning and AI strategist, marketing economist, and applied researcher. He leads Vector1, an independent research studio working on cognitive AI, causal inference, marketing economics, and knowledge engineering, with a focus on simulation-driven intelligence systems.

Over more than twenty years, Brian has led data science and analytics initiatives at enterprises ranging from $4B to $125B in revenue, including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, and AT&T. His work applies economics and causal inference to connect marketing investment to structural outcomes, operational decisions to economic value, and AI adoption to durable competitive advantage.

He is also the founder of KC AI Lab, a 1,500-member AI and ML education community in the Midwest. He lives in Kansas City and is an active part of the Kansas City community.

Research and Open-Source Projects

The modeling approach in this paper draws on PyCausalSim and Papilon.

Causal Inference and Economics

• PyCausalSim: Simulation-driven causal discovery and evaluation, using counterfactual scenarios to isolate structural drivers.
• Papilon: Complex-system simulation and optimization, with applications in media mix modeling and adaptive architectures.
• Daedalus: Causal analytics for money-flow mapping and macroeconomic tracking.
• The Autonomy Economy: A macroeconomic framework for the combined effects of rapid productivity growth and structural price deflation in an AI-automated economy.

Forecasting and Applied Modeling

• Regimecast: Time-series regime detection, structural breaks, and economic forecasting.
• CarryNet: Financial modeling for commodity carry trade analysis.
• Foodmesh-ml: Machine learning, optimization, and simulation for distributed food systems.

Cognitive AI and Semantic Architecture

• Memory-Node Encapsulation (MNE): A cognitive AI architecture and data structure for artificial episodic memory (NeoCortex-M).
• MeaningFlow: Semantic content modeling, embeddings, clustering, and coverage gap analysis.

Contact: brian@vector1.ai | GitHub @Bodhi8 | Substack

The text of this article may be shared with attribution under Creative Commons Attribution 4.0 International (CC BY 4.0). Code and models referenced here are available separately to paid subscribers.