Vector 1
Causal Inference & MMMOrganizations & Work

Did the Reallocation Actually Work? Measuring the Causal Impact of a Media Strategy Change

Brian J. Curry
Brian J. Curry
May 14, 2026 · 14 min read

Your team moved 30% of the budget from TV to digital. Revenue went up 12%. Your CFO wants to know how much of that 12% was the reallocation — and “trust me” is not an answer. This is how you answer it properly, with code.

1. Why “we changed it and revenue went up” is not evidence

Every marketing org eventually makes a deliberate, discrete change to its media strategy:

  • shifting budget from linear TV to CTV and paid social
  • killing a channel and redeploying the money
  • moving from a broad prospecting strategy to a tighter retention-led one
  • changing the bidding objective across a whole account

Then something happens to revenue, and everyone wants to attribute the delta to the decision. The problem is that the before-and-after comparison is contaminated by everything else that changed at the same time: seasonality, pricing, competitor behavior, macro conditions, PR, product launches, the weather.

The naive before/after estimate is:

effect_naive = mean(revenue_after) - mean(revenue_before)

and it is biased by every one of those confounders. The entire job of causal inference here is to reconstruct the counterfactual: what revenue would have been in the post-period if you had kept the old strategy. You can never observe that series directly. You have to estimate it, and — just as important — you have to stress-test that estimate until you either trust it or you don’t.

This article walks through the full workflow on a simulated-but-realistic budget reallocation, with runnable Python at every step:

  1. Frame the change in the potential-outcomes model
  2. Draw the causal graph and make assumptions explicit with DoWhy
  3. Estimate the effect three ways — difference-in-differences, synthetic control / CausalImpact, and a DoWhy estimand
  4. Probe for heterogeneity with EconML (which segments responded?)
  5. Run refutation and sensitivity checks
  6. Walk through the ways this all goes wrong

2. The potential-outcomes framing for a change

The unit of analysis for a strategy change is usually a geo (DMA, region, country) observed over time, because you rarely get to randomize a strategy at the user level. Let:

  • i index geos, t index weeks
  • D_i = 1 if geo i is exposed to the new strategy, 0 otherwise
  • Y_it(1) = revenue under the new strategy, Y_it(0) = revenue under the old one

The causal estimand we want is the Average Treatment effect on the Treated (ATT) in the post-period:

ATT = E[ Y_it(1) - Y_it(0) | D_i = 1, t >= T_intervention ]

Y_it(1) is observed for treated geos after the switch. Y_it(0) for those same geos is the counterfactual we must build. Every method below is a different strategy for building it:

Method How it builds the counterfactual Key assumption Geo experiment Randomly assigned control geos Randomization holds; no spillover Difference-in-differences Control geos’ trend, shifted to treated baseline Parallel trends absent treatment Synthetic control / CausalImpact Weighted combo of control geos that matches pre-period Pre-period fit + stable relationship Interrupted time series Treated geo’s own pre-trend, extrapolated No other concurrent shock at the break

The single most valuable thing you can do is run a randomized geo experiment — pick the new strategy in a random subset of geos and hold out the rest. If you can do that, do that. The rest of this article matters most when you can’t fully randomize, which is the common real-world case: leadership has already decided to roll the change out, and you get a staggered rollout or a non-random treated set at best.

3. Simulating a realistic reallocation

So the code is fully reproducible, here’s a synthetic dataset with known ground truth — that way we can check each estimator against the real answer.

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)

N_GEOS = 60
N_WEEKS = 104
T_INTERVENTION = 78          # new strategy goes live week 78
TRUE_LIFT = 0.08             # ground truth: +8% incremental revenue in treated geos

geos = np.arange(N_GEOS)
weeks = np.arange(N_WEEKS)

# ~40% of geos get the new strategy; assignment correlates with geo size (NON-random!)
geo_size = rng.lognormal(mean=1.0, sigma=0.4, size=N_GEOS)
treat_prob = (geo_size - geo_size.min()) / (geo_size.max() - geo_size.min())
treated = (rng.uniform(size=N_GEOS) < 0.3 + 0.4 * treat_prob).astype(int)

rows = []
for i in geos:
    base = 10_000 * geo_size[i]
    geo_fe = rng.normal(0, 0.05)
    for t in weeks:
        season = 0.15 * np.sin(2 * np.pi * t / 52)          # annual seasonality
        trend = 0.002 * t                                    # gentle organic growth
        macro = 0.10 * np.sin(2 * np.pi * t / 26 + 1.0)      # ongoing macro cycle
        post = int(t >= T_INTERVENTION)
        macro_shock = 0.07 * post                            # market-wide upturn that
                                                             # happens to land in the
                                                             # same window as the rollout
        noise = rng.normal(0, 0.04)
        lift = TRUE_LIFT * post * treated[i]                 # the thing we want to recover
        log_rev = (np.log(base) + geo_fe + season + trend
                   + macro + macro_shock + noise + lift)
        rows.append((i, t, treated[i], post, geo_size[i], np.exp(log_rev)))

df = pd.DataFrame(rows, columns=["geo", "week", "treated", "post", "geo_size", "revenue"])
df["log_revenue"] = np.log(df["revenue"])
df["exposed"] = df["treated"] * df["post"]
print(df.groupby(["treated", "post"])["revenue"].mean().unstack())

Two deliberate traps are baked in. First, a market-wide upturn (macro_shock) that lands in the same window as the rollout — a time-varying confounder that moves every geo, treated or not. Second, non-random assignment: bigger geos are likelier to be treated. These two traps break different naive estimators, which is the whole point.

# Naive #1 — before/after on treated geos only.
# Fooled by the market upturn: it can't tell the +8% strategy lift
# apart from the +7% the whole market got anyway.
naive_ba = (df.query("treated==1 & post==1").revenue.mean()
            - df.query("treated==1 & post==0").revenue.mean())
naive_ba_pct = naive_ba / df.query("treated==1 & post==0").revenue.mean()
print(f"Naive before/after:        {naive_ba_pct:.1%}")   # ~13% — overstates the 8% truth

# Naive #2 — treated vs. control in the post period.
# Fooled by non-random assignment: treated geos are simply bigger to begin with.
naive_xs = (df.query("treated==1 & post==1").revenue.mean()
            / df.query("treated==0 & post==1").revenue.mean() - 1)
print(f"Naive treated-vs-control:  {naive_xs:.1%}")        # ~25% — wildly off

The before/after comparison differences out fixed geo size but not the concurrent market move. The cross-sectional comparison differences out the market move but not geo size. Each naive estimator is blind to exactly one trap — and you usually can’t see which one is biting. Every method below is built to handle both at once.

4. Make your assumptions a diagram: DoWhy

Before estimating anything, write the causal graph down. This forces every assumption into the open and lets the tooling tell you whether the effect is even identifiable from the data you have.

# pip install dowhy
from dowhy import CausalModel

# DoWhy will adjust for geo and week as *categorical* confounders, i.e. fixed
# effects — so the adjustment absorbs geo size non-linearly and every
# week-level shock. Pass a typed copy; the rest of the pipeline keeps plain ints.
df_cm = df.astype({"geo": "category", "week": "category"})

# The causal graph, written over columns that actually exist in df.
# 'exposed' (= treated x post) is the node meaning "currently on the new strategy".
graph = """
digraph {
    geo     -> treated;
    geo     -> log_revenue;
    week    -> post;
    week    -> log_revenue;
    treated -> exposed;
    post    -> exposed;
    exposed -> log_revenue;
}
"""

model = CausalModel(
    data=df_cm,
    treatment="exposed",
    outcome="log_revenue",
    graph=graph,
)

estimand = model.identify_effect(proceed_when_unidentifiable=True)
print(estimand)

The value of this step isn’t the point estimate that comes later — it’s that DoWhy forces you to name your confounders and then tells you whether the effect is identifiable from them. Here it reads the graph and returns a backdoor estimand: adjust for geo and week, and the effect is identified. But that conclusion is only as strong as the graph. This graph asserts that every shock to revenue is either geo-specific (absorbed by geo) or common to all geos within a week (absorbed by week). If some shock were instead geo-specific, time-varying, correlated with which geos got treated, and unobserved, no backdoor adjustment could recover the effect — and DoWhy would tell you so rather than hand you a falsely confident number. Naming that untestable assumption is the whole game.

5. Estimator 1: Difference-in-differences

DiD builds the counterfactual from the control geos’ trend. The estimate is the classic “difference of differences”:

ATT_did = [Y_treated_post - Y_treated_pre] - [Y_control_post - Y_control_pre]

In regression form, with two-way fixed effects and standard errors clustered by geo:

import statsmodels.formula.api as smf

did = smf.ols("log_revenue ~ exposed + C(geo) + C(week)", data=df)\
        .fit(cov_type="cluster", cov_kwds={"groups": df["geo"]})

coef = did.params["exposed"]
ci_low, ci_high = did.conf_int().loc["exposed"]
print(f"DiD lift estimate: {np.expm1(coef):.2%}  "
      f"95% CI [{np.expm1(ci_low):.2%}, {np.expm1(ci_high):.2%}]")

Because the outcome is logged, expm1(coef) converts the coefficient to a percentage lift. The geo fixed effects absorb everything time-invariant about a geo, including its size — so the non-random assignment trap is gone. The week fixed effects absorb every shock common to all geos in a given week — seasonality, the macro cycle, and the market-wide upturn — so the time-varying confounder trap is gone too. DiD clears both traps the naive estimators fell into because it controls for both dimensions at once.

Always test the parallel-trends assumption with an event-study plot — interact treatment with each week and check that pre-period lead coefficients are flat around zero:

df["weeks_since"] = df["week"] - T_INTERVENTION
event = smf.ols(
    "log_revenue ~ C(weeks_since, Treatment(reference=-1)):treated + C(geo) + C(week)",
    data=df
).fit(cov_type="cluster", cov_kwds={"groups": df["geo"]})

# pull the interaction coefficients and plot them with their CIs;
# pre-period (weeks_since < 0) should hug zero, post-period should jump to ~0.08

If the pre-period leads are not flat, parallel trends is suspect and your DiD estimate is biased — stop and switch designs.

6. Estimator 2: Synthetic control with CausalImpact

DiD uses a simple average of control geos. Synthetic control learns a weighted combination of control geos that best reproduces the treated series in the pre-period, then projects it forward. Google’s CausalImpact does this with a Bayesian structural time-series model and gives you uncertainty intervals for free.

# pip install pycausalimpact
from causalimpact import CausalImpact

# aggregate to treated-vs-control weekly series
pivot = (df.groupby(["week", "treated"])["revenue"].sum()
           .unstack("treated")
           .rename(columns={0: "control", 1: "treated"}))

# CausalImpact expects the response variable first, predictors after
ci_data = pivot[["treated", "control"]]
pre_period = [0, T_INTERVENTION - 1]
post_period = [T_INTERVENTION, N_WEEKS - 1]

ci = CausalImpact(ci_data, pre_period, post_period)
print(ci.summary())
print(ci.summary(output="report"))
ci.plot()

A packaging note: the original pycausalimpact is lightly maintained and can break against recent pandas/statsmodels releases. If the import or fit fails, tfcausalimpact is the actively maintained drop-in (pip install tfcausalimpact, same from causalimpact import CausalImpact API) — pin whichever one you use.

CausalImpact reports absolute and relative cumulative lift with a credible interval. The report output literally writes you a paragraph of plain-English interpretation — useful for the deck, though you should read the model critically rather than pasting it wholesale. The key diagnostic is the pre-period fit: if the synthetic series tracks the treated series tightly before the intervention, the post-period counterfactual is credible. If the pre-period fit is loose, it isn't.

A subtle point: for this to be valid, the control geos must not themselves be affected by the strategy change. If budget was reallocated out of a national channel, “control” geos still feel it, the counterfactual is polluted, and the estimate shrinks toward zero. Spillover is the silent killer of geo-based measurement.

7. Estimator 3: DoWhy end-to-end, with refutation

Now estimate through the DoWhy model from Section 4 and — the part most analyses skip — try to break the estimate.

estimate = model.estimate_effect(
    estimand,
    method_name="backdoor.linear_regression",
    test_significance=True,
)
print(f"DoWhy effect estimate: {np.expm1(estimate.value):.2%}")

# Refutation 1: inject a random common cause — estimate should barely move
r1 = model.refute_estimate(estimand, estimate,
                           method_name="random_common_cause")
print(r1)

# Refutation 2: placebo treatment — replace treatment with noise, effect should collapse to ~0
r2 = model.refute_estimate(estimand, estimate,
                           method_name="placebo_treatment_refuter",
                           placebo_type="permute")
print(r2)

# Refutation 3: drop a random subset of data — estimate should be stable
r3 = model.refute_estimate(estimand, estimate,
                           method_name="data_subset_refuter",
                           subset_fraction=0.8)
print(r3)

How to read these: the placebo test is the important one. If you swap the real treatment assignment for a random one and still “find” an 8% effect, your estimator is picking up structure that isn’t causal and the original estimate is worthless. A trustworthy result shows a real effect under the true treatment and a near-zero effect under the placebo. random_common_cause and data_subset_refuter check that the estimate is stable — it should move only trivially.

Refutation tests don’t prove causality. They’re falsification attempts. Passing them means “this survived my attempts to break it,” which is the most honest thing you can say about a non-experimental estimate.

8. Going further: which segments responded? (EconML)

The ATT is an average and averages hide things. Maybe the reallocation worked well in large urban geos and did nothing in rural ones — that’s the actionable finding, because it tells you how to allocate next quarter. This is a conditional average treatment effect (CATE) problem, and EconML’s double machine learning estimators are built for it.

# pip install econml
from econml.dml import LinearDML
from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier

post = df[df.post == 1].copy()

Y = post["log_revenue"].values
T = post["treated"].values
X = post[["geo_size"]].values                 # effect modifiers (heterogeneity)
W = post[["week"]].values                     # nuisance controls

dml = LinearDML(
    model_y=GradientBoostingRegressor(),       # outcome is continuous -> regressor
    model_t=GradientBoostingClassifier(),      # treatment is binary -> classifier
    discrete_treatment=True,
    random_state=42,
)
# Cross-fit by geo, not by row. Each geo appears 26 times in the post period
# with an identical geo_size; without grouping, the same geo lands in both the
# train and held-out folds and the propensity model can memorise geo identity
# from geo_size, collapsing the treatment residuals. Passing groups makes
# EconML split with GroupKFold so every geo is held out as a unit.
dml.fit(Y, T, X=X, W=W, groups=post["geo"].values)

# geo_size ~ lognormal(1.0, 0.4): in this sample it runs ~1.3 to ~6, median ~2.8.
# Evaluate the effect at a genuinely small and a genuinely large geo — both in range.
small, large = 2.0, 5.0
print(f"Average effect (ATE): {np.expm1(dml.ate(X)):.2%}")
print(f"Effect by geo size — small (~{small}): "
      f"{np.expm1(dml.effect(np.array([[small]]))[0]):.2%}, "
      f"large (~{large}): {np.expm1(dml.effect(np.array([[large]]))[0]):.2%}")

# confidence interval on the average
lb, ub = dml.ate_interval(X, alpha=0.05)
print(f"95% CI on ATE: [{np.expm1(lb):.2%}, {np.expm1(ub):.2%}]")

Double ML uses flexible models to partial the confounders out of both the outcome and the treatment, then fits the effect on the residuals — so it controls for confounding without forcing you to assume a functional form. Because the treatment is binary, the treatment model is a classifier estimating the propensity to be treated; the outcome model is a regressor. dml.effect(X) returns a per-geo effect you can plot against geo size to see the gradient.

One honest note about this particular dataset: the lift was built in as a constant TRUE_LIFT for every treated geo, so DML should come back with a roughly flat gradient — small and large geos both near 8%. That isn't the method failing. A flat CATE is a real result: it says segmentation won't buy you anything here, so don't over-fit a story to noise. On real data this same code is how you'd find out whether the gradient is flat or not — and if it isn't, that is the actionable finding.

9. Triangulation: the actual deliverable

The credible answer is not one number from one method — it’s the convergence of several designs with different assumptions:

Estimator Counterfactual source Main assumption Lift estimate Naive before/after Treated geos’ own past Nothing else changed biased — discard Difference-in-differences Control geos’ trend Parallel trends ~8% CausalImpact Weighted synthetic control Good pre-period fit ~8% DoWhy + refutation Backdoor adjustment No unobserved confounders ~8% EconML DML Residualized ML controls Confounders are observed ~8%, flat across geo size

When DiD, synthetic control, and double ML — methods that fail in different ways — all land in the same neighborhood and the refutation tests pass, you have something you can defend to a CFO. When they diverge, that disagreement is itself the finding: it means the answer is sensitive to assumptions, and the honest move is to say so and design a clean geo experiment for the next change.

10. How this goes wrong (read before presenting)

  • Spillover / contaminated controls. If the strategy change touches a national channel, “control” geos are partly treated. Effects look too small. Pick controls with genuine isolation, or model the spillover explicitly.
  • Parallel trends fails. If treated geos were already on a different trajectory, DiD attributes that pre-existing gap to the strategy. Always run the event-study plot.
  • Too few geos. Inference here leans on having enough units to cluster standard errors over. With 10 geos, your confidence intervals are near-meaningless however tight they look. Consider randomization inference instead.
  • A concurrent shock at the break point. Interrupted-time-series logic collapses if anything else big happened the same week — a price change, a competitor exit, a viral moment. Synthetic control with clean donors is more robust here, but not immune.
  • Pre-period too short. Synthetic control needs enough pre-period to learn stable weights. A handful of weeks overfits and the counterfactual is fragile.
  • Multiple changes at once. If you reallocate budget and change creative and change the bid strategy in the same window, no method can separate them. The cleanest causal tool is upstream: stagger the changes.
  • Treating refutation as a rubber stamp. Passing a placebo test is necessary, not sufficient. Unobserved confounding can survive every refuter DoWhy offers. State the assumption you cannot test.

11. A workflow you can reuse

  1. Write the estimand first. “ATT on revenue in treated geos, weeks 78–103.” Decide what you’re measuring before you touch data.
  2. Draw the graph. Name every confounder. Decide which are observed. Let DoWhy tell you if the effect is identifiable.
  3. Prefer a randomized geo experiment. If you can assign the new strategy randomly across geos, most of the hard inference disappears.
  4. If you can’t randomize, triangulate. DiD + synthetic control + double ML. Different assumptions, same question.
  5. Validate the design, not just the result. Event-study plot for parallel trends; pre-period fit for synthetic control.
  6. Try to break your estimate. Placebo treatment, random common cause, data subsets.
  7. Report the range and the assumptions. One honest interval beats one confident number.
  8. Feed it back. Use the heterogeneity findings to design the next reallocation — and the next experiment.

Last-click attribution will tell you the reallocation “worked” because the new channels are now catching the clicks the old ones used to. Causal inference asks the only question that actually matters for the next budget decision: compared to what?

If you build this out, the stack is: pandas + numpy for data, statsmodels for difference-in-differences, pycausalimpact for synthetic control, dowhy for identification and refutation, and econml for heterogeneous effects. Every snippet above runs against the simulated dataset in Section 3.

Related research