Before You Model Anything: Defining the Causal Question (Part 1)
This is Part 1 of a series where we walk through causal inference end-to-end — with real code, real pitfalls, and the kind of analysis you’d actually do at work. If you’ve ever been asked “did this campaign work?” and jumped straight to comparing group averages, this series is for you.
The Slack Message That Starts Every Causal Inference Project
You’re a data scientist at an e-commerce company. Marketing just ran a big email campaign promoting a seasonal sale. They targeted about 40% of the customer base and left the rest alone. A week later, your VP of Marketing sends you a message:
“The campaign was a huge success — customers who got the email spent 2x more than those who didn’t. Can you validate this for the board?”
This is the moment where the project begins. Not when you open a notebook. Not when you start fitting models. Right here, when someone hands you a vague business question and expects a number back.
The most important thing you do in a causal inference project happens before you write a single line of code: you define exactly what causal question you’re trying to answer.
From Business Question to Causal Question
“Did the campaign work?” is a business question. It’s natural language. It’s ambiguous. And it’s not something you can estimate directly. You need to translate it into something precise — a causal question with five components nailed down:
1. Treatment: What is the intervention? In our case, receiving the promotional email.
2. Outcome: What are we measuring the effect on? Customer spend — but over what period? Spend in the 7 days following the email send.
3. Population: Who are we making claims about? All customers in the database at the time of the campaign, or only those eligible for the promotion?
4. Timeframe: When did treatment happen and when do we measure the outcome? The email went out on March 1st; we’re measuring spend from March 1–7.
5. Counterfactual: What’s the comparison? What would these customers have spent if they hadn’t received the email? This is the one you can never observe directly. It’s the core problem of causal inference.
Put it all together:
What is the average effect of receiving the promotional email on customer spend in the 7 days following the campaign, compared to what those same customers would have spent had they not received it?
This might feel pedantic. It’s not. Every word is doing work. Change “average effect” to “effect on our best customers” and you need a different estimand. Change “7 days” to “30 days” and you’re measuring something else entirely. Change “those same customers” to “customers in general” and your identification strategy might change.
Write the question down. Put it at the top of your notebook. Every modeling decision downstream flows from it.
# Good practice: pin your causal question in the notebook itself. CAUSAL_QUESTION = """ What is the average treatment effect (ATE) of receiving the promotional email on customer spend in the 7 days following the campaign, compared to not receiving the email? Treatment: Receiving the promotional email (binary: yes/no) Outcome: Total spend ($) in the 7-day window post-send Population: All customers in the database at time of campaign Estimand: ATE = E[Y(1)] - E[Y(0)] """ print(CAUSAL_QUESTION)
That E[Y(1)] - E[Y(0)] notation is the Average Treatment Effect — the expected difference in outcome between the world where everyone gets treated and the world where no one does. You'll see it everywhere in the causal inference literature. It's the quantity we're trying to estimate for the rest of this series.
Understanding How the Data Was Generated
Once you’ve defined the question, the next thing to do — still before modeling — is to understand how the data came into existence. Not the schema. Not the column types. The process by which people ended up in the treatment or control group.
This is the conversation you need to have with your marketing team:
- How did you decide who got the email? (“We used our internal scoring model — targeted customers most likely to convert.”)
- What does that scoring model use? (“Purchase history, engagement, loyalty tier.”)
- Did anyone opt out? (“Yeah, customers who unsubscribed.”)
- Was there any randomization? (“No, we wanted to maximize ROI so we targeted our best customers.”)
These answers matter enormously. They tell you:
- There was no randomization. So a simple treated-vs-control comparison will be biased.
- Treatment assignment depended on customer characteristics — specifically, things like purchase history and loyalty that also drive spending. Those are confounders.
- There may be selection effects — unsubscribed customers are different in ways that could matter.
Let’s encode what we’ve learned into a simulated dataset so we can see these dynamics play out. Simulation is underrated as a tool for causal inference: when you build the data-generating process yourself, you know the true effect and can check your methods against ground truth.
import numpy as np
import pandas as pd
np.random.seed(42)
n = 10_000
# --- The data-generating process ---
# Each customer has a "loyalty score" — think recency, frequency, monetary value
# rolled into one number. This is the confounder we identified in our
# conversation with the marketing team: it drives BOTH who gets the email
# AND how much they spend.
loyalty = np.random.beta(2, 5, size=n) * 100 # skewed: most customers are low-loyalty
# Marketing targets high-loyalty customers (the selection mechanism
# we learned about by asking how targeting worked)
propensity = 1 / (1 + np.exp(-(loyalty - 50) / 10))
treatment = np.random.binomial(1, propensity)
# The TRUE causal effect of the email is $10.
# A modest, realistic bump. This is what we're trying to recover.
TRUE_EFFECT = 10
# Spending depends on loyalty (strongly) and the email (modestly)
base_spend = 20 + 1.5 * loyalty + np.random.normal(0, 15, size=n)
spend = base_spend + TRUE_EFFECT * treatment
spend = np.clip(spend, 0, None) # no negative spending
df = pd.DataFrame({
"customer_id": range(n),
"loyalty_score": loyalty,
"received_email": treatment,
"spend": spend,
})
print(f"Customers: {n:,}")
print(f"Treated (got email): {treatment.sum():,} ({treatment.mean():.1%})")
print(f"True causal effect: ${TRUE_EFFECT}")Notice how the simulation directly reflects what we learned from the marketing team. The loyalty score is the confounder — it drives both treatment assignment (via the propensity function) and the outcome (via the spending equation). The true effect is $10, embedded in the data-generating process. Everything downstream is about recovering that number.
What Happens When You Skip Step 1
Now let’s see what the VP’s analysis actually looks like — the group comparison that “validated” the campaign:
naive = df.groupby("received_email")["spend"].agg(["mean", "median", "count"])
naive.index = ["No Email", "Got Email"]
print(naive.round(2))
print(f"\nNaive 'lift': ${naive.loc['Got Email', 'mean'] - naive.loc['No Email', 'mean']:.2f}")
print(f"True causal effect: ${TRUE_EFFECT}")
mean median count
No Email 51.23 48.67 6234
Got Email 116.45 113.89 3766
Naive 'lift': $65.22
True causal effect: $10Sixty-five dollars. Off by more than 6x. This is what happens when you answer a causal question with a descriptive comparison.
The VP wasn’t wrong that the email group spent more. That’s a true descriptive fact. But the causal question — did the email cause the difference? — requires accounting for why those groups were different in the first place.
The naive estimate overstates the campaign’s effect by more than 6x.
If this were a real company, that inflated $65 number would flow into ROI calculations, budget decisions, and next quarter’s marketing plan. Millions of dollars allocated based on a number that’s mostly fiction.
Diagnosing the Problem: What We Predicted
Because we did the work upfront — defining our question and understanding the assignment mechanism — we already know what’s wrong. The marketing team told us they targeted high-loyalty customers. Let’s verify:
print("Average loyalty score:")
print(f" No Email group: {df[df.received_email == 0].loyalty_score.mean():.1f}")
print(f" Got Email group: {df[df.received_email == 1].loyalty_score.mean():.1f}")
Average loyalty score:
No Email group: 23.4
Got Email group: 51.7
Exactly what we expected. The email group has more than double the loyalty score. These are fundamentally different populations. Comparing their spending is like comparing marathon runners to couch sitters and concluding that running shoes make you fast.
Left: Marketing targeted high-loyalty customers. Right: At any given loyalty level, the true gap is only ~$10.
The scatter plot is the key visual. Look at customers with similar loyalty scores in both groups. The gap between the red and blue dots at any given loyalty level is close to the true $10 effect. The massive overall gap between the groups? That’s loyalty doing the heavy lifting, not the email.
This diagnosis wasn’t a surprise — it was a prediction. We asked the marketing team how they targeted, they told us, and we identified loyalty as a confounder before looking at a single number. That’s the payoff of doing Step 1 properly.
The Counterfactual Frame: What We Actually Need
Let’s come back to the hardest part of our causal question — the counterfactual. We want to know what email recipients would have spent if they hadn’t received the email. We can never observe this directly. It’s the fundamental problem of causal inference.
But we can get a taste of why it matters. Here’s a thought experiment embedded in the data:
# What if we could peek at the counterfactual?
# Since we built the simulation, we can.
counterfactual_spend = base_spend # what everyone would have spent WITHOUT the email
actual_spend_treated = spend[treatment == 1]
counterfactual_treated = counterfactual_spend[treatment == 1]
true_ate_treated = (actual_spend_treated - counterfactual_treated).mean()
print(f"True effect on the treated: ${true_ate_treated:.2f}")
print(f"(Should be close to ${TRUE_EFFECT})")In real data you never get to do this. You have one outcome per person — the one that actually happened. The entire field of causal inference is a collection of strategies for estimating this unobservable counterfactual under different assumptions.
A Quick Sanity Check: What If They’d Randomized?
To close the loop, let’s verify that if marketing had randomized, we wouldn’t need any of this:
random_treatment = np.random.binomial(1, 0.4, size=n)
spend_rct = base_spend + TRUE_EFFECT * random_treatment
spend_rct = np.clip(spend_rct, 0, None)
df_rct = pd.DataFrame({"received_email": random_treatment, "spend": spend_rct})
rct_means = df_rct.groupby("received_email")["spend"].mean()
print(f"RCT lift: ${rct_means[1] - rct_means[0]:.2f}")
print(f"True effect: ${TRUE_EFFECT}")
RCT lift: $10.14
True effect: $10With random assignment, the simple comparison nails it. Randomization breaks the link between the confounder and treatment, so loyalty is balanced across groups by construction. No fancy methods needed. This is why A/B tests are the gold standard.
But in our scenario — and in most real-world situations — you can’t randomize after the fact. Marketing already ran the campaign. The doctor already prescribed the treatment. The policy already went into effect. We need methods that recover the experimental answer from observational data.
Where We Stand
Here’s what we accomplished in this first step — all before fitting a single model:
Defined a precise causal question with a clear treatment, outcome, population, timeframe, and counterfactual. This is our north star for the rest of the project.
Understood the data-generating process by talking to the people who created the data. We learned that treatment assignment was non-random and depended on customer loyalty.
Identified the core threat to a naive analysis: loyalty is a confounder that creates a spurious association between the email and spending.
Quantified how bad the bias is — a 6x overestimate of the true effect — giving us concrete motivation for doing this properly.
Established the counterfactual frame — the quantity we need to estimate and why it’s fundamentally unobservable.
We haven’t “controlled for” anything yet. We haven’t adjusted, matched, weighted, or instrumented. That’s intentional. The most common mistake in applied causal inference isn’t using the wrong estimator — it’s estimating the answer to the wrong question, or not understanding the assignment mechanism well enough to know what to adjust for.
What’s Coming Next
Part 2: Drawing the Map — We’ll formalize our causal thinking with Directed Acyclic Graphs (DAGs). Instead of relying on intuition about what’s a confounder, we’ll use the graph to derive exactly what we need to condition on — and what we should leave alone (controlling for the wrong thing can create bias). We’ll use dowhy and networkx to build and reason about causal graphs for our marketing example.
Part 3: Methods That Work When Regression Doesn’t — Propensity score matching, inverse probability weighting, and stratification.
Part 4: When Simple Methods Break Down — Instrumental variables, difference-in-differences, regression discontinuity.
Part 5: Sensitivity Analysis — How fragile is your causal estimate? What if you missed a confounder?
Part 6: The Modern Toolkit — Double/debiased machine learning, EconML, and bringing ML into causal estimation.
All code from this series is available as a notebook on GitHub. Built with Python 3.11, numpy, pandas, and matplotlib.
Part 2 drops next week. Follow along if you want to stop lying to your VP of Marketing — or at least lie with better error bars.
Brian James Curry is an Enterprise AI Strategist, Data and Marketing Scientist, and Applied Researcher. He is the founder of Vector1 Research, an independent research lab focused on marketing economics, causal systems, cognitive AI architecture, and the economic implications of intelligent systems. He also founded KC AI Lab, a data science community and incubator for applied machine learning education.
