Three Methods, One Answer: Estimating the Causal Effect (Part 3)
This is Part 3 of a series on causal inference applied to a real-world marketing problem. In Part 1, we defined our causal question — “what’s the true effect of a promotional email on customer spend?” — and showed the naive estimate was off by 6x. In Part 2, we drew the causal DAG and used the backdoor criterion to determine that loyalty score is the confounder we need to adjust for. Now we adjust.
We Know What to Control For. Now How?
At the end of Part 2, we had a clear plan: condition on loyalty score to block the backdoor path from Email to Spend. We even did a quick stratified analysis and got $11.14 — close to the true $10 effect.
But in practice, “condition on the confounder” isn’t a single operation. There are multiple ways to do it, and they make different assumptions, handle different situations, and fail in different ways. A real causal inference project doesn’t just pick one — it runs several and checks whether they agree.
In this article, we’ll implement three methods from scratch on our marketing dataset:
- Stratification — the simplest approach, and we’ll do it more carefully this time.
- Inverse Probability Weighting (IPW) — reweight the data so it looks like a randomized experiment.
- Propensity Score Matching — pair each treated customer with a similar untreated one.
All three rely on the same underlying assumption (no unmeasured confounders — the backdoor criterion from Part 2), but they get there by different paths. When they converge on the same answer, we can be more confident. When they don’t, that tells us something useful too.
Before Anything: The Propensity Score
Two of our three methods use a quantity called the propensity score — the probability of receiving treatment given the observed covariates. In our case: what’s the probability a customer received the email, given their loyalty score?
This might seem like a detour — we want to estimate the effect of the email on spending, so why are we modeling who got the email? The key insight, due to Rosenbaum and Rubin (1983), is that if the propensity score is correctly specified, conditioning on it is sufficient to remove confounding. Instead of matching on every confounder directly (which gets hard with many covariates), you can match on this single number.
Let’s estimate it. With one confounder, this is just logistic regression:
import numpy as np
import pandas as pd
from scipy.optimize import minimize
from scipy.special import expit
# Recreate dataset from Parts 1 and 2
np.random.seed(42)
n = 10_000
loyalty = np.random.beta(2, 5, size=n) * 100
propensity = 1 / (1 + np.exp(-(loyalty - 50) / 10))
treatment = np.random.binomial(1, propensity)
TRUE_EFFECT = 10
base_spend = 20 + 1.5 * loyalty + np.random.normal(0, 15, size=n)
spend = base_spend + TRUE_EFFECT * treatment
spend = np.clip(spend, 0, None)
df = pd.DataFrame({
"loyalty_score": loyalty,
"received_email": treatment,
"spend": spend,
})
# Estimate propensity score via logistic regression
X = np.column_stack([np.ones(n), loyalty])
def neg_log_likelihood(beta, X, y):
p = expit(X @ beta)
p = np.clip(p, 1e-10, 1 - 1e-10)
return -np.sum(y * np.log(p) + (1 - y) * np.log(1 - p))
result = minimize(neg_log_likelihood, x0=np.zeros(2), args=(X, treatment), method='BFGS')
ps = expit(X @ result.x)
df['propensity_score'] = ps
print(f"Propensity score model coefficients:")
print(f" Intercept: {result.x[0]:.3f}")
print(f" Loyalty: {result.x[1]:.3f}")
print(f" PS range: [{ps.min():.4f}, {ps.max():.4f}]")
Propensity score model coefficients:
Intercept: -5.104
Loyalty: 0.102
PS range: [0.0061, 0.9692]The propensity score ranges from near-zero (low-loyalty customers who almost certainly didn’t get the email) to near-one (high-loyalty customers who almost certainly did). This makes sense — it mirrors the targeting mechanism we learned about from the marketing team.
But before we use this for estimation, we need to check something critical: overlap.
Checking Overlap (Positivity)
For any causal method that adjusts on the propensity score to work, we need overlap between the treated and control groups — there must be customers at every level of the propensity score who both did and did not receive the email. If all customers with PS > 0.8 got the email, we have no way to know what a high-PS customer would have spent without the email.
The formal name for this is the positivity assumption: 0 < P(Treatment | Covariates) < 1 for all covariate values.
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(10, 5.5))
ax.hist(ps[treatment == 0], bins=50, alpha=0.6, color='#4878CF', label='Control (no email)', density=True)
ax.hist(ps[treatment == 1], bins=50, alpha=0.6, color='#D65F5F', label='Treated (got email)', density=True)
ax.set_xlabel('Propensity Score')
ax.set_ylabel('Density')
ax.set_title('Propensity Score Distributions — Checking Overlap')
ax.axvline(0.05, color='#999', linestyle='--', linewidth=1)
ax.axvline(0.95, color='#999', linestyle='--', linewidth=1)
ax.legend()
plt.tight_layout()
plt.show()
The distributions overlap substantially in the middle — good. But notice the tails: lots of control units have very low propensity scores (they were never going to get the email), and some treated units have high scores (they were almost certain to get it). These extreme regions are where our estimates will be most uncertain.
The dashed lines mark a common trimming zone — we’ll revisit this when we get to IPW.
Method 1: Stratification
We previewed this in Part 2. The idea: split the data into bins based on the confounder (or propensity score), compute the treatment effect within each bin, and take a weighted average.
Within each bin, treated and control customers have similar loyalty scores, so the within-bin comparison approximates a mini-experiment. Cochran (1968) showed that five strata on the propensity score can remove over 90% of confounding bias. We’ll use ten for finer resolution.
df['ps_decile'] = pd.qcut(df.propensity_score, q=10, labels=False)
stratum_effects = []
print("Stratum-level effects:")
print(f"{'Bin':>4} {'Effect':>8} {'N_treat':>8} {'N_ctrl':>8} {'PS range':>20}")
print("-" * 52)
for b in range(10):
sub = df[df.ps_decile == b]
treated = sub[sub.received_email == 1]
control = sub[sub.received_email == 0]
if len(treated) > 5 and len(control) > 5:
effect = treated.spend.mean() - control.spend.mean()
stratum_effects.append((effect, len(sub)))
ps_lo = sub.propensity_score.min()
ps_hi = sub.propensity_score.max()
print(f"{b:>4} {effect:>8.2f} {len(treated):>8} {len(control):>8} [{ps_lo:.3f}, {ps_hi:.3f}]")
strat_ate = sum(e * w for e, w in stratum_effects) / sum(w for _, w in stratum_effects)
print(f"\nStratified ATE: ${strat_ate:.2f}")
print(f"True effect: ${TRUE_EFFECT}")
Bin Effect N_treat N_ctrl PS range
----------------------------------------------------
0 6.87 10 990 [0.006, 0.019]
1 8.55 19 981 [0.019, 0.032]
2 11.98 37 963 [0.032, 0.050]
...
9 10.52 551 449 [0.453, 0.969]
Stratified ATE: $10.83
True effect: $10This works, and it’s transparent — you can see the effect in each stratum and verify that the groups are comparable. But notice the first few bins: there are only 10–37 treated units among nearly 1,000 controls. The effect estimates in those bins are noisy because there are so few treated customers to compare against.
Strengths of stratification: Simple, interpretable, easy to inspect visually. You can see immediately if certain strata are problematic.
Weaknesses: Choosing the number of bins is arbitrary. With many confounders, stratifying on all of them simultaneously becomes infeasible (the curse of dimensionality). And bins at the extremes of the propensity score distribution are unreliable.
Method 2: Inverse Probability Weighting (IPW)
IPW takes a different approach. Instead of binning, it creates a pseudo-population where treatment assignment is independent of the confounders — effectively recreating what a randomized experiment would have looked like.
The intuition: a treated customer with a low propensity score is unusual — they got the email even though they weren’t the typical target. This person is highly informative because they’re similar to the control group. IPW gives them more weight. Conversely, a treated customer with a high propensity score is typical of the treated group and gets less weight.
The weights are:
- Treated units: weight = 1 / propensity_score
- Control units: weight = 1 / (1 — propensity_score)
# Compute IPW weights
weights = np.where(treatment == 1, 1 / ps, 1 / (1 - ps))
print(f"Weight distribution:")
print(f" Mean: {weights.mean():.2f}")
print(f" Median: {np.median(weights):.2f}")
print(f" Max: {weights.max():.2f}")
print(f" 99th: {np.percentile(weights, 99):.2f}")
Weight distribution:
Mean: 2.04
Median: 1.10
Max: 112.28
99th: 20.87
See that max weight of 112? That’s a single customer with an extreme propensity score contributing 112 times their natural influence to the estimate. This is the practical achilles heel of IPW: extreme weights. One or two outlier weights can dominate the entire estimate.
The standard fix is trimming — exclude observations with propensity scores too close to 0 or 1, where the weights blow up. This trades a small amount of bias (we’re no longer estimating the ATE over the full population) for a large reduction in variance.
# IPW with trimming: exclude PS < 0.05 or PS > 0.95
trim_mask = (ps > 0.05) & (ps < 0.95)
ps_trim = ps[trim_mask]
t_trim = treatment[trim_mask]
s_trim = spend[trim_mask]
print(f"Trimming removed {n - trim_mask.sum()} observations ({(1-trim_mask.mean())*100:.1f}%)")
# Compute weights on trimmed data
w_trim = np.where(t_trim == 1, 1 / ps_trim, 1 / (1 - ps_trim))
# Hajek (normalized) estimator — more stable than Horvitz-Thompson
w_treated = w_trim * t_trim
w_control = w_trim * (1 - t_trim)
ipw_treated_mean = np.sum(w_treated * s_trim) / np.sum(w_treated)
ipw_control_mean = np.sum(w_control * s_trim) / np.sum(w_control)
ipw_ate = ipw_treated_mean - ipw_control_mean
print(f"\nIPW (trimmed) ATE: ${ipw_ate:.2f}")
print(f"True effect: ${TRUE_EFFECT}")
Trimming removed 3696 observations (37.0%)
IPW (trimmed) ATE: $10.06
True effect: $10$10.06 — essentially the true effect. Trimming removed customers at the extremes where we can’t reliably estimate counterfactuals, and the remaining weighted population gives us a clean estimate.
Strengths of IPW: Uses all the data (within the trimmed region) rather than binning. Straightforward to extend to multiple confounders. Provides a natural weighted pseudo-population you can reuse for other analyses.
Weaknesses: Extremely sensitive to propensity score model misspecification. If the PS model is wrong, the weights are wrong, and the estimate can be wildly off. Extreme weights require trimming, which changes the target population. And the variance can be high even after trimming.
Method 3: Propensity Score Matching
Matching is the most intuitive method. For each treated customer, find the most similar untreated customer (based on propensity score), pair them up, and compare their outcomes. The difference within each pair estimates the treatment effect for that treated individual. Average across pairs to get the ATT (Average Treatment effect on the Treated).
treated_idx = np.where(treatment == 1)[0]
control_idx = np.where(treatment == 0)[0]
ps_control = ps[control_idx]
# Nearest-neighbor matching on propensity score
matched_pairs = []
for t_idx in treated_idx:
distances = np.abs(ps_control - ps[t_idx])
best_match = np.argmin(distances)
c_idx = control_idx[best_match]
matched_pairs.append({
'treated_idx': t_idx,
'control_idx': c_idx,
'ps_treated': ps[t_idx],
'ps_control': ps[c_idx],
'spend_treated': spend[t_idx],
'spend_control': spend[c_idx],
'effect': spend[t_idx] - spend[c_idx],
})
matched_df = pd.DataFrame(matched_pairs)
matching_att = matched_df.effect.mean()
print(f"Matched pairs: {len(matched_df)}")
print(f"Mean PS distance: {(matched_df.ps_treated - matched_df.ps_control).abs().mean():.4f}")
print(f"\nMatching ATT: ${matching_att:.2f}")
print(f"True effect: ${TRUE_EFFECT}")
Matched pairs: 1790
Mean PS distance: 0.0021
Matching ATT: $9.17
True effect: $10$9.17 — right in the neighborhood of the true effect. But we should check that matching actually did its job: are the matched groups balanced on the confounder?
# Balance check: loyalty scores before and after matching
loyalty_treated = loyalty[treated_idx]
loyalty_all_control = loyalty[control_idx]
loyalty_matched_control = loyalty[matched_df.control_idx.values]
smd_before = abs(loyalty_treated.mean() - loyalty_all_control.mean()) / loyalty.std()
smd_after = abs(loyalty_treated.mean() - loyalty_matched_control.mean()) / loyalty.std()
print(f"Loyalty Score Balance:")
print(f" Before matching - Treated: {loyalty_treated.mean():.1f}, Control: {loyalty_all_control.mean():.1f}, SMD: {smd_before:.3f}")
print(f" After matching - Treated: {loyalty_treated.mean():.1f}, Control: {loyalty_matched_control.mean():.1f}, SMD: {smd_after:.3f}")
print(f"\n (SMD < 0.1 is generally considered good balance)")
Loyalty Score Balance:
Before matching - Treated: 49.6, Control: 22.7, SMD: 1.314
After matching - Treated: 49.6, Control: 49.3, SMD: 0.013
(SMD < 0.1 is generally considered good balance
Before matching, the groups were wildly different — an SMD of 1.31 means the treated group’s loyalty was more than a full standard deviation higher than the control. After matching, the SMD drops to 0.01. The matched control group now looks like the treated group on the confounder. That’s exactly what we want.
Matching With a Caliper
One refinement: what if the nearest match is still far away? A caliper sets a maximum acceptable distance. If no match is within the caliper, the treated unit is dropped. This sacrifices some treated observations for better match quality.
caliper = 0.05
matched_cal = []
dropped = 0
for t_idx in treated_idx:
distances = np.abs(ps_control - ps[t_idx])
best_match = np.argmin(distances)
if distances[best_match] <= caliper:
c_idx = control_idx[best_match]
matched_cal.append(spend[t_idx] - spend[c_idx])
else:
dropped += 1
matching_att_cal = np.mean(matched_cal)
print(f"Caliper matching ATT: ${matching_att_cal:.2f}")
print(f"Dropped: {dropped} treated units ({dropped/len(treated_idx)*100:.1f}%)")
Caliper matching ATT: $9.17
Dropped: 0 treated units (0.0%)In our case, the caliper didn’t drop anyone because the overlap between groups is good. In messier datasets, the caliper becomes essential for avoiding bad matches.
Strengths of matching: Highly intuitive — you’re literally comparing like with like. Easy to inspect individual matches and verify quality. The balance check provides a clear diagnostic. Works well even when the outcome model is complex.
Weaknesses: With replacement means some control units are reused, which understates variance. Without replacement introduces sensitivity to match order. Discards treated units that can’t be matched, which changes the target population. And like IPW, it’s only as good as the propensity score model.
Putting It All Together
Let’s compare all three methods side by side — including the naive estimate from Part 1:
naive = df[df.received_email == 1].spend.mean() - df[df.received_email == 0].spend.mean()
print("=" * 45)
print(f"{'Method':<25} {'Estimate':>10}")
print("=" * 45)
print(f"{'Naive comparison':<25} {'$'+f'{naive:.2f}':>10}")
print(f"{'Stratification':<25} {'$'+f'{strat_ate:.2f}':>10}")
print(f"{'IPW (trimmed)':<25} {'$'+f'{ipw_ate:.2f}':>10}")
print(f"{'PS Matching':<25} {'$'+f'{matching_att:.2f}':>10}")
print("-" * 45)
print(f"{'TRUE EFFECT':<25} {'$'+f'{TRUE_EFFECT:.2f}':>10}")
print("=" * 45)
=============================================
Method Estimate
=============================================
Naive comparison $41.38
Stratification $10.83
IPW (trimmed) $10.06
PS Matching $9.17
---------------------------------------------
TRUE EFFECT $10.00
=============================================
Three different methods, three estimates within about a dollar of the true effect. The naive comparison — the number that would have gone in the board deck — was off by $31.
The convergence is the important part. When multiple methods with different mechanics all land in the same place, that’s a strong signal. If they disagreed — say stratification gave $12 but IPW gave $25 — that would be a red flag that something is wrong with either the propensity score model or the identifying assumptions.
Which Method Should You Actually Use?
This is the question everyone asks, and the honest answer is: run all of them.
But if you need guidance on where to start:
Use stratification when you have one or two confounders and want something transparent enough to explain to a non-technical stakeholder. It’s the easiest to debug and the hardest to get catastrophically wrong.
Use IPW when you have many confounders and the propensity score model is well-specified. It’s more efficient than stratification and naturally extends to time-varying treatments. But watch the weights — if they’re extreme, your estimate is fragile.
Use matching when explainability matters and you want to show “here are the specific comparable pairs.” It’s also a good choice when the treated group is much smaller than the control group. But be careful about match quality, and always check balance.
In practice, run all three plus a regression adjustment, and report them together. If they agree, you’re in good shape. If they disagree, investigate why — the disagreement is information.
What All Three Methods Assume (and What Can Go Wrong)
All three methods rest on the same two assumptions, both inherited from the backdoor criterion in Part 2:
1. No unmeasured confounders (conditional ignorability). Every variable that affects both treatment and outcome is observed and included. If there’s a confounder we missed — say customer intent to purchase, which correlates with loyalty but isn’t captured by it — all three methods will be biased, and no amount of statistical cleverness fixes it.
2. Positivity (overlap). At every level of the confounders, there’s a non-zero chance of being in both the treated and control groups. We checked this with the propensity score overlap plot. Where positivity breaks down (extremely high or low PS), our estimates are unreliable.
These are the assumptions we documented in our DAG in Part 2. They’re untestable from the data alone — they require domain knowledge and judgment. Part 5 of this series will show how to stress-test them with sensitivity analysis: “how big would an unmeasured confounder have to be to change our conclusion?”
Where We Stand
We went from a vague business question ($65 lift!) to a rigorous causal analysis in three steps:
Part 1: Define the causal question. Understand the data-generating process.
Part 2: Draw the DAG. Identify confounders. Derive the adjustment set via the backdoor criterion.
Part 3 (this article): Estimate the causal effect using three methods that all confirm the email’s true impact is around $10.
That’s the core workflow of applied causal inference. For many real-world problems, this is sufficient. You define, you map, you estimate, and you report with the appropriate caveats about assumptions.
But real data is messier than our simulation, and sometimes the backdoor criterion can’t be satisfied — you can’t observe all the confounders. That’s where the next set of methods comes in.
What’s Coming Next
Part 4: When You Can’t Observe the Confounder — Instrumental variables, difference-in-differences, and regression discontinuity. These methods work when the standard approach fails — when your DAG has an unblockable backdoor path.
Part 5: Sensitivity Analysis — Your DAG is a set of assumptions. How fragile are they? We’ll quantify how much unmeasured confounding would be needed to change our conclusion.
Part 6: The Modern Toolkit — Double/debiased machine learning and EconML for when the relationship between confounders and outcome is too complex for parametric models.
All code from this series is available as a notebook on GitHub. Built with Python 3.11, numpy, pandas, scipy, and matplotlib.
Part 4 drops next week. Things are about to get harder — and more interesting.
Brian James Curry is an Enterprise AI Strategist, Data and Marketing Scientist, and Applied Researcher. He is the founder of Vector1 Research, an independent research lab focused on marketing economics, causal systems, cognitive AI architecture, and the economic implications of intelligent systems. He also founded KC AI Lab, a data science community and incubator for applied machine learning education.
