Causal Modeling : When to Discover, When to Infer

Part II : Measuring structural knowledge in bits, and why no amount of experimentation can recover a causal graph on its own

TL;DR Part I argued that causal discovery and causal inference are different problems. This piece answers the operational question: which one does your situation call for?

The answer turns out to be quantitative. Structural knowledge can be measured in bits, and once you do that, three things fall out of the arithmetic:

On a 5-variable system, observational discovery resolves 9.84 of 14.84 bits of structural uncertainty at roughly 800× the information-per-dollar of a randomized experiment.
Experiments alone cannot finish the job, ever. Run the optimal experiment sequence with no discovery step and it plateaus at 64 surviving graphs with 64 distinct skeletons a hard identifiability wall, not a budget problem.
Discovery alone cannot finish either. It leaves the Markov equivalence class untouched.

They are complementary, not competing — and the division of labor between them is formal and provable, not a matter of taste. Everything below is computed by exhaustive enumeration over all 29,281 DAGs on five nodes.

1. The wrong question

“Should we use causal discovery or causal inference?” is malformed in the same way as “should we use a map or a speedometer?”

But practitioners keep asking it, and they ask it because both disciplines market themselves as the answer to “what causes what.” So let me replace it with a question that has an actual answer:

How much of your structural uncertainty is already resolved, and what is the cheapest instrument for resolving the rest?

This is answerable, and the units are bits.

2. Structural information accounting

2.1 The prior

Before any analysis, your uncertainty about the causal structure over d variables is uncertainty over the space of DAGs. Uniform over that space, the entropy is H₀ = log₂ | DAGs(d) |.

Using Robinson’s recurrence from Part I, here is how structural ignorance scales as you add variables:

3 Variables: 25 DAGs (4.64 bits of uncertainty)
4 Variables: 543 DAGs (9.08 bits of uncertainty)
5 Variables: 29,281 DAGs (14.84 bits of uncertainty)
10 Variables: 4.18 × 1⁰¹⁸ DAGs (62.0 bits of uncertainty)
20 Variables: 2.34 × 1⁰⁷² DAGs (240.3 bits of uncertainty)

Note the growth: H₀ scales as O(d²), because the number of possible edges does. Twenty variables is 240 bits of structural ignorance.

2.2 What observational discovery buys

Discovery cannot return a DAG. It returns a Markov equivalence class (MEC)the set of graphs with identical conditional independence structure. By the Verma Pearl characterization, two DAGs are Markov equivalent exactly when they share a skeleton and a set of v-structures.

For the conversion-rate system from Part I:

Hypothesis space (all DAGs): 29,281 (H = 14.84 bits)
After observational discovery: 32 candidates remain (H = 5.00 bits
DISCOVERY YIELDS: 9.84 bits (66% of the problem solved)
Residual ambiguity (the MEC): 5.00 bits

Discovery, using no experiments at all, eliminates 29,249 of 29,281 candidate structures.

2.3 What an experiment buys

Model a perfect intervention the way a real test reads out: randomize X, then observe which variables respond. What you learn is the descendant set of X. Applied to the 32 graphs discovery left standing, here is the expected information gain for testing each variable:

Intervening on conversion: yields 3.078 expected bits (10 distinct outcomes)
Intervening on promo_depth: yields 3.078 expected bits (10 distinct outcomes)
Intervening on traffic_quality: yields 2.680 expected bits (8 distinct outcomes)
Intervening on cart_adds: yields 2.680 expected bits (8 distinct outcomes)
Intervening on support_tickets: yields 1.186 expected bits (4 distinct outcomes)

2.4 Bits per dollar

Price them. Discovery is compute plus analyst time call it $200. A single adequately-powered experiment, loaded with engineering, traffic cost, and three weeks of calendar, is conservatively $50,000.

Discovery: 9.84 bits / $200 = 49.19 bits per $1k
Experiment: 3.08 bits / $50,000 = 0.062 bits per $1k

Discovery is ~800x more information-efficient AT THIS STAGE.

The qualifier is essential and I want to be emphatic about it, because the number is otherwise easy to abuse. That ratio is a function of where you are, not a property of the methods. Discovery is enormously efficient when structural uncertainty is high, and its marginal value goes to zero the moment you hit the equivalence class. Which brings us to the part that actually matters.

3. The complementarity result

Here is the experiment worth running on the methods themselves. Take two paths to the true graph and give both an unlimited budget.

Path A : experiments only. No discovery. Adaptively choose the maximally informative intervention at each step:

Start: 29,281 candidates (14.84 bits)
Step 1: intervene traffic_quality (+3.549 bits) $\rightarrow$ 3,377 candidates
Step 2: intervene promo_depth (+2.689 bits) $\rightarrow$ 632 candidates
Step 3: intervene cart_adds (+1.848 bits) $\rightarrow$ 160 candidates
Step 4: intervene conversion (+0.971 bits) $\rightarrow$ 64 candidates
Step 5: NO intervention yields any information. PLATEAU.
Result: Stuck at 64 candidates (6.00 bits irreducible).

Path B — discovery, then experiments.

Start: 32 candidates (5.00 bits)
Step 1: intervene promo_depth (+3.078 bits) $\rightarrow$ 2 candidates
Step 2: intervene cart_adds (+1.000 bits) $\rightarrow$ 1 candidate
Result: Identified after 2 experiments.

Path A: 4 experiments, $200,000 spent, NOT IDENTIFIED — and no further experiment can help.

Path B: 2 experiments + discovery, $100,200, fully identified. Path B is the optimal strategy. Causal discovery draws the map so that causal inference knows exactly where to drive.

3.1 Why Path A hits a wall

This is not a failure of my search procedure. It is an identifiability limit: all 64 survivors have different edge sets. Interventions reveal descendants (the transitive closure), and the transitive closure does not determine adjacency. No experiment distinguishes them, because there is nothing in “which variables responded” that separates a direct edge from a longer route.

So the division of labor is formal:

Observational Discovery gives you Adjacency (skeleton + v-structures)but not full orientation.
Intervention gives you Orientation (descendant sets) but not adjacency.

Each supplies precisely what the other cannot. This is why “we’ll just A/B test our way there” and “we’ll just run causal discovery” are both wrong, and wrong in exactly opposite directions.

3.2 How the split scales

Enumerating exhaustively for every graph size we can compute:

3 Variables: 4.64 bits prior $\rightarrow$ Discovery resolves 69.2% (leaves 1.43 residual bits)
4 Variables: 9.08 bits prior $\rightarrow$ Discovery resolves 78.1% (leaves 1.99 residual bits)
5 Variables: 14.84 bits prior $\rightarrow$ Discovery resolves 84.5% (leaves 2.30 residual bits)

The residual grows glacially while H₀ grows quadratically. The strategic implication is the opposite of most teams’ instinct:

The more variables you have, the larger the fraction of the structural problem that discovery can solve, and the smaller the fraction requiring experiments. At $d = 5$ discovery does 85% of the work. At $d = 20$ it does over 99%. Organizations run this backwards they scale up experimentation as systems get more complex, precisely when its relative value is collapsing.

4. A selection criterion

I’ll name the quantity, since it’s the practical output of all this.

Structural Information Ratio (SIR) the fraction of structural uncertainty already resolved:

SIR = ( H₀ − H_current ) / H₀

And the decision rule it implies:

SIR < 0.5 (DISCOVERY REGIME): Experiments are near-worthless here. You do not yet know what to test. Run discovery. Add domain constraints.
SIR 0.5–0.9 (HYBRID REGIME): Discovery has done its job. Use the surviving candidates to DESIGN the experiment (§5). Highest-leverage zone.
SIR > 0.9 (INFERENCE REGIME): Structure is effectively settled. Stop discovering. Estimate magnitudes: backdoor, IV, DiD, DML, RCT.

Most teams that believe they are “doing causal work” are operating at SIR ≈ 0.1 while using SIR > 0.9 tooling. That mismatch is the pathology described in Part I flawless experiments on arbitrarily chosen variables.

5. Experiment design as an output of discovery

Back to §2.3. The spread between the best and worst intervention:

BEST: conversion (3.078 bits)
WORST: support_tickets (1.186 bits)

Same budget, same three weeks, 2.6× the structural information. And support_tickets — the worst choice — is exactly what a correlation-ranked test roadmap would have prioritized: from Part I, it has the highest marginal correlation with conversion (r = 0.754) of any variable in the system.

The variable most correlated with your outcome was the least informative thing you could have tested. It is a collider. It causes nothing, and intervening on it merely blocks paths rather than revealing downstream descendants. Testing it burns $50,000 to learn 1.186 bits when 3.078 were available.

Selecting the intervention that maximally splits your surviving candidate set is a small amount of code and is the single highest-ROI addition available to most experimentation programs. Discovery does not replace your experimentation program — it tells that program what to test.

6. Use-case taxonomy

Mapping this onto the domains I work in:

Marketing Mix & Attribution
Discovery’s Role (High): Do channels interact? Does paid search cannibalize organic? Is the brand-to-performance path real? (SIR is near zero).
Inference’s Role: Given the structure: incrementality, saturation curves, marginal ROAS, geo-holdouts.
The Common Pitfall: Estimating channel contribution from a regression whose structure nobody ever specified.
Conversion-Rate Optimization
Discovery’s Role (High): Which of 40 tracked metrics are causes, consequences, or colliders? Nearly all value is here, yet it’s usually skipped.
Inference’s Role: A/B testing the specific element. Well-solved.
The Common Pitfall: Testing whatever ranked highest in a correlation matrix (often a collider).
Economic & Money-Flow Analysis
Discovery’s Role (Absolute): Where does value flow? Intervention is often impossible or unethical, so discovery is the only instrument available.
Inference’s Role: Natural experiments: policy discontinuities, staggered rollouts, DiD, synthetic control.
The Common Pitfall: Treating “no experiment possible” as “no causal claim possible.” A counsel of despair, and false.
Product Analytics & Retention
Discovery’s Role (High): Does the feature drive retention, or do retained users find the feature? Direction is the entire question.
Inference’s Role: Feature-flag rollouts once the causal direction is established.
The Common Pitfall: Reverse causation shipped as a product roadmap.
Time Series & Regime Detection
Discovery’s Role (High): Lagged structure, regime shifts. Temporal order is a free, powerful domain constraint that collapses $H_0$.
Inference’s Role: Interrupted time series, synthetic control.
The Common Pitfall: Reporting Granger causality as structural causality. It is a predictive-precedence test, not a structural one.

7. Four ways to pick wrong

Testing at SIR ≈ 0. The experiment is clean and the variable was chosen from a correlation ranking. You paid full freight for a third of the available information, and possibly for a collider that causes nothing.
Shipping a CPDAG as if it were a DAG. Discovery returned an equivalence class; someone read the arrows off the plot. Report the equivalence class, or report that you resolved it and how.
“Control for everything” as a causal method. Adjusting for all measured covariates often flips the sign of the estimate because one covariate was a collider. This is invisible in regression output.
Experimenting on a mediator and calling it the total effect. Randomizing cart_adds gives you a real, unbiased effect — of cart_adds. It tells you nothing about promo_depth, whose total effect flows partly through it.

8. The decision procedure

Do you have a specific, pre-specified X and Y, and a defensible account of the confounding structure?

YES $\rightarrow$ INFERENCE. Identify (backdoor / IV / DiD / DML / RCT), estimate, refute. Report sensitivity to the structure you assumed. You are done.
NO $\rightarrow$ Compute SIR.

2. Can you enumerate candidate structures?

Apply domain constraints FIRST. Temporal order is a free, powerful domain constraint. If you know X happens before Y, the edge Y $\rightarrow$ X is impossible, instantly cutting the hypothesis space for that pair in half and exponentially reducing $H_0$.

3. Route by SIR:

SIR < 0.5 $\rightarrow$ DISCOVERY. PC / GES / LiNGAM / NOTEARS. Bootstrap for edge stability. Do not run experiments yet.
SIR 0.5–0.9 $\rightarrow$ HYBRID. Rank interventions by expected bits. Run the top one. Recompute. Repeat.
SIR > 0.9 $\rightarrow$ INFERENCE. Proceed as step 1.

The recursion matters: after each experiment, SIR moves, and the correct method may change. This is a loop, not a pipeline.

9. Limitations

The bit accounting assumes a uniform prior over DAGs. Domain knowledge is a prior, and a good one. Encoding domain constraints is free, and it is the highest-return step in the entire procedure.
Perfect interventions. Real experiments are imperfect. When an experiment fails to reach significance, organizations often blame sample size when they should actually be questioning the causal graph (e.g., they intervened on an isolated node).
Exhaustive enumeration stops at about d = 6. Beyond that, MEC sizes must be sampled or bounded.
Causal sufficiency, again. Latent confounders push you to FCI and PAGs, and the clean adjacency/orientation split above becomes messier.
The dollar figures are illustrative. $200 and $50,000 are my estimates for a mid-size organization. Substitute your own; the ordering rarely changes.

10. The Playbook: A Step-by-Step for Practitioners

If entropy math isn’t your daily language, here is the exact translation of what this framework looks like in practice.

Phase 1: Causal Discovery (Mapping the System)

Think of this phase as drawing the map before you decide where to drive. You don’t need a budget for this; you just need your existing observational data.

Step 1: List your variables. Gather the metrics you actually track and suspect are related (e.g., ad spend, organic traffic, cart adds, final sales).
Step 2: Apply common sense. Tell the algorithm what is physically impossible. If variable A happens on Monday and variable B happens on Friday, B cannot cause A. This alone eliminates millions of wrong assumptions.
Step 3: Let the algorithm find the shape. Feed the data into a discovery algorithm (like PC or GES). The algorithm looks at patterns in the data to figure out which variables are directly connected (the “skeleton”) and which ones are definitely causing others.
Step 4: Find the blind spots. The algorithm will hand you a mostly finished map, but a few arrows will still be double-sided (ambiguous). It has narrowed the universe of possibilities down to a tiny handful of remaining options. Stop here. Discovery has done its job.

Phase 2: Causal Inference (Testing the Map)

Now that you know exactly where the map is ambiguous, you run targeted tests to force the system to reveal the final answers.

Step 1: Pick the right test. Look at the handful of possible maps left over from Phase 1. Find the specific variable that, if you changed it, would instantly prove which map is correct. This is the only thing you should A/B test right now.
Step 2: Run the experiment. Intervene. Change budgets, hide a feature, or run an A/B test on the exact variable you identified in Step 1.
Step 3: Measure the flow. Watch what happens downstream. Because Phase 1 already gave you the structure, you aren’t just looking at “did sales go up?” You are looking at the exact path the impact traveled.
Step 4: Update the map. Use the results of your test to draw the final, definitive arrows. You now have a solved causal graph and can accurately measure ROI across the entire system.

11. Closing

The two disciplines are not competitors. They are instruments that read different parts of the same object, and the reason to keep them terminologically distinct is that each is blind exactly where the other sees.

Observational data cannot orient the equivalence class. Interventions cannot recover adjacency. Run only experiments and you stall at 64 graphs; run only discovery and you stall at 32. Run them in the right order and two experiments finish what four could not.

Discovery tells you what to test. Testing tells you what discovery could not see. The organizations that get causality right are not the ones with the biggest experimentation programs — they are the ones that know which of the two problems they are solving on any given day.

All figures computed by exhaustive enumeration over the 29,281 labeled DAGs on five nodes. The companion script reproduces every number, including the plateau. NumPy and SciPy only. Script available upon request.

Part I: “Causal Discovery Is Not Causal Inference” — the formal distinction, with the worked example this piece builds on.

References

Chickering, D. M. (2002). Optimal Structure Identification with Greedy Search. JMLR 3, 507–554.
Eberhardt, F., Glymour, C., & Scheines, R. (2005). On the Number of Experiments Sufficient and in the Worst Case Necessary to Identify All Causal Relations Among N Variables. UAI.
Gillispie, S. B., & Perlman, M. D. (2001). Enumerating Markov Equivalence Classes of Acyclic Digraph Models. UAI.
Hauser, A., & Bühlmann, P. (2012). Characterization and Greedy Learning of Interventional Markov Equivalence Classes of Directed Acyclic Graphs. JMLR 13, 2409–2464.
Hauser, A., & Bühlmann, P. (2014). Two Optimal Strategies for Active Learning of Causal Models from Interventional Data. International Journal of Approximate Reasoning 55(4).
He, Y.-B., & Geng, Z. (2008). Active Learning of Causal Networks with Intervention Experiments and Optimal Designs. JMLR 9, 2523–2547.
Lindley, D. V. (1956). On a Measure of the Information Provided by an Experiment. Annals of Mathematical Statistics 27(4).
Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press.
Peters, J., Janzing, D., & Schölkopf, B. (2017). Elements of Causal Inference. MIT Press.
Robinson, R. W. (1977). Counting Unlabeled Acyclic Digraphs. Combinatorial Mathematics V, Springer.
Runge, J., et al. (2019). Detecting and Quantifying Causal Associations in Large Nonlinear Time Series Datasets. Science Advances 5(11).
Shimizu, S., Hoyer, P. O., Hyvärinen, A., & Kerminen, A. (2006). A Linear Non-Gaussian Acyclic Model for Causal Discovery. JMLR 7, 2003–2030.
Spirtes, P., Glymour, C., & Scheines, R. (2000). Causation, Prediction, and Search (2nd ed.). MIT Press.
Verma, T., & Pearl, J. (1990). Equivalence and Synthesis of Causal Models. UAI.

About the Author

Brian Curry is a Kansas City–based AI Strategist, Marketing Economist, and Applied Researcher. He drives Vector1, a personal open-source research initiative advancing marketing economics, causal inference, cognitive AI architectures, and the economics of intelligent automation.

He has led advanced data science and analytics initiatives across major enterprises including Koch Industries, Tractor Supply, Hallmark, Garmin, and AT&T. His work applies economic thinking and causal inference to connect marketing investment to outcomes, operational decisions to economic value, and AI adoption to competitive advantage.

Key Research & Open-Source Projects (GitHub: @Bodhi8):

PyCausalSim — A Python framework for causal discovery and inference through simulation. Enables testing of counterfactual scenarios to identify true structural drivers.
Papilon — A complex system and causal discovery library for Media Mix Modeling (MMM) and adaptive systems.
Regimecast — Applied research and tooling for time-series regime detection and forecasting.
CarryNet — Open-source research and modeling focused on commodity carry trade analysis.
Foodmesh-ml — Machine learning, optimization, and simulation framework for modeling distributed food systems.
Memory-Node Encapsulation (MNE) — A cognitive AI architecture and novel data structure for artificial episodic memory.
The Autonomy Economy — A macroeconomic framework projecting the dual impacts of massive productivity growth and extreme price deflation in an AI-automated world.
Meaning-Flow — Research focused on semantic content modeling and automated coverage gap analysis.

Contact: brian at vector1.ai | Vector1 | GitHub | LinkedIn

This article may be shared with attribution under Creative Commons Attribution 4.0 International (CC BY 4.0).