Finding Signal: A Practitioner’s Guide to EDA Methods for Alpha Discovery

Correlation, mutual information, tree importances, SHAP, and the statistical traps that turn noise into false discoveries.

What “signal” actually means

Most exploratory data analysis tutorials treat EDA as a hygiene step: check distributions, plot a correlation heatmap, drop nulls, move on. That framing is fine for cleaning data and useless for the harder question every quant and applied data scientist actually faces: which of these features carry predictive information about the target, how much, and is it real?

It helps to define terms precisely, because they generalize far beyond trading:

Signal is the component of a feature’s relationship with the target that is stable and reproducible — it survives resampling, regime changes, and out-of-sample testing.
Noise is the component that looks like a relationship in your sample but is an artifact of finite data.
Alpha, generalized out of its finance origin, is predictive edge beyond a naive baseline. A feature has alpha if a model using it beats a model that ignores it — net of the complexity you spent to find it. The “net of complexity” clause is the entire game. Almost any feature improves in-sample fit; very few improve honest out-of-sample performance.

EDA for signal discovery is therefore an exercise in measuring dependence while controlling for self-deception. The methods below are organized into four tiers: measuring pairwise dependence, attributing importance inside models, understanding structure and redundancy, and — most importantly — the statistical machinery that tells you whether anything you found is real.

Tier 1 — Measuring dependence between a feature and the target

Pearson correlation: the linear baseline

Pearson’s r measures the strength of a linear relationship. It is the right first tool precisely because it is restrictive: if a strong linear relationship exists, you want to know immediately, and r is cheap, interpretable, and has a closed-form significance test.

import numpy as np
from scipy import stats

r, p = stats.pearsonr(x, y)
print(f"Pearson r = {r:.4f}, p = {p:.2e}")

Its failure modes are well known but worth stating because they motivate everything that follows. Pearson’s r is blind to nonlinear structure (a perfect parabola can have r ≈ 0), it is badly distorted by outliers and heavy tails (a single extreme point can manufacture or destroy a correlation), and it assumes a roughly bivariate-normal relationship for its significance test to be exact. Financial returns, click distributions, biological measurements, and most real-world features violate at least one of these. Treat a high Pearson r as a strong lead and a low one as uninformative, not as evidence of independence.

Spearman and Kendall: rank correlations

Spearman’s ρ is Pearson’s r computed on the ranks of the data. This single change buys robustness to outliers and sensitivity to any monotonic relationship, not just linear ones. Kendall’s τ measures the same monotonic association via concordant/discordant pair counts; it is more robust still and has better small-sample behavior, at higher computational cost.

rho, p_s = stats.spearmanr(x, y)
tau, p_k = stats.kendalltau(x, y)

A practical heuristic: report Spearman alongside Pearson by default. A large gap between them is itself diagnostic — it usually means heavy tails, outliers, or a monotonic-but-curved relationship. In quantitative finance, the rank correlation between a predictor and forward returns is essentially the Information Coefficient (IC), the field’s native measure of signal strength; the same quantity under a different name shows up as a ranking-quality metric in recommender systems and learning-to-rank.

Distance correlation: catching nonlinear dependence

Rank correlations still only detect monotonic structure. To catch arbitrary dependence — including the parabola, the periodicity, the conditional-variance relationship — you need measures that go to zero if and only if the variables are statistically independent.

Distance correlation has exactly this property: it is zero iff the variables are independent, captures nonlinear and non-monotonic dependence, and works for multivariate inputs.

from scipy.spatial.distance import pdist, squareform

def distance_correlation(x, y):
    x = np.atleast_2d(x).reshape(len(x), -1)
    y = np.atleast_2d(y).reshape(len(y), -1)
    a = squareform(pdist(x)); b = squareform(pdist(y))
    A = a - a.mean(0) - a.mean(1)[:, None] + a.mean()
    B = b - b.mean(0) - b.mean(1)[:, None] + b.mean()
    dcov2 = (A * B).mean()
    dvar_x = (A * A).mean(); dvar_y = (B * B).mean()
    return np.sqrt(dcov2 / np.sqrt(dvar_x * dvar_y))

Mutual information: dependence as reduction in uncertainty

Mutual information is the most general dependence measure in this tier, and it deserves its own treatment because it is both the most powerful and the most frequently misused.

The idea. MI measures how much knowing the feature reduces your uncertainty about the target, where uncertainty is Shannon entropy. Formally, for variables $X$ and $Y$,

$$I(X;Y) = H(Y) — H(Y \mid X) = \sum_{x,y} p(x,y),\log\frac{p(x,y)}{p(x),p(y)}$$

Read the second form directly: it is the average log-ratio between the joint distribution and the product of marginals. If $X$ and $Y$ are independent, $p(x,y) = p(x)p(y)$, the ratio is 1, the log is 0, and $I(X;Y) = 0$ exactly. Any departure from independence — linear, monotonic, periodic, heteroskedastic, or a relationship that only exists in the tails — pushes the joint away from the product of marginals and makes MI positive. This is the property correlation lacks: MI is zero if and only if the variables are statistically independent, with no functional-form assumption whatsoever.

Why it earns its place in signal discovery. MI is model-free (it commits to no functional form, so it cannot be fooled the way Pearson is by a parabola), it handles continuous, discrete, and mixed variable types, and it extends naturally to the multivariate and conditional cases you actually need for feature selection. It is, in a precise information-theoretic sense, an upper bound on what any model could extract from that feature in isolation — which makes it a principled ceiling on a feature’s standalone predictive value.

The hard part is estimation. MI is defined on densities you do not have; you have a finite sample. How you estimate it dominates the result.

Binning / histogram estimators discretize each variable and count. Simple and fast, but the answer is extremely sensitive to bin count, and binning systematically overestimates MI in finite samples — pure noise produces apparently positive MI, and more bins produce more spurious MI. Avoid histogram MI for continuous features unless you correct the bias explicitly.
k-nearest-neighbor (Kraskov–Stögbauer–Grassberger, “KSG”) estimators work directly in continuous space without binning and are the practical default. This is what scikit-learn’s mutual_info_regression / mutual_info_classif implement. They are far better behaved than binning but still carry a small positive bias and depend on the neighbor count k (a low k reduces bias but inflates variance; k = 3–5 is the usual compromise).

from sklearn.feature_selection import mutual_info_regression, mutual_info_classif

# Continuous target. discrete_features tells the estimator which columns
# are categorical so it uses the right (mixed) estimator per feature.
mi = mutual_info_regression(
    X, y, discrete_features=is_discrete_mask,
    n_neighbors=5, random_state=0,
)
mi = pd.Series(mi, index=X.columns).sort_values(ascending=False)

Make it comparable: normalization. Raw MI is in nats (or bits) and has no fixed upper bound, so a value of 0.21 means nothing in isolation and MI scores are not comparable across targets. Normalize when you need an interpretable, bounded number — a common choice rescales MI by the entropies of the two variables to land in [0, 1], analogous to a correlation coefficient:

$$\mathrm{NMI}(X;Y) = \frac{I(X;Y)}{\sqrt{H(X),H(Y)}}$$

The pitfall that matters most: MI has no free null. Unlike Pearson or Spearman, MI has no simple analytic significance test, and because every estimator is positively biased, a positive MI estimate is not evidence of dependence. The correct test is a permutation null: shuffle the target, recompute MI many times to trace out the distribution of MI under genuine independence for your sample size and estimator, and judge the observed value against that distribution. Skipping this step is the single most common way MI manufactures false signal.

def mi_permutation_test(x, y, n_perm=500, random_state=0):
    rng = np.random.default_rng(random_state)
    obs = mutual_info_regression(x.reshape(-1, 1), y, random_state=0)[0]
    null = np.array([
        mutual_info_regression(x.reshape(-1, 1),
                               rng.permutation(y), random_state=0)[0]
        for _ in range(n_perm)
    ])
    p_value = (1 + (null >= obs).sum()) / (1 + n_perm)
    return obs, null.mean(), p_value   # subtract null.mean() for a debiased MI

Conditional MI and feature selection. MI’s real power for signal discovery is the conditional form, $I(X; Y \mid Z)$ — the information $X$ carries about $Y$ beyond what $Z$ already provides. This is what lets you distinguish a feature that is genuinely additive from one that is merely redundant with features you already have, addressing the univariate blind spot directly. The classic operationalization is mRMR (minimum-redundancy, maximum-relevance): greedily select features with high MI to the target but low MI to the already-selected set. It is a far better filter than ranking by marginal MI alone, though it remains a filter, not a substitute for the model-based importance in Tier 2.

One sign-related caveat. MI is unsigned: it tells you a feature is informative but not the direction of the effect. Pair a high-MI finding with a SHAP dependence plot or a simple conditional-mean plot to recover direction before you trust it as a signal.

A clean default screening stack: Spearman for monotonic signal with a free significance test, distance correlation as an independence test, and MI to rank nonlinear features against each other. Never rely on a single dependence measure — each is blind to something.

A caution on screening by univariate dependence

Univariate screening has a structural blind spot: a feature can be useless alone but powerful in combination (the canonical example is XOR, where each input has zero marginal correlation with the output but the pair is perfectly predictive). Conversely, a feature with strong marginal correlation can be entirely redundant given features you already have. Univariate dependence is a cheap first filter, not a feature-selection method. That is what the next tier is for.

Tier 2 — Importance and attribution inside models

To capture interactions and conditional structure you must put features into a model and ask what the model relies on. Tree ensembles are the workhorse here because they capture nonlinearities and interactions automatically, need little preprocessing, and expose several importance measures.

Impurity (Gini / “MDI”) importance — and why to distrust it

The default feature_importances_ on a fitted Random Forest or gradient-boosting model is Mean Decrease in Impurity: how much each feature reduced split impurity, averaged over trees. It is free (computed during training) but carries a serious, well-documented bias: it inflates the apparent importance of high-cardinality and continuous features, because such features offer more candidate split points and can overfit noise. A purely random continuous column can outrank a genuinely useful binary one.

from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(n_estimators=500, max_features="sqrt",
                           min_samples_leaf=20, n_jobs=-1, random_state=0)
rf.fit(X_train, y_train)
mdi = pd.Series(rf.feature_importances_, index=X.columns).sort_values()

Use MDI for a fast first look, never as your final word on importance.

Permutation importance — model-agnostic, but watch correlated features

Permutation importance measures how much a performance metric degrades on held-out data when one feature’s values are shuffled, breaking its relationship with the target while preserving its marginal distribution. It is model-agnostic, tied directly to predictive performance, and computed out-of-sample.

from sklearn.inspection import permutation_importance

r = permutation_importance(rf, X_valid, y_valid, n_repeats=30,
                           scoring="r2", random_state=0, n_jobs=-1)
perm = pd.Series(r.importances_mean, index=X.columns)

Its key pitfall: with correlated features, permuting one in isolation lets the model recover the information from its correlated partner, so both look unimportant and you wrongly conclude neither matters. Mitigations: cluster features and permute clusters jointly, or use conditional/grouped permutation. Always compute permutation importance on a validation set, never on training data — on training data it measures memorization.

SHAP — consistent attribution with local explanations

SHAP values assign each feature its game-theoretic Shapley contribution to each individual prediction, relative to a baseline. SHAP gives consistent global rankings (a feature the model genuinely relies on more cannot receive a smaller attribution), local explanations (why this prediction came out as it did), and a view of interactions and effect direction that bare importance scores cannot provide. For tree models, TreeExplainer computes exact SHAP values efficiently.

import shap

explainer = shap.TreeExplainer(rf)
sv = explainer.shap_values(X_valid)

shap.summary_plot(sv, X_valid)                 # global: ranking + direction + spread
shap.dependence_plot("feature_42", sv, X_valid) # nonlinearity + interaction for one feature

Reading the summary plot is where the actual signal discovery happens: a feature with high mean |SHAP| is influential; the color gradient tells you the direction (does a high feature value push predictions up or down?); the spread of points reveals interaction effects (if the same feature value produces wildly different SHAP values, its effect is conditional on something else). Two caveats: SHAP explains what the model learned, not ground truth — a leaky or overfit model produces confident, wrong attributions — and correlated features can have attribution split between them somewhat arbitrarily. SHAP makes a model interpretable; it does not make a bad model true.

Triangulation, not a single number

MDI, permutation importance, and SHAP measure different things — split contribution, performance degradation, and game-theoretic attribution. Features that rank highly under all three are credible signal candidates. Disagreements are informative, not annoying: a feature with high MDI but low permutation importance is often a high-cardinality column overfitting noise; a feature with high SHAP but low standalone correlation is often carrying a genuine interaction. The disagreements point you at exactly the features worth investigating.

Tier 3 — Structure, redundancy, and the geometry of your feature set

A list of “important” features is misleading if half of them are near-duplicates. Redundancy inflates variance, destabilizes importances, and wastes model capacity.

Multicollinearity: VIF and correlation clustering

The Variance Inflation Factor quantifies how much a feature is explained by a linear combination of the others; rules of thumb flag VIF > 5–10 as problematic. More useful in practice is to cluster features on a correlation- or distance-correlation-derived distance and keep one representative per cluster.

from scipy.cluster.hierarchy import linkage, fcluster
from scipy.spatial.distance import squareform

corr = pd.DataFrame(X, columns=cols).corr(method="spearman").abs()
dist = 1 - corr
Z = linkage(squareform(dist.values, checks=False), method="average")
clusters = fcluster(Z, t=0.3, criterion="distance")  # tune the threshold

Keep, per cluster, the member with the strongest dependence on the target or the cheapest/most-interpretable one. This single step often does more for model stability than any amount of hyperparameter tuning.

PCA and eigenstructure

The eigenvalue spectrum of the feature covariance/correlation matrix reveals intrinsic dimensionality. A spectrum dominated by a few large eigenvalues means most features are repackaging a handful of latent factors — common in markets (a dominant market factor), sensor arrays, and survey data. PCA is excellent for diagnosis (how redundant is this feature set, really?). Be cautious using principal components directly as model features: they are linear, global, fit without reference to the target, and unstable under distribution shift, which makes them poor primary alpha sources even when they are good diagnostics.

Tier 4 — The statistical machinery that decides if any of it is real

This is the tier that separates practitioners from people who generate plausible-looking nonsense. Every method above will happily report a strong relationship in pure noise if you let it.

Effect size vs. statistical significance

With large n, everything is statistically significant. A p-value of 1e-12 on a correlation of 0.02 means you have precisely measured a relationship too weak to matter. Always report and reason about effect size (the magnitude — r, MI in nats, mean |SHAP|, IC) and treat the p-value only as a crude noise filter, never as evidence of importance. In high-frequency or large-panel settings, every feature clears any significance threshold; ranking by effect size is the only thing that carries information.

The multiple-comparisons problem (a.k.a. data dredging)

If you test 1,000 random features against a target at α = 0.05, you expect 50 false positives by construction — and the ones that survive will be precisely those that fit the noise best. This is the single biggest source of false alpha in both quant finance and applied ML. Defenses, in rough order of strength:

Correct the threshold. Bonferroni (conservative) or Benjamini–Hochberg false discovery rate control (preferred when testing many features).
Hold out data you never look at until a single final evaluation. Every peek spends statistical power.
Track the effective number of trials. In finance this motivates the Deflated Sharpe Ratio; the general principle is that a result’s credibility must be discounted by how many things you tried to get it.
Demand a mechanism. A feature with a plausible causal or structural story survives out-of-sample far more often than one found by pure search. This is not statistics, but it is the most reliable filter in practice.

Cross-validation that respects your data’s structure

Naive k-fold cross-validation leaks whenever observations are not exchangeable, which is almost always in real signal problems:

Temporal data: random folds train on the future to predict the past. Use forward-chaining / TimeSeriesSplit. For overlapping labels (a return computed over a forward window), add purging and embargoing of train samples adjacent to the validation block, or the validation set is contaminated by construction.
Grouped data (multiple rows per user, asset, patient, device): split by group so the same entity never appears in both train and validation, otherwise the model memorizes the entity rather than learning signal.

from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=5, gap=20)  # 'gap' embargoes leakage across the split

An “alpha” that exists under random k-fold but vanishes under structure-aware CV was never alpha — it was leakage. This is the most common way good-looking research dies, and finding out at the EDA stage is enormously cheaper than finding out in production.

Non-stationarity and regime dependence

A feature–target relationship is rarely a constant. Compute your dependence measure in a rolling window and plot it over time. A signal that is strong on average but flips sign across regimes is not one signal — it is a regime-detection problem wearing a feature’s clothes. Stability over time is often a better indicator of real, exploitable signal than peak in-sample strength, because a moderate, persistent edge survives deployment while a large, unstable one does not.

ic = (pd.DataFrame({"f": feature, "y": target})
        .rolling(252)
        .corr(method="spearman")
        .unstack()["f"]["y"]
        .dropna())
ic.plot(title="Rolling rank-IC — look for stability, not just height")

A pragmatic workflow

Clean and frame first. Fix the target definition, handle missingness deliberately (the pattern of missingness is often itself a signal), and split off a final test set you will not touch.
Univariate screen with Spearman + distance correlation + MI. Treat this as a coarse filter, never as selection.
Kill redundancy via correlation clustering before importance analysis, so importances are interpretable.
Model-based importance, triangulating MDI, permutation importance, and SHAP — all computed out-of-sample. Investigate the disagreements; that is where the interesting structure lives.
Stress-test survivors: structure-aware CV, rolling-window stability, multiple-testing correction, and a demand for a plausible mechanism.
One final evaluation on the held-out set. If it disappoints, the honest move is to resist the urge to re-loop on the same data — that is exactly how overfitting launders itself into a backtest.

The unifying idea: in low signal-to-noise problems — which describes most genuine alpha discovery, in finance and well beyond it — your tooling will always find something. The discipline that matters is not finding patterns; it is the relentless, slightly paranoid effort to prove that the pattern you found is not an artifact of your own search. The methods in Tiers 1–3 generate hypotheses. Tier 4 is what makes the difference between research and self-deception.