Harness Validation on Synthetic Data: What the Modeling Ladder Actually Produces

Part 3 of the corn forecasting series — running the full pipeline against a synthetic data generating process where ground truth is known, and what the results reveal about each tier.

Context

Parts 1 and 2 of this series built two layers of forecasting infrastructure: a modeling ladder spanning naive baselines through ML and foundation models, and an uncertainty-quantification layer of calibration diagnostics, conformal prediction, and ensembling. The remaining question is empirical: when the ladder runs end-to-end, does it produce the behavior the design predicts?

Real commodity data carries an inherent limitation for methodology validation: when models disagree, there is no way to tell which one is right. Synthetic data with a known data-generating process (DGP) removes that ambiguity. A model that fails to recover known structure has a definable failure mode. A model that beats its peers on data where ground truth is available has earned its place. Synthetic validation is what working forecasting groups run before trusting a pipeline on production data — and skipping it is one of the more common reasons forecasting deployments produce silent failures in their first months.

This piece runs the full ladder against a synthetic corn-like series, reports the actual results, and isolates four findings worth carrying back to real-data work.

Implementation note. The runtime environment used here lacked the LightGBM and statsforecast packages. Methodologically equivalent substitutes were used: sklearn.ensemble.GradientBoostingRegressor with the quantile-loss objective (same mechanism LightGBM uses internally) for the ML tier; hand-rolled Holt-Winters and ridge-AR(p) for the classical tier. Results are illustrative of the ladder structure rather than calibrated to specific library defaults.

The synthetic DGP

The DGP was designed to be honest about its structure rather than easy. Daily business-day frequency, 2012–01–02 through 2026–05–01 (3,740 observations). The target series combines:

Mild upward trend (~2 cents per year, base level $5/bushel)
Annual seasonality with a $0.45 peak-to-trough amplitude, peaking in late June (crop-cycle reflex)
Three regime shifts modeled as smooth squared-exponential bumps: 2012 drought (amplitude $3.0), 2020 COVID recovery ($1.8), 2022 Ukraine spike ($2.8)
GARCH-like persistent volatility with elevated levels during regime transitions
Five exogenous drivers with documented correlation structure to the target: USD index (target r = −0.25), WTI crude (+0.35), natural gas (+0.22), soybean (+0.55), wheat (+0.40)

The series spans a range of $4.28–$8.55 with three clearly visible regime spikes. The smooth squared-exponential bumps are easier than real regime shifts (no discontinuities, no volatility clustering during transitions) but harder than a pure stationary process. Ground truth components — trend, seasonal, regime, volatility, noise — are preserved alongside the realized series so that any model’s predictions can be compared to known structural truth.

Running the ladder

The walk-forward evaluation uses an expanding window with 2-year initial training, 21-day forecast horizon, and 84-day (quarterly) step between folds — 38 folds total across the dataset. Five models from the ladder were trained from scratch on each fold:

Naive — last observed value
Seasonal naive — value from 252 trading days ago (one trading year)
Holt-Winters — additive triple exponential smoothing with annual seasonality
AR(21) — autoregressive with 21 lags, ridge-regularized
Gradient boosting — multi-quantile GBR on engineered features (lags, rolling stats, calendar, return-volatility, lagged drivers)

The complete results, MAE at each horizon, averaged across all 38 folds:

Model h=1 h=5 h=10 h=21 gbm 0.092 0.113 0.113 0.117 ar 0.094 0.110 0.114 0.132 naive 0.158 0.157 0.164 0.180 seasonal_naive 0.652 0.671 0.674 0.672 holt_winters 0.537 0.813 0.910 0.971

The single-fold view shows the qualitative behavior. The aggregate distribution across all folds:

Three patterns deserve attention.

Finding 1: Seasonal naive fails because regime drift dominates seasonality

The conventional intuition for a seasonal commodity is that seasonal naive should beat plain naive — the calendar carries information, and using a year-ago value should outperform yesterday’s value. The result here is the opposite: seasonal naive (MAE 0.672) is nearly four times worse than plain naive (MAE 0.180).

This is a feature of the DGP, not a bug in the metric. The seasonal component has $0.45 peak-to-trough amplitude. The three regime shifts have amplitudes of $1.8, $2.8, and $3.0. When the seasonal-naive forecast pulls a value from 252 days ago, it is frequently pulling that value across a regime boundary. The annual reflex is real but smaller than the macro drift, and any forecaster that ignores the latter loses badly.

This generalizes beyond synthetic data. In commodity series where macro factors (geopolitics, weather, currency, policy) drive multi-year regimes, seasonal models that anchor to year-ago values systematically miss the regime, even on series that exhibit clear annual cycles. The seasonal-naive bar from Part 1 — “if your model can’t beat seasonal naive, it isn’t doing useful work” — is necessary but not sufficient. A more demanding bar is to beat plain naive when seasonality is small relative to regime drift, which holds for most macro-linked commodities.

Finding 2: Misspecified classical models can fail worse than naive

Holt-Winters with reasonable smoothing parameters performed catastrophically here (MAE 0.971 at h=21, well above naive’s 0.180). The structural reason: with additive seasonality and constant smoothing parameters, the trend term β cannot adapt fast enough through regime transitions. After a regime shift the trend estimate overshoots in the direction of the recent move, persists through the horizon, and produces forecasts that diverge from the post-shift level.

This is not a Holt-Winters failure mode; it is a misspecification failure mode. Damped trend, regime-aware variants, or proper grid search over smoothing parameters would all repair it. But the failure illustrates a broader point about classical methods: their interpretability is conditional on the specification being correct. An off-the-shelf ETS with default parameters carries no automatic guarantee of beating baselines. The “classical methods are robust” intuition holds only for series whose generative structure matches the model’s assumptions.

The same pattern appears with off-the-shelf ARIMA on series with structural breaks, with seasonal decomposition on series with multiplicative interactions, and with state-space models when state dimensionality is wrong. The defense is the modeling ladder itself: if a tier-1 method fails to beat tier-0 baselines in walk-forward CV, it should not be deployed regardless of how principled its derivation is.

Finding 3: GBM wins, but its calibration is honest only after conformal wrapping

Gradient boosting on engineered features (lags, rolling stats, calendar features, lagged exogenous drivers) achieved the best mean point accuracy (MAE 0.117 at h=21), narrowly ahead of AR(21) (0.132). This is the expected pattern when exogenous drivers carry signal and the ML model can exploit nonlinear interactions among lag features and drivers.

The interesting result is not the point accuracy but the distributional accuracy. GBM was trained with quantile objectives at α = 0.1 and α = 0.9, producing nominal 80% prediction intervals directly from the model. On the held-out evaluation folds, empirical coverage of those intervals was 70.2% — nearly 10 percentage points below the nominal level. This is silent miscalibration of exactly the type Part 2 warned about. Point accuracy is excellent; the uncertainty quantification lies.

Applying split-conformal calibration on the first 60% of folds, then evaluating on the remaining 40%:

Method Empirical coverage Mean interval width GBM raw quantiles 0.702 0.359 GBM + conformal wrapper 0.807 0.421

The conformal layer recovered nominal coverage almost exactly (0.807 vs target 0.800) at a 17% width inflation. The intervals are about 6 cents wider on a $5–8 series. This is the trade-off Part 2 described in the abstract: distribution-free coverage guarantees cost some sharpness, but the cost is modest and the result is intervals that mean what they say.

The figure tells the full calibration story across the nominal range. Raw GBM quantiles undercover at every level. Conformal-wrapped quantiles track the diagonal closely. This is exactly the failure mode that goes unnoticed when the reporting infrastructure shows only RMSE: the model is accurate, and silently overconfident.

Finding 4: Ensembling everything makes things worse

The conventional wisdom holds that ensembling improves over the best individual model. The empirical result here, on this DGP:

Strategy MAE vs. best individual Best individual (GBM) 0.117 — Equal average, all 5 models 0.294 +150% Inverse-MAE weighted, all 5 0.150 +27.5% Equal average, top 3 (naive+AR+GBM) 0.130 +10.5% Equal average, top 2 (AR+GBM) 0.117 −0.8%

The naive equal-average ensemble of all five models was substantially worse than the best individual — 150% worse, in fact. Inverse-error weighting recovers most of the gap but still loses to the best single model. Only when the ensemble was pre-filtered to the top performers did it match (top-2) or come close to matching (top-3) the best individual.

The mechanism is straightforward. The two broken models (seasonal naive and the misspecified Holt-Winters) have errors of MAE 0.67 and 0.97 respectively. Including them in an equal-weighted ensemble drags the ensemble forecast toward those wrong forecasts. Inverse-error weighting reduces but does not eliminate this contamination, because even small weights times large errors compound.

The takeaway is not “ensembles don’t work.” Top-2 ensembling slightly beat the best individual, which is the textbook result when component models have decorrelated errors and roughly comparable skill. The takeaway is ensembles work only after filtering. Adding a bad model to an ensemble degrades performance approximately in proportion to that model’s error magnitude and its weight. In production forecasting infrastructure, the ensembling step should be preceded by an explicit skill-floor check: any component with walk-forward CV error worse than some threshold (e.g., 2× the best component, or above plain naive) should be excluded from the ensemble.

This is a different operational rule than the typical “throw everything in and let the meta-model figure it out” approach taught in many tutorials. The latter is fine when component models are of comparable skill, which is the case in standardized competitions but not in real production stacks where some tiers may be misspecified.

Decision-grade view

The pinball loss for GBM at quantile 0.9 on the evaluation folds is 0.031. In a hedging context where the cost of being short physical supply is 5× the cost of carrying excess inventory, the newsvendor calculus says to target the 83rd percentile of the demand forecast distribution rather than the median. The relevant metric becomes pinball loss at q ≈ 0.83, not RMSE at the median.

The point of running this metric is not to declare a model “good” at q=0.9. The point is that the model selected on RMSE may not be the model selected on the loss-relevant quantile. In production work where the cost structure is known, decision-grade evaluation has to be built into the model-selection step, not bolted on afterward. The infrastructure built here supports that — every model that produces quantile output has its pinball loss computed at each quantile in walk-forward CV — but the ranking is left to the application.

What the synthetic exercise validates

The harness behaves correctly. Specifically:

Models with access to true structural drivers (the exogenous features, in the GBM tier) outperform models without that access by a meaningful margin.
Models with structural assumptions that match the DGP (AR for the local dynamics) come second.
Naive baselines outperform misspecified or unhelpful models (seasonal-naive, broken Holt-Winters).
Calibration diagnostics correctly identify silent miscalibration in the GBM intervals.
Conformal wrapping recovers nominal coverage at modest width cost.
Ensembling produces predictable behavior — improvement when components are skilled and comparable, degradation when components include broken models.

These are unsurprising in retrospect, but the fact that the harness produces them correctly is the point. A harness that doesn’t pass this kind of validation on synthetic data cannot be trusted to deliver clean diagnostics on production data, where the truth is unknown and ambiguity favors silent failure.

What the exercise does not address

The synthetic DGP is materially easier than real corn futures in three ways. First, the regime shifts are smooth squared-exponential bumps rather than discontinuous breaks; real regime changes (Ukraine invasion in early 2022, COVID lockdowns in March 2020) come with sharp jumps and volatility regime shifts that are not modeled here. Second, the volatility process is GARCH-like but does not model fat tails realistically — real corn returns exhibit kurtosis and occasional skewness that simple GARCH does not capture. Third, the exogenous drivers were generated with a fixed correlation structure to the target; in reality, those correlations are time-varying and structurally non-stationary.

A harness that performs well on this DGP is necessary but not sufficient for trust on real corn data. The next step — running the same ladder against yfinance-pulled ZC=F data with realistic exogenous drivers — is what would deliver the empirical capstone the series was originally building toward.

Conclusion

The three-part series has now built and validated the methodology end-to-end: ladder design (Part 1), uncertainty quantification (Part 2), and empirical behavior on a controlled DGP (Part 3). The results carry forward to real-data work directly. The harness ranks models correctly, diagnoses calibration accurately, repairs miscalibration honestly, and ensembles intelligently when filtered. These are the components of a forecasting practice that produces decision-grade output rather than accuracy-grade demos.

What the series has not delivered is a real corn forecast. That is the practical next step: run the same pipeline against actual ZC=F and the actual exogenous drivers, with a publishable GitHub repo, and report what the ladder produces on real data. The methodology is now validated; the data is what's missing.

For practitioners who have followed the series and want to apply this approach to their own forecasting problems, the recommended sequence is exactly the one used here: synthetic DGP first to validate the harness, then real data once the harness is trusted, then production deployment with continuous calibration monitoring. The skip-to-real-data path remains popular and remains the most common source of silent forecasting failures in production. The methodological cost of an extra week on synthetic validation is small. The cost of deploying a miscalibrated forecasting model into a commercial decision pipeline is not.

About the Author

Brian Curry is a Kansas City-based Data Scientist, AI expert, and founder of Vector1 Research, where he explores the frontier of Artificial Intelligence, Causal Inference, Agentic Systems, and Economic Modeling.

His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics. He is the creator of PyCausalSim (causal discovery through simulation), Papilon (complex system optimization), Daedalus (causal and economic analysis), and Memory-Node Encapsulation (data structures for artificial episodic memory).