Your Energy Price Forecast Fails Exactly When It Matters Most. Here’s How I Fixed That.

An open-source system that reads the market’s regime — volatility, spikes, curve behavior — and re-weights its forecasting models before turbulence hits.

Every energy forecaster has lived some version of this: your model posts respectable error metrics for months, management is happy, and then a cold snap hits, a unit trips, or gas storage draws harder than expected — and the model that averaged 5% error misses the one week that determined the quarter’s P&L by a factor of ten. Winter Storm Uri sent ERCOT to $9,000/MWh while models trained on years of $25 power shrugged. TTF gas in 2022 spent months trading in a range that pre-crisis models assigned essentially zero probability. WTI printed negative in April 2020 and broke half the parametric models in the industry.

The uncomfortable truth is that calm-period accuracy in these markets is nearly worthless. Power, gas, and most commodities spend the majority of their time being boring — mean-reverting, seasonal, forgiving — and the entire economic value of a forecast concentrates in the scarcity events, the ramps, and the regime breaks. That’s precisely where standard approaches fall apart.

This article is about why that failure is structural, the architecture I built to address it, and the open-source repo where you can run all of it yourself.

Why ensembles — the industry’s best practice — die in exactly these moments

Decades of forecasting competitions keep delivering the same verdict: combinations of simple, diverse models beat sophisticated single models. The M4 competition was won by FFORMA, a meta-model that learned to weight nine textbook methods. Most serious energy forecasting shops internalized this long ago and run ensembles.

But ensembles have a failure mode tailor-made for commodity markets. Averaging works by cancelling uncorrelated errors — and in calm regimes it does. A regime break is precisely the moment your models’ errors become correlated: when ERCOT tightens toward scarcity pricing, everything mean-reverting misses low together. When a heatwave load ramp arrives, the seasonal-naive model and the smoother both anchor to last week’s mild profile. The ensemble average then confidently splits the difference between five wrong answers. Diversification protects you in exactly the conditions where you didn’t need protection.

The fix isn’t a better model. It’s a meta-layer that knows which market it’s standing in — calm shoulder-season mean reversion, or a tightening system one outage away from scarcity.

The architecture

The system, open-sourced as regimecast, has four layers:

1. A pool of diverse, simple base models. Seasonal naive, damped-trend exponential smoothing, a ridge-regularised autoregression, the classical theta method, and a gradient-boosted model on lags and calendar features (the one member built to consume exogenous drivers — temperature, load forecasts, fuel prices, reservoir levels). Nothing exotic, and that’s the point. The design principle is diversity of inductive bias: the AR model believes in mean reversion (true for gas storage cycles, most of the time), the naive model anchors to recent seasonality (true for load-driven power shape, most of the time), the GBM believes in fundamentals. Their disagreement is information.

2. A regime state vector. Before every forecast, the system computes meta-features describing the market’s current mood: realized volatility, spike intensity (robust z-score exceedances — the statistical signature of scarcity events), trend strength, seasonal strength, level shifts, short-vs-long volatility ratio, and autocorrelation. Seven numbers answering one question: is this the calm market or the dangerous one? For production use on real markets, this vector is where domain knowledge enters — forward curve shape (contango vs. backwardation), storage levels versus five-year norms, reserve margins, weather anomalies. The framework treats the feature list as pluggable for exactly that reason.

3. A meta-model mapping regime → weights. Trained on out-of-fold forecasts generated by rolling-origin evaluation, one gradient-boosted regressor per pool member learns to predict that member’s error given the regime — learning rules like “when volatility is elevated and spikes are appearing, stop trusting the mean-reverters.” Weights are a softmax over predicted errors, always convex, so a confused meta-model degrades toward an average rather than extrapolating into nonsense. And there’s an explicit out-of-distribution fallback: when the regime vector drifts outside anything seen in training, weights blend back toward equal. Energy markets reliably produce genuinely unprecedented days — Uri was outside every training distribution in the industry — and on those days, humility is a feature.

4. Two recursions.

Recursion one: the ensemble studies its own mistakes

The residual cascade is boosting applied at the ensemble level. Fit the stacked ensemble, collect its out-of-fold errors, train a specialist to predict those errors from the regime features, add the correction, repeat until a validation set says stop.

This works because ensemble errors in energy markets aren’t random — they cluster in identifiable places: morning and evening ramps, the hours around sunset in solar-heavy markets, weekend-to-weekday transitions, the first days after a regime break when every model is still anchored to the old world. A model trained only on what the ensemble gets wrong becomes a specialist in exactly those hard cases. In my runs the cascade accepts one or two correction stages before the validation gate closes. The gate matters: unbounded recursion here is an overfitting machine.

Recursion two: weights that never stop updating

Between retrains, markets drift — storage fills, spreads compress, a new interconnector changes the price formation. So the second recursion is a feedback loop: recursive least squares over the base forecasts with a forgetting factor, projected onto the probability simplex after every update. Each settled price nudges the combination weights; stale regimes decay exponentially. It’s recursion in the control-theory sense — a thermostat, not a genealogy — and it’s what keeps the system honest between the calm of a mild November and the chaos of a January cold snap.

The satisfying part: in my benchmark, the online weights independently converged to trusting the same model the offline meta-layer favored. Two different recursions, same discovery.

Results (with the caveat most benchmarks hide)

I benchmarked on a deliberately nasty synthetic series built to mimic hourly power prices: daily and weekly seasonality, a mean-reverting stochastic level, switching volatility regimes, and exponential price spikes. Strict 70/30 holdout, 44 rolling origins, 24-hour-ahead horizon — the day-ahead auction problem.

• The stacked-plus-cascade system beat the equal-weight ensemble by roughly 25% on MAE and posted the best RMSE of anything tested — and RMSE is the metric that punishes missing spikes, which is the miss that costs money.
• Split by volatility regime, it was the best performer in turbulent periods — the conditions the architecture exists for, and the conditions where a day-ahead bid or a hedge ratio actually depends on the forecast.
• The caveat: a plain AR model won overall MAE. Why? Because my synthetic generator is itself AR-based — the model enjoys home-field advantage no single model gets on real market data, where the data-generating process changes with the weather, the fuel stack, and policy. I’m telling you this because a benchmark that hides its most inconvenient row is advertising, not evidence.

The honest summary: the meta-layer does real, measurable work, and its edge concentrates in turbulence — which for anyone bidding day-ahead, sizing a hedge, or managing storage is the entire point.

What I’d warn you about

Out-of-fold discipline is everything. If your meta-model ever sees a base model’s in-sample fit, it learns to trust whichever model overfits hardest, and your backtest becomes fiction. Rolling origin, always, at every layer — including the cascade.

Every recursive layer spends your data budget. An hourly power series sounds like a lot of data, but it contains maybe a handful of genuine regime cycles — a few winters, one or two real scarcity episodes, perhaps one structural break like a fuel-switching shift. A meta-meta-model sounds clever and fits noise. I tried; skip it.

Build the fallback before you need it. The out-of-distribution blend toward equal weights isn’t decoration. The expensive failures in this industry happen on unprecedented days, and an adaptive system with no humility mechanism is an amplifier pointed at exactly those days.

Respect the plumbing. Real commodity data will punish a clean-room framework: futures need roll-adjustment or the meta-model learns your roll schedule instead of the market; power data has missing and doubled hours at DST transitions; European power and gas can print negative; hub granularity matters (nodal ERCOT is not zonal ERCOT). The repo deliberately ships market-agnostic — the plumbing is where market-specific work belongs.

What’s next

The roadmap: conformal prediction intervals (a trader needs a calibrated P95 far more than a slightly better point forecast — interval quality is what sizes positions), real-data loaders for specific markets, and evolutionary configuration search — treating the entire system’s design as a genome and running selection pressure over generations, with a quarantined holdout the evolution never touches.

The repo is MIT-licensed, pure numpy/scikit-learn, and trains in minutes on a laptop: https://github.com/Bodhi8/regimecast. Run python examples/demo.py, then break it on your own market and tell me where it fails. The regime features that matter for Nordic hydro (reservoir levels, snowpack) are not the ones that matter for ERCOT (reserve margins, temperature extremes) or for TTF (storage, LNG arrivals, interconnector flows) — and I'd rather learn that from your issues than from my own losses.

If you forecast power, gas, or commodities for a living, I’d genuinely like to hear which regimes your models die in. That’s where the next meta-feature comes from.

Brian James Curry is a Kansas City–based data scientist and applied Machine Learning and AI researcher working at the intersection of economic modeling, causal inference, and machine learning — with a standing preference for models where the economics is load-bearing. He is the founder of Vector1 Research, an independent lab exploring marketing economics, cognitive AI architecture, and AI-driven enterprise transformation, with research contributions including Papilon, PyCausalSim, and Memory-Node Encapsulation. His twenty-plus years of enterprise work span Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, and AT&T.