Forecasting at Scale: A Hybrid Architecture Where Classical Models, Deep Learning, Foundation Models, and LLMs Each Do What They Do Best
By Brian Curry | Vector1 Research
“The question is never ‘which model is best.’ It is ‘which model is best for this series, at this horizon, under this regime, with this much history, given this cost of being wrong.’ Any system that answers with a single model is answering the wrong question.”
Vector1 Research maintains a private reference implementation of this architecture used in production forecasting engagements. The open-source components it builds on — StatsForecast, NeuralForecast, the foundation model families — are all publicly available and cited above.
Abstract
The discourse around forecasting in 2026 has been captured by two competing hype narratives. The first holds that time-series foundation models — Chronos-2, TimesFM, MOIRAI-2 — have made everything else obsolete, that zero-shot forecasting is the future, and that classical statistics belongs in a museum. The second holds that you can simply hand a large language model your time series and ask it to predict the future. Both narratives are wrong, and building a production forecasting system on either one will cost a company dearly.
This article presents a defensible, production-grade reference architecture for forecasting at web scale — the kind that forecasts metrics like monthly active users, revenue, demand, or capacity for an organization where being wrong has real consequences. The central thesis is that a serious forecasting system is a plural system: classical statistical models, gradient-boosted trees, modern deep learning architectures, and time-series foundation models all coexist in a governed model zoo, competing and ensembling per series. Large language models are not the forecasting engine. They are the operations intelligence layer — helping build features, reason about model selection, narrate drift, diagnose anomalies, generate deployment configuration, and translate forecasts into decisions. They wrap the system; they do not replace its statistics.
We walk the complete lifecycle: from use-case framing through feature engineering, model development across the full zoo, selection and ensembling, production deployment, monitoring and drift detection, and the iteration loop — with the LLM operations layer woven through each stage. The goal is an architecture a senior practitioner could actually defend in a design review and actually run at scale.
Keywords: Time-Series Forecasting, Foundation Models, Production ML, Model Zoo, Ensemble Forecasting, Drift Detection, MLOps, LLM Operations, Backtesting, Probabilistic Forecasting, Explainability
I. Start With the Decision, Not the Model
Every forecasting system that fails in production fails for the same reason: it was built to optimize a metric instead of to inform a decision. The MAPE looked great in the notebook, and then the forecast was useless to the people who had to act on it.
Before any model is chosen, before any feature is engineered, the system must answer four questions about the decision the forecast serves:
What decision does this forecast drive, and what is the cost structure of being wrong? Forecasting monthly active users for a board deck has a symmetric, relatively forgiving error tolerance — being off by 2% in either direction is fine. Forecasting server capacity has a profoundly asymmetric cost: under-provisioning causes outages (catastrophic), over-provisioning wastes money (merely bad). The loss function must encode this asymmetry. A system that optimizes symmetric RMSE for an asymmetric decision is optimizing the wrong thing, no matter how sophisticated the model.
What horizon and granularity actually matter? A forecast that is accurate at a 1-day horizon but needed at 90 days is worthless. A daily forecast when the decision is made monthly is false precision. The horizon and granularity are properties of the decision, not the data.
What is the required form of the output — a point, a distribution, or a scenario? Most production decisions need a distribution, not a point. “MAU will be 47.2M” is far less useful than “MAU will be between 45.1M and 49.3M with 80% confidence, with the downside scenario driven primarily by the seasonal dip we see every Q3.” Probabilistic forecasting is not a luxury; it is the default requirement for any forecast that informs a real decision under uncertainty.
What is the explainability requirement? A forecast that leadership cannot interrogate will not be trusted, and an untrusted forecast does not inform decisions regardless of its accuracy. If a VP asks “why did the forecast drop 8% versus last month’s run,” the system must be able to answer. This requirement shapes model selection as much as accuracy does — and it is where the LLM layer earns much of its keep.
This is the first place the LLM operations layer enters, and it enters as a structuring tool, not a predictor:
from canary.llm import LLMClient
class ForecastFraming:
"""
Uses an LLM to help structure a forecasting problem from a business
request before any modeling begins. The LLM does not forecast — it
interrogates the decision, surfaces the implicit loss function, and
proposes an evaluation protocol.
"""
def __init__(self, llm: LLMClient):
self.llm = llm
def frame_problem(self, business_request: str,
available_history: dict) -> dict:
"""
Translate a vague business request into a precise forecasting spec.
Example input: "We need to know how many users we'll have next year
so finance can plan headcount."
Output: structured spec including horizon, granularity, loss
asymmetry, output form, and explainability requirements.
"""
response = self.llm.complete(f"""
A stakeholder made this forecasting request:
"{business_request}"
Available data: {available_history}
Produce a structured forecasting specification. Specifically reason about:
1. The actual decision this forecast drives
2. Whether the cost of over-forecasting vs under-forecasting is symmetric
(and if not, which direction is more costly and roughly by how much)
3. The decision-relevant horizon and granularity
4. Whether a point forecast suffices or a full predictive distribution
is required
5. The explainability requirement based on who consumes the forecast
Return structured JSON. Flag any place where the request is
underspecified and a human should clarify before proceeding.
""")
return self._parse_spec(response)The output of this stage is not a model. It is a contract: the precise specification the rest of the system is built to satisfy. Everything downstream is judged against it.
II. Features: The Part Foundation Models Don’t Make Obsolete
One of the most damaging claims in the foundation-model hype is that feature engineering is dead — that you can hand a raw series to a pretrained transformer and skip the work. This is true for exactly the class of problems where it doesn’t matter much, and false for most problems where forecasting is hard and valuable.
Foundation models are remarkable at capturing the endogenous structure of a series — its own seasonality, trend, and autocorrelation. They are weak precisely where most business value lives: in the exogenous drivers. The marketing campaign that launches next Tuesday, the pricing change, the competitor’s product launch, the holiday calendar specific to your business, the macroeconomic covariate — these are the things that move the metric and that the foundation model has never seen.
A production feature layer therefore has to serve the entire zoo, because different model families consume features differently:
import pandas as pd
import numpy as np
from dataclasses import dataclass
from typing import Optional
@dataclass
class FeatureSpec:
"""Defines the feature surface available to every model in the zoo."""
# Calendar features (consumed by tree models, deep learning, TiDE)
calendar: list[str] # day_of_week, week_of_year, month, is_holiday, ...
# Lag features (consumed by tree models, Lag-Llama)
lags: list[int] # [1, 7, 28, 364] — daily/weekly/monthly/yearly
# Rolling statistics (consumed by tree models)
rolling_windows: list[int] # [7, 28, 90] day windows for mean/std/min/max
# Known future covariates (consumed by TFT, TiDE, TSMixer, MOIRAI)
future_covariates: list[str] # planned_promotions, price, known_events
# Past-only covariates (consumed by most multivariate models)
past_covariates: list[str] # competitor_signal, weather, web_traffic
# Static covariates (consumed by global models)
static: list[str] # region, product_category, customer_segment
class FeaturePipeline:
"""
Constructs the full feature surface from raw series + covariates.
Critically, this pipeline produces features in the formats required by
EVERY model family, because the model zoo will consume the same
underlying data in different shapes:
- Classical models: just the target series (+ optional exog regressors)
- Tree models: wide tabular feature matrix (lags, rolling, calendar)
- Deep learning: tensor windows with covariate channels
- Foundation models: context windows + covariate conditioning
"""
def __init__(self, spec: FeatureSpec):
self.spec = spec
def build_tabular(self, df: pd.DataFrame,
target_col: str) -> pd.DataFrame:
"""
Build the wide tabular feature matrix for gradient-boosted models.
This is where domain expertise still dominates foundation models.
"""
features = df.copy()
# Lag features
for lag in self.spec.lags:
features[f"{target_col}_lag_{lag}"] = df[target_col].shift(lag)
# Rolling statistics (computed on lagged values to avoid leakage)
for window in self.spec.rolling_windows:
shifted = df[target_col].shift(1) # shift first — no leakage
features[f"{target_col}_roll_mean_{window}"] = (
shifted.rolling(window).mean()
)
features[f"{target_col}_roll_std_{window}"] = (
shifted.rolling(window).std()
)
# Calendar features
features["day_of_week"] = df.index.dayofweek
features["week_of_year"] = df.index.isocalendar().week
features["month"] = df.index.month
features["quarter"] = df.index.quarter
# Fourier terms for smooth seasonality (helps tree models)
for period, order in [(7, 3), (365.25, 5)]:
for k in range(1, order + 1):
t = np.arange(len(df))
features[f"sin_{period}_{k}"] = np.sin(2 * np.pi * k * t / period)
features[f"cos_{period}_{k}"] = np.cos(2 * np.pi * k * t / period)
return features
def build_tensor_windows(self, df: pd.DataFrame, target_col: str,
context_length: int, horizon: int):
"""
Build sliding tensor windows for deep learning and foundation models.
Returns context windows, future covariate windows, and targets.
"""
# Implementation produces (n_windows, context_length, n_channels) tensors
# consumed by TFT, TiDE, TSMixer, and the foundation model adapters
...The LLM layer contributes here too — not by engineering features directly, but by reasoning about which features are likely to matter for a given series and surfacing candidate exogenous drivers a human might have missed:
class FeatureAdvisor:
"""
LLM-assisted feature discovery. The LLM proposes candidate features
and exogenous drivers based on the domain context. A human reviews;
the pipeline implements. The LLM never silently adds features to a
production model.
"""
def suggest_features(self, series_description: str,
current_features: list[str]) -> dict:
response = self.llm.complete(f"""
We are forecasting: {series_description}
Current features: {current_features}
Propose additional candidate features or exogenous drivers that
domain knowledge suggests would improve this forecast. For each,
explain the hypothesized causal mechanism and how to source the data.
Prioritize known-future covariates (things we'll know in advance for
the forecast horizon) since these are most valuable. Flag any
proposed feature that risks target leakage.
""")
return self._parse_suggestions(response)The discipline here is essential: the LLM proposes, a human disposes, and the pipeline implements deterministically. An LLM that silently injects features into a production forecasting model is a source of untraceable drift, not a feature.
III. The Model Zoo: Five Families, Each With a Job
This is the heart of the architecture and the place where the article diverges most sharply from the single-model hype. A production forecasting system at scale does not pick a model. It maintains a governed zoo of model families and lets the data decide, per series, which wins.
3.1 Why Plurality Is Non-Negotiable
The empirical reality, established across the M-competitions and confirmed repeatedly in production, is that no model family dominates across all series. The factors that determine which family wins are properties of the individual series:
- History length: a series with 60 observations cannot support a deep learning model; a well-specified ETS or Theta model will beat a transformer every time. A series with 50,000 observations and rich covariates is where deep learning and foundation models shine.
- Signal-to-noise and regularity: clean, strongly seasonal series are often won by classical statistical models that encode that structure directly. Noisy, irregular series with complex covariate interactions favor gradient-boosted trees or deep learning.
- Covariate richness: if the forecast is driven by known-future covariates (promotions, pricing, events), models that consume them — TFT, TiDE, TSMixer, MOIRAI — have a structural advantage.
- Cold-start: a brand-new series with no history is precisely where foundation models earn their existence, providing reasonable zero-shot forecasts where every trained model would fail.
A system that uses one family is leaving accuracy on the table for most of its series. The cost of maintaining the zoo is real, but at scale it is dwarfed by the cost of systematically worse forecasts.
3.2 The Five Families
Family 1 — Classical Statistical (StatsForecast). ARIMA, ETS, Theta, TBATS, Croston (for intermittent demand). Fast, interpretable, well-calibrated, and frequently the winner on short, regular series. The StatsForecast library makes these trivially parallelizable across millions of series. These models are not legacy — they are the correct answer for a large fraction of real series, and any architecture that excludes them is worse for it.
Family 2 — Gradient-Boosted Trees (LightGBM / XGBoost). Trained on the wide tabular feature matrix, a single global LightGBM model across many series, with categorical static features identifying each series, remains one of the strongest and most cost-effective approaches in existence. It dominates when there are rich engineered features and complex nonlinear covariate interactions. It is cheap to train, cheap to serve, and naturally explainable through feature importance and SHAP.
Family 3 — Modern Deep Learning (NeuralForecast). Temporal Fusion Transformer (TFT) for interpretable attention over covariates, N-BEATS / N-HiTS for pure univariate power, TiDE and TSMixer for efficient covariate-aware MLP architectures. These win on long, complex, multivariate series with sufficient history. TiDE and TSMixer specifically are MLP-based and far cheaper to train than transformers while remaining competitive — important at scale.
Family 4 — Time-Series Foundation Models. Chronos-2 (T5-based, tokenizes values, production-mature with AWS integration), TimesFM (Google, decoder-only, enterprise-reliable), MOIRAI-2 (Salesforce, strong multivariate and covariate support), Lag-Llama (decoder-only, explicit lag features, strong probabilistic output). Their unique value is zero-shot forecasting: reasonable forecasts on series they have never seen, with no training. This makes them the answer for cold-start, for the long tail of low-volume series where training a bespoke model is uneconomical, and as a strong baseline that any trained model must beat to justify its existence.
Family 5 — LLM-Adapter Models (Time-LLM). A genuinely distinct family worth including in the zoo as a forecasting tool, not just as the ops layer. Time-LLM reprograms a frozen LLM (GPT-2, LLaMA) to forecast by translating time-series patches into text prototypes, injecting domain knowledge through a “prompt-as-prefix.” Its niche is the situation where forecasting must incorporate unstructured contextual information — a series whose future is shaped by described events that resist encoding as numeric covariates. It is the most experimental family and should be governed accordingly, but it earns a place in the zoo for problems where text context genuinely drives the target.
from statsforecast import StatsForecast
from statsforecast.models import AutoARIMA, AutoETS, AutoTheta, CrostonOptimized
from neuralforecast import NeuralForecast
from neuralforecast.models import TFT, NHITS, TiDE, TSMixer
import lightgbm as lgb
class ModelZoo:
"""
Governed registry of forecasting model families.
Each family is registered with metadata describing where it is
appropriate, so the selection layer (Section IV) can reason about
which candidates to run for a given series rather than running all
of them — critical for cost control at scale.
"""
def __init__(self):
self.registry = {}
self._register_defaults()
def _register_defaults(self):
# Family 1: Classical statistical
self.register(
name="statistical",
builder=lambda: StatsForecast(
models=[AutoARIMA(), AutoETS(), AutoTheta()],
freq='D',
n_jobs=-1
),
appropriate_when={
"min_history": 30,
"max_history": None,
"strengths": ["short_series", "regular_seasonality",
"fast", "interpretable", "well_calibrated"],
"cost": "very_low"
}
)
# Family 1b: Intermittent demand
self.register(
name="intermittent",
builder=lambda: StatsForecast(
models=[CrostonOptimized()], freq='D', n_jobs=-1
),
appropriate_when={
"min_history": 30,
"strengths": ["intermittent_demand", "sparse_series"],
"cost": "very_low"
}
)
# Family 2: Gradient-boosted trees (global model)
self.register(
name="gbdt_global",
builder=lambda: LightGBMForecaster(
params={"objective": "quantile", "alpha": 0.5,
"num_leaves": 128, "learning_rate": 0.05}
),
appropriate_when={
"min_history": 100,
"strengths": ["rich_covariates", "nonlinear",
"many_series", "cheap_to_serve", "explainable"],
"cost": "low"
}
)
# Family 3: Deep learning
self.register(
name="deep_learning",
builder=lambda horizon: NeuralForecast(
models=[
TFT(h=horizon, input_size=horizon*4, max_steps=500),
NHITS(h=horizon, input_size=horizon*4, max_steps=500),
TiDE(h=horizon, input_size=horizon*4, max_steps=500),
],
freq='D'
),
appropriate_when={
"min_history": 1000,
"strengths": ["long_complex_series", "multivariate",
"covariate_interactions"],
"cost": "high"
}
)
# Family 4: Foundation models (zero-shot)
self.register(
name="foundation",
builder=lambda: FoundationModelAdapter(
models=["chronos-2", "timesfm", "moirai-2"]
),
appropriate_when={
"min_history": 0, # zero-shot — works with no history
"strengths": ["cold_start", "zero_shot", "long_tail",
"strong_baseline"],
"cost": "medium"
}
)
# Family 5: LLM-adapter
self.register(
name="llm_adapter",
builder=lambda: TimeLLMForecaster(backbone="llama"),
appropriate_when={
"min_history": 200,
"strengths": ["text_context_drivers", "described_events"],
"cost": "high",
"governance": "experimental"
}
)
def register(self, name: str, builder, appropriate_when: dict):
self.registry[name] = {
"builder": builder,
"metadata": appropriate_when
}
def candidates_for_series(self, series_profile: dict) -> list[str]:
"""
Return the subset of model families appropriate for a series,
based on its profile. This prevents running every model on every
series — essential for cost control at web scale.
"""
candidates = []
history_len = series_profile["history_length"]
for name, entry in self.registry.items():
meta = entry["metadata"]
if history_len >= meta.get("min_history", 0):
if meta.get("max_history") is None or \
history_len <= meta["max_history"]:
candidates.append(name)
return candidatesIV. Selection and Ensembling: Letting the Data Decide
With a zoo of candidate models, the system needs a principled way to decide which model — or which combination — to deploy for each series. This is done through rigorous backtesting, and it is where most forecasting systems quietly cheat.
4.1 Backtesting Without Lying to Yourself
The single most common failure in forecasting is temporal leakage in evaluation — letting information from the future contaminate the assessment of past performance. The only honest evaluation is time-series cross-validation with expanding or rolling windows, where the model is repeatedly trained on data up to time t and evaluated on t+1…t+h, walked forward across the series.
from dataclasses import dataclass
import numpy as np
@dataclass
class BacktestResult:
model_name: str
series_id: str
fold_errors: list[dict] # per-fold metrics
mean_metrics: dict # aggregated across folds
calibration: dict # coverage of prediction intervals
cost_estimate: float # training + inference cost
class Backtester:
"""
Walk-forward time-series cross-validation.
Critically evaluates not just point accuracy but:
- Probabilistic calibration (do the 80% intervals actually contain
the truth 80% of the time?)
- The asymmetric, decision-relevant loss from Section I
- Cost, so the selection layer can trade accuracy against expense
"""
def __init__(self, n_windows: int = 5, horizon: int = 28,
step_size: int = 28, loss_fn=None):
self.n_windows = n_windows
self.horizon = horizon
self.step_size = step_size
self.loss_fn = loss_fn or self._default_loss
def evaluate(self, model, series, covariates=None) -> BacktestResult:
"""Walk forward through the series, training and evaluating each fold."""
fold_errors = []
n = len(series)
for w in range(self.n_windows):
# Expanding window: train on everything up to the cutoff
cutoff = n - (self.n_windows - w) * self.step_size
train = series[:cutoff]
actual = series[cutoff:cutoff + self.horizon]
if len(actual) < self.horizon:
continue
# Fit and forecast (probabilistic where supported)
model.fit(train, covariates)
forecast = model.predict(self.horizon, level=[80, 95])
fold_errors.append({
"fold": w,
"mae": self._mae(actual, forecast["point"]),
"rmse": self._rmse(actual, forecast["point"]),
"mase": self._mase(actual, forecast["point"], train),
"decision_loss": self.loss_fn(actual, forecast["point"]),
"coverage_80": self._coverage(actual, forecast, 80),
"coverage_95": self._coverage(actual, forecast, 95),
})
return BacktestResult(
model_name=model.name,
series_id=series.name,
fold_errors=fold_errors,
mean_metrics=self._aggregate(fold_errors),
calibration=self._calibration_summary(fold_errors),
cost_estimate=model.cost_estimate()
)
@staticmethod
def _mase(actual, predicted, train):
"""
Mean Absolute Scaled Error — the right default metric.
Scale-free, interpretable (MASE < 1 beats naive seasonal),
and comparable across series. Far superior to MAPE, which
explodes near zero and is asymmetric.
"""
naive_errors = np.abs(np.diff(train))
scale = naive_errors.mean() + 1e-8
return np.mean(np.abs(actual - predicted)) / scale
@staticmethod
def _coverage(actual, forecast, level):
"""
Empirical coverage of prediction intervals — the calibration check
that almost everyone skips and that matters more than point accuracy
for decisions under uncertainty.
"""
lower = forecast[f"lower_{level}"]
upper = forecast[f"upper_{level}"]
inside = (actual >= lower) & (actual <= upper)
return float(inside.mean())Two things in this backtester deserve emphasis because they are routinely neglected. First, MASE over MAPE: MAPE is the default in most organizations and it is a bad default — it is undefined at zero, explodes near zero, and asymmetrically penalizes over- versus under-forecasting. MASE is scale-free, interpretable against a naive baseline, and comparable across series. Second, calibration: a forecast whose 80% intervals only contain the truth 60% of the time is lying about its own uncertainty, and decisions made on that false confidence will be systematically too aggressive. Coverage is not optional.
4.2 Selection: Per-Series, Cost-Aware
With honest backtest results across the candidate zoo, selection becomes an optimization that trades accuracy against cost and respects the decision’s requirements:
class ModelSelector:
"""
Selects the deployed model (or ensemble) per series from backtest results.
Selection is multi-objective:
- Accuracy on the decision-relevant loss
- Calibration (intervals must be honest)
- Cost (at web scale, a 0.5% accuracy gain is not worth 10x the cost)
- Robustness (consistency across folds, not just mean performance)
"""
def select(self, backtest_results: list[BacktestResult],
requirements: dict) -> dict:
scored = []
for result in backtest_results:
# Reject models that fail the calibration requirement outright
if requirements.get("requires_calibration", True):
cov_80 = result.calibration.get("mean_coverage_80", 0)
if abs(cov_80 - 0.80) > 0.10: # more than 10pp miscalibrated
continue
# Composite score: accuracy, robustness, cost
accuracy_score = 1.0 / (1.0 + result.mean_metrics["decision_loss"])
robustness = 1.0 / (1.0 + result.mean_metrics.get("loss_std", 0))
cost_penalty = result.cost_estimate / requirements.get(
"cost_budget", 1.0)
score = (
0.6 * accuracy_score +
0.25 * robustness -
0.15 * cost_penalty
)
scored.append((result, score))
scored.sort(key=lambda x: x[1], reverse=True)
# Decide: single model or ensemble?
return self._decide_single_or_ensemble(scored, requirements)
def _decide_single_or_ensemble(self, scored, requirements):
"""
Ensembling almost always improves accuracy and calibration, but
costs more to run. The decision depends on the value of the series.
For high-value series: ensemble the top 3 families (diversity matters
more than individual strength — a classical + GBDT + foundation model
ensemble is robust precisely because the families fail differently).
For long-tail low-value series: single best model, usually the
foundation model or a cheap classical model.
"""
top = scored[0][0]
if requirements.get("series_value", "low") == "high" and len(scored) >= 3:
# Ensemble the top 3 — but only if they're diverse families
top_3 = [s[0] for s in scored[:3]]
families = set(r.model_name.split("_")[0] for r in top_3)
if len(families) >= 2: # require family diversity
return {
"strategy": "ensemble",
"models": [r.model_name for r in top_3],
"weights": self._optimize_ensemble_weights(top_3),
"rationale": "high-value series, diverse family ensemble"
}
return {
"strategy": "single",
"model": top.model_name,
"rationale": f"best composite score, "
f"MASE={top.mean_metrics['mase']:.3f}"
}The ensembling insight worth dwelling on: the value of an ensemble comes from diversity of failure modes, not from averaging strong models. A classical model, a gradient-boosted tree, and a foundation model fail on different series for different reasons. Their errors are weakly correlated, so combining them reduces variance more than combining three deep learning models would. This is why the zoo’s plurality is not just about picking winners — it is the raw material for robust ensembles.
V. The LLM Operations Layer: Connective Tissue, Not Engine
Here is where large language models do their real work in this architecture — and it is everywhere except the forecast itself. The LLM is the system’s operations intelligence: it reasons about the system, narrates it, and connects it to humans. It never produces the number.
5.1 Build-Time: Model Selection Reasoning
The selection logic in Section IV is quantitative, but its output needs to be explained and sometimes overridden by human judgment. The LLM translates the backtest evidence into reasoning a human can audit:
class SelectionExplainer:
"""
The LLM reads the backtest results and the selection decision and
produces a human-auditable rationale. This is not the LLM making the
decision — the decision is made quantitatively. The LLM makes the
decision legible and flags cases where a human should look.
"""
def explain_selection(self, backtest_results, selection) -> str:
return self.llm.complete(f"""
Forecasting model selection for series {selection.get('series_id')}.
Backtest results across the model zoo:
{self._format_results(backtest_results)}
Selected: {selection}
Explain in 3-4 sentences why this selection is reasonable given the
evidence. Specifically note: was the decision close? Is there a
cheaper model that was nearly as good? Are there signs the winning
model is overfit to the backtest folds? Flag for human review if
the top two models are within 2% on the decision loss but differ
significantly in cost or interpretability.
""")5.2 Deploy-Time: Configuration and Backtest Orchestration
LLMs are genuinely useful at generating and validating the deployment configuration — the glue code, the pipeline DAGs, the monitoring thresholds — from the validated model spec. This is code generation against a known schema, which is what LLMs are reliably good at, and it always passes through validation and human review before it touches production:
class DeploymentConfigGenerator:
"""
Generate deployment configuration (orchestration DAG, monitoring
thresholds, retraining triggers) from the validated model selection.
Output is always validated against a schema and reviewed before deploy.
"""
def generate_monitoring_config(self, selection, backtest) -> dict:
suggested = self.llm.complete(f"""
We are deploying {selection} to production.
Its backtest performance: {backtest.mean_metrics}
Its fold-to-fold variance: {backtest.calibration}
Propose monitoring thresholds:
- At what MASE degradation should we alert? (consider its backtest
variance — a noisy model needs wider thresholds)
- At what prediction-interval coverage breach should we alert?
- What retraining cadence fits this model family and series volatility?
Return JSON. These are SUGGESTIONS that a human will review and
the system will validate against safe bounds before applying.
""")
return self._validate_against_safe_bounds(self._parse(suggested))5.3 Monitor-Time: Drift Narration and Root-Cause Analysis
This is the highest-value application of the LLM layer. Drift detection itself is statistical (Section VI). But a drift alert is just a number — and a number does not tell an on-call engineer what happened or what to do. The LLM turns the statistical signal into an actionable narrative:
class DriftAnalyst:
"""
When statistical drift detection fires, the LLM synthesizes the
quantitative signals into a root-cause hypothesis and recommended action.
The detection is statistical; the diagnosis and communication are the LLM's.
"""
def analyze_drift(self, drift_signal: dict, recent_data: dict,
forecast_errors: dict, known_events: list) -> dict:
return self.llm.complete(f"""
A forecasting model has triggered a drift alert.
Drift signals: {drift_signal}
Recent actuals vs forecast: {forecast_errors}
Known business events in the window: {known_events}
Feature distribution changes: {recent_data['feature_drift']}
Produce:
1. A ranked set of root-cause hypotheses (data quality issue?
genuine regime change? known event the model didn't account for?
upstream pipeline break?)
2. For each, the evidence for and against
3. A recommended action (retrain? hotfix? wait and monitor?
escalate to human?)
4. A one-paragraph summary suitable for an on-call engineer at 3am
Distinguish clearly between data drift (input distribution changed)
and concept drift (input-output relationship changed) — they require
different responses.
""")5.4 Decision-Time: Executive Narrative Generation
The final mile of any forecasting system is translating the forecast into a decision, and that translation is a communication problem. The LLM is genuinely good at turning a probabilistic forecast plus its drivers into a concise executive narrative — provided it is given the actual numbers and explanations and constrained to report them faithfully:
class ForecastNarrator:
"""
Translate a forecast + its explainability outputs into an executive
narrative. The LLM reports the numbers and the model's own attributions;
it does not invent or editorialize beyond the provided evidence.
"""
def narrate(self, forecast: dict, attributions: dict,
prior_forecast: dict, audience: str) -> str:
return self.llm.complete(f"""
Current forecast: {forecast} (point + 80% and 95% intervals)
Change vs last forecast: {self._diff(forecast, prior_forecast)}
Model's driver attributions: {attributions}
Audience: {audience}
Write a concise narrative (3-4 sentences for an executive audience):
- The headline number and the uncertainty around it
- What changed versus the prior forecast and why (use the attributions)
- The key risk to the forecast and the scenario that would break it
Report only what the numbers and attributions support. Do not
speculate beyond the provided evidence. State uncertainty plainly.
""")The discipline running through all four applications is the same: the LLM operates on the system’s outputs and reasons about the system, but it never produces the forecast and it never acts without validation. This is what makes the architecture defensible. An LLM hallucinating a narrative is an annoyance; an LLM hallucinating a forecast that drives a capacity decision is an outage.
VI. Monitoring and Drift: Detecting Decay Before It Costs
A forecasting model in production does not fail loudly. It degrades silently as the world drifts away from the conditions it was trained on. The research consensus is stark: static models suffer 15–22% performance degradation under non-stationary conditions, and 91% of model failures trace to drift. The monitoring layer exists to catch this before business metrics move.
There are two distinct phenomena, and conflating them is a common and costly error:
Data drift — the input distribution changes. Feature values shift statistically away from the training distribution. A MAU model trained on pre-launch traffic patterns sees different patterns post-launch. The features changed.
Concept drift — the input-output relationship changes, even when inputs look statistically stable. An economic regime change alters the relationship between your drivers and your target. The map changed, not just the territory.
They require different responses — data drift often calls for retraining on recent data, while concept drift may require rethinking the model or features entirely — so the monitoring layer must distinguish them:
import numpy as np
from scipy import stats
class ForecastMonitor:
"""
Multi-layer production monitoring for the forecasting system.
Four monitoring layers (the production-monitoring consensus for 2026):
1. Infrastructure: latency, throughput, errors
2. Data quality: input drift, missingness, schema changes
3. Model performance: rolling accuracy, calibration decay
4. Business impact: does the forecast still drive good decisions?
"""
def detect_data_drift(self, reference_dist: np.ndarray,
production_dist: np.ndarray) -> dict:
"""
Detect input distribution drift using a population stability index
plus a distributional test. PSI is the production standard; the
KS test adds statistical rigor.
"""
psi = self._population_stability_index(reference_dist, production_dist)
ks_stat, ks_p = stats.ks_2samp(reference_dist, production_dist)
return {
"psi": psi,
"ks_statistic": ks_stat,
"ks_p_value": ks_p,
# PSI > 0.2 is the conventional "significant drift" threshold
"data_drift_detected": psi > 0.2 or ks_p < 0.05,
"severity": "high" if psi > 0.25 else
"moderate" if psi > 0.1 else "low"
}
def detect_concept_drift(self, recent_errors: np.ndarray,
baseline_errors: np.ndarray) -> dict:
"""
Concept drift manifests as the model's error distribution worsening
even when inputs look stable. We detect it through a change in the
forecast error distribution, using both a magnitude test and a
change-point check.
"""
# Has the error distribution shifted upward significantly?
t_stat, t_p = stats.ttest_ind(recent_errors, baseline_errors,
alternative='greater')
error_inflation = recent_errors.mean() / (baseline_errors.mean() + 1e-8)
return {
"error_inflation_ratio": float(error_inflation),
"significance_p": float(t_p),
"concept_drift_detected": t_p < 0.05 and error_inflation > 1.15,
"interpretation": (
"Model relationship has degraded — inputs may look normal "
"but the model no longer maps them correctly to outputs"
)
}
@staticmethod
def _population_stability_index(ref, prod, bins=10) -> float:
"""PSI: the production-standard data drift metric."""
breakpoints = np.percentile(ref, np.linspace(0, 100, bins + 1))
breakpoints[0], breakpoints[-1] = -np.inf, np.inf
ref_pct = np.histogram(ref, breakpoints)[0] / len(ref) + 1e-6
prod_pct = np.histogram(prod, breakpoints)[0] / len(prod) + 1e-6
return float(np.sum((prod_pct - ref_pct) * np.log(prod_pct / ref_pct)))When drift fires, the system does not blindly retrain. The research is clear that proactive, policy-governed retraining outperforms reactive retraining by roughly 4x, but indiscriminate retraining on drifted data can bake in the very anomaly that triggered the alert. The drift signal goes to the LLM drift analyst (Section 5.3), which produces a root-cause hypothesis and recommended action, and a human reviews the policy decision for anything above a low-severity threshold. This is the autonomous-retraining-with-human-oversight pattern that defines mature MLOps in 2026: the system detects, evaluates whether retraining is cost-justified, retrains, validates, and deploys — with humans reviewing policies and exceptions rather than executing each step.
VII. The Iteration Loop: Anomaly Detection, Attribution, Early Warning
Point forecasts are the beginning, not the end, of a mature forecasting system’s value. The same infrastructure that produces forecasts produces several higher-order capabilities that are often where the system delivers its greatest business impact.
Anomaly detection as a forecasting byproduct. A forecast plus its prediction interval is an anomaly detector for free: an actual that falls outside the 99% interval is, by definition, an anomaly worth investigating. Running this across all monitored series produces an early-warning system for business health — the forecast that suddenly misses badly is often the first signal of a problem upstream.
Automated root-cause analysis. When a metric deviates from forecast, the question is always “why.” The combination of the feature attributions (from the explainable models in the zoo — TFT’s attention weights, the GBDT’s SHAP values) and the LLM’s reasoning layer produces a first-pass root-cause analysis automatically, turning a multi-hour analyst investigation into a reviewed hypothesis in minutes.
Channel and campaign attribution. For series driven by known interventions (marketing campaigns, product launches, pricing changes), the covariate-aware models in the zoo — TFT and TiDE especially — produce attributions of how much each driver contributed to the forecast. This connects forecasting directly to the decisions that the business controls.
Scenario analysis. Because the covariate-aware models accept known-future covariates, they natively support “what-if” scenarios: forecast MAU under three different marketing-spend scenarios, present the distribution of outcomes for each, and let leadership choose. This is where forecasting stops being a reporting function and becomes a decision-support function — and it is the capability that most justifies the investment in the full architecture.
class ScenarioEngine:
"""
Generate forecast scenarios under different assumptions about
controllable future covariates. The covariate-aware models in the zoo
(TFT, TiDE, TSMixer, MOIRAI) make this native: vary the known-future
covariates, re-forecast, present the distribution of outcomes.
"""
def run_scenarios(self, model, base_covariates: dict,
scenarios: list[dict]) -> dict:
results = {}
for scenario in scenarios:
modified = {**base_covariates, **scenario["covariate_overrides"]}
forecast = model.predict(
horizon=scenario["horizon"],
future_covariates=modified,
level=[80, 95]
)
results[scenario["name"]] = {
"forecast": forecast,
"expected_value": forecast["point"].sum(),
"downside_80": forecast["lower_80"].sum(),
"upside_80": forecast["upper_80"].sum(),
}
return resultsVIII. The Reference Architecture, Assembled
Pulling the layers together, the complete system has a clear shape:
┌──────────────────────────────────────────────────────────────────────┐ │ DECISION LAYER │ │ Use-case framing · Loss function definition · Output requirements │ │ [LLM: problem framing, narrative generation, scenario presentation] │ ├──────────────────────────────────────────────────────────────────────┤ │ FEATURE LAYER │ │ Calendar · Lags · Rolling stats · Known-future & past covariates │ │ [LLM: feature discovery suggestions — human reviewed] │ ├──────────────────────────────────────────────────────────────────────┤ │ MODEL ZOO │ │ Classical (StatsForecast) · GBDT (LightGBM) · Deep (NeuralForecast) │ │ Foundation (Chronos-2, TimesFM, MOIRAI-2) · LLM-adapter (Time-LLM) │ ├──────────────────────────────────────────────────────────────────────┤ │ SELECTION & ENSEMBLE LAYER │ │ Walk-forward backtesting · Calibration check · Cost-aware selection │ │ Diversity-driven ensembling │ │ [LLM: selection rationale, human-review flagging] │ ├──────────────────────────────────────────────────────────────────────┤ │ SERVING LAYER │ │ Batch + on-demand inference · Probabilistic output · Scenario engine │ │ [LLM: deployment config generation — schema-validated] │ ├──────────────────────────────────────────────────────────────────────┤ │ MONITORING LAYER │ │ Data drift (PSI/KS) · Concept drift · Calibration decay · Anomaly │ │ [LLM: drift narration, root-cause hypotheses, on-call summaries] │ ├──────────────────────────────────────────────────────────────────────┤ │ ITERATION LAYER │ │ Policy-governed retraining · Attribution · Early-warning · Scenarios │ │ [LLM: root-cause analysis, retraining recommendations — reviewed] │ └──────────────────────────────────────────────────────────────────────┘
The defining characteristic of this architecture — the thing that makes it defensible in a design review and survivable in production — is the clean separation of concerns between the statistical layer and the LLM layer. The forecasts come from forecasting models: classical, tree-based, deep, and foundation, selected by honest backtesting. The LLMs reason about the system, narrate it, generate its configuration, diagnose its failures, and connect it to the humans who make decisions. Neither layer pretends to be the other.
This separation is not a compromise. It is the correct factoring of the problem. Forecasting is a statistical task with decades of hard-won theory, and the foundation models are the latest and a genuinely valuable addition to that lineage — but they are forecasting models, evaluated as forecasting models, not language models doing arithmetic. Operations, communication, and reasoning about the system are language tasks, and that is where LLMs are genuinely transformative. A system that respects this distinction gets the best of both. A system that collapses it — that asks an LLM to forecast, or that excludes LLMs from operations out of purism — gets the worst of one.
IX. What This Buys You
A forecasting organization that builds on this architecture gets several things that single-model systems cannot provide:
It gets accuracy across the full diversity of its series, because the zoo means every series is forecast by a model suited to it — short series by classical models, rich-covariate series by deep learning, cold-start series by foundation models, the long tail cheaply by zero-shot.
It gets honest uncertainty, because calibration is a first-class selection criterion and probabilistic output is the default, so the distributions that drive decisions actually mean what they claim.
It gets resilience, because the monitoring layer catches drift before it costs and distinguishes the kind that needs retraining from the kind that needs rethinking.
It gets trust, because every forecast can be interrogated — the model selection is explained, the drivers are attributed, the changes are narrated — and a forecast that can be interrogated is a forecast that gets used.
And it gets leverage from LLMs without betting the system on them, because the language models do the operations and communication work they excel at while the statistical models do the forecasting work they excel at, with a clean line between the two that survives contact with production.
The forecasting systems that will matter over the next several years are not the ones that bet everything on the newest model class. They are the ones that treat every model class — classical, tree, deep, foundation, and LLM-adapter — as a tool with a job, governed by honest evaluation and connected by an intelligence layer that knows its place. That is the system worth building.
Technical Stack Reference
The architecture in this article is implementable today with the following open-source and commercial components:
- Classical forecasting:
StatsForecast(Nixtla) — ARIMA, ETS, Theta, Croston, TBATS at scale - Deep learning forecasting:
NeuralForecast(Nixtla) — TFT, N-HiTS, TiDE, TSMixer - Gradient boosting:
LightGBM,XGBoostwith custom quantile objectives - Foundation models: Chronos-2 (Amazon), TimesFM (Google), MOIRAI-2 (Salesforce), Lag-Llama
- LLM-adapter: Time-LLM (reprogramming framework)
- Orchestration: Airflow or Prefect for pipeline DAGs
- Feature store: Feast or a warehouse-native store (Snowflake, BigQuery)
- Monitoring: custom PSI/KS detectors + Arize, WhyLabs, or Fiddler for observability
- Compute: Spark for parallel classical fitting; GPU cluster for deep + foundation models
References
- Das, A., Kong, W., Sen, R., & Zhou, Y. (2024). A decoder-only foundation model for time-series forecasting. Google Research. arXiv:2310.10688. [TimesFM]
- Ansari, A. F. et al. (2024). Chronos: Learning the language of time series. Amazon Science. [Chronos]
- Woo, G. et al. (2024). Unified training of universal time series forecasting transformers. Salesforce Research. [MOIRAI]
- Rasul, K. et al. (2023). Lag-Llama: Towards foundation models for probabilistic time series forecasting. arXiv:2310.08278.
- Jin, M. et al. (2024). Time-LLM: Time series forecasting by reprogramming large language models. ICLR 2024. arXiv:2310.01728.
- Lim, B. et al. (2021). Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4). [TFT]
- Das, A. et al. (2023). Long-term forecasting with TiDE: Time-series dense encoder. TMLR. [TiDE]
- Chen, S. et al. (2023). TSMixer: An all-MLP architecture for time series forecasting. TMLR.
- Oreshkin, B. et al. (2020). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR 2020.
- Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4). [MASE]
- Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4).
- Gama, J. et al. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4).
- Shi, X. et al. (2025). Time-MoE: Billion-scale time series foundation models with mixture of experts. ICLR 2025.
- Curry, B. (2024–2026). Vector1 Research Series. Medium / vector1.ai.
Forecasting at Scale: A Hybrid Architecture Where Classical Models, Deep Learning, Foundation Models, and LLMs Each Do What They Do Best
By Brian Curry | Vector1 Research
“The question is never ‘which model is best.’ It is ‘which model is best for this series, at this horizon, under this regime, with this much history, given this cost of being wrong.’ Any system that answers with a single model is answering the wrong question.”
Abstract
The discourse around forecasting in 2026 has been captured by two competing hype narratives. The first holds that time-series foundation models — Chronos-2, TimesFM, MOIRAI-2 — have made everything else obsolete, that zero-shot forecasting is the future, and that classical statistics belongs in a museum. The second holds that you can simply hand a large language model your time series and ask it to predict the future. Both narratives are wrong, and building a production forecasting system on either one will cost a company dearly.
This article presents a defensible, production-grade reference architecture for forecasting at web scale — the kind that forecasts metrics like monthly active users, revenue, demand, or capacity for an organization where being wrong has real consequences. The central thesis is that a serious forecasting system is a plural system: classical statistical models, gradient-boosted trees, modern deep learning architectures, and time-series foundation models all coexist in a governed model zoo, competing and ensembling per series. Large language models are not the forecasting engine. They are the operations intelligence layer — helping build features, reason about model selection, narrate drift, diagnose anomalies, generate deployment configuration, and translate forecasts into decisions. They wrap the system; they do not replace its statistics.
We walk the complete lifecycle: from use-case framing through feature engineering, model development across the full zoo, selection and ensembling, production deployment, monitoring and drift detection, and the iteration loop — with the LLM operations layer woven through each stage. The goal is an architecture a senior practitioner could actually defend in a design review and actually run at scale.
Keywords: Time-Series Forecasting, Foundation Models, Production ML, Model Zoo, Ensemble Forecasting, Drift Detection, MLOps, LLM Operations, Backtesting, Probabilistic Forecasting, Explainability
I. Start With the Decision, Not the Model
Every forecasting system that fails in production fails for the same reason: it was built to optimize a metric instead of to inform a decision. The MAPE looked great in the notebook, and then the forecast was useless to the people who had to act on it.
Before any model is chosen, before any feature is engineered, the system must answer four questions about the decision the forecast serves:
What decision does this forecast drive, and what is the cost structure of being wrong? Forecasting monthly active users for a board deck has a symmetric, relatively forgiving error tolerance — being off by 2% in either direction is fine. Forecasting server capacity has a profoundly asymmetric cost: under-provisioning causes outages (catastrophic), over-provisioning wastes money (merely bad). The loss function must encode this asymmetry. A system that optimizes symmetric RMSE for an asymmetric decision is optimizing the wrong thing, no matter how sophisticated the model.
What horizon and granularity actually matter? A forecast that is accurate at a 1-day horizon but needed at 90 days is worthless. A daily forecast when the decision is made monthly is false precision. The horizon and granularity are properties of the decision, not the data.
What is the required form of the output — a point, a distribution, or a scenario? Most production decisions need a distribution, not a point. “MAU will be 47.2M” is far less useful than “MAU will be between 45.1M and 49.3M with 80% confidence, with the downside scenario driven primarily by the seasonal dip we see every Q3.” Probabilistic forecasting is not a luxury; it is the default requirement for any forecast that informs a real decision under uncertainty.
What is the explainability requirement? A forecast that leadership cannot interrogate will not be trusted, and an untrusted forecast does not inform decisions regardless of its accuracy. If a VP asks “why did the forecast drop 8% versus last month’s run,” the system must be able to answer. This requirement shapes model selection as much as accuracy does — and it is where the LLM layer earns much of its keep.
This is the first place the LLM operations layer enters, and it enters as a structuring tool, not a predictor:
from canary.llm import LLMClient
class ForecastFraming:
"""
Uses an LLM to help structure a forecasting problem from a business
request before any modeling begins. The LLM does not forecast — it
interrogates the decision, surfaces the implicit loss function, and
proposes an evaluation protocol.
"""
def __init__(self, llm: LLMClient):
self.llm = llm
def frame_problem(self, business_request: str,
available_history: dict) -> dict:
"""
Translate a vague business request into a precise forecasting spec.
Example input: "We need to know how many users we'll have next year
so finance can plan headcount."
Output: structured spec including horizon, granularity, loss
asymmetry, output form, and explainability requirements.
"""
response = self.llm.complete(f"""
A stakeholder made this forecasting request:
"{business_request}"
Available data: {available_history}
Produce a structured forecasting specification. Specifically reason about:
1. The actual decision this forecast drives
2. Whether the cost of over-forecasting vs under-forecasting is symmetric
(and if not, which direction is more costly and roughly by how much)
3. The decision-relevant horizon and granularity
4. Whether a point forecast suffices or a full predictive distribution
is required
5. The explainability requirement based on who consumes the forecast
Return structured JSON. Flag any place where the request is
underspecified and a human should clarify before proceeding.
""")
return self._parse_spec(response)The output of this stage is not a model. It is a contract: the precise specification the rest of the system is built to satisfy. Everything downstream is judged against it.
II. Features: The Part Foundation Models Don’t Make Obsolete
One of the most damaging claims in the foundation-model hype is that feature engineering is dead — that you can hand a raw series to a pretrained transformer and skip the work. This is true for exactly the class of problems where it doesn’t matter much, and false for most problems where forecasting is hard and valuable.
Foundation models are remarkable at capturing the endogenous structure of a series — its own seasonality, trend, and autocorrelation. They are weak precisely where most business value lives: in the exogenous drivers. The marketing campaign that launches next Tuesday, the pricing change, the competitor’s product launch, the holiday calendar specific to your business, the macroeconomic covariate — these are the things that move the metric and that the foundation model has never seen.
A production feature layer therefore has to serve the entire zoo, because different model families consume features differently:
import pandas as pd
import numpy as np
from dataclasses import dataclass
from typing import Optional
@dataclass
class FeatureSpec:
"""Defines the feature surface available to every model in the zoo."""
# Calendar features (consumed by tree models, deep learning, TiDE)
calendar: list[str] # day_of_week, week_of_year, month, is_holiday, ...
# Lag features (consumed by tree models, Lag-Llama)
lags: list[int] # [1, 7, 28, 364] — daily/weekly/monthly/yearly
# Rolling statistics (consumed by tree models)
rolling_windows: list[int] # [7, 28, 90] day windows for mean/std/min/max
# Known future covariates (consumed by TFT, TiDE, TSMixer, MOIRAI)
future_covariates: list[str] # planned_promotions, price, known_events
# Past-only covariates (consumed by most multivariate models)
past_covariates: list[str] # competitor_signal, weather, web_traffic
# Static covariates (consumed by global models)
static: list[str] # region, product_category, customer_segment
class FeaturePipeline:
"""
Constructs the full feature surface from raw series + covariates.
Critically, this pipeline produces features in the formats required by
EVERY model family, because the model zoo will consume the same
underlying data in different shapes:
- Classical models: just the target series (+ optional exog regressors)
- Tree models: wide tabular feature matrix (lags, rolling, calendar)
- Deep learning: tensor windows with covariate channels
- Foundation models: context windows + covariate conditioning
"""
def __init__(self, spec: FeatureSpec):
self.spec = spec
def build_tabular(self, df: pd.DataFrame,
target_col: str) -> pd.DataFrame:
"""
Build the wide tabular feature matrix for gradient-boosted models.
This is where domain expertise still dominates foundation models.
"""
features = df.copy()
# Lag features
for lag in self.spec.lags:
features[f"{target_col}_lag_{lag}"] = df[target_col].shift(lag)
# Rolling statistics (computed on lagged values to avoid leakage)
for window in self.spec.rolling_windows:
shifted = df[target_col].shift(1) # shift first — no leakage
features[f"{target_col}_roll_mean_{window}"] = (
shifted.rolling(window).mean()
)
features[f"{target_col}_roll_std_{window}"] = (
shifted.rolling(window).std()
)
# Calendar features
features["day_of_week"] = df.index.dayofweek
features["week_of_year"] = df.index.isocalendar().week
features["month"] = df.index.month
features["quarter"] = df.index.quarter
# Fourier terms for smooth seasonality (helps tree models)
for period, order in [(7, 3), (365.25, 5)]:
for k in range(1, order + 1):
t = np.arange(len(df))
features[f"sin_{period}_{k}"] = np.sin(2 * np.pi * k * t / period)
features[f"cos_{period}_{k}"] = np.cos(2 * np.pi * k * t / period)
return features
def build_tensor_windows(self, df: pd.DataFrame, target_col: str,
context_length: int, horizon: int):
"""
Build sliding tensor windows for deep learning and foundation models.
Returns context windows, future covariate windows, and targets.
"""
# Implementation produces (n_windows, context_length, n_channels) tensors
# consumed by TFT, TiDE, TSMixer, and the foundation model adapters
...The LLM layer contributes here too — not by engineering features directly, but by reasoning about which features are likely to matter for a given series and surfacing candidate exogenous drivers a human might have missed:
class FeatureAdvisor:
"""
LLM-assisted feature discovery. The LLM proposes candidate features
and exogenous drivers based on the domain context. A human reviews;
the pipeline implements. The LLM never silently adds features to a
production model.
"""
def suggest_features(self, series_description: str,
current_features: list[str]) -> dict:
response = self.llm.complete(f"""
We are forecasting: {series_description}
Current features: {current_features}
Propose additional candidate features or exogenous drivers that
domain knowledge suggests would improve this forecast. For each,
explain the hypothesized causal mechanism and how to source the data.
Prioritize known-future covariates (things we'll know in advance for
the forecast horizon) since these are most valuable. Flag any
proposed feature that risks target leakage.
""")
return self._parse_suggestions(response)The discipline here is essential: the LLM proposes, a human disposes, and the pipeline implements deterministically. An LLM that silently injects features into a production forecasting model is a source of untraceable drift, not a feature.
III. The Model Zoo: Five Families, Each With a Job
This is the heart of the architecture and the place where the article diverges most sharply from the single-model hype. A production forecasting system at scale does not pick a model. It maintains a governed zoo of model families and lets the data decide, per series, which wins.
3.1 Why Plurality Is Non-Negotiable
The empirical reality, established across the M-competitions and confirmed repeatedly in production, is that no model family dominates across all series. The factors that determine which family wins are properties of the individual series:
- History length: a series with 60 observations cannot support a deep learning model; a well-specified ETS or Theta model will beat a transformer every time. A series with 50,000 observations and rich covariates is where deep learning and foundation models shine.
- Signal-to-noise and regularity: clean, strongly seasonal series are often won by classical statistical models that encode that structure directly. Noisy, irregular series with complex covariate interactions favor gradient-boosted trees or deep learning.
- Covariate richness: if the forecast is driven by known-future covariates (promotions, pricing, events), models that consume them — TFT, TiDE, TSMixer, MOIRAI — have a structural advantage.
- Cold-start: a brand-new series with no history is precisely where foundation models earn their existence, providing reasonable zero-shot forecasts where every trained model would fail.
A system that uses one family is leaving accuracy on the table for most of its series. The cost of maintaining the zoo is real, but at scale it is dwarfed by the cost of systematically worse forecasts.
3.2 The Five Families
Family 1 — Classical Statistical (StatsForecast). ARIMA, ETS, Theta, TBATS, Croston (for intermittent demand). Fast, interpretable, well-calibrated, and frequently the winner on short, regular series. The StatsForecast library makes these trivially parallelizable across millions of series. These models are not legacy — they are the correct answer for a large fraction of real series, and any architecture that excludes them is worse for it.
Family 2 — Gradient-Boosted Trees (LightGBM / XGBoost). Trained on the wide tabular feature matrix, a single global LightGBM model across many series, with categorical static features identifying each series, remains one of the strongest and most cost-effective approaches in existence. It dominates when there are rich engineered features and complex nonlinear covariate interactions. It is cheap to train, cheap to serve, and naturally explainable through feature importance and SHAP.
Family 3 — Modern Deep Learning (NeuralForecast). Temporal Fusion Transformer (TFT) for interpretable attention over covariates, N-BEATS / N-HiTS for pure univariate power, TiDE and TSMixer for efficient covariate-aware MLP architectures. These win on long, complex, multivariate series with sufficient history. TiDE and TSMixer specifically are MLP-based and far cheaper to train than transformers while remaining competitive — important at scale.
Family 4 — Time-Series Foundation Models. Chronos-2 (T5-based, tokenizes values, production-mature with AWS integration), TimesFM (Google, decoder-only, enterprise-reliable), MOIRAI-2 (Salesforce, strong multivariate and covariate support), Lag-Llama (decoder-only, explicit lag features, strong probabilistic output). Their unique value is zero-shot forecasting: reasonable forecasts on series they have never seen, with no training. This makes them the answer for cold-start, for the long tail of low-volume series where training a bespoke model is uneconomical, and as a strong baseline that any trained model must beat to justify its existence.
Family 5 — LLM-Adapter Models (Time-LLM). A genuinely distinct family worth including in the zoo as a forecasting tool, not just as the ops layer. Time-LLM reprograms a frozen LLM (GPT-2, LLaMA) to forecast by translating time-series patches into text prototypes, injecting domain knowledge through a “prompt-as-prefix.” Its niche is the situation where forecasting must incorporate unstructured contextual information — a series whose future is shaped by described events that resist encoding as numeric covariates. It is the most experimental family and should be governed accordingly, but it earns a place in the zoo for problems where text context genuinely drives the target.
from statsforecast import StatsForecast
from statsforecast.models import AutoARIMA, AutoETS, AutoTheta, CrostonOptimized
from neuralforecast import NeuralForecast
from neuralforecast.models import TFT, NHITS, TiDE, TSMixer
import lightgbm as lgb
class ModelZoo:
"""
Governed registry of forecasting model families.
Each family is registered with metadata describing where it is
appropriate, so the selection layer (Section IV) can reason about
which candidates to run for a given series rather than running all
of them — critical for cost control at scale.
"""
def __init__(self):
self.registry = {}
self._register_defaults()
def _register_defaults(self):
# Family 1: Classical statistical
self.register(
name="statistical",
builder=lambda: StatsForecast(
models=[AutoARIMA(), AutoETS(), AutoTheta()],
freq='D',
n_jobs=-1
),
appropriate_when={
"min_history": 30,
"max_history": None,
"strengths": ["short_series", "regular_seasonality",
"fast", "interpretable", "well_calibrated"],
"cost": "very_low"
}
)
# Family 1b: Intermittent demand
self.register(
name="intermittent",
builder=lambda: StatsForecast(
models=[CrostonOptimized()], freq='D', n_jobs=-1
),
appropriate_when={
"min_history": 30,
"strengths": ["intermittent_demand", "sparse_series"],
"cost": "very_low"
}
)
# Family 2: Gradient-boosted trees (global model)
self.register(
name="gbdt_global",
builder=lambda: LightGBMForecaster(
params={"objective": "quantile", "alpha": 0.5,
"num_leaves": 128, "learning_rate": 0.05}
),
appropriate_when={
"min_history": 100,
"strengths": ["rich_covariates", "nonlinear",
"many_series", "cheap_to_serve", "explainable"],
"cost": "low"
}
)
# Family 3: Deep learning
self.register(
name="deep_learning",
builder=lambda horizon: NeuralForecast(
models=[
TFT(h=horizon, input_size=horizon*4, max_steps=500),
NHITS(h=horizon, input_size=horizon*4, max_steps=500),
TiDE(h=horizon, input_size=horizon*4, max_steps=500),
],
freq='D'
),
appropriate_when={
"min_history": 1000,
"strengths": ["long_complex_series", "multivariate",
"covariate_interactions"],
"cost": "high"
}
)
# Family 4: Foundation models (zero-shot)
self.register(
name="foundation",
builder=lambda: FoundationModelAdapter(
models=["chronos-2", "timesfm", "moirai-2"]
),
appropriate_when={
"min_history": 0, # zero-shot — works with no history
"strengths": ["cold_start", "zero_shot", "long_tail",
"strong_baseline"],
"cost": "medium"
}
)
# Family 5: LLM-adapter
self.register(
name="llm_adapter",
builder=lambda: TimeLLMForecaster(backbone="llama"),
appropriate_when={
"min_history": 200,
"strengths": ["text_context_drivers", "described_events"],
"cost": "high",
"governance": "experimental"
}
)
def register(self, name: str, builder, appropriate_when: dict):
self.registry[name] = {
"builder": builder,
"metadata": appropriate_when
}
def candidates_for_series(self, series_profile: dict) -> list[str]:
"""
Return the subset of model families appropriate for a series,
based on its profile. This prevents running every model on every
series — essential for cost control at web scale.
"""
candidates = []
history_len = series_profile["history_length"]
for name, entry in self.registry.items():
meta = entry["metadata"]
if history_len >= meta.get("min_history", 0):
if meta.get("max_history") is None or \
history_len <= meta["max_history"]:
candidates.append(name)
return candidatesIV. Selection and Ensembling: Letting the Data Decide
With a zoo of candidate models, the system needs a principled way to decide which model — or which combination — to deploy for each series. This is done through rigorous backtesting, and it is where most forecasting systems quietly cheat.
4.1 Backtesting Without Lying to Yourself
The single most common failure in forecasting is temporal leakage in evaluation — letting information from the future contaminate the assessment of past performance. The only honest evaluation is time-series cross-validation with expanding or rolling windows, where the model is repeatedly trained on data up to time t and evaluated on t+1…t+h, walked forward across the series.
from dataclasses import dataclass
import numpy as np
@dataclass
class BacktestResult:
model_name: str
series_id: str
fold_errors: list[dict] # per-fold metrics
mean_metrics: dict # aggregated across folds
calibration: dict # coverage of prediction intervals
cost_estimate: float # training + inference cost
class Backtester:
"""
Walk-forward time-series cross-validation.
Critically evaluates not just point accuracy but:
- Probabilistic calibration (do the 80% intervals actually contain
the truth 80% of the time?)
- The asymmetric, decision-relevant loss from Section I
- Cost, so the selection layer can trade accuracy against expense
"""
def __init__(self, n_windows: int = 5, horizon: int = 28,
step_size: int = 28, loss_fn=None):
self.n_windows = n_windows
self.horizon = horizon
self.step_size = step_size
self.loss_fn = loss_fn or self._default_loss
def evaluate(self, model, series, covariates=None) -> BacktestResult:
"""Walk forward through the series, training and evaluating each fold."""
fold_errors = []
n = len(series)
for w in range(self.n_windows):
# Expanding window: train on everything up to the cutoff
cutoff = n - (self.n_windows - w) * self.step_size
train = series[:cutoff]
actual = series[cutoff:cutoff + self.horizon]
if len(actual) < self.horizon:
continue
# Fit and forecast (probabilistic where supported)
model.fit(train, covariates)
forecast = model.predict(self.horizon, level=[80, 95])
fold_errors.append({
"fold": w,
"mae": self._mae(actual, forecast["point"]),
"rmse": self._rmse(actual, forecast["point"]),
"mase": self._mase(actual, forecast["point"], train),
"decision_loss": self.loss_fn(actual, forecast["point"]),
"coverage_80": self._coverage(actual, forecast, 80),
"coverage_95": self._coverage(actual, forecast, 95),
})
return BacktestResult(
model_name=model.name,
series_id=series.name,
fold_errors=fold_errors,
mean_metrics=self._aggregate(fold_errors),
calibration=self._calibration_summary(fold_errors),
cost_estimate=model.cost_estimate()
)
@staticmethod
def _mase(actual, predicted, train):
"""
Mean Absolute Scaled Error — the right default metric.
Scale-free, interpretable (MASE < 1 beats naive seasonal),
and comparable across series. Far superior to MAPE, which
explodes near zero and is asymmetric.
"""
naive_errors = np.abs(np.diff(train))
scale = naive_errors.mean() + 1e-8
return np.mean(np.abs(actual - predicted)) / scale
@staticmethod
def _coverage(actual, forecast, level):
"""
Empirical coverage of prediction intervals — the calibration check
that almost everyone skips and that matters more than point accuracy
for decisions under uncertainty.
"""
lower = forecast[f"lower_{level}"]
upper = forecast[f"upper_{level}"]
inside = (actual >= lower) & (actual <= upper)
return float(inside.mean())Two things in this backtester deserve emphasis because they are routinely neglected. First, MASE over MAPE: MAPE is the default in most organizations and it is a bad default — it is undefined at zero, explodes near zero, and asymmetrically penalizes over- versus under-forecasting. MASE is scale-free, interpretable against a naive baseline, and comparable across series. Second, calibration: a forecast whose 80% intervals only contain the truth 60% of the time is lying about its own uncertainty, and decisions made on that false confidence will be systematically too aggressive. Coverage is not optional.
4.2 Selection: Per-Series, Cost-Aware
With honest backtest results across the candidate zoo, selection becomes an optimization that trades accuracy against cost and respects the decision’s requirements:
class ModelSelector:
"""
Selects the deployed model (or ensemble) per series from backtest results.
Selection is multi-objective:
- Accuracy on the decision-relevant loss
- Calibration (intervals must be honest)
- Cost (at web scale, a 0.5% accuracy gain is not worth 10x the cost)
- Robustness (consistency across folds, not just mean performance)
"""
def select(self, backtest_results: list[BacktestResult],
requirements: dict) -> dict:
scored = []
for result in backtest_results:
# Reject models that fail the calibration requirement outright
if requirements.get("requires_calibration", True):
cov_80 = result.calibration.get("mean_coverage_80", 0)
if abs(cov_80 - 0.80) > 0.10: # more than 10pp miscalibrated
continue
# Composite score: accuracy, robustness, cost
accuracy_score = 1.0 / (1.0 + result.mean_metrics["decision_loss"])
robustness = 1.0 / (1.0 + result.mean_metrics.get("loss_std", 0))
cost_penalty = result.cost_estimate / requirements.get(
"cost_budget", 1.0)
score = (
0.6 * accuracy_score +
0.25 * robustness -
0.15 * cost_penalty
)
scored.append((result, score))
scored.sort(key=lambda x: x[1], reverse=True)
# Decide: single model or ensemble?
return self._decide_single_or_ensemble(scored, requirements)
def _decide_single_or_ensemble(self, scored, requirements):
"""
Ensembling almost always improves accuracy and calibration, but
costs more to run. The decision depends on the value of the series.
For high-value series: ensemble the top 3 families (diversity matters
more than individual strength — a classical + GBDT + foundation model
ensemble is robust precisely because the families fail differently).
For long-tail low-value series: single best model, usually the
foundation model or a cheap classical model.
"""
top = scored[0][0]
if requirements.get("series_value", "low") == "high" and len(scored) >= 3:
# Ensemble the top 3 — but only if they're diverse families
top_3 = [s[0] for s in scored[:3]]
families = set(r.model_name.split("_")[0] for r in top_3)
if len(families) >= 2: # require family diversity
return {
"strategy": "ensemble",
"models": [r.model_name for r in top_3],
"weights": self._optimize_ensemble_weights(top_3),
"rationale": "high-value series, diverse family ensemble"
}
return {
"strategy": "single",
"model": top.model_name,
"rationale": f"best composite score, "
f"MASE={top.mean_metrics['mase']:.3f}"
}The ensembling insight worth dwelling on: the value of an ensemble comes from diversity of failure modes, not from averaging strong models. A classical model, a gradient-boosted tree, and a foundation model fail on different series for different reasons. Their errors are weakly correlated, so combining them reduces variance more than combining three deep learning models would. This is why the zoo’s plurality is not just about picking winners — it is the raw material for robust ensembles.
V. The LLM Operations Layer: Connective Tissue, Not Engine
Here is where large language models do their real work in this architecture — and it is everywhere except the forecast itself. The LLM is the system’s operations intelligence: it reasons about the system, narrates it, and connects it to humans. It never produces the number.
5.1 Build-Time: Model Selection Reasoning
The selection logic in Section IV is quantitative, but its output needs to be explained and sometimes overridden by human judgment. The LLM translates the backtest evidence into reasoning a human can audit:
class SelectionExplainer:
"""
The LLM reads the backtest results and the selection decision and
produces a human-auditable rationale. This is not the LLM making the
decision — the decision is made quantitatively. The LLM makes the
decision legible and flags cases where a human should look.
"""
def explain_selection(self, backtest_results, selection) -> str:
return self.llm.complete(f"""
Forecasting model selection for series {selection.get('series_id')}.
Backtest results across the model zoo:
{self._format_results(backtest_results)}
Selected: {selection}
Explain in 3-4 sentences why this selection is reasonable given the
evidence. Specifically note: was the decision close? Is there a
cheaper model that was nearly as good? Are there signs the winning
model is overfit to the backtest folds? Flag for human review if
the top two models are within 2% on the decision loss but differ
significantly in cost or interpretability.
""")5.2 Deploy-Time: Configuration and Backtest Orchestration
LLMs are genuinely useful at generating and validating the deployment configuration — the glue code, the pipeline DAGs, the monitoring thresholds — from the validated model spec. This is code generation against a known schema, which is what LLMs are reliably good at, and it always passes through validation and human review before it touches production:
class DeploymentConfigGenerator:
"""
Generate deployment configuration (orchestration DAG, monitoring
thresholds, retraining triggers) from the validated model selection.
Output is always validated against a schema and reviewed before deploy.
"""
def generate_monitoring_config(self, selection, backtest) -> dict:
suggested = self.llm.complete(f"""
We are deploying {selection} to production.
Its backtest performance: {backtest.mean_metrics}
Its fold-to-fold variance: {backtest.calibration}
Propose monitoring thresholds:
- At what MASE degradation should we alert? (consider its backtest
variance — a noisy model needs wider thresholds)
- At what prediction-interval coverage breach should we alert?
- What retraining cadence fits this model family and series volatility?
Return JSON. These are SUGGESTIONS that a human will review and
the system will validate against safe bounds before applying.
""")
return self._validate_against_safe_bounds(self._parse(suggested))5.3 Monitor-Time: Drift Narration and Root-Cause Analysis
This is the highest-value application of the LLM layer. Drift detection itself is statistical (Section VI). But a drift alert is just a number — and a number does not tell an on-call engineer what happened or what to do. The LLM turns the statistical signal into an actionable narrative:
class DriftAnalyst:
"""
When statistical drift detection fires, the LLM synthesizes the
quantitative signals into a root-cause hypothesis and recommended action.
The detection is statistical; the diagnosis and communication are the LLM's.
"""
def analyze_drift(self, drift_signal: dict, recent_data: dict,
forecast_errors: dict, known_events: list) -> dict:
return self.llm.complete(f"""
A forecasting model has triggered a drift alert.
Drift signals: {drift_signal}
Recent actuals vs forecast: {forecast_errors}
Known business events in the window: {known_events}
Feature distribution changes: {recent_data['feature_drift']}
Produce:
1. A ranked set of root-cause hypotheses (data quality issue?
genuine regime change? known event the model didn't account for?
upstream pipeline break?)
2. For each, the evidence for and against
3. A recommended action (retrain? hotfix? wait and monitor?
escalate to human?)
4. A one-paragraph summary suitable for an on-call engineer at 3am
Distinguish clearly between data drift (input distribution changed)
and concept drift (input-output relationship changed) — they require
different responses.
""")5.4 Decision-Time: Executive Narrative Generation
The final mile of any forecasting system is translating the forecast into a decision, and that translation is a communication problem. The LLM is genuinely good at turning a probabilistic forecast plus its drivers into a concise executive narrative — provided it is given the actual numbers and explanations and constrained to report them faithfully:
class ForecastNarrator:
"""
Translate a forecast + its explainability outputs into an executive
narrative. The LLM reports the numbers and the model's own attributions;
it does not invent or editorialize beyond the provided evidence.
"""
def narrate(self, forecast: dict, attributions: dict,
prior_forecast: dict, audience: str) -> str:
return self.llm.complete(f"""
Current forecast: {forecast} (point + 80% and 95% intervals)
Change vs last forecast: {self._diff(forecast, prior_forecast)}
Model's driver attributions: {attributions}
Audience: {audience}
Write a concise narrative (3-4 sentences for an executive audience):
- The headline number and the uncertainty around it
- What changed versus the prior forecast and why (use the attributions)
- The key risk to the forecast and the scenario that would break it
Report only what the numbers and attributions support. Do not
speculate beyond the provided evidence. State uncertainty plainly.
""")The discipline running through all four applications is the same: the LLM operates on the system’s outputs and reasons about the system, but it never produces the forecast and it never acts without validation. This is what makes the architecture defensible. An LLM hallucinating a narrative is an annoyance; an LLM hallucinating a forecast that drives a capacity decision is an outage.
VI. Monitoring and Drift: Detecting Decay Before It Costs
A forecasting model in production does not fail loudly. It degrades silently as the world drifts away from the conditions it was trained on. The research consensus is stark: static models suffer 15–22% performance degradation under non-stationary conditions, and 91% of model failures trace to drift. The monitoring layer exists to catch this before business metrics move.
There are two distinct phenomena, and conflating them is a common and costly error:
Data drift — the input distribution changes. Feature values shift statistically away from the training distribution. A MAU model trained on pre-launch traffic patterns sees different patterns post-launch. The features changed.
Concept drift — the input-output relationship changes, even when inputs look statistically stable. An economic regime change alters the relationship between your drivers and your target. The map changed, not just the territory.
They require different responses — data drift often calls for retraining on recent data, while concept drift may require rethinking the model or features entirely — so the monitoring layer must distinguish them:
import numpy as np
from scipy import stats
class ForecastMonitor:
"""
Multi-layer production monitoring for the forecasting system.
Four monitoring layers (the production-monitoring consensus for 2026):
1. Infrastructure: latency, throughput, errors
2. Data quality: input drift, missingness, schema changes
3. Model performance: rolling accuracy, calibration decay
4. Business impact: does the forecast still drive good decisions?
"""
def detect_data_drift(self, reference_dist: np.ndarray,
production_dist: np.ndarray) -> dict:
"""
Detect input distribution drift using a population stability index
plus a distributional test. PSI is the production standard; the
KS test adds statistical rigor.
"""
psi = self._population_stability_index(reference_dist, production_dist)
ks_stat, ks_p = stats.ks_2samp(reference_dist, production_dist)
return {
"psi": psi,
"ks_statistic": ks_stat,
"ks_p_value": ks_p,
# PSI > 0.2 is the conventional "significant drift" threshold
"data_drift_detected": psi > 0.2 or ks_p < 0.05,
"severity": "high" if psi > 0.25 else
"moderate" if psi > 0.1 else "low"
}
def detect_concept_drift(self, recent_errors: np.ndarray,
baseline_errors: np.ndarray) -> dict:
"""
Concept drift manifests as the model's error distribution worsening
even when inputs look stable. We detect it through a change in the
forecast error distribution, using both a magnitude test and a
change-point check.
"""
# Has the error distribution shifted upward significantly?
t_stat, t_p = stats.ttest_ind(recent_errors, baseline_errors,
alternative='greater')
error_inflation = recent_errors.mean() / (baseline_errors.mean() + 1e-8)
return {
"error_inflation_ratio": float(error_inflation),
"significance_p": float(t_p),
"concept_drift_detected": t_p < 0.05 and error_inflation > 1.15,
"interpretation": (
"Model relationship has degraded — inputs may look normal "
"but the model no longer maps them correctly to outputs"
)
}
@staticmethod
def _population_stability_index(ref, prod, bins=10) -> float:
"""PSI: the production-standard data drift metric."""
breakpoints = np.percentile(ref, np.linspace(0, 100, bins + 1))
breakpoints[0], breakpoints[-1] = -np.inf, np.inf
ref_pct = np.histogram(ref, breakpoints)[0] / len(ref) + 1e-6
prod_pct = np.histogram(prod, breakpoints)[0] / len(prod) + 1e-6
return float(np.sum((prod_pct - ref_pct) * np.log(prod_pct / ref_pct)))When drift fires, the system does not blindly retrain. The research is clear that proactive, policy-governed retraining outperforms reactive retraining by roughly 4x, but indiscriminate retraining on drifted data can bake in the very anomaly that triggered the alert. The drift signal goes to the LLM drift analyst (Section 5.3), which produces a root-cause hypothesis and recommended action, and a human reviews the policy decision for anything above a low-severity threshold. This is the autonomous-retraining-with-human-oversight pattern that defines mature MLOps in 2026: the system detects, evaluates whether retraining is cost-justified, retrains, validates, and deploys — with humans reviewing policies and exceptions rather than executing each step.
VII. The Iteration Loop: Anomaly Detection, Attribution, Early Warning
Point forecasts are the beginning, not the end, of a mature forecasting system’s value. The same infrastructure that produces forecasts produces several higher-order capabilities that are often where the system delivers its greatest business impact.
Anomaly detection as a forecasting byproduct. A forecast plus its prediction interval is an anomaly detector for free: an actual that falls outside the 99% interval is, by definition, an anomaly worth investigating. Running this across all monitored series produces an early-warning system for business health — the forecast that suddenly misses badly is often the first signal of a problem upstream.
Automated root-cause analysis. When a metric deviates from forecast, the question is always “why.” The combination of the feature attributions (from the explainable models in the zoo — TFT’s attention weights, the GBDT’s SHAP values) and the LLM’s reasoning layer produces a first-pass root-cause analysis automatically, turning a multi-hour analyst investigation into a reviewed hypothesis in minutes.
Channel and campaign attribution. For series driven by known interventions (marketing campaigns, product launches, pricing changes), the covariate-aware models in the zoo — TFT and TiDE especially — produce attributions of how much each driver contributed to the forecast. This connects forecasting directly to the decisions that the business controls.
Scenario analysis. Because the covariate-aware models accept known-future covariates, they natively support “what-if” scenarios: forecast MAU under three different marketing-spend scenarios, present the distribution of outcomes for each, and let leadership choose. This is where forecasting stops being a reporting function and becomes a decision-support function — and it is the capability that most justifies the investment in the full architecture.
class ScenarioEngine:
"""
Generate forecast scenarios under different assumptions about
controllable future covariates. The covariate-aware models in the zoo
(TFT, TiDE, TSMixer, MOIRAI) make this native: vary the known-future
covariates, re-forecast, present the distribution of outcomes.
"""
def run_scenarios(self, model, base_covariates: dict,
scenarios: list[dict]) -> dict:
results = {}
for scenario in scenarios:
modified = {**base_covariates, **scenario["covariate_overrides"]}
forecast = model.predict(
horizon=scenario["horizon"],
future_covariates=modified,
level=[80, 95]
)
results[scenario["name"]] = {
"forecast": forecast,
"expected_value": forecast["point"].sum(),
"downside_80": forecast["lower_80"].sum(),
"upside_80": forecast["upper_80"].sum(),
}
return resultsVIII. The Reference Architecture, Assembled
Pulling the layers together, the complete system has a clear shape:
┌──────────────────────────────────────────────────────────────────────┐ │ DECISION LAYER │ │ Use-case framing · Loss function definition · Output requirements │ │ [LLM: problem framing, narrative generation, scenario presentation] │ ├──────────────────────────────────────────────────────────────────────┤ │ FEATURE LAYER │ │ Calendar · Lags · Rolling stats · Known-future & past covariates │ │ [LLM: feature discovery suggestions — human reviewed] │ ├──────────────────────────────────────────────────────────────────────┤ │ MODEL ZOO │ │ Classical (StatsForecast) · GBDT (LightGBM) · Deep (NeuralForecast) │ │ Foundation (Chronos-2, TimesFM, MOIRAI-2) · LLM-adapter (Time-LLM) │ ├──────────────────────────────────────────────────────────────────────┤ │ SELECTION & ENSEMBLE LAYER │ │ Walk-forward backtesting · Calibration check · Cost-aware selection │ │ Diversity-driven ensembling │ │ [LLM: selection rationale, human-review flagging] │ ├──────────────────────────────────────────────────────────────────────┤ │ SERVING LAYER │ │ Batch + on-demand inference · Probabilistic output · Scenario engine │ │ [LLM: deployment config generation — schema-validated] │ ├──────────────────────────────────────────────────────────────────────┤ │ MONITORING LAYER │ │ Data drift (PSI/KS) · Concept drift · Calibration decay · Anomaly │ │ [LLM: drift narration, root-cause hypotheses, on-call summaries] │ ├──────────────────────────────────────────────────────────────────────┤ │ ITERATION LAYER │ │ Policy-governed retraining · Attribution · Early-warning · Scenarios │ │ [LLM: root-cause analysis, retraining recommendations — reviewed] │ └──────────────────────────────────────────────────────────────────────┘
The defining characteristic of this architecture — the thing that makes it defensible in a design review and survivable in production — is the clean separation of concerns between the statistical layer and the LLM layer. The forecasts come from forecasting models: classical, tree-based, deep, and foundation, selected by honest backtesting. The LLMs reason about the system, narrate it, generate its configuration, diagnose its failures, and connect it to the humans who make decisions. Neither layer pretends to be the other.
This separation is not a compromise. It is the correct factoring of the problem. Forecasting is a statistical task with decades of hard-won theory, and the foundation models are the latest and a genuinely valuable addition to that lineage — but they are forecasting models, evaluated as forecasting models, not language models doing arithmetic. Operations, communication, and reasoning about the system are language tasks, and that is where LLMs are genuinely transformative. A system that respects this distinction gets the best of both. A system that collapses it — that asks an LLM to forecast, or that excludes LLMs from operations out of purism — gets the worst of one.
IX. What This Buys You
A forecasting organization that builds on this architecture gets several things that single-model systems cannot provide:
It gets accuracy across the full diversity of its series, because the zoo means every series is forecast by a model suited to it — short series by classical models, rich-covariate series by deep learning, cold-start series by foundation models, the long tail cheaply by zero-shot.
It gets honest uncertainty, because calibration is a first-class selection criterion and probabilistic output is the default, so the distributions that drive decisions actually mean what they claim.
It gets resilience, because the monitoring layer catches drift before it costs and distinguishes the kind that needs retraining from the kind that needs rethinking.
It gets trust, because every forecast can be interrogated — the model selection is explained, the drivers are attributed, the changes are narrated — and a forecast that can be interrogated is a forecast that gets used.
And it gets leverage from LLMs without betting the system on them, because the language models do the operations and communication work they excel at while the statistical models do the forecasting work they excel at, with a clean line between the two that survives contact with production.
The forecasting systems that will matter over the next several years are not the ones that bet everything on the newest model class. They are the ones that treat every model class — classical, tree, deep, foundation, and LLM-adapter — as a tool with a job, governed by honest evaluation and connected by an intelligence layer that knows its place. That is the system worth building.
Technical Stack Reference
The architecture in this article is implementable today with the following open-source and commercial components:
- Classical forecasting:
StatsForecast(Nixtla) — ARIMA, ETS, Theta, Croston, TBATS at scale - Deep learning forecasting:
NeuralForecast(Nixtla) — TFT, N-HiTS, TiDE, TSMixer - Gradient boosting:
LightGBM,XGBoostwith custom quantile objectives - Foundation models: Chronos-2 (Amazon), TimesFM (Google), MOIRAI-2 (Salesforce), Lag-Llama
- LLM-adapter: Time-LLM (reprogramming framework)
- Orchestration: Airflow or Prefect for pipeline DAGs
- Feature store: Feast or a warehouse-native store (Snowflake, BigQuery)
- Monitoring: custom PSI/KS detectors + Arize, WhyLabs, or Fiddler for observability
- Compute: Spark for parallel classical fitting; GPU cluster for deep + foundation models
References
- Das, A., Kong, W., Sen, R., & Zhou, Y. (2024). A decoder-only foundation model for time-series forecasting. Google Research. arXiv:2310.10688. [TimesFM]
- Ansari, A. F. et al. (2024). Chronos: Learning the language of time series. Amazon Science. [Chronos]
- Woo, G. et al. (2024). Unified training of universal time series forecasting transformers. Salesforce Research. [MOIRAI]
- Rasul, K. et al. (2023). Lag-Llama: Towards foundation models for probabilistic time series forecasting. arXiv:2310.08278.
- Jin, M. et al. (2024). Time-LLM: Time series forecasting by reprogramming large language models. ICLR 2024. arXiv:2310.01728.
- Lim, B. et al. (2021). Temporal Fusion Transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4). [TFT]
- Das, A. et al. (2023). Long-term forecasting with TiDE: Time-series dense encoder. TMLR. [TiDE]
- Chen, S. et al. (2023). TSMixer: An all-MLP architecture for time series forecasting. TMLR.
- Oreshkin, B. et al. (2020). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR 2020.
- Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4). [MASE]
- Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2022). M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4).
- Gama, J. et al. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4).
- Shi, X. et al. (2025). Time-MoE: Billion-scale time series foundation models with mixture of experts. ICLR 2025.
- Curry, B. (2024–2026). Vector1 Research Series. Medium / vector1.ai.
About the Author
Brian Curry is a Kansas City–based AI researcher, data scientist, and founder of Vector1 Research, where he works at the intersection of forecasting, causal inference, production ML systems, and cognitive AI architecture. He is the creator of Papilon (marketing mix modeling and causal inference) and MeaningFlow (semantic content modeling), and the author of Memory-Node Encapsulation (MNE).
Connect: Medium · LinkedIn · vector1.ai · brian@vector1.ai
