Demand-Synchronized Media: The Optimal Execution Theory of Marketing Spend
Your MMM tells you where the next dollar goes. It says almost nothing about when. Borrowing the mathematics traders use to execute large orders against fluctuating liquidity, this piece builds a framework for timing spend against latent demand — weather, seasonality, and all — and puts a dollar value on a weather forecast.
Walk into any media planning meeting and you’ll hear a sophisticated conversation about allocation: which channels, what mix, how the response curves bend. Then watch what happens to the time dimension. The quarterly budget gets divided by thirteen, a seasonality multiplier gets applied from last year’s curve, someone bumps the holiday weeks, and the plan ships. The cross-sectional problem gets stochastic optimization; the temporal problem gets a spreadsheet ritual.
This is backwards, because the temporal structure of demand is the richest exploitable signal most brands have. Demand is not a flat line with holiday bumps. It breathes — with weather fronts, paydays, school calendars, sporting events, category news cycles — and it does so partly predictably, on horizons long enough to act on. A brand that spends into rising demand states and throttles in dead ones, holding total budget constant, is arbitraging its own calendar.
The reason this arbitrage goes uncollected isn’t ignorance — every practitioner knows ice cream sells in heat waves. It’s that the problem has never been properly formalized. When should spend lead the demand peak, and by how much? How does a forecast whose skill decays over ten days change this week’s pacing? What is the option value of holding budget back? These are quantitative questions, and the marketing literature mostly hasn’t asked them since the media-pulsing debates of the 1980s and 90s.
Quantitative finance, meanwhile, solved a structurally identical problem twenty-five years ago. A trader who must execute a large order over a fixed window, against liquidity that fluctuates partly predictably, facing market impact that penalizes spending too fast, is doing exactly what a brand does executing a quarterly budget against fluctuating demand with saturating response curves. The finance solution — Almgren-Chriss optimal execution and its descendants — is a trajectory optimization under a stochastic state process. Nobody has ported it to media.
This article does. The framework has four layers: estimate the latent demand state, condition the response curve on it, optimize the spend trajectory against it, and price the information that feeds it. Reference implementations are inline; the full library (working name demandsync, a Vector1 Research project in the Papilon family) follows in a companion release.
The Mapping: Execution Theory → Media Spend
The Almgren-Chriss setup: liquidate X shares over T periods. Trading too fast incurs market impact (you move the price against yourself); trading too slow incurs timing risk (the price wanders). The optimal policy is a schedule — a trajectory through time — that balances impact against risk, adapting to the liquidity state.
Every object in that problem has a media twin. The order size is the quarterly budget. Market impact — the penalty for trading too fast — is the saturation curve: dollars crowded into one week buy diminishing response. Liquidity — how much the market absorbs cheaply — is the demand state: the same dollar buys more response when demand is hot. Timing risk is demand uncertainty beyond the forecast horizon. And the execution schedule is the pacing plan.
One asymmetry makes the media version richer than the finance version: adstock. Media impact persists — this week’s spend echoes into future weeks with decay λ. Finance has no true equivalent (impact decays, but doesn’t deliver value later). Adstock means spend is an investment in a decaying stock of goodwill — the insight is as old as Nerlove and Arrow (1962) — and it changes the optimal timing in a specific, calculable way: you don’t spend at the demand peak, you spend ahead of it, by roughly the adstock half-life. More on that below, because it’s the most actionable single result in the framework.
Layer 1: Demand Is a Latent State — Estimate It Like One
The first discipline: stop treating demand drivers as a pile of regressors and start treating demand as a hidden state to be filtered. You never observe “demand.” You observe noisy, lagged, partially overlapping symptoms of it — search volume, store traffic, weather, site sessions, category sales indices. The correct machinery is a state-space model:
d_{t+1} = φ·d_t + Γ'z_t + w_t (state evolution)
y_t = H·d_t + v_t (observed signals)where d_t is the latent demand state, z_t are exogenous drivers (weather, calendar, macro), y_t is the vector of observable proxies, and the Kalman filter delivers both a real-time estimate d̂_t and — critically — its uncertainty. The uncertainty is not a nuisance; it’s an input to the optimizer in Layer 3.
Weather deserves its special seat at this table, for a reason that’s underappreciated: weather is the only major demand driver with a genuinely skillful public forecast. Modern numerical weather prediction delivers real predictive skill out to roughly 7–10 days — the “quiet revolution” documented by Bauer, Thorpe & Brunet (2015) — and useful probabilistic guidance beyond that. Search trends nowcast; macro data lags; but weather forecasts. That means weather enters the model twice: as a contemporaneous state driver (today’s heat moves today’s demand) and as a forward signal (the 8-day forecast moves the optimal spend plan today). A driver you can see coming is a fundamentally different asset than one you can only react to.
A minimal working nowcaster, using nothing beyond NumPy:
import numpy as np
def kalman_demand_filter(y, z, phi=0.9, gamma=None, q=0.05, r=None):
"""Filter a latent demand state from proxy signals and drivers.
y : (T, m) observed proxies (search index, traffic, category sales)
z : (T, k) exogenous drivers (temp anomaly, precip, holiday flags)
Returns d_hat (T,), d_var (T,) - the demand state and its variance.
"""
T, m = y.shape
k = z.shape[1]
gamma = np.zeros(k) if gamma is None else gamma
H = np.ones(m) # each proxy loads on demand
R = np.eye(m) * (0.1 if r is None else r)
d_hat = np.zeros(T); d_var = np.zeros(T)
d, P = 0.0, 1.0
for t in range(T):
# predict
d = phi * d + gamma @ z[t]
P = phi**2 * P + q
# update
S = H * P @ H + np.diag(R) if m > 1 else P + R[0,0]
K = P * H / (P + np.diag(R)) # per-signal gains
innov = y[t] - H * d
d = d + (K * innov).sum() / max(m, 1)
P = (1 - K.mean()) * P
d_hat[t], d_var[t] = d, P
return d_hat, d_var(The production version uses a proper multivariate update, time-varying observation noise, and a particle filter when the demand process is regime-switching — but the skeleton is this.) The output is a demand curve with error bars, updated weekly or daily, that every downstream layer consumes.
Implementation: the filtering stack. Don’t hand-roll the production filter. The workhorse is statsmodels.tsa.statespace — subclass MLEModel for the custom demand specification (or start from UnobservedComponents with a local-linear trend + regression effects, which covers 80% of cases), and you get maximum-likelihood estimation of φ, Γ, and the noise variances via the Kalman filter's prediction-error decomposition for free, with EM available for poor starting values. Two operational details that separate a working system from a demo: use the filtered state d̂_{t|t} for real-time decisioning but the Rauch-Tung-Striebel smoothed state d̂_{t|T} when estimating Layer 2's elasticities on history — the smoother uses all data and cuts attenuation bias from state-measurement error; and estimate observation noise per-proxy rather than pooled, because search indices are far cleaner signals than store traffic. When demand is genuinely regime-switching (fashion, entertainment) or the proxies are heavy-tailed, upgrade to a particle filter — the particles library (Chopin & Papaspiliopoulos's companion package) gives you bootstrap and guided filters with minimal ceremony, and Särkkä's Bayesian Filtering and Smoothing is the reference for choosing between them. A pragmatic middle path is the unscented Kalman filter via filterpy if the only nonlinearity is in the observation map.
Implementation: weather data. Three tiers, all programmatically accessible. Historical actuals: Meteostat (free Python API, station-level) or NOAA GHCN-Daily for ground truth; ERA5 reanalysis via the Copernicus CDS API when you need gridded consistency across markets. Forecasts: Open-Meteo’s free API serves GFS/ECMWF-blend forecasts up to 16 days with hourly resolution and — critically — offers historical forecast archives; for US retail, the herbie package pulls archived GFS/HRRR model output directly from NOAA's AWS buckets. Derived features: don't feed raw temperature — feed anomalies against the market's climatological norm (demand responds to unusual weather, not weather), degree-day transforms (CDD/HDD with category-appropriate base temperatures), and event flags (first hot weekend of the year moves grills; the fourth doesn't). Spatially, aggregate station/grid data to your trading geography (DMA or state) weighted by store footprint or population, not by simple averaging — a heat wave in a market with no distribution is noise.
The vintage discipline. This is the detail that invalidates most backtests in this space: you must store forecasts as of the date they were issued, not just actuals. If your backtest gives the week-10 pacing decision access to the week-10 actual temperature — or even to a forecast issued later than the decision date — you’ve committed lookahead bias and your value-of-information estimate in Layer 4 is fiction. Build the forecast-vintage table from day one: (issue_date, target_date, horizon, variable, value). Open-Meteo's previous-runs endpoint and herbie's archive access exist precisely so you can reconstruct honest vintages for history you didn't capture live.
Layer 2: Your Response Curve Is a Family of Curves
Standard MMM practice fits one saturation curve per channel:
R(s) = β · s^α / (s^α + κ^α) (Hill response)
The framework’s core empirical claim is that this object is misspecified in the time dimension: effectiveness is state-conditional. The same $50K in paid social buys different incremental response when the demand state is +1.5σ (heat wave, event week, payday alignment) than when it’s −1σ. Formally, the curve’s ceiling and half-saturation point become functions of the demand state:
R(s, d) = β(d) · s^α / (s^α + κ(d)^α) β(d) = β₀ · (1 + γ_β · d) (ceiling scales with demand) κ(d) = κ₀ · (1 − γ_κ · d) (saturation arrives later in hot states)
γ_β and γ_κ are the demand elasticities of effectiveness, and they are estimable with machinery your MMM already has: interaction terms between adstocked spend and the filtered demand state, or more flexibly a varying-coefficient model (Hastie & Tibshirani, 1993) letting β and κ move smoothly with d̂_t. Using the filtered state from Layer 1 rather than raw weather variables matters — it pools many weak drivers into one well-measured index and avoids the multiple-testing swamp of interacting spend with every weather variable separately.
This layer is where an old ideological fight becomes an empirical parameter. “Spend into strength” (procyclical: advertise when demand is hot, γ_β > 0 dominates) versus “spend to stimulate” (countercyclical: advertising works hardest when organic demand is absent) is not a philosophy — it’s the sign and size of γ, and it differs by category, channel, and funnel position. Ice cream search ads are violently procyclical. Upper-funnel brand video for considered purchases may be nearly state-neutral, which itself is a finding: state-neutral channels are your ballast, spent in demand troughs when procyclical channels are throttled.
Two subtleties before estimating. First, endogeneity: if your current buying already chases demand (branded search auto-scales, sales-triggered boosts), naive interactions overstate γ — the fix is instrumenting with the exogenous drivers (weather is a gift here: nothing is more exogenous to your ad ops than a cold front) or at minimum controlling for the spend-setting rule. Second, don’t confuse auction state with demand state: hot demand often brings competitor crowding, so the net γ you estimate is demand lift minus CPM inflation — which is exactly the number the optimizer needs, but the decomposition matters for strategy.
Implementation: estimating γ. The estimation stack, in ascending order of sophistication. The baseline is the interaction specification inside your existing MMM — in PyMC or NumPyro, the state-conditional Hill is a ten-line change to a standard Bayesian MMM, and hierarchical pooling across channels is what makes weakly-identified brand-channel elasticities usable:
import pymc as pm
with pm.Model() as state_mmm:
# channel-level demand elasticities, partially pooled
mu_g = pm.Normal("mu_gamma", 0.0, 0.5)
sd_g = pm.HalfNormal("sd_gamma", 0.5)
gamma_b = pm.Normal("gamma_beta", mu_g, sd_g, shape=n_channels)
beta0 = pm.HalfNormal("beta0", 1.0, shape=n_channels)
kappa = pm.HalfNormal("kappa", 1.0, shape=n_channels)
alpha = pm.Gamma("alpha", 3.0, 1.0, shape=n_channels)
# a_ct: adstocked spend; d_t: smoothed demand state (Layer 1)
beta_t = beta0 * (1 + gamma_b * d_t[:, None]) # (T, C)
resp = (beta_t * a_ct**alpha / (a_ct**alpha + kappa**alpha)).sum(axis=1)
sigma = pm.HalfNormal("sigma", 1.0)
pm.Normal("y", resp + controls, sigma, observed=revenue)The priors on γ centered at zero are doing real work: the model must be convinced by data that a channel is demand-elastic, which is the correct default. NumPyro on JAX is the drop-in when T×C gets large — same model, an order of magnitude faster NUTS. The flexible alternative when you don’t want to commit to the linear-in-d parameterization is a varying-coefficient GAM — mgcv in R (s(d, by = adstock)) remains the gold standard; pyGAM covers the basic case in Python — which lets β(d) be an arbitrary smooth and will show you if the elasticity itself saturates. The causal upgrade is double machine learning for heterogeneous effects: econml's CausalForestDML or LinearDML with spend as treatment, response as outcome, and the demand state in the effect-modifier slot X estimates CATE(d) = ∂R/∂s | d directly under weaker functional-form assumptions (Chernozhukov et al., 2018) — cross-fitted, honest confidence intervals included, and the shape of CATE against d is a nonparametric read on γ that will embarrass your Hill parameterization if it's wrong. For the endogeneity correction specifically, the control-function approach is the practical one: first stage regresses spend on the weather instruments and calendar (via linearmodels.IV2SLS or manually), and the first-stage residual enters the response model as a covariate — significant residual coefficient is itself the diagnostic that your historical pacing was demand-chasing.
Validation of γ deserves its own experiment, not just observational fit: a weather-stratified geo test. Pick matched geo pairs, hold spend flat in controls while treatment geos run amplified pacing, and — the key design element — schedule the analysis windows around forecast weather events so the test has power precisely where the elasticity lives. GeoLift-style synthetic control analysis (or Papilon’s DiD/synthetic-control methods) on the readout. One clean geo test on your highest-γ channel converts the whole program from “modeling exercise” to “measured effect” in one quarter.
Layer 3: The Trajectory — Spend as Optimal Execution
Now the centerpiece. Given: a budget B for the quarter, the filtered demand state and its forecast distribution, state-conditional response curves, and adstock carryover λ. Find: the spend path s₁…s_T maximizing expected response.
max E[ Σ_t R(a_t, d_t) ]
s.t. a_t = s_t + λ·a_{t−1} (adstock stock)
Σ_t s_t ≤ B, s_t ≥ 0This is a stochastic dynamic program with a three-dimensional state — adstock level a, demand state d, remaining budget b — and a clean Bellman recursion:
V_t(a, d, b) = max_{0 ≤ s ≤ b} R(s + λa, d) + E_{d'|d}[ V_{t+1}(λa + s·λ⁰, d', b − s) ]Backward induction over a discretized grid solves it exactly at weekly resolution — the state space is small enough that this runs in seconds:
def optimal_trajectory(T, budget, demand_forecast, demand_var,
beta0=1.0, g_beta=0.5, kappa0=50.0, g_kappa=0.3,
alpha=1.6, lam=0.6, n_grid=41):
"""Backward-induction spend trajectory against a demand forecast.
demand_forecast : (T,) mean path of the demand state (from Layer 1)
demand_var : (T,) forecast variance (skill decay -> rising var)
Returns spend path (T,).
"""
import numpy as np
b_grid = np.linspace(0, budget, n_grid) # remaining budget
a_grid = np.linspace(0, budget * 0.6, n_grid) # adstock stock
def hill(s, d):
beta = beta0 * (1 + g_beta * d)
kappa = kappa0 * max(1 - g_kappa * d, 0.2)
return beta * s**alpha / (s**alpha + kappa**alpha + 1e-9)
# certainty-equivalent demand path with a risk haircut on far weeks
d_ce = demand_forecast - 0.5 * g_beta * demand_var
V = np.zeros((n_grid, n_grid)) # V[a, b] at t+1
policy = np.zeros((T, n_grid, n_grid))
for t in range(T - 1, -1, -1):
V_new = np.zeros_like(V)
for i, a in enumerate(a_grid):
for j, b in enumerate(b_grid):
s_opts = np.linspace(0, b, 21)
best, s_best = -1e18, 0.0
for s in s_opts:
reward = hill(s + lam * a, d_ce[t])
a_next = np.clip(lam * (a + s), a_grid[0], a_grid[-1])
b_next = b - s
ia = np.searchsorted(a_grid, a_next)
ib = np.searchsorted(b_grid, b_next)
ia, ib = min(ia, n_grid-1), min(ib, n_grid-1)
val = reward + V[ia, ib]
if val > best:
best, s_best = val, s
V_new[i, j] = best
policy[t, i, j] = s_best
V = V_new
# roll forward
path, a, b = [], 0.0, budget
for t in range(T):
ia = min(np.searchsorted(a_grid, a), n_grid-1)
ib = min(np.searchsorted(b_grid, b), n_grid-1)
s = float(policy[t, ia, ib])
path.append(s); a = lam * (a + s); b -= s
return np.array(path)Run this against any demand forecast with a bump in it and the solution exhibits the framework’s signature result: optimal spend leads the demand peak. The intuition is pure adstock arithmetic — response at the peak is driven by the adstock stock, which takes time to build; the stock peaks one carryover timescale after spend peaks. The back-of-envelope version every planner should carry:
lead time ≈ adstock half-life = ln(0.5) / ln(λ)
For λ = 0.6 weekly, that’s about 1.4 weeks; for λ = 0.8, over three weeks. Most seasonal plans I’ve audited spend at the peak — front-loading exactly zero — which means their goodwill stock crests as demand is already receding. The DP recovers the lead automatically, and it also produces the less obvious behaviors: holding a budget reserve as option value when forecast variance is high, deeper throttling in troughs when γ_β is large, and earlier, flatter ramps when saturation (low κ, steep α) punishes concentration.
In production you don’t solve this once; you wrap it in model predictive control: re-solve the DP every week as the Kalman filter updates the demand state and the weather forecast rolls forward, execute only the first week of the plan, repeat. MPC is what turns a plan into a policy — the plan is always wrong by week three; the policy of re-planning is what carries the value. This is also the natural coupling point to the previous article in this series: the RegimeMonitor's alarm is a state variable this controller should condition on, because trajectory optimization within a regime is exactly the layer that must yield when the regime itself is destabilizing.
Implementation: making the optimizer production-grade. The reference DP above makes three simplifications you should undo in order. First, replace nearest-grid lookup with linear interpolation of the value function (scipy.interpolate.RegularGridInterpolator over the (a, b) grid) — nearest-neighbor value lookups create policy chatter at grid boundaries that shows up as spend oscillation; interpolated value iteration kills it and lets you cut the grid to 25 points per dimension. Second, vectorize: the inner loops collapse to array ops over the full (a × b × s) tensor, and in JAX with vmap + jit a 13-week, 3-state DP re-solves in tens of milliseconds — fast enough to solve per posterior draw of the Layer 2 parameters, which is how you propagate estimation uncertainty into the plan rather than pretending γ is known. Third, when you scale to multi-channel trajectories the DP's state space explodes (one adstock state per channel), and you have three escape routes, in order of preference: (a) convex reformulation — if you operate on the concave region of each Hill curve (usually true above minimum viable spend levels), the whole multi-channel trajectory problem is a concave maximization under linear constraints, and cvxpy with the MOSEK or Clarabel backend solves 5 channels × 52 weeks exactly, in seconds, with the demand path entering as coefficients; (b) fitted value iteration — approximate V with a small MLP or gradient-boosted regressor over sampled states, standard approximate-DP practice (Bertsekas covers the convergence caveats); (c) scenario-based stochastic programming — sample K demand paths from the Layer 1 filter's predictive distribution (which correctly widens with forecast horizon), solve the deterministic problem per scenario with non-anticipativity constraints on the first weeks, and take the here-and-now decision; this is the rigorous replacement for the certainty-equivalent haircut in the reference code.
Why not reinforcement learning? Worth addressing because it’s the first question every ML team asks. You can pose this as an episodic RL problem — gymnasium environment, adstock and budget in the observation, PPO from stable-baselines3 — and it will work. But RL is the wrong default here: the transition dynamics are known (adstock arithmetic and a fitted demand process), the horizon is short, and the state is low-dimensional, which is precisely the regime where model-based DP/MPC dominates model-free RL on sample efficiency, auditability, and constraint handling. The honest case for RL arrives later, when you add competitor response or auction feedback loops that you can simulate but not write in closed form. Until then, PPO is a benchmark to beat in the appendix, not the production controller.
MPC operations. The cadence details that matter in practice: re-solve weekly (daily for large DR-heavy accounts) on a rolling 13-week horizon with the terminal value approximated by the infinite-horizon average — a naked finite horizon teaches the optimizer to spend down adstock in week 13, a classic end-effect artifact. Warm-start each re-solve from last week’s trajectory. Impose ramp constraints |s_t − s_{t−1}| ≤ Δ directly in the optimizer (trivial in the cvxpy formulation) rather than post-hoc clipping, because platforms’ learning phases and creative ops both punish step changes. And log every plan with its input vintage — plan-versus-realized becomes your controller’s own diagnostic time series.
Layer 4: What Is a Forecast Worth in Dollars?
The framework’s last layer answers the question that gets this program funded: the value of information. Run three trajectories through the same simulated demand realizations:
- Flat pacing — budget ÷ T. The null strategy, and closer to common practice than anyone admits.
- Forecast-informed — the MPC policy using forecasts with realistic skill decay.
- Perfect foresight — the DP against the realized demand path. The unreachable ceiling.
The spread between (1) and (3) is the total value locked in demand timing. The fraction of that spread captured by (2) is the value of your actual forecast — and its complement is the residual worth of better information. This decomposition produces the two numbers executives actually need: “demand-synchronized pacing is worth X% of quarterly response at current forecast skill,” and “a marginal improvement in forecast skill (buying premium weather data, building a better nowcaster) is worth $Y.” In categories with strong weather elasticity — beverages, HVAC, apparel, QSR, lawn & garden, allergy OTC — γ_β is large and the forecast horizon covers a meaningful share of demand variance, so X lands in visibly material territory. In weather-inert categories the same arithmetic honestly returns “don’t bother,” which is equally valuable to know before someone buys a data subscription.
The finance intuition transfers cleanly: forecast skill is alpha decay. Weather information is worth the most in exactly the window where its skill exceeds the market’s default assumption of climatology — days 2 through 8 — and the optimizer should be greediest with information precisely there.
Backtesting Without Lying to Yourself
The Layer 4 numbers are only as honest as the backtest that produces them, and this class of backtest has three standard ways to flatter itself. The protocol that avoids them:
Walk-forward everything. At each simulated decision date t, the pipeline may touch only: proxy data through t, forecast vintages issued on or before t, and Layer 1/Layer 2 parameters estimated on data through t − 1 (refit quarterly in the loop, exactly as production would). Expanding-window refits, never a single full-history fit — a γ estimated on the whole history has quietly seen the future’s weather-demand covariance.
Score against the response model’s uncertainty, not its point estimate. The counterfactual response of the optimized trajectory is computed by the same Layer 2 model that drove the optimization — an incestuous loop that inflates the reported lift. Break it two ways: evaluate each strategy under posterior draws of the response parameters (report the lift distribution, and note how much of its mass sits above zero), and where possible anchor with the geo-test result as an external calibration point.
Report the placebo. Run the identical MPC machinery with the demand forecast replaced by climatology (day-of-year averages). The gap between forecast-driven and climatology-driven pacing isolates the value of dynamic information from the value of merely knowing your seasonality — which most organizations conflate, and which flatters the fancy system. If your MPC only beats flat pacing but not climatology pacing, you’ve built an expensive seasonal curve.
Production architecture. The weekly loop, in the stack most retail data teams already run: raw proxies and weather land in Snowflake (Meteostat/Open-Meteo pulls via a scheduled ingestion job; forecast vintages appended, never overwritten); dbt models build the feature tables — demand proxies, degree-days, anomaly transforms, the vintage-correct forecast view; an Airflow (or Dagster) DAG runs the Kalman update, re-solves the MPC, and writes the recommended trajectory plus diagnostics back to a decisions table; Snowpark keeps the Python close to the data if you’d rather not ship it out. The decisions table is append-only with full input lineage — every recommendation reproducible from its vintage. Downstream, the trajectory lands wherever humans approve it (a Streamlit page suffices) before platform APIs execute, because a controller that moves money should spend its first two quarters in recommendation mode, building the trust ledger of plan-versus-outcome.
The stack at a glance:
Layer Job Primary tools Escalation path 1. Demand state Filter latent demand + uncertainty statsmodels state space (MLEModel/UnobservedComponents) particles (regime-switching), filterpy UKF 1. Weather Actuals, forecasts, vintages Open-Meteo, Meteostat, NOAA GHCN ERA5/CDS, herbie for archived GFS/HRRR 2. Elasticities Estimate γ per channel PyMC hierarchical MMM interaction NumPyro (scale), mgcv/pyGAM (shape), econml DML (causal), linearmodels IV (endogeneity) 3. Trajectory Optimal spend path Exact DP + RegularGridInterpolator, JAX cvxpy (multi-channel concave), fitted VI, scenario stochastic program 3. Control Weekly re-planning MPC loop, ramp constraints, 13-wk rolling horizon RL (stable-baselines3) only once feedback loops enter 4. Valuation Price the forecast Walk-forward backtest on vintages Geo test for external calibration
The Playbook
Getting from here to production, in order of dependency: instrument the demand state first (the Kalman layer needs a few quarters of proxies and drivers — start logging now); estimate γ per channel from history you already have, with weather as the instrument; shadow-run the MPC against your actual pacing for a quarter and score the counterfactual before moving a dollar; then flip channels to synchronized pacing in order of their γ magnitude.
And the honest limitations, stated plainly. The DP’s certainty-equivalent treatment of demand risk is a simplification — full stochastic DP with the filtered variance is the rigorous version, at real computational cost. Competitive reaction is unmodeled: if the whole category learns to surf the same weather, the hot-state CPM inflation eats part of γ (the equilibrium version of this problem is a genuinely open research question — mean-field games for media markets, for anyone hunting a dissertation). Creative and operational latency put a floor on how fast pacing can actually move; the controller must carry ramp constraints. And short-horizon response data will always estimate γ more confidently for lower-funnel channels — treat brand-channel elasticities with the skepticism they deserve.
None of these dents the core claim: the time dimension of media spend is an optimization surface, it has structure you can forecast, and the mathematics for exploiting it has been sitting in a neighboring field for two decades, fully solved, waiting for someone to carry it across.
The cross-sectional question — where does the dollar go — had its revolution. The temporal question — when — is next.
Brian Curry is the founder of Vector1 Research, an independent applied AI lab publishing open-source frameworks in causal inference, marketing economics, and cognitive AI architecture. The companion library, demandsync, joins Papilon and mediaphase in the Vector1 open-source family.
References
Optimal execution and trajectory optimization
- Almgren, R., & Chriss, N. (2001). Optimal execution of portfolio transactions. Journal of Risk, 3(2), 5–39.
- Bertsekas, D. P. (2017). Dynamic Programming and Optimal Control (4th ed.). Athena Scientific.
- Rawlings, J. B., Mayne, D. Q., & Diehl, M. (2017). Model Predictive Control: Theory, Computation, and Design. Nob Hill.
Advertising dynamics and timing
- Nerlove, M., & Arrow, K. J. (1962). Optimal advertising policy under dynamic conditions. Economica, 29(114), 129–142.
- Sethi, S. P. (1977). Dynamic optimal control models in advertising: A survey. SIAM Review, 19(4), 685–725.
- Feinberg, F. M. (2001). On continuous-time optimal advertising under S-shaped response. Management Science, 47(11), 1476–1487.
- Naik, P. A., Mantrala, M. K., & Sawyer, A. G. (1998). Planning media schedules in the presence of dynamic advertising quality. Marketing Science, 17(3), 214–235.
State-space methods and varying coefficients
- Durbin, J., & Koopman, S. J. (2012). Time Series Analysis by State Space Methods (2nd ed.). Oxford University Press.
- Särkkä, S. (2013). Bayesian Filtering and Smoothing. Cambridge University Press.
- Chopin, N., & Papaspiliopoulos, O. (2020). An Introduction to Sequential Monte Carlo. Springer.
- Hastie, T., & Tibshirani, R. (1993). Varying-coefficient models. Journal of the Royal Statistical Society B, 55(4), 757–796.
Causal estimation
- Chernozhukov, V., Chetverikov, D., Demirer, M., et al. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68.
Convex and approximate optimization
- Boyd, S., & Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.
Weather forecasting and value of information
- Bauer, P., Thorpe, A., & Brunet, G. (2015). The quiet revolution of numerical weather prediction. Nature, 525, 47–55.
- Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22–26.
MMM carryover and saturation
- Jin, Y., Wang, Y., Sun, Y., Chan, D., & Koehler, J. (2017). Bayesian methods for media mix modeling with carryover and shape effects. Google Research.
Software
- Curry, B. (2026). mediaphase: Phase-transition detection for media portfolios. github.com/Bodhi8/mediaphase
- Curry, B. (2025). Papilon: Complex systems and causal discovery for marketing. github.com/Bodhi8/papilon
Two ecosystems are converging on the same problem from opposite ends. On one side sits the open tooling stack for knowledge representation and retrieval: personal knowledge graphs like Obsidian, property-graph and RDF databases, vector stores, and the hybrid retrieval architectures that stitch them together for LLM applications. On the other side sits Palantir’s Foundry and AIP: a vertically integrated enterprise platform whose Ontology, Pipeline Builder, AIP Logic, and Workshop implement — in a governed, opinionated form — essentially the same architecture that teams otherwise assemble from open components.
Understanding both is valuable because they illuminate each other. The open stack teaches you what is actually happening under the hood: how graphs are stored and traversed, how approximate nearest-neighbor indexes trade recall for latency, why lexical and dense retrieval fail in complementary ways. The Palantir stack teaches you what the assembled system has to look like to survive contact with an enterprise: typed schemas, permission-aware retrieval, human-in-the-loop writeback, lineage, and evaluation. This article works through both stacks in technical depth and then maps one onto the other.
Part I: The Open Knowledge and Retrieval Stack
1. Obsidian: the emergent, file-first knowledge graph
Obsidian is often dismissed as a note-taking app, but architecturally it is a local-first, schema-optional knowledge graph whose storage layer is nothing more than a directory of Markdown files. That design choice has deep consequences.
The data model. Every note is a node. Edges are created implicitly by [[wikilinks]] in the note body, giving you a directed, unlabeled, untyped graph. Obsidian maintains a link index over the vault, which is what powers backlinks (the reverse-adjacency view), the local graph (k-hop neighborhood of the current note), and the global graph view. Because links are just text in files, the graph is fully reconstructible from the vault — there is no proprietary database to corrupt or migrate. The trade-off is that edges carry no semantics: [[Alice]] → [[Acme Corp]] cannot natively distinguish works at from founded from sued.
Typed attributes via frontmatter. YAML frontmatter (surfaced in the UI as Properties) bolts a property model onto nodes:
--- type: person employer: "[[Acme Corp]]" role: CTO start_date: 2021-03-01 tags: [contact, engineering] ---
Links inside frontmatter values are indexed as graph edges too, which gives you a poor man’s typed edge: the property key (employer) acts as the edge label. This is exactly the pattern that community query plugins exploit.
Query layers. The Dataview plugin (and its successor-generation tools like Datacore, plus the now-native Bases feature) turns the vault into a queryable database. Dataview parses frontmatter and inline fields into an in-memory index and exposes a SQL-ish DQL plus a JavaScript API:
TABLE role, start_date FROM #contact WHERE employer = [[Acme Corp]] SORT start_date ASC
This is a property-graph query in disguise: filter nodes by label (#contact), traverse a typed edge (employer), project properties. What you cannot do is multi-hop traversal with path semantics ("people who worked at companies that Acme acquired") — Dataview has no recursive traversal, no shortest-path, no pattern matching across edges. The moment your questions become genuinely graph-shaped, you have outgrown the tool.
Retrieval and embeddings. A second plugin ecosystem (Smart Connections, Copilot for Obsidian, various local-RAG plugins) adds dense retrieval: notes are chunked, embedded (locally via small models or through API embeddings), stored in a local vector index, and used either for “related notes” surfacing or as RAG context for a chat interface over the vault. This makes an Obsidian vault a surprisingly complete microcosm of the full stack discussed below: Markdown files as the document store, the link index as the graph, a plugin-managed HNSW or flat index as the vector store, and full-text search as the lexical channel.
When to graduate. Obsidian’s graph is emergent — structure accretes from writing, not from schema design. That is its superpower for personal knowledge work and its ceiling for systems work. When you need typed edges with properties, concurrent multi-user writes, transactional guarantees, or traversal queries, you export (the vault is just files; parsing wikilinks and frontmatter into nodes/edges is a hundred lines of Python) and move to a real graph database.
Implementation guide: vault → graph export, and local RAG over a vault
The export script below turns a vault into nodes.csv and edges.csv suitable for bulk-loading into Neo4j (used again in §2). Frontmatter keys whose values are wikilinks become typed edges; body wikilinks become generic LINKS_TO edges.
import csv, re, pathlib, yaml
VAULT = pathlib.Path("~/vault").expanduser()
WIKILINK = re.compile(r"\[\[([^\]\|#]+)")
FM = re.compile(r"^---\n(.*?)\n---\n", re.S)nodes, edges = {}, []
for f in VAULT.rglob("*.md"):
name = f.stem
text = f.read_text(encoding="utf-8")
meta = {}
if (m := FM.match(text)):
try:
meta = yaml.safe_load(m.group(1)) or {}
except yaml.YAMLError:
pass
text = text[m.end():]
nodes[name] = {"name": name, "type": meta.get("type", "note"),
"tags": ";".join(meta.get("tags", []) or [])}
for key, val in meta.items(): # typed edges from frontmatter
for v in (val if isinstance(val, list) else [val]):
if isinstance(v, str) and (lm := WIKILINK.search(v)):
edges.append((name, lm.group(1).strip(), key.upper()))
for lm in WIKILINK.finditer(text): # untyped body links
edges.append((name, lm.group(1).strip(), "LINKS_TO"))with open("nodes.csv", "w", newline="") as fh:
w = csv.DictWriter(fh, ["name", "type", "tags"]); w.writeheader()
w.writerows(nodes.values())
with open("edges.csv", "w", newline="") as fh:
csv.writer(fh).writerows([("src", "dst", "rel"), *edges])For retrieval over the vault without leaving it, the plugin route is Smart Connections or Copilot for Obsidian (point them at a local Ollama embedding model such as nomic-embed-text if the vault shouldn't leave the machine). The script route — more controllable, and reusable as the indexing half of §4 — chunks by heading and embeds into a local Chroma store:
import chromadb, re
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
col = chromadb.PersistentClient("./vault_index").get_or_create_collection(
"vault", metadata={"hnsw:space": "cosine"})for f in VAULT.rglob("*.md"):
body = f.read_text(encoding="utf-8")
for i, sec in enumerate(re.split(r"\n(?=#{1,3} )", body)): # heading-aware chunks
if len(sec.strip()) < 40:
continue
col.upsert(ids=[f"{f.stem}::{i}"],
embeddings=[model.encode(sec).tolist()],
documents=[sec],
metadatas=[{"note": f.stem}])hits = col.query(query_embeddings=[model.encode("zero-trust rollout decisions").tolist()],
n_results=5)Re-run on a schedule or a filesystem watcher; upserting by stable chunk ID makes re-indexing idempotent.
2. Graph databases: property graphs, RDF, and when traversal beats joins
Graph databases exist because certain query shapes are pathological in relational systems. A relational join resolves relationships at query time via index lookups or hash joins; cost grows with the size of the tables being joined. Native graph engines like Neo4j use index-free adjacency: each node physically stores pointers to its adjacent relationships, so traversing an edge is a pointer dereference — O(1) per hop regardless of total graph size. A five-hop traversal over a billion-edge graph touches only the neighborhoods along the path. The same query as a five-way self-join in SQL can be catastrophic.
The property graph model. Nodes and relationships both carry labels and arbitrary key-value properties. Relationships are first-class, directed, and typed:
// Cypher: who at Acme worked with people who later joined competitors?
MATCH (a:Person)-[:WORKS_AT]->(:Company {name: 'Acme'}),
(a)-[:COLLABORATED_WITH]-(b:Person)-[:WORKS_AT]->(c:Company)
WHERE c.industry = 'Enterprise Software' AND c.name <> 'Acme'
RETURN b.name, c.name, count(*) AS sharedProjects
ORDER BY sharedProjects DESCCypher’s pattern-matching syntax (ASCII-art arrows for edges) was the basis for ISO GQL, standardized in 2024 as the first new ISO database query language since SQL — a sign the model has matured. Implementations span Neo4j, Memgraph (in-memory, streaming-oriented), Amazon Neptune, ArangoDB and TigerGraph, with Kùzu and FalkorDB serving the embedded/lightweight niche that pairs well with LLM experiments.
The RDF alternative. The W3C stack — RDF triples (subject–predicate–object), SPARQL for querying, OWL for ontology semantics, SHACL for validation — optimizes for different goals: global identifiers (IRIs), federation across datasets, and formal reasoning (an OWL reasoner can infer triples: if worksAt is transitive over subsidiaryOf, inferred employment relationships materialize automatically). RDF shines in data-integration and interoperability contexts (life sciences, cultural heritage, linked open data) and tends to lose to property graphs on developer ergonomics and raw traversal performance. Notably, "ontology" in this stack means a formal, machine-interpretable schema with inference semantics — which is worth holding in mind when we reach Palantir's much looser use of the same word.
Graph algorithms as a retrieval primitive. Beyond queries, graph libraries (Neo4j GDS, NetworkX at small scale, cuGraph on GPU) provide algorithms that matter directly for knowledge retrieval: PageRank and betweenness centrality for importance weighting, Louvain/Leiden community detection for clustering entities into topics (the backbone of GraphRAG’s “global” mode, below), node2vec/GraphSAGE for structural embeddings, and shortest-path/k-hop expansion for assembling context around a seed entity.
Vectors inside the graph. The major graph engines now ship native vector indexes (Neo4j added HNSW-backed vector indexes; Neptune, Memgraph, and others followed). This matters because it collapses a two-system architecture into one: you can vector-search to find entry-point nodes, then traverse typed edges to expand context — the canonical graph-RAG motion — inside a single query.
Implementation guide: Neo4j from zero to graph-plus-vector retrieval
Stand up Neo4j with the Graph Data Science plugin and load the CSVs exported in §1 (any node/edge CSVs work the same way):
docker run -d --name neo4j -p 7474:7474 -p 7687:7687 \
-e NEO4J_AUTH=neo4j/localdevpass \
-e NEO4J_PLUGINS='["graph-data-science"]' \
-v $PWD/import:/import neo4j:5
CREATE CONSTRAINT note_name IF NOT EXISTS
FOR (n:Note) REQUIRE n.name IS UNIQUE;
LOAD CSV WITH HEADERS FROM 'file:///import/nodes.csv' AS row
MERGE (n:Note {name: row.name})
SET n.type = row.type, n.tags = split(row.tags, ';');LOAD CSV WITH HEADERS FROM 'file:///import/edges.csv' AS row
MATCH (a:Note {name: row.src})
MERGE (b:Note {name: row.dst}) // create stubs for unresolved links
WITH a, b, row
CALL apoc.merge.relationship(a, row.rel, {}, {}, b) YIELD rel
RETURN count(rel);Attach embeddings (compute them with the same sentence-transformers model as §1, write via the Python driver as a n.embedding list property), then create the vector index and run the graph-RAG motion — vector entry, typed expansion — as one query:
CREATE VECTOR INDEX note_vec IF NOT EXISTS
FOR (n:Note) ON (n.embedding)
OPTIONS {indexConfig: {`vector.dimensions`: 384,
`vector.similarity_function`: 'cosine'}};
// $q is the query embedding passed as a parameter
CALL db.index.vector.queryNodes('note_vec', 5, $q) YIELD node, score
MATCH (node)-[r]-(nbr:Note)
RETURN node.name AS entry, score, type(r) AS rel, nbr.name AS neighbor
ORDER BY score DESC;Community detection for topic clustering (the Leiden step GraphRAG relies on) is two GDS calls:
CALL gds.graph.project('kg', 'Note', '*');
CALL gds.leiden.write('kg', {writeProperty: 'community'})
YIELD communityCount;From Python, neo4j (official driver) or neomodel (OGM) cover application access; for LLM tool use, expose parameterized Cypher templates as tools rather than letting a model write free-form Cypher — the same closed-mutation principle §6 formalizes. If Docker is overkill, Kùzu gives you an embedded, in-process property graph with Cypher and vector indexes (pip install kuzu), the SQLite of graph databases.
3. Vector stores: ANN indexes and the engineering beneath “semantic search”
A vector store solves one problem: given a query embedding q ∈ ℝᵈ (d typically 384–3072), return the k stored vectors closest under cosine/dot-product/L2 — fast, at scale, with metadata filtering. Exact search is a linear scan: fine to a few hundred thousand vectors (NumPy or FAISS flat indexes are genuinely underrated), untenable beyond. Everything else is approximate nearest neighbor (ANN), which trades a controllable amount of recall for orders of magnitude of latency.
HNSW (Hierarchical Navigable Small World) is the workhorse. It builds a multi-layer proximity graph: sparse upper layers for coarse routing, dense bottom layer for precision. Search greedily descends layers, maintaining a candidate beam of size efSearch. Key parameters: M (edges per node — higher M, better recall, more memory), efConstruction (build-time beam — build quality), efSearch (query-time beam — the recall/latency dial you turn in production). HNSW delivers >95–99% recall at millisecond latency but keeps the graph and vectors in RAM, which is the cost driver.
IVF + product quantization is the memory-constrained alternative: cluster vectors into nlist cells (k-means), search only the nprobe nearest cells, and compress vectors via PQ (split into subvectors, quantize each against a small codebook) so a 1536-d float32 vector (6 KB) shrinks to 32–64 bytes. Recall suffers; a re-ranking pass over full-precision vectors for the top candidates buys most of it back. DiskANN-family designs (Vamana graphs with SSD-resident vectors) and scalar/binary quantization round out the design space when the corpus outgrows RAM.
The filtering problem is the sharpest practical edge. Real queries are “nearest neighbors where tenant_id = X and date > Y.” Post-filtering (ANN first, filter after) silently under-returns when the filter is selective; pre-filtering degenerates to a scan; the good engines (Qdrant, Weaviate, Vespa, pgvector with iterative scans) implement filter-aware traversal that applies predicates during graph search. When you evaluate a vector database, filtered-search recall is the benchmark to run — unfiltered ANN numbers are marketing.
The landscape splits into dedicated engines (Pinecone as managed serverless; Qdrant, Weaviate, Milvus as open-source servers; Chroma and LanceDB embedded for prototyping) and vector capabilities inside databases you already run — pgvector in Postgres, Elasticsearch/OpenSearch, Redis, MongoDB Atlas. The boring advice is usually right: pgvector with HNSW handles tens of millions of vectors with transactional metadata, joins, and one fewer system to operate; dedicated engines earn their place at hundreds of millions of vectors, high QPS, or when their filtering/multitenancy machinery is decisive. Two operational facts get discovered late in every project: embeddings are coupled to a specific model version, so re-embedding the whole corpus on model change must be a designed-for pipeline, not an emergency; and chunk-level vectors need bidirectional links back to source documents and their ACLs, or you will build permission leaks into retrieval.
Implementation guide: production-shaped pgvector, with a Qdrant alternative
The Postgres route keeps vectors next to your relational metadata and ACLs. Schema, HNSW index, and the two tuning knobs that matter:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
doc_id text NOT NULL,
tenant_id text NOT NULL,
section text,
content text NOT NULL,
embedding vector(1024) NOT NULL, -- must match your model's dimension
model_ver text NOT NULL DEFAULT 'bge-m3@1'
);CREATE INDEX chunks_hnsw ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 200);CREATE INDEX chunks_tenant ON chunks (tenant_id);
import psycopg
from pgvector.psycopg import register_vector
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-m3")
conn = psycopg.connect("dbname=rag"); register_vector(conn)def search(query: str, tenant: str, k: int = 20):
q = model.encode(query, normalize_embeddings=True)
with conn.cursor() as cur:
cur.execute("SET hnsw.ef_search = 100") # recall/latency dial
cur.execute("""
SELECT id, doc_id, section, content,
1 - (embedding <=> %s) AS score
FROM chunks
WHERE tenant_id = %s
ORDER BY embedding <=> %s
LIMIT %s""", (q, tenant, q, k))
return cur.fetchall()Three practices worth adopting from day one: batch inserts inside transactions and build the HNSW index after bulk load (order-of-magnitude faster); store model_ver on every row so a model upgrade is WHERE model_ver = old re-embedding, not a guess; and validate filtered recall — run your gold queries with the tenant_id predicate and compare against an exact scan (SET enable_indexscan = off), because selective filters are where HNSW quietly drops relevant results (raise ef_search, or use pgvector's iterative scan mode, if it does).
The same search against Qdrant, whose filter-aware HNSW handles selective predicates natively and which adds server-side quantization when RAM matters:
from qdrant_client import QdrantClient, models
qc = QdrantClient("localhost", port=6333)
qc.create_collection("chunks",
vectors_config=models.VectorParams(size=1024, distance=models.Distance.COSINE),
quantization_config=models.ScalarQuantization(
scalar=models.ScalarQuantizationConfig(type=models.ScalarType.INT8)))hits = qc.query_points("chunks",
query=model.encode("q3 churn drivers", normalize_embeddings=True),
query_filter=models.Filter(must=[
models.FieldCondition(key="tenant_id", match=models.MatchValue(value="acme"))]),
limit=20)4. Hybrid retrieval: why one index is never enough
Dense and lexical retrieval fail in complementary ways, and this is measurable, not folklore. Dense (bi-encoder) retrieval captures paraphrase and semantics but is unreliable for exact identifiers — part numbers, error codes, names, rare terms — which are exactly what BM25 handles perfectly. BM25, conversely, scores zero for a relevant document sharing no query terms. Production systems therefore run both and fuse.
Fusion. The robust default is Reciprocal Rank Fusion: each document scores Σ 1/(k + rankᵢ) across result lists (k ≈ 60). RRF needs no score normalization — critical, because BM25 scores and cosine similarities live on incomparable scales — and is hard to beat without tuning. Weighted score fusion (normalize, then α·dense + (1−α)·sparse) can edge it out when you have evaluation data to fit α. Elasticsearch, OpenSearch, Weaviate, Qdrant, and Vespa all ship hybrid+RRF natively.
The retrieval ladder. A modern pipeline is staged, cheap-to-expensive: (1) first-stage recall — BM25 and/or ANN pulls ~100–1000 candidates; (2) optional learned-sparse (SPLADE — term expansion with learned weights, served on inverted indexes) or late-interaction (ColBERT — per-token embeddings with MaxSim scoring) as stronger first stages; (3) cross-encoder reranking of the top 50–100, where a transformer reads query and passage jointly — the single highest-leverage quality upgrade in most RAG systems, at ~10–100 ms; (4) query-side transforms: rewriting conversational queries into standalone ones, decomposing multi-part questions, HyDE (embed a hypothetical answer instead of the query). Underneath all of it sits chunking — structure-aware splitting (Markdown headings, table integrity), 200–800 tokens with overlap, and metadata (title, section path, doc type) attached to every chunk — which quietly determines the ceiling of the whole system.
Evaluation turns retrieval from vibes into engineering: build a query→relevant-chunks gold set (even 50 queries helps; LLM-as-judge with human spot-checks scales it), track recall@k and nDCG/MRR per pipeline stage, and tune α, efSearch, rerank depth, and chunking against it. Every parameter above is only tunable relative to such a set.
GraphRAG: the stacks converge. Vanilla RAG answers “find passages similar to the query” and fails at questions whose answers are distributed — “what are the main themes across this corpus?”, “how is A connected to C?”. GraphRAG (popularized by Microsoft’s implementation) uses an LLM at index time to extract entities and typed relations from chunks, assembles them into a knowledge graph, runs hierarchical community detection (Leiden), and pre-summarizes each community. Local search embeds the query, lands on entity nodes, traverses the graph to pull neighbors, relations, and their source chunks. Global search map-reduces over community summaries to answer corpus-level questions. The cost is real (an LLM pass over the whole corpus at indexing, plus graph maintenance on updates), so the decision rule is: entity-centric, multi-hop, or thematic question load justifies GraphRAG; needle-in-haystack lookup does not. Architecturally, note what happened: documents, a vector index, a property graph, lexical search, and an LLM orchestration layer all became components of one retrieval system. That composite is exactly what Palantir sells as a product.
Implementation guide: a complete hybrid pipeline with RRF, reranking, and evaluation
This is the whole §4 ladder in one runnable file — BM25 and dense first stages, RRF fusion, cross-encoder rerank, and a recall@k harness. Swap the in-memory pieces for Elasticsearch/pgvector without changing the structure.
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np
chunks = load_chunks() # [{id, content, meta}, ...]
texts = [c["content"] for c in chunks]bm25 = BM25Okapi([t.lower().split() for t in texts])
enc = SentenceTransformer("BAAI/bge-m3")
dense = enc.encode(texts, normalize_embeddings=True) # persist in pgvector in prod
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")def rrf(rank_lists, k=60):
scores = {}
for ranks in rank_lists:
for r, cid in enumerate(ranks):
scores[cid] = scores.get(cid, 0) + 1.0 / (k + r + 1)
return sorted(scores, key=scores.get, reverse=True)def retrieve(query, k_first=100, k_final=8):
lex = np.argsort(-bm25.get_scores(query.lower().split()))[:k_first]
sem = np.argsort(-(dense @ enc.encode(query, normalize_embeddings=True)))[:k_first]
fused = rrf([lex.tolist(), sem.tolist()])[:50]
ce = reranker.predict([(query, texts[i]) for i in fused])
return [chunks[fused[i]] for i in np.argsort(-ce)[:k_final]]# --- evaluation: recall@k against a gold set ---
gold = [{"q": "what triggered the Q3 rollback", "relevant": {"doc7::2", "doc7::3"}}, ...]def recall_at_k(retr_fn, k=8):
hits = [len({c["id"] for c in retr_fn(g["q"])[:k]} & g["relevant"]) / len(g["relevant"])
for g in gold]
return sum(hits) / len(hits)print("hybrid+rerank:", recall_at_k(retrieve))The harness is the point: define retrieve_bm25, retrieve_dense, retrieve_hybrid, and the reranked variant, and print recall@k for each. That four-row table tells you what each stage buys on your corpus and is the only defensible basis for tuning k_first, RRF's k, rerank depth, or chunk size. Expect the typical ordering — dense ≈ BM25 alone, hybrid clearly better, rerank the largest single jump — but verify it; identifier-heavy corpora invert the first comparison.
Implementation guide: GraphRAG in anger
Microsoft’s reference implementation is the fastest path to a working entity-graph index:
pip install graphrag mkdir -p ragproj/input && cp corpus/*.txt ragproj/input/ graphrag init --root ragproj # writes settings.yaml + .env
In settings.yaml, set the chat and embedding models (any OpenAI-compatible endpoint, including a local vLLM/Ollama server), chunk size (~300 tokens with overlap is the usual start), and entity types to extract (organization, person, system, event — constraining these to your domain measurably improves the graph). Then:
graphrag index --root ragproj # LLM pass: entities → graph → Leiden → summaries graphrag query --root ragproj --method global "What are the recurring failure themes?" graphrag query --root ragproj --method local "How is the billing service connected to the outage on May 3?"
Budget the index run before committing: cost scales with corpus tokens × extraction passes, and it re-runs on updates (incremental indexing exists but graph merge quality needs review). For a lighter, more controllable variant, skip the framework: run your own extraction prompt per chunk emitting (head, relation, tail, evidence_chunk_id) JSON, load the triples into the Neo4j instance from §2, reuse its Leiden call for communities, and implement local search as vector entry (§2's note_vec pattern) plus 1–2 hop expansion. You lose the polish, keep the schema, and the graph lives where the rest of your queries do.
Part II: Foundry and AIP — the Same Architecture as a Governed Platform
Palantir Foundry is an end-to-end data operations platform: connect source systems, transform data in versioned pipelines, model it semantically, and build operational applications on top. AIP (Artificial Intelligence Platform, launched 2023) is the LLM layer woven through it. The pieces to understand, in dependency order, are Pipeline Builder (data in), the Ontology (the semantic/graph layer), AIP Logic (LLM functions over the Ontology), and Workshop (applications for end users). Two platform-wide properties condition everything: every dataset and transform is versioned with full lineage, and a mandatory security model (granular, propagating access controls) travels with data everywhere — including into LLM context.
5. Pipeline Builder: the ingestion and transformation layer
Pipeline Builder is Foundry’s visual pipeline environment: a node graph of datasets and transforms compiling down to managed Spark (batch or incremental) or streaming (Flink-based) execution. Functionally it occupies the slot that dbt + Airflow + Spark jobs occupy in an open stack, with a few properties that matter for the retrieval story.
First, pipelines are declarative graphs with typed schemas propagated edge to edge, branch-based development (Global Branching lets you change a pipeline, the Ontology, and downstream apps on one branch and merge atomically — monorepo semantics for a data platform), and lineage recorded automatically. Second, LLM operations are pipeline nodes: a Use LLM node applies a prompted model to every row (classification, extraction, summarization at dataset scale, with schema-constrained outputs), and a Text to embeddings expression embeds string columns into vector columns using platform-provided embedding models. Third, document processing is first-class: AIP Document Intelligence handles OCR/vision-model extraction, and as of 2026 also chunking and embedding in-platform, with a chunker optimized for Markdown structure, bullets, and tables — the platform equivalent of the structure-aware chunking discussed in §4, run as governed pipeline steps whose outputs are versioned datasets.
The significance is architectural: in the open stack, your embedding job is a script whose failure modes (partial re-embeds, model-version drift, chunks orphaned from ACLs) you own. In Foundry the embedding step is a transform inside a lineage graph; re-embedding on model change is re-running a declared pipeline, and the resulting vectors inherit the source data’s security markings because everything downstream of a dataset does.
Implementation guide: a document-to-embeddings pipeline in Pipeline Builder
The following builds the ingestion half of a RAG workflow — raw documents in, chunked-and-embedded dataset out — entirely in the UI.
- Land the source. In Data Connection, configure a sync from the source system (SharePoint, S3, a JDBC source, or direct upload) into a raw dataset. For PDFs/scans, run AIP Document Intelligence first to produce a dataset of extracted text (it emits Markdown-ish structured text, which the chunker exploits).
- Create the pipeline. New Pipeline Builder pipeline (batch; choose incremental if the source appends), add the raw dataset as input. Its schema appears on the node graph.
- Clean and shape. Add transform nodes: filter empty documents, normalize whitespace, derive a stable
chunk_source_id. Every transform previews on sample data before you commit it; the AIP Generate assist can draft transform logic from a prompt, and Explain documents existing nodes. - Chunk. Use the Document Intelligence chunking transform (heading/table-aware) or a text-splitting expression, producing one row per chunk with
document_rid,chunk_index,section_path,content— the metadata §4 says every chunk needs. - Embed. Add the Text to embeddings expression on
content, selecting a platform-provided embedding model; the output is a new vector column. Embedding is the expensive node — expect slower previews, and keep it downstream of all filtering so you only pay for rows you keep. (For row-wise LLM enrichment — classification, entity tagging — add a Use LLM node here with a schema-constrained output.) - Output and schedule. Add an output dataset node (this becomes the Ontology backing dataset in §6), configure the build schedule or trigger-on-input-update, and deploy. Do all of this on a branch; the proposal that merges it shows the full diff, and lineage now records every hop from source system to vector column.
The single most important design choice is made in step 4: chunk identity. Give chunks deterministic IDs derived from document + position so incremental builds update in place rather than duplicating, and so Ontology objects (next section) keyed on them remain stable across rebuilds.
6. The Ontology: an operational knowledge graph with opinions
The Ontology is Foundry’s semantic layer and the platform’s center of gravity. Strip away the branding and it is a governed property graph materialized from pipeline outputs, with an action system attached. Its primitives:
Object types are node schemas — Aircraft, Customer, WorkOrder — with typed properties, a primary key, and a backing dataset (the pipeline output that hydrates instances). Properties can be scalars, structs, media references, time series, and — critically for this article — vector types holding embeddings, plus semantic-search-enabled string properties. Link types are typed, directional edges between object types (Customer — submits→ Ticket), backed by join keys or join tables in the underlying datasets. Interfaces add polymorphism (multiple object types implementing a shared shape). Together these give you the property-graph model of §2, with two deviations: the schema is centrally governed rather than emergent (ontology changes go through proposals and review), and the graph is materialized from pipelines rather than written directly — the graph is a projection of governed data, perpetually rebuildable from lineage.
Action types are the piece with no direct analog in Neo4j or Weaviate: declared, typed, permission-checked mutations (Reassign WorkOrder, Approve Exception) with validation rules, side effects, and audit trails. The Ontology is read-write — “two-way interoperability,” with writeback to source systems — but only through actions. This is the platform’s answer to the question every agentic-RAG builder eventually faces: how does the model do something, safely? By invoking a typed action whose preconditions and permissions are enforced by the platform, with optional human approval, never by emitting raw SQL.
Functions (TypeScript or Python against the Ontology SDK) are server-side logic over objects, and they expose the retrieval machinery. The object search API supports lexical/keyword search, structured filters, link traversal, aggregation, and KNN over vector properties. A semantic-search function is ~15 lines: embed the query with the same platform model used at indexing, call nearestNeighbors on the vector property, order by relevance. Because that runs inside the security model, a user's KNN search can only ever return objects that user can see — filtered ANN (§3's hardest problem) enforced by construction rather than by application code. Hybrid retrieval is composed the same way: keyword search plus KNN plus link expansion in a single function, fused however you like.
The word “ontology” here would make a W3C semanticist wince — there is no OWL, no inference engine, no open-world reasoning. It is a pragmatic ontology: a shared, typed vocabulary of an organization’s entities, relations, and actions, designed so that humans, applications, and LLMs operate on the same objects. The bet is that governed shared vocabulary plus actions delivers more operational value than formal reasoning, and for enterprise workflows that bet has largely paid off.
Implementation guide: modeling the retrieval graph in the Ontology
Continuing from the pipeline output, this creates the object model and the search function the LLM layer will consume.
- Object types. In Ontology Manager, create Document (backing dataset: the document-level pipeline output; primary key
document_rid; title property: filename) and DocumentChunk (backing dataset: the chunk output from §5, step 6; primary key: the deterministic chunk ID). On DocumentChunk, map the embedding column to a vector property, declaring its dimension and similarity function; markcontentas searchable to get lexical indexing too. - Link types. Create Document — has chunk→ DocumentChunk on the
document_ridforeign key. If your pipeline also extracts entities (a Use LLM node emitting entity IDs per chunk), add DocumentChunk —mentions→ Entity — this link is what later turns flat RAG into graph-RAG-style expansion. - An action type. Create Flag Chunk Inaccurate on DocumentChunk: parameters (reason string), a rule setting
review_status = 'flagged', submission criteria restricting who may invoke it. This gives both humans and Logic functions a governed feedback channel. - Propose and merge. Ontology changes ship through a proposal; reviewers see the schema diff alongside any pipeline changes on the same global branch.
The retrieval function is a short TypeScript Function against the generated bindings — embed the query with the same model used in the pipeline, KNN over the vector property, relevance-ordered:
import { Function, Integer } from "@foundry/functions-api";
import { Objects } from "@foundry/ontology-api";
import { TextEmbeddingModel } from "@foundry/models-api/language-models";
export class Retrieval {
@Function()
public async searchChunks(query: string, k: Integer): Promise<DocumentChunk[]> {
const [emb] = (await TextEmbeddingModel.createEmbeddings({ inputs: [query] })).embeddings;
return Objects.search()
.documentChunk()
.nearestNeighbors(c => c.embedding.near(emb, { kValue: k }))
.orderByRelevance()
.take(k);
}
}Hybrid retrieval is the same function grown: run a keyword where search and the KNN in parallel, fuse with the RRF function from §4 (it is ten lines of TypeScript), and optionally expand each hit through the mentions link to pull sibling chunks about the same entities. Publish the function; note that a Workshop user and an AIP Logic call invoking it get results already filtered to their permissions — the filtered-ANN guarantee no open-stack component gives you for free.
7. AIP Logic: LLM functions with the Ontology as tool-space
AIP Logic is a no-code/low-code environment for building, testing, and shipping LLM-powered functions. A Logic function takes typed inputs (strings, objects, object sets), runs a sequence of blocks — the central one being Use LLM, with prompt, model selection from a multi-provider catalog (GPT, Claude, Llama, others), and tools — and returns typed outputs or proposed Ontology edits.
The tool list is where the architecture shows. The LLM can be granted: Ontology query tools (search/filter/traverse specific object types), the semantic search functions from §6, action invocation (create/modify objects through the governed action layer), and arbitrary published Functions. Scoping is explicit and enforced: the model can only see object types it was handed, only within the calling user’s permissions, and only mutate through declared actions. This is the ReAct/tool-calling agent pattern — the same shape as a LangGraph agent wired to retrievers and mutation endpoints — but with tool-space defined by the governed Ontology instead of ad-hoc Python functions, which changes the failure modes materially.
Around the core sit the operational trimmings that distinguish a demo from a system: AIP Evals provides evaluation suites — test cases, exact-match and LLM-judged metrics, variance tracking across runs — the platform analog of the retrieval gold sets in §4, applied to end-to-end functions. Automate triggers Logic functions on conditions (new object arrives, property crosses threshold) and routes proposed Ontology edits into apply-or-stage-for-human-review flows. Logic functions publish as ordinary Functions, so they compose: callable from Workshop apps, other Logic, pipelines, or externally through the Ontology SDK. AIP Chatbot Studio (née Agent Studio) covers the conversational/multi-agent configuration on the same substrate, and AIP Analyst (GA 2026) provides a general conversational interface over the Ontology with traceable derivations.
A representative RAG-over-the-Ontology build, end to end: Pipeline Builder ingests documents → Document Intelligence extracts, chunks, embeds → chunks land as a DocumentChunk object type with a vector property, linked to Document and to referenced entities → a TypeScript semantic-search function wraps KNN → an AIP Logic function receives a user question, calls the search tool plus structured Ontology queries, synthesizes an answer with citations to objects, and optionally proposes an action → Evals scores it against a suite → Automate wires it to triggers. Every arrow in that chain is versioned, branchable, permissioned, and visible in lineage.
Implementation guide: a grounded Q&A function in AIP Logic, with Evals and Automate
- Create the function. New AIP Logic file; define inputs —
question: string, optionally aDocumentobject set to scope retrieval — and output type (a string, or a struct withanswerandcitedChunkIdsfor downstream rendering). - Add the Use LLM block. Pick a model from the catalog (you can A/B models later, and Workflow Lineage can swap the model across many Logic functions at once). Write the system prompt: role, grounding rules (“answer only from tool results; cite chunk IDs; say ‘not found’ otherwise”), output format.
- Attach tools, minimally. Add the published
searchChunksfunction as a tool with a usage instruction ("call with k=8 for every question"); add the DocumentChunk and Document object types as Ontology query tools if the model should also filter/traverse structurally; add the Flag Chunk Inaccurate action only if you want the model proposing flags. Everything not attached is invisible to the model — scope is the security boundary, so start narrow. - Debug interactively. The Logic debugger runs the function on sample inputs showing each block’s inputs/outputs, every tool call, and token usage — fix prompt and tool-instruction issues here, not in production.
- Build the Evals suite. In AIP Evals, create test cases (question → expected answer or expected cited chunks), add evaluators — exact/contains checks for citation IDs, an LLM-judged rubric for answer quality — and run the suite across model or prompt variants. Gate releases on it: a Logic edit that drops citation accuracy should fail review the way a failing unit test does.
- Publish and automate. Publish a version (Logic functions are ordinary Functions — callable from Workshop, OSDK apps, other Logic). In Automate, add a trigger — e.g., on new Document objects, run a summarization Logic function and stage its proposed Ontology edits for human review rather than auto-applying. The apply-vs-stage decision per automation is where you encode your risk tolerance.
Two prompt-engineering specifics that matter more here than in generic RAG: instruct the model to pass the user’s question verbatim as the tool query (models love to “helpfully” rewrite queries into worse ones — if you want rewriting, make it an explicit prior block), and return typed structs rather than prose whenever a downstream widget or automation consumes the output, since typed outputs are what make Evals assertions cheap.
8. Workshop: the application layer
Workshop is the low-code builder for the operational apps end users actually touch: pages composed of widgets (object tables, filters, charts, maps, forms, detail views) bound to object sets — live, declarative queries against the Ontology — with variables and events wiring widgets into interactive flows. Action forms render the Ontology’s action types as validated UI, so the same governed mutation an LLM proposes is the one a human executes by button. AIP surfaces in Workshop through chat and AIP-backed widgets that embed Logic functions or chatbots in context — the pattern where an operator views a WorkOrder object and an embedded function drafts a resolution from linked history is a few hours of configuration, because retrieval, reasoning, and writeback were all already defined one layer down. Object Views (default per-object pages), Slate (freeform HTML/JS apps), and the Ontology SDK (full-code React/Python apps against generated typed bindings) cover the rest of the application spectrum.
Implementation guide: a review-and-ask app in Workshop
Wiring the previous three sections into an end-user application takes roughly this shape:
- Module and data. New Workshop module; define an object-set variable over Document and a dependent one over its linked DocumentChunks.
- Browse layer. Add a Filter List widget (facets on document type, date, review status) driving an Object Table of Document; selection events populate a
selectedDocumentvariable. - Detail layer. Add an Object View or property panel bound to
selectedDocument, with a child table of its chunks via the has chunk link. - Ask layer. Embed an AIP-backed widget bound to the Q&A Logic function, passing
selectedDocument(or the filtered object set) as the scoping input; rendercitedChunkIdsfrom the typed output as links that select the corresponding chunk rows — citations become navigable, which is what makes grounding inspectable by non-engineers. - Act layer. Add a button rendering the Flag Chunk Inaccurate action form on the selected chunk. The same action the Logic function can propose is now a validated human control; both paths land in the audit trail.
- Ship. Preview as a user with narrower permissions than yours (retrieval, chat, and actions all silently scope down — verify the app degrades sensibly), then publish, or package the module + ontology + pipeline as a Marketplace product for redeployment.
For teams that outgrow low-code, the same app in OSDK is npx @osdk/create-app, which scaffolds a React project with generated typed clients for the object types, the search function, and the action — the Ontology remains the API.
Part III: Mapping the Stacks onto Each Other
The correspondence is close enough to state as a table:
Concern Open stack Foundry/AIP Ingestion & transforms dbt, Airflow, Spark, custom ETL Pipeline Builder / Transforms Document processing Unstructured.io, custom chunkers AIP Document Intelligence Embeddings OpenAI/local models + scripts Text-to-embeddings nodes, platform models Graph layer Neo4j / RDF store; GraphRAG index Ontology object & link types Vector search Qdrant, pgvector, Weaviate… Vector properties + KNN object search Lexical/hybrid Elasticsearch + RRF Object search + functions composing both Typed mutations Hand-built APIs + auth Action types LLM orchestration LangGraph, LlamaIndex, custom agents AIP Logic / Chatbot Studio Evaluation RAGAS, custom harnesses AIP Evals Triggers/automation Airflow, queues, cron Automate UI React, Streamlit, Retool Workshop / Slate / OSDK Governance & lineage Assembled: Ranger, OpenLineage, IAM glue Native, mandatory, propagating
Three observations fall out of the mapping.
The graph is the contract. Obsidian demonstrates the emergent extreme: zero schema cost, structure discovered by writing, ceiling reached when queries need types. Neo4j/RDF sit in the middle: schema by convention or by formal ontology, full traversal power, governance DIY. Palantir sits at the governed extreme: schema changes are reviewed proposals, and in exchange every consumer — human analyst, Workshop app, LLM tool call — shares one typed vocabulary. The consistent lesson across GraphRAG research and enterprise deployments alike is that LLMs perform dramatically better against a typed graph with named relations than against either raw text or an unlabeled link soup: the schema functions as a prompt, a validator, and an API surface simultaneously.
Retrieval quality lives in the same three places regardless of stack. Chunking/indexing decisions (structure-aware splitting, metadata, embedding model choice), the hybrid composition (dense + lexical + graph expansion, fused and reranked), and evaluation discipline (a gold set and per-stage metrics, or Evals suites). Foundry does not exempt you from any of these — a badly chunked corpus retrieves badly on any platform — it changes who operates the machinery and how failures surface (lineage and Evals rather than log archaeology).
Actions are the frontier vanilla RAG ignores. The open stack has mature retrieval and immature, ad-hoc mutation: most agent frameworks let a model call arbitrary Python, and safety is whatever the developer wrote. The Ontology’s action types — typed, validated, permissioned, auditable, human-approvable mutations as the only write path for both people and models — are the most instructive design in the platform, and the pattern most worth copying into open-stack systems: define a closed vocabulary of mutations with preconditions, and give the model that, never a database handle.
Choosing, honestly
For a personal corpus or a team wiki, Obsidian plus an embedding plugin is a complete system, and its ceiling is real but distant. For a product whose core competency is retrieval, the open stack — pgvector or Qdrant, hybrid search with RRF and a cross-encoder, possibly a GraphRAG layer on Neo4j or Kùzu, LangGraph or plain code for orchestration — offers control, portability, and unit economics no platform matches, at the cost of owning governance, evaluation, and operations yourself. Foundry/AIP earns its (substantial) cost where the problem is operational rather than informational: many source systems, strict security and audit requirements, human-in-the-loop decisions, and LLM outputs that must turn into governed actions against real assets. Its lock-in is structural — an organization that has encoded its operations as an Ontology with thousands of dependent functions does not migrate casually — which is precisely why the architecture is worth understanding independently of the vendor: every idea in Part II can be implemented, with more effort and more freedom, from the parts in Part I.
The convergence is the takeaway. Personal knowledge graphs, GraphRAG pipelines, and billion-dollar operational platforms have all arrived at the same shape: a typed graph of entities and relations at the center, vector and lexical indexes at the edges for entry, an LLM orchestration layer that treats retrieval and mutation as tools, and evaluation wrapped around the whole thing. Learn the shape once and both ecosystems become legible.
Brian Curry is a Kansas City–based cognitive systems architect and applied AI researcher working at the intersection of AI architecture, knowledge engineering, and intelligent-system design. He is the founder of Vector1 Research, an applied AI lab publishing on agent systems, AI evaluation, cognitive architecture, and the macroeconomics of AI-driven transformation. He has led applied AI and data initiatives across major enterprises including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, AT&T, and McClatchy, and founded KC AI Lab, a 1,500-member AI/ML community. His open-source work includes Papilon (causal inference and counterfactual simulation), PyCausalSim (causal discovery via agent-based simulation), MeaningFlow (semantic modeling and ontology), and Memory-Node Encapsulation (episodic memory for agentic systems). Contact: brian at vector1.ai · LinkedIn
