Is Your Corpus Compounding?
A Measurement Framework for Organizational Knowledge in the Age of Augmentation
Last week I made the case that the four-layer Augmentation Stack — structured retrieval grounding, causal reasoning, persistent verifiable memory, and agentic verification loops — is what turns AI from a slop-remixer into an evolutionary selection mechanism on the organizational knowledge corpus. The piece closed with the question every executive should be asking right now:
Is our knowledge corpus compounding, or accumulating?
That’s where the article ended. This one is where the answer starts. Because the uncomfortable truth about the last three years of enterprise AI investment is that almost no organization can answer the question. They can tell you how many documents are in their vector store. They can tell you how many queries the chatbot got last quarter. They can tell you the average user satisfaction score on the answers. None of those numbers — not one — tells you whether the corpus is getting smarter or simply getting bigger.
This is a measurement problem, not a technology problem. And it’s the measurement problem that separates serious knowledge engineering from glorified document indexing.
What follows is the framework I’ve been refining through Vector1 Research and applying in production environments — a four-family metric system mapped to the four layers of the stack, anchored by a causal lift estimate that ties the whole thing back to organizational outcomes. Code is in Python because that’s where the practitioner ecosystem lives. References are at the end.
Why Most Knowledge Management Metrics Are Vanity Metrics
Walk into the typical AI-augmented knowledge function in 2026 and you’ll see roughly the same dashboard. Total documents indexed. Queries per day. Top searched terms. Average response time. Maybe a thumbs-up rate on answers. If the team is sophisticated, a retrieval recall@k score on a frozen evaluation set.
Every one of those metrics is a vanity metric in the precise technical sense (Croll & Yoskovitz, 2013). They measure activity, not outcomes. They tell you the system is being used; they tell you nothing about whether the corpus is improving or whether the answers are connecting to better organizational decisions. Worse, they’re all gameable. Document count goes up automatically as people write more docs. Query volume goes up when you mandate use. Thumbs-up rates correlate more with answer fluency than answer correctness — confidently wrong answers get the same thumbs-up as confidently right ones, because users don’t usually have the ground truth to discriminate (Zhou et al., 2024).
The retrieval-quality literature has known about this gap for years. Standard IR metrics — MRR, NDCG@k, MAP — measure whether the retrieval system surfaces relevant documents but say nothing about whether the corpus those documents come from is internally consistent, current, or connected. The newer RAG-specific evaluation frameworks like RAGAS (Es et al., 2023), ARES (Saad-Falcon et al., 2023), and TruLens add faithfulness, answer relevance, and context precision/recall on top of retrieval quality, which is genuine progress, but they still operate within a single corpus snapshot. None of them measure whether the corpus is evolving in a useful direction.
The problem is structural. Knowledge management has historically been measured the way librarianship was measured — circulation statistics, holdings size, patron satisfaction. Those metrics make sense for a passive collection. They make no sense for a system whose entire value proposition is that it’s supposed to be evolving.
A measurement framework for the Augmentation Stack has to answer a different set of questions. Is the corpus getting more accurate over time, or just larger? Are the recommendations the system makes connecting to verifiable outcomes? Are validated outputs being retained as next-generation training signal? Is the AI-augmented version of knowledge access actually causing better organizational decisions, or just being correlated with them? And on the executive level, the rolled-up question: what’s the causal lift of this entire investment, and how do we know?
The framework that follows organizes around those questions. It maps directly onto the four layers of the Augmentation Stack, with one metric family per layer, and closes with the cross-cutting causal lift estimate that ties the whole thing back to organizational outcomes.
Family 1 — Corpus Health (Layer 1: Structured Retrieval Grounding)
Corpus health metrics measure whether the substrate the system retrieves from is actually getting better as a knowledge object. The literature on knowledge graph quality and refinement (Paulheim, 2017; Färber et al., 2018) gives us a good starting point, which I extend here for the AI-augmented retrieval setting.
Entity coverage
Define the corpus’s entity coverage as the fraction of named entity mentions in the corpus that successfully resolve to canonical entities in the knowledge graph:
Σ over docs d: count of entities in d that resolve to KG
EC = ─────────────────────────────────────────────────────────
Σ over docs d: total entities in dEntity coverage near 1.0 means retrieval can disambiguate confidently; entity coverage at 0.4 means your system retrieves the wrong document about half the time when entity names overlap.
Operationally this is a NER + entity resolution pipeline. spaCy’s en_core_web_trf or GLiNER (Zaratiana et al., 2023) handle the recognition. Entity resolution against the KG benefits from string-similarity baselines (RapidFuzz, dedupe) augmented with embedding-based candidate generation (Tedeschi et al., 2021) and a cross-encoder ranker. Production targets I've seen work in well-instrumented organizations: EC ≥ 0.85 for canonical business entities (customers, products, projects); EC ≥ 0.70 for general domain entities.
Contradiction density
Define contradiction density as the rate of NLI-detected contradictions across same-topic document pairs, normalized per thousand documents. The standard tool is a fine-tuned NLI model (DeBERTa-v3-large-mnli is the current strong open baseline; Honovich et al., 2022 review the broader space) applied pairwise to claim-bearing sentences clustered by topic.
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
from itertools import combinations
from collections import defaultdict
class ContradictionDetector:
def __init__(self, model_id="MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli"):
self.tok = AutoTokenizer.from_pretrained(model_id)
self.model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
# label order for this checkpoint: 0=entail, 1=neutral, 2=contradict
self.contra_idx = 2
@torch.no_grad()
def score_pair(self, premise: str, hypothesis: str) -> float:
x = self.tok(premise, hypothesis, return_tensors="pt", truncation=True)
probs = torch.softmax(self.model(**x).logits, dim=-1)[0]
return probs[self.contra_idx].item()
def density(self, claims_by_topic: dict, threshold: float = 0.85) -> dict:
"""Return contradictions per 1k document-pairs, by topic."""
out = defaultdict(lambda: {"pairs": 0, "contradictions": 0})
for topic, claims in claims_by_topic.items():
for a, b in combinations(claims, 2):
out[topic]["pairs"] += 1
if self.score_pair(a, b) >= threshold:
out[topic]["contradictions"] += 1
return {
t: 1000.0 * v["contradictions"] / max(v["pairs"], 1)
for t, v in out.items()
}The trend matters more than the level. Contradictions should decrease over time as the resolution and curation processes catch up. If contradiction density is flat or rising quarter over quarter, your corpus is accumulating, not compounding. For benchmarking, internal corpora I’ve evaluated cluster around 12–25 contradictions per 1,000 document-pairs at baseline, with mature programs driving this below 5 within a year of disciplined curation.
Freshness decay
Track the fraction of the corpus older than the domain half-life. Half-lives differ by content type — engineering RFCs decay in months, regulatory contracts in years, scientific literature on a per-domain basis (Larivière et al., 2008, on citation half-lives, gives a defensible empirical anchor). A weighted version, where each document’s contribution to retrieval is weighted by its hit rate in production, is more useful in practice than the raw count.
A retrieval-weighted freshness metric: of the chunks actually returned by the system over the last 30 days, what fraction came from documents past their domain half-life? If 60% of your retrievals are pulling from documents past their half-life, your AI is fluently giving you advice from three years ago.
Citation graph density
A corpus where every document is an island is a corpus that doesn’t compound. Three sub-metrics worth tracking, drawn from network science (Barabási, 2016):
- Mean inbound degree — average number of inbound references per document
- Largest connected component fraction — fraction of corpus in the LCC of the citation graph
- Average shortest path length — mean hops between any two documents in the LCC
Healthy corpora have mean inbound degree of at least 3, LCC fraction of at least 0.7, and average shortest path length in the 4–6 range (small-world structure). Most enterprise corpora I’ve audited have mean inbound degree around 0.3 and LCC fraction below 0.2 — they are document piles, not knowledge networks. Tools: NetworkX for analysis; the citation extraction is a one-time NER+linking pass.
Schema coverage
The fraction of the corpus with structured metadata — entity tags, document type, authoring team, validation status, supersession links, expiration date. Structured coverage is what makes the corpus machine-tractable beyond raw embedding similarity. Target ≥ 0.80; most enterprises sit at 0.10–0.20 when they first measure.
A corpus health evaluator that pulls these together:
from dataclasses import dataclass
from datetime import datetime, timedelta
import networkx as nx
@dataclass
class CorpusHealth:
entity_coverage: float
contradiction_density: float # per 1k pairs, weighted across topics
freshness_decay: float # retrieval-weighted
mean_inbound_degree: float
lcc_fraction: float
schema_coverage: float
class CorpusHealthEvaluator:
def __init__(self, corpus, kg, ner, contradiction_detector,
retrieval_log, half_life_days=365):
self.corpus = corpus
self.kg = kg
self.ner = ner
self.cd = contradiction_detector
self.retrieval_log = retrieval_log # last-30d chunk retrieval events
self.h = timedelta(days=half_life_days)
def evaluate(self) -> CorpusHealth:
return CorpusHealth(
entity_coverage=self._entity_coverage(),
contradiction_density=self._weighted_contradiction_density(),
freshness_decay=self._retrieval_weighted_freshness(),
mean_inbound_degree=self._mean_inbound_degree(),
lcc_fraction=self._lcc_fraction(),
schema_coverage=self._schema_coverage(),
)
def _entity_coverage(self):
total = resolved = 0
for doc in self.corpus.iter_documents():
for e in self.ner(doc.text):
total += 1
if self.kg.resolve(e) is not None:
resolved += 1
return resolved / total if total else 0.0
def _retrieval_weighted_freshness(self):
now = datetime.now()
weighted_stale = weighted_total = 0.0
for event in self.retrieval_log.events():
doc = self.corpus.get(event.doc_id)
w = event.weight
weighted_total += w
if now - doc.last_updated > self.h:
weighted_stale += w
return weighted_stale / weighted_total if weighted_total else 0.0
def _mean_inbound_degree(self):
g = self.corpus.citation_graph()
return sum(dict(g.in_degree()).values()) / max(g.number_of_nodes(), 1)
def _lcc_fraction(self):
g = self.corpus.citation_graph().to_undirected()
if g.number_of_nodes() == 0:
return 0.0
lcc = max(nx.connected_components(g), key=len)
return len(lcc) / g.number_of_nodes()
def _weighted_contradiction_density(self):
claims = self.corpus.claims_by_topic()
densities = self.cd.density(claims)
weights = {t: len(c) for t, c in claims.items()}
total_w = sum(weights.values())
return sum(densities[t] * weights[t] for t in densities) / max(total_w, 1)
def _schema_coverage(self):
required = {"doc_type", "owner", "valid_from", "valid_to", "supersedes"}
ok = sum(1 for d in self.corpus.iter_documents()
if required.issubset(d.metadata.keys()))
return ok / len(self.corpus)This is the easiest of the four families to instrument and the one most teams skip first. Entity coverage and schema coverage can be computed on day one. Contradiction density needs the NLI model but is a fixed-cost integration. Freshness and citation density are arithmetic.
Family 2 — Decision Quality (Layer 2: Causal Reasoning)
Decision quality metrics measure whether the system’s outputs are epistemically defensible — whether the recommendations it makes are causally grounded, calibrated, and resilient to scrutiny. Most enterprise AI fails this entire category by quietly producing confident answers with no causal claims.
Causal claim rate
The fraction of recommendations the system produces that are accompanied by an explicit causal mechanism, an effect-size estimate, and a confidence interval. The default LLM output — “based on customer behavior, we recommend X” — has none of these and should score zero. The grounded output — “under our causal model, X has an estimated retention effect of +2.3% ± 0.6%, driven primarily by reducing onboarding friction in the third session” — has all three. This metric forces the system architecture to actually pair the LLM with a causal model rather than letting it freelance.
Refutation test pass rate
Measures the fraction of causal claims that survive a structured set of refutation tests (Sharma & Kıcıman, 2020, in DoWhy). Standard refutations:
- Placebo treatment — replace the cause with a random variable; effect should be ≈ 0
- Random common cause — add a random unobserved confounder; effect should be stable
- Data subset robustness — re-estimate on random subsets; effect should be consistent
- Add unobserved confounder — Rosenbaum-style sensitivity analysis (Rosenbaum, 2002)
If your system’s causal claims fail these tests at high rates, the claims are decorative, not real. DoWhy gives you these out of the box; the operational discipline is running them on every causal claim before publishing, which most teams skip.
Calibration error
Measures whether the system’s stated confidence intervals are honest. Expected Calibration Error (ECE) is the standard metric (Guo et al., 2017):
M |B_m|
ECE = Σ ───── × | acc(B_m) − conf(B_m) |
m=1 n
where:
M = number of confidence bins
B_m = predictions whose stated confidence falls in bin m
|B_m|/n = fraction of predictions in bin m
acc(B_m) = empirical accuracy of predictions in bin m
conf(B_m) = mean stated confidence of predictions in bin mA well-calibrated system that says “80% confident” should be right 80% of the time.
Most LLMs are dramatically over-confident (Kadavath et al., 2022; Tian et al., 2023; Xiong et al., 2024). The remediation literature is now mature: temperature scaling and Platt scaling (Guo et al., 2017) for parametric calibration, isotonic regression for non-parametric, conformal prediction (Angelopoulos & Bates, 2023) for distribution-free coverage guarantees. A practitioner-grade ECE calculator with reliability diagram:
import numpy as np
def expected_calibration_error(
confidences: np.ndarray, # stated confidence in [0,1]
correctness: np.ndarray, # 0/1 outcomes
n_bins: int = 15,
) -> dict:
"""
Compute ECE and produce reliability diagram data.
confidences and correctness are 1-D arrays of same length.
"""
bins = np.linspace(0.0, 1.0, n_bins + 1)
bin_idx = np.digitize(confidences, bins) - 1
bin_idx = np.clip(bin_idx, 0, n_bins - 1)
n = len(confidences)
ece = 0.0
diagram = []
for m in range(n_bins):
mask = bin_idx == m
if mask.sum() == 0:
diagram.append({"conf": (bins[m]+bins[m+1])/2, "acc": None, "n": 0})
continue
acc_m = correctness[mask].mean()
conf_m = confidences[mask].mean()
ece += (mask.sum() / n) * abs(acc_m - conf_m)
diagram.append({"conf": conf_m, "acc": acc_m, "n": int(mask.sum())})
return {"ece": ece, "reliability_diagram": diagram}Out-of-the-box LLM confidence (verbalized self-reported) typically scores ECE of 0.15–0.30 on knowledge-intensive tasks (Tian et al., 2023). Post-calibration target: ECE < 0.05. That’s a hard but achievable bar with conformal methods or trained verifier heads.
Decision-outcome attribution coverage
The percentage of system recommendations that are followed up with a recorded outcome. This metric ties forward into Layer 3 memory and Layer 4 verification, but it lives here because it’s also a decision quality signal. A system that recommends but never closes the loop is generating untestable opinion at scale.
Family 3 — Memory Health (Layer 3: Persistent Verifiable Memory)
Memory health metrics measure whether the institutional memory is actually persistent, retrievable, and growing in usefulness. This is where most KM functions are weakest because traditional KM never had to instrument memory in this way.
The relevant academic anchor here is the agent memory and continual-learning literature (Park et al., 2023, on generative agents; Packer et al., 2023, on MemGPT; Zhong et al., 2024, on MemoryBank), plus the older information-systems work on records management and provenance (Moreau et al., 2011, on the W3C PROV standard).
Outcome attribution rate
The fraction of decisions, recommendations, or system actions that have an associated outcome record. Without this linkage, the system is decision-blind — it can’t learn from its own track record because it doesn’t know its own track record. Operational target: ≥ 0.60 within six months of instrumentation, ≥ 0.85 within eighteen months. Below 0.30 is what most organizations measure on first audit.
Provenance trace completeness
Measures whether every output the system produces can be traced back to specific source documents, specific causal models, and specific memory entries that contributed to it. The W3C PROV data model (Moreau et al., 2011) gives the standard schema; OpenLineage and Marquez are the production implementations worth knowing. Auditability requires this. EU AI Act compliance increasingly requires it. Debugging absolutely requires it.
A trace is complete if for any output produced at time t, you can recover four things: the retrieved chunk set, the memory entries consulted, the causal model invoked, and the model version. Score completeness as the fraction of outputs for which all four are recoverable.
Episodic recall precision
Measures whether the system can answer “what happened the last time we tried X” with a verifiable, specific, dated answer rather than a synthetic-sounding generality. Hold out a set of known organizational episodes — past projects, decisions, incidents — and query the system about them. Grade responses against ground truth on three axes: factual accuracy, temporal precision, and provenance correctness.
The benchmark to use is your own episodes; published agent-memory benchmarks (LoCoMo from Maharana et al., 2024; LongMemEval from Wu et al., 2024) give you a methodology and baseline comparisons but are domain-general.
Memory promotion rate
The fraction of working-memory entries that are promoted into long-term retention based on outcome verification. This is the rate at which the system is actively curating its memory based on what worked. A system with high working-memory volume but low promotion rate is hoarding; a system with disciplined promotion is learning. The MemGPT-style hierarchical memory architecture (Packer et al., 2023) makes this metric naturally observable; ad-hoc memory implementations often don’t.
Family 4 — Evolutionary Fitness (Layer 4: Agentic Verification Loops)
The fourth family is the highest-leverage and the least instrumented in current practice. These are the metrics that tell you whether the selection layer is actually performing selection. The conceptual anchor is the program-search and self-improvement literature (Romera-Paredes et al., 2024, on FunSearch; Trinh et al., 2024, on AlphaGeometry; Zelikman et al., 2022, on STaR; Yuan et al., 2024, on self-rewarding language models).
Verifier coverage
The percentage of system outputs that pass through some external verification mechanism — code execution, formal proof, simulation, A/B test, structured human review with rubric — before being eligible for retention as next-generation signal. Verifier coverage of zero means no selection is happening at all; the system is generation-only, which is the model-collapse pattern documented by Shumailov et al. (2024). Verifier coverage approaching 100% means every output gets graded against external reality.
A taxonomy of verifiers worth standardizing on:
- Executable verifiers — code tests, formal proof (Lean, Coq), simulation outcomes. Highest signal, lowest false-positive rate.
- Reference verifiers — comparison against curated ground truth. High signal in covered cases, no signal outside.
- Model-as-judge verifiers — LLM grading another LLM’s output (Zheng et al., 2023). Cheap and scalable but noisy and gameable; calibrate against human or executable verifiers regularly.
- Human verifiers — structured rubric review. Gold standard for subjective domains; expensive.
Don’t conflate them. Track verifier coverage by type.
Validated retention rate
Of the outputs that pass verification, the fraction that actually feed back into the corpus, the memory, or the model fine-tuning signal as next-generation substrate. Verification without retention is empty exercise. Many organizations have evaluation pipelines that grade outputs and then throw the grades away.
Generation lift
The signal-to-noise improvement of the corpus from one generation to the next. Operationally:
GenLift(t → t+1) = E[ V( y_{t+1}(x) ) ] − E[ V( y_t(x) ) ]
x ∈ D_eval
where:
V = verifier scoring function (output → score in [0,1])
D_eval = held-out evaluation distribution
y_t(x) = system output for input x at generation t
E[ . ] = expectation over the eval distributionPositive generation lift over multiple successive generations is the empirical signature of compounding. Flat or negative generation lift over multiple generations is the empirical signature of accumulation or collapse.
A practical evaluator with bootstrap confidence intervals:
import numpy as np
from typing import Callable
def generation_lift(
eval_set: list,
gen_t_outputs: list, # list of system outputs at generation t
gen_t1_outputs: list, # list of system outputs at generation t+1
verifier: Callable, # verifier(input, output) -> float in [0,1]
n_bootstrap: int = 1000,
seed: int = 0,
) -> dict:
rng = np.random.default_rng(seed)
n = len(eval_set)
assert len(gen_t_outputs) == len(gen_t1_outputs) == n
scores_t = np.array([verifier(eval_set[i], gen_t_outputs[i]) for i in range(n)])
scores_t1 = np.array([verifier(eval_set[i], gen_t1_outputs[i]) for i in range(n)])
diffs = scores_t1 - scores_t
# Paired bootstrap CI on the lift
boots = np.empty(n_bootstrap)
for b in range(n_bootstrap):
idx = rng.integers(0, n, size=n)
boots[b] = diffs[idx].mean()
return {
"lift": diffs.mean(),
"ci_95": (np.percentile(boots, 2.5), np.percentile(boots, 97.5)),
"p_value_one_sided": float((boots <= 0).mean()),
"n": n,
}The same statistical machinery (paired bootstrap, BCa intervals from Efron & Tibshirani, 1993) applies whether the verifier is executable, reference-based, model-as-judge, or human. The bootstrap CI is what lets you separate “the system genuinely got better” from “noise.”
Selection pressure strength
How aggressively the verification layer is filtering. Pure pass-through (everything retained) is no selection; harsh filtering (only top decile retained) is strong selection. The right level depends on the verifier’s accuracy and the cost of false retention; track it explicitly as the retention-rate distribution by verifier score, and tune.
The Causal Lift Question
The four metric families above are diagnostic. They tell you whether the components of the Augmentation Stack are functioning. They do not, on their own, answer the executive question: is this whole investment causing better organizational outcomes?
That question requires a different kind of measurement. Not a dashboard metric, but a causal estimate. The toolkit comes from the program evaluation and applied econometrics literatures (Imbens & Rubin, 2015; Angrist & Pischke, 2008; Pearl, 2009).
Method selection by rollout pattern
Staggered rollout across teams or regions. Use difference-in-differences. The teams that received access to the augmented system early are the treatment group; the teams receiving access later are the control during the lag period. Two-way fixed effects regression with team and time fixed effects:
Y_it = α_i + λ_t + β · D_it + X_it · γ + ε_it where: Y_it = outcome for team i at time t α_i = team fixed effect λ_t = time fixed effect D_it = 1 if team i has system access at time t, else 0 X_it = control covariates β = the causal effect of interest
Recent econometric work (Goodman-Bacon, 2021; Callaway & Sant’Anna, 2021; Roth et al., 2023) shows the standard TWFE estimator can be biased under heterogeneous treatment effects with staggered timing — use the Callaway–Sant’Anna or de Chaisemartin–D’Haultfœuille estimators, both implemented in differences and did Python packages.
All-at-once rollout. Bayesian structural time series via CausalImpact (Brodersen et al., 2015), accessible in Python through tfcausalimpact or pycausalimpact. The validity rests on having clean pre-period covariate series unaffected by the intervention.
import pandas as pd
from causalimpact import CausalImpact
def estimate_augmentation_lift(
outcome: pd.Series,
covariates: pd.DataFrame,
intervention_date: str,
horizon_days: int = 90,
):
"""BSTS-based causal lift estimate (Brodersen et al., 2015)."""
df = pd.concat([outcome.rename("y"), covariates], axis=1).asfreq("D")
pre = [df.index.min(), pd.Timestamp(intervention_date) - pd.Timedelta(days=1)]
post = [pd.Timestamp(intervention_date),
pd.Timestamp(intervention_date) + pd.Timedelta(days=horizon_days)]
ci = CausalImpact(df, pre, post)
s = ci.summary_data
return {
"abs_effect": s.loc["abs_effect", "average"],
"rel_effect": s.loc["rel_effect", "average"],
"ci_95": (s.loc["abs_effect_lower", "average"],
s.loc["abs_effect_upper", "average"]),
"p_value": ci.p_value,
"report": ci.summary(output="report"),
}Single treated organization, comparable peers available. Synthetic control (Abadie & Gardeazabal, 2003; Abadie et al., 2010), implemented in Python via pysyncon or SparseSC. Construct a weighted combination of donor units that matches the treated unit's pre-period trajectory; the post-period gap is the causal estimate.
Per-user adoption variation. Propensity score matching or instrumental variables (Imbens & Rubin, 2015), with causalinference, dowhy, or econml. EconML's DoubleML estimators (Chernozhukov et al., 2018) handle high-dimensional confounder control with machine-learning nuisance estimation.
Planned in advance. Stepped wedge cluster-randomized rollout (Hussey & Hughes, 2007; Hemming et al., 2015). This is the gold standard — sequential, randomized release of the system to teams over time, designed from the start to enable causal estimation. Most organizations don’t plan ahead this way, but they should, because the alternative is spending seven figures on a system whose business case is forever unfalsifiable.
Robustness reporting
A causal lift estimate is only as defensible as its sensitivity analysis. Standard kit:
- Placebo tests. Run the same analysis with a fake intervention date; the estimated effect should be ≈ 0.
- Permutation inference. For synthetic control, randomly assign the treatment to donor units and rebuild the null distribution.
- Rosenbaum bounds. Quantify how strong an unobserved confounder would need to be to overturn the conclusion (Rosenbaum, 2002).
- Pre-trend tests. For DiD, demonstrate parallel pre-trends; for BSTS, validate the fit on the pre-period.
A defensible quarterly executive update reads: “The augmentation stack rollout is associated with a +14.2% lift in cross-team decision velocity, 95% CI [+8.1%, +20.3%], p = 0.003, robust to placebo controls (effect = +0.4%, p = 0.71) and Rosenbaum bounds tolerating Γ ≤ 1.8.” That is a fundamentally different kind of communication than “the chatbot got 12,000 queries last quarter.”
For my own work I’ve been extending this kind of analysis through Papilon, which adds agent-based counterfactual simulation on top of the standard observational toolkit — useful when historical data is sparse or when the intervention’s mechanism involves complex feedback loops that pure time-series methods miss. The causal-inference toolkit is mature enough now that there’s no excuse for not running these analyses; the gap is methodological will, not capability.
A Reference Dashboard Architecture
Putting this together into something an organization actually uses requires three layers of view, calibrated to three audiences.
Executive view. Four normalized metrics (one per layer, 0–100 scale) plus the causal lift estimate with its confidence interval. Recommended headline metrics: corpus health → schema coverage; decision quality → calibration error (inverted, so higher = better); memory health → outcome attribution rate; evolutionary fitness → cumulative generation lift over the last four quarters. Five numbers, four trends. If three of them are red and the causal lift is non-significant, the program is in trouble regardless of how popular the chatbot is.
Manager view. Each layer expanded into its full metric family with two-quarter trend lines. Drill-downs to corpus segments, recommendation classes, memory pools, and verifier types. This is the level at which a Director of Knowledge Engineering reports to leadership.
Practitioner view. Full instrumentation. Every metric, every breakdown, every alert. Retrieval anomalies, contradiction spikes traced to specific document classes, memory-promotion bottlenecks, verifier disagreement rates. This is where engineers and analysts work day-to-day.
The architectural decision worth making explicit: build this on top of an existing observability stack, not as a dedicated AI observability platform. Most metrics here are scheduled jobs writing to your data warehouse (Snowflake, BigQuery, Databricks) and visualized in whatever BI tool you already use (Looker, Tableau, Mode, Metabase, Hex). Causal lift runs as a quarterly batch in a notebook. Specialized AI observability vendors (Arize, Fiddler, WhyLabs, LangSmith, Patronus) are useful for production model monitoring but oversolved for the corpus-level, organizational-outcome-level questions this framework targets.
A Maturity Model
The four metric families also serve as a maturity model.
Level 0 — Unmeasured. Activity metrics only. Document count, query volume, thumbs-up rate. The majority of enterprise AI deployments in 2026 sit here.
Level 1 — Component instrumented. Some metrics from one or two of the four families are tracked, usually corpus health basics. Decision quality, memory, and verification are unmeasured.
Level 2 — Multi-layer measured. All four families have at least baseline instrumentation. The team can speak to corpus health, decision quality, memory function, and verification fitness as separate concerns. Causal lift is not yet estimated.
Level 3 — Causally estimated. Full instrumentation plus quarterly causal lift estimates with confidence intervals on at least one organizational outcome. The team can defend the program’s ROI against scrutiny. Reviews include refutation tests on causal claims.
Level 4 — Compounding demonstrable. Generation lift over multiple successive generations is positive and statistically significant. The corpus is provably getting better. Recommendations are causally linked to outcomes with documented effect sizes. Quarter-over-quarter compounding curve presentable to the board.
A maturity score is the fastest way to clarify where an organization actually is and what the next investment should target. Most organizations want to be at Level 3 by year one and Level 4 by year three. Most are honestly at Level 0 or 1 today and underestimate how much foundational work the climb requires.
Where to Start
Order of operations matters.
Start with corpus health. Schema coverage and entity coverage are computable on day one and immediately reveal the structural gaps in your substrate. Most teams discover that their corpus is far less structured than they thought and that retrieval quality issues they were blaming on the model are actually substrate issues. Fix the substrate first.
Add decision quality next. Calibration error and refutation test pass rates can be added to any system that produces recommendations and immediately raise the epistemic floor. The first time your system fails a placebo test, you’ll know more about its actual reliability than you did from a year of thumbs-up data.
Memory health is third. Once outputs are better-grounded, ensure they’re retained with provenance and connected to outcomes. This is the layer where the loop between system action and organizational learning gets closed. Adopt the W3C PROV standard for the schema; it’ll save rework later.
Evolutionary fitness is fourth and hardest. Verifier coverage and generation lift require disciplined verification protocols and cross-generational evaluation methodology. Most teams won’t be able to instrument this fully until they’ve done the first three. That’s fine — the value compounds even without it for a while, as long as you know it’s the next destination.
The causal lift analysis runs in parallel with all four, on whatever cadence your data permits. Quarterly is reasonable; annual is the absolute minimum for a serious program.
The Discipline
Knowledge management has spent thirty years measuring the wrong things — usage statistics, satisfaction surveys, repository size — because the underlying systems were too static to instrument any other way. The Augmentation Stack changes that. The system is now dynamic, the corpus is now evolving, and the evolutionary trajectory is now measurable. What’s required is the discipline to actually measure it, the rigor to demand causal evidence rather than activity metrics, and the architectural patience to instrument all four layers rather than the easy one.
The organizations that build this measurement discipline will be the ones that can prove, quantitatively and defensibly, that their knowledge corpus is compounding. They will be measurably ahead of their competitors a year from now and dramatically ahead three years from now, because the compounding curve is exponential and the gap widens every quarter. The organizations that ship dashboards full of activity metrics will spend the next decade arguing about whether their AI investment was worth it, with no evidence either way.
This is generation zero. The instrumentation you build now is the instrumentation you’ll be running across every subsequent generation. Build it like you mean it.
That’s the job.
References
Causal inference and program evaluation
- Abadie, A. & Gardeazabal, J. (2003). The Economic Costs of Conflict: A Case Study of the Basque Country. American Economic Review.
- Abadie, A., Diamond, A. & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies. JASA.
- Angrist, J. & Pischke, J. (2008). Mostly Harmless Econometrics. Princeton.
- Brodersen, K. et al. (2015). Inferring Causal Impact Using Bayesian Structural Time-Series Models. Annals of Applied Statistics.
- Callaway, B. & Sant’Anna, P. (2021). Difference-in-Differences with Multiple Time Periods. Journal of Econometrics.
- Chernozhukov, V. et al. (2018). Double/Debiased Machine Learning. Econometrics Journal.
- Goodman-Bacon, A. (2021). Difference-in-Differences with Variation in Treatment Timing. Journal of Econometrics.
- Hemming, K. et al. (2015). The Stepped Wedge Cluster Randomised Trial. BMJ.
- Hussey, M. & Hughes, J. (2007). Design and Analysis of Stepped Wedge Cluster Randomized Trials. Contemporary Clinical Trials.
- Imbens, G. & Rubin, D. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge.
- Pearl, J. (2009). Causality (2nd ed.). Cambridge.
- Rosenbaum, P. (2002). Observational Studies (2nd ed.). Springer.
- Roth, J. et al. (2023). What’s Trending in Difference-in-Differences? Journal of Econometrics.
- Sharma, A. & Kıcıman, E. (2020). DoWhy: An End-to-End Library for Causal Inference. arXiv.
Calibration and uncertainty
- Angelopoulos, A. & Bates, S. (2023). Conformal Prediction: A Gentle Introduction. Foundations and Trends in ML.
- Guo, C. et al. (2017). On Calibration of Modern Neural Networks. ICML.
- Kadavath, S. et al. (2022). Language Models (Mostly) Know What They Know. arXiv.
- Tian, K. et al. (2023). Just Ask for Calibration. EMNLP.
- Xiong, M. et al. (2024). Can LLMs Express Their Uncertainty? ICLR.
RAG, retrieval, and evaluation
- Es, S. et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv.
- Honovich, O. et al. (2022). TRUE: Re-evaluating Factual Consistency Evaluation. NAACL.
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP. NeurIPS.
- Saad-Falcon, J. et al. (2023). ARES: Automated Evaluation Framework for RAG Systems. arXiv.
- Tedeschi, S. et al. (2021). NER4EL: Named Entity Recognition for Entity Linking. EMNLP.
- Zaratiana, U. et al. (2023). GLiNER: Generalist Model for NER. arXiv.
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench. NeurIPS.
Knowledge graphs and corpus quality
- Färber, M. et al. (2018). Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO. Semantic Web.
- Larivière, V. et al. (2008). The Decline in the Concentration of Citations. JASIST.
- Paulheim, H. (2017). Knowledge Graph Refinement: A Survey. Semantic Web.
- Moreau, L. et al. (2011). The PROV Data Model. W3C Recommendation.
Memory, agents, and self-improvement
- Maharana, A. et al. (2024). Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo). ACL.
- Packer, C. et al. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv.
- Park, J. et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. UIST.
- Romera-Paredes, B. et al. (2024). Mathematical Discoveries from Program Search with LLMs (FunSearch). Nature.
- Trinh, T. et al. (2024). Solving Olympiad Geometry without Human Demonstrations (AlphaGeometry). Nature.
- Wu, D. et al. (2024). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. arXiv.
- Yuan, W. et al. (2024). Self-Rewarding Language Models. arXiv.
- Zelikman, E. et al. (2022). STaR: Bootstrapping Reasoning with Reasoning. NeurIPS.
- Zhong, W. et al. (2024). MemoryBank: Enhancing LLMs with Long-Term Memory. AAAI.
Model collapse and synthetic data
- Shumailov, I. et al. (2024). AI Models Collapse When Trained on Recursively Generated Data. Nature.
Statistics and methodology
- Croll, A. & Yoskovitz, B. (2013). Lean Analytics. O’Reilly.
- Efron, B. & Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall.
- Barabási, A.-L. (2016). Network Science. Cambridge.
Other
- Zhou, K. et al. (2024). Relying on the Unreliable: The Impact of Language Models’ Reluctance to Express Uncertainty. ACL.
Brian Curry is an applied AI researcher and the founder of Vector1 Research, an independent lab focused on knowledge engineering, causal inference, agent evaluation, and the architecture of compounding organizational intelligence. He is the author of open-source frameworks including MeaningFlow (semantic content modeling), Papilon (causal inference and complex systems simulation), and Memory-Node Encapsulation (autobiographical memory architecture for agentic systems). He writes regularly on applied AI, knowledge architecture, and decision science.
