Measuring Corpus Magnetism: Instrumenting Content for the Retrieval Layer

You cannot optimize what you cannot observe, and share of answer — the fraction of category prompts where a model cites you — is the wrong place to start observing. It is lagging, sparse, and tells you nothing about why. By the time it moves, the experiment is already over.

The question worth instrumenting is upstream: is your corpus magnetic? Define magnetism as a corpus’s propensity to be retrieved and cited per unit of content, net of the domain authority that would predict it. A magnetic corpus punches above its weight — small, dense, disproportionately pulled into answers. A heavy one does the opposite: thousands of pages crawled, almost nothing surfaced. The Cloudflare crawl-to-referral ratios from the companion piece are the symptom; magnetism is the latent variable.

Measuring it means instrumenting three planes — ingestion, retrieval, emission — and then fusing them. The first two are engineering. The fusion is where the efficacy lives, and it is the part this piece does not hand over.

Plane 1 — Ingestion telemetry

Before a chunk can be cited it has to be crawled, and your access logs already record exactly which AI operators are reading what, and how recently. Mining them gives you ingestion coverage — and, more usefully, the ingested-but-dark set: content the models have read but never surface.

import re, gzip
from collections import defaultdict
from datetime import datetime

AI_AGENTS = re.compile(
    r"(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|anthropic-ai|"
    r"PerplexityBot|Perplexity-User|Google-Extended|CCBot|Bytespider|"
    r"Meta-ExternalAgent|Amazonbot)", re.I)

last_seen, hits = {}, defaultdict(int)
for line in gzip.open("access.log.gz", "rt", errors="ignore"):
    m = AI_AGENTS.search(line)
    if not m:
        continue
    agent = m.group(1).lower()
    path  = line.split('"')[1].split(" ")[1]          # GET <path> HTTP/x
    ts    = datetime.strptime(line[line.find("[")+1:line.find("]")].split()[0],
                              "%d/%b/%Y:%H:%M:%S")
    hits[(agent, path)] += 1
    last_seen[(agent, path)] = max(last_seen.get((agent, path), ts), ts)

# coverage = crawled∩corpus / corpus ; recency = freshness of last crawl
# dark_set = corpus − {paths ever crawled}

That much is mechanical. What turns it into an ingestion score is a recency-decay kernel over last_seen and a dedupe model that separates genuine re-ingestion from churn — we are not unpacking the kernel here. The point of the plane is the diagnostic gap it exposes: most corpora discover that 30–60% of crawled pages are dark. Crawl budget spent, zero magnetism.

Plane 2 — Retrieval simulation

Emission is downstream of retrieval, and retrieval is simulable before any engine cites you. Embed your own chunks, embed a query manifold that approximates the sub-query fan-out of your category, and measure each chunk’s retrieval density — how often it clears top-k:

ρ(c) = (1/|Q|) · Σ_{q ∈ Q}  1[ rank(c | q) ≤ k ]

import numpy as np
from sentence_transformers import SentenceTransformer

model  = SentenceTransformer("all-MiniLM-L6-v2")
chunks = load_corpus_chunks()                  # atomic, self-contained units
Q      = synthesize_query_manifold(domain)     # <-- withheld: see methodology

C = model.encode([c.text for c in chunks], normalize_embeddings=True)
Qe = model.encode(Q, normalize_embeddings=True)

sims = Qe @ C.T                                # |Q| x |chunks| cosine matrix
topk = np.argsort(-sims, axis=1)[:, :K]
density = np.bincount(topk.ravel(), minlength=len(chunks)) / len(Q)

# density ≈ 0  -> dark chunk (embedding too diffuse or off-manifold)
# density high -> magnetic chunk (dense in the right neighborhoods)

The harness is trivial. The signal is almost entirely in synthesize_query_manifold. A keyword list produces a manifold that flatters your existing content and predicts nothing; a manifold that actually mirrors how models decompose real intent is the difference between a retrieval simulator that anticipates citation and one that just reranks your sitemap. How that manifold is generated — and how K is set per engine rather than guessed — is the part we keep. What the plane buys you is a per-chunk magnetism prior you can act on a week before any engine has re-crawled.

Plane 3 — Emission probing

The third plane closes the loop against the live ecosystem: fan a probe set across the engines, parse what they actually cite, and compute per-domain share of answer and time-to-citation.

import asyncio

ENGINES = ["openai", "anthropic", "gemini", "perplexity"]

async def probe(prompt, engine):
    resp = await call_engine(engine, prompt, web=True)
    return parse_citations(resp, engine)       # per-engine citation schemas

async def run(prompt_set):
    tasks = [probe(p, e) for p in prompt_set for e in ENGINES]
    results = await asyncio.gather(*tasks)
    return aggregate_share_of_answer(results)  # domain -> {engine: rate}

Run on a fixed cadence, this yields the observables that matter: which of your URLs are cited for which sub-queries, on which engines, and the latency from publish to first citation (near-real-time on live-retrieval engines, index-gated on others). But emission probing earns its cost only when it is run differentially: perturb a single chunk — add a sourced statistic, restructure into an answer-first unit, tighten its self-containment — re-probe, and attribute the delta. That counterfactual loop is the empirical engine the Princeton GEO work gestures at; operationalizing it across engines without confounds is non-trivial, and the controls we use to do it cleanly are not in this article.

The fusion (the part that isn’t here)

Three planes, three signal families:

Magnetism(corpus) = F( Ingestion, RetrievalDensity, EmissionRate )

The reason generic GEO checklists underperform is that they stop at the planes — they tell you to add schema and write FAQs and call it a strategy. The efficacy is not in the planes; it is in F: the functional form that fuses them, the per-vertical calibration of the fusion weights, the manifold synthesis that makes Plane 2 predictive, and the optimization feedback that turns the whole thing into a loop rather than a dashboard. Those four components are what separate a corpus that is measured from one that is magnetic — and they are exactly what we are not printing here.

The instrumentation above is real and it is enough to show you whether your content is dark or pulling. What it does on its own is diagnose. Making it optimize is a different document.

About the Author

Brian Curry is a Kansas City-based data scientist and applied AI researcher with two decades of enterprise experience across retail, agriculture, energy, hospitality, telecommunications, and media. He is the founder of Vector1 Research, an independent lab focused on applied AI, causal inference, agentic systems, and the economics of intelligent organizations. His work centers on building systems that surface the causal and structural drivers of performance — bridging advanced analytics, machine learning, and executive decision-making. He writes regularly on Medium about applied AI, knowledge engineering, decision science, and the engineering disciplines that make machine learning actually work in production.