The Retrieval Twin: Simulating Query Fan-Out to Engineer AI Search Visibility
Every GEO tool on the market tells you whether you were cited. None of them can tell you why — or predict whether you will be. This article shows how to build that instrument yourself, in Python, by constructing a local simulation of the retrieval layer that AI search engines run against your content.
In The Answer Layer, I argued that a new layer has inserted itself between your content and your audience — an AI layer that retrieves, synthesizes, and answers, often without a click. And when I released MeaningFlow, the thesis was that content strategy has to be modeled in semantic space: embeddings, clusters, and graphs, not keywords and pages.
This article closes the loop between those two ideas. Because in mid-2026, we finally know — from Google’s own documentation — how the answer layer mechanically works. And once you know the mechanics, you can simulate them.
What Google finally told us
In May 2026, Google published its official guidance on optimizing for generative AI features in Search. Strip away the reassurances and two architectural admissions remain:
- AI Overviews and AI Mode are RAG systems. Retrieval-augmented generation running on top of the core Search index. The generative layer doesn’t know your content; it retrieves it, passage by passage, at answer time.
- The retrieval step uses query fan-out. The user’s question is decomposed into a set of concurrent, related sub-queries — Google’s own example expands “how to fix a lawn that’s full of weeds” into sub-queries like “best herbicides for lawns” and “how to prevent weeds in lawn.” Independent analyses put the fan-out at roughly 8–20 sub-queries per complex question, each retrieving its own source set before synthesis.
The same guide told everyone to stop doing things: llms.txt, artificial micro-chunking, AI-specific rewrites, and exotic schema are, per Google, not ranking factors for its generative features. AEO and GEO, in Google’s framing, are “still SEO.”
That framing is half right and strategically incomplete. Yes — the retrieval candidates come from the same index, shaped by the same quality systems. But the unit of selection changed. Multiple 2025 studies found that roughly two-thirds of pages cited in AI Mode sit outside the organic top ten, because fan-out retrieves passages matched to sub-queries, not pages matched to the head query. Your page can rank #4 for the head term and contribute nothing to the answer, while a competitor’s paragraph — buried on a page that ranks #23 — gets extracted and cited because it happens to be the best vector-space match for sub-query #7.
That is a fundamentally different optimization target. And it is a measurable one.
The instrumentation gap
The AI-visibility tool market is crowded and well-funded — Profound, Bluefish, Otterly, Peec, plus the AI modules bolted onto Semrush, Ahrefs, and Conductor. They all answer the same question: did my brand appear in the answer?
That’s presence measurement. It’s observational, it’s after-the-fact, and it’s noisy — cited-source sets churn 40–60% month to month on some surfaces. What none of these tools give you is the diagnostic layer:
- Which specific passages of mine are retrievable, and for which sub-queries?
- Where does my content lose to a competitor’s in embedding space — before any answer is ever generated?
- If I restructure this page, does its retrievability improve — before I publish?
Presence tools can’t answer these because they sit downstream of a black box. But the box is less black than it used to be. We know it’s dense retrieval over passages, driven by fan-out queries. Both of those components can be approximated locally with open-source parts. I call the result a retrieval twin: a local, inspectable model of how the answer layer sees your site.
Is it the same model Google or OpenAI runs? No — their rerankers, hybrid lexical/dense scoring, and quality signals are proprietary. But bi-encoder retrieval is the common architectural core of every RAG system, and the research literature is blunt about why it matters: a passage that isn’t retrieved cannot be cited, summarized, or reasoned over, no matter how good the generator is. Retrieval failure is unrecoverable downstream. If your content fails in a generic embedding space against plausible fan-out queries, that’s a strong prior that it fails in the production systems too — and unlike the production systems, the twin tells you exactly where and why.
Architecture of the twin
The pipeline has five stages, and if you’ve used MeaningFlow, the first three will feel familiar — this is the same semantic modeling stack pointed at a new question. MeaningFlow asks “where does demand exist without supply?” The retrieval twin asks “where does supply exist but fail to surface?” Together they cover both failure modes of AI visibility.
┌────────────────────────────────────────────────────┐ │ 1. INGEST crawl site → extract → clean │ ├────────────────────────────────────────────────────┤ │ 2. CHUNK heading-aware passage segmentation │ ├────────────────────────────────────────────────────┤ │ 3. EMBED bi-encoder → vector index │ ├────────────────────────────────────────────────────┤ │ 4. FAN-OUT LLM decomposes head queries into │ │ sub-query sets │ ├────────────────────────────────────────────────────┤ │ 5. SIMULATE retrieve per sub-query → score │ │ & SCORE retrievability, coverage, gaps │ └────────────────────────────────────────────────────┘
Let’s build it.
Stage 1–2: Ingest and chunk the way engines chunk
RAG systems don’t embed your page; they embed passages. Empirical work on AI Overview extraction converges on self-contained passages in roughly the 100–200 word range, bounded by semantic structure — which is why heading-aware chunking beats fixed-window chunking for this purpose. (Note this is modeling how engines segment, not the discredited tactic of artificially fragmenting your published content — Google explicitly says you don’t need to chunk your pages for it. Your twin chunks them to mirror what retrieval does internally.)
import re
import requests
from dataclasses import dataclass, field
from bs4 import BeautifulSoup
@dataclass
class Chunk:
url: str
heading_path: str # e.g. "Guide > Choosing skis > Ski length"
text: str
n_words: int = field(init=False)
def __post_init__(self):
self.n_words = len(self.text.split())
def extract_chunks(url: str, min_words: int = 40, max_words: int = 180) -> list[Chunk]:
"""Heading-aware passage extraction that approximates how
RAG pipelines segment a page for dense retrieval."""
html = requests.get(url, timeout=20).text
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "aside"]):
tag.decompose()
chunks, heading_stack, buffer = [], [], []
def flush():
text = " ".join(" ".join(buffer).split())
if len(text.split()) >= min_words:
# split oversized sections into passage-sized windows
words = text.split()
for i in range(0, len(words), max_words):
passage = " ".join(words[i:i + max_words])
if len(passage.split()) >= min_words:
chunks.append(Chunk(url, " > ".join(heading_stack), passage))
buffer.clear()
body = soup.body or soup
for el in body.find_all(["h1", "h2", "h3", "h4", "p", "li", "td"]):
if el.name in ("h1", "h2", "h3", "h4"):
flush()
level = int(el.name[1])
heading_stack[:] = heading_stack[:level - 1]
heading_stack.append(el.get_text(strip=True))
else:
buffer.append(el.get_text(" ", strip=True))
flush()
return chunksTwo design choices matter here. First, the heading_path travels with every chunk — prepending it at embedding time restores the context that segmentation destroys, mimicking how production systems carry document metadata into the passage representation. Second, the word-count window (40–180) is deliberately tuned to the passage sizes that extraction studies observe in the wild, not to what's convenient.
Stage 3: Embed into a retrieval index
Any strong open-source bi-encoder works; the e5 and bge families consistently benchmark at or near the top for retrieval, and — critically for a diagnostic tool — they’re reproducible. Same model, same vectors, forever.
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("intfloat/e5-base-v2")
def embed_chunks(chunks: list[Chunk]) -> np.ndarray:
# e5 convention: passages get a "passage: " prefix
texts = [f"passage: {c.heading_path}. {c.text}" for c in chunks]
vecs = model.encode(texts, normalize_embeddings=True,
batch_size=64, show_progress_bar=True)
return np.asarray(vecs, dtype=np.float32)
def embed_queries(queries: list[str]) -> np.ndarray:
texts = [f"query: {q}" for q in queries]
return np.asarray(model.encode(texts, normalize_embeddings=True),
dtype=np.float32)With normalized vectors, retrieval is a matrix multiply. For a site of even 50,000 chunks this runs in milliseconds on a laptop — no vector database required for the diagnostic use case:
def retrieve(query_vecs: np.ndarray, chunk_vecs: np.ndarray, k: int = 10):
"""Return (indices, scores) of top-k chunks per query."""
sims = query_vecs @ chunk_vecs.T # cosine similarity
idx = np.argsort(-sims, axis=1)[:, :k]
scores = np.take_along_axis(sims, idx, axis=1)
return idx, scoresStage 4: Simulate the fan-out
This is the stage that makes the twin a 2026 instrument rather than a 2023 semantic-search demo. You are not testing your content against the queries users type. You’re testing it against the queries the model generates on the user’s behalf.
Use an LLM to decompose each head query the way AI Mode does — into parallel sub-intents spanning definitional, comparative, procedural, evaluative, and constraint-specific angles:
import json
import anthropic
client = anthropic.Anthropic() # or any LLM client you prefer
FANOUT_PROMPT = """You are simulating the query fan-out step of an AI search
engine. Given a user question, generate the {n} concurrent sub-queries the
engine would issue to retrieve passages for a synthesized answer.
Cover distinct sub-intents: definitions, comparisons, how-to steps,
selection criteria, constraints (budget/skill/location), and freshness.
Sub-queries should read like search queries, not sentences.
User question: "{query}"
Respond with a JSON array of {n} strings only."""
def fan_out(head_query: str, n: int = 12) -> list[str]:
msg = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1000,
messages=[{"role": "user",
"content": FANOUT_PROMPT.format(query=head_query, n=n)}],
)
text = msg.content[0].text.strip().removeprefix("```json").removesuffix("```")
return json.loads(text)For a head query like “what skis should an intermediate skier buy,” a realistic fan-out set looks like: ski length guide intermediate, all-mountain vs carving skis, ski waist width explained, best intermediate skis 2026, ski flex rating meaning, how much do good skis cost, demo skis before buying, and so on. Each of those is an independent retrieval event. Each is a chance to be included in the answer — or excluded from it.
This also reframes keyword research. A head term with 1,000 monthly searches, routed through AI Mode at a 12-query fan-out, represents on the order of 12,000 retrieval events across a semantically related surface. The demand landscape MeaningFlow maps from query data is, in the fan-out era, an underestimate of the true retrieval surface — the sub-queries the models generate are demand that never appears in any keyword tool.
Stage 5: Score retrievability
Now run the simulation and compute the metrics that presence tools structurally cannot give you.
from collections import defaultdict
def simulate(head_queries: list[str], chunks: list[Chunk],
chunk_vecs: np.ndarray, k: int = 10, n_fanout: int = 12):
report = {
"chunk_retrievals": defaultdict(list), # chunk -> (subquery, rank, score)
"subquery_best": {}, # subquery -> best (score, chunk)
"uncovered": [], # sub-queries with weak best match
}
for hq in head_queries:
subs = fan_out(hq, n=n_fanout)
qv = embed_queries(subs)
idx, scores = retrieve(qv, chunk_vecs, k=k)
for si, sub in enumerate(subs):
best = float(scores[si, 0])
report["subquery_best"][sub] = (best, chunks[idx[si, 0]])
if best < 0.80: # calibrate per model; see note below
report["uncovered"].append((hq, sub, best))
for rank in range(k):
ci = int(idx[si, rank])
report["chunk_retrievals"][ci].append(
(sub, rank + 1, float(scores[si, rank])))
return reportFrom that raw output, three diagnostics fall out directly:
1. Chunk Retrievability Score (CRS). For each passage: across the full simulated fan-out surface, how often does it appear in a top-k retrieval set, and at what rank? A discounted sum works well:
def crs(retrievals: list[tuple]) -> float:
# reciprocal-rank credit, summed across all sub-queries that surfaced it
return sum(1.0 / rank for _, rank, _ in retrievals)Sort your entire site by CRS and the distribution is invariably brutal and clarifying: a small minority of passages does all the retrieval work, a long tail of passages is never retrieved for anything. Those dead passages aren’t necessarily bad writing — they’re usually passages whose meaning is smeared across page-level context that segmentation destroyed. They fail as self-contained units. That’s an editing instruction, not a mystery.
2. Fan-out coverage. For each head query: what fraction of its sub-queries found a strong match (above your calibrated similarity floor) somewhere on your site? This is the number that explains partial citation — why you appear in the answer’s pricing paragraph but not its comparison paragraph. You covered sub-queries 1–4 and 9; a competitor owned 5–8.
3. Semantic gaps, now retrieval-verified. This is where the twin composes with MeaningFlow. MeaningFlow’s pipeline — Sentence-BERT embeddings, UMAP reduction, HDBSCAN clustering, NetworkX graph — identifies clusters where audience demand exists without matching content supply. The retrieval twin adds the verification layer: feed the uncovered sub-queries back through the same embedding space, cluster them, and you get a ranked map of retrieval-verified content gaps — not “we don’t have a page about X” but “across N simulated retrieval events in this semantic region, our best passage scored 0.71 while inclusion demands ~0.85.”
# Compose with MeaningFlow-style clustering to turn misses into a roadmap
import umap, hdbscan
def gap_map(uncovered: list[tuple]):
subs = [u[1] for u in uncovered]
vecs = embed_queries(subs)
reduced = umap.UMAP(n_components=5, metric="cosine",
random_state=42).fit_transform(vecs)
labels = hdbscan.HDBSCAN(min_cluster_size=4).fit_predict(reduced)
clusters = defaultdict(list)
for sub, label in zip(subs, labels):
if label >= 0:
clusters[label].append(sub)
# each cluster = one coherent content brief, sized by retrieval demand
return sorted(clusters.values(), key=len, reverse=True)Each resulting cluster is, in effect, a machine-generated content brief with a built-in business case: here are 9 sub-queries the answer engines will issue in this territory, here’s our current best score against each, here’s the passage structure that would win them.
The pre-publish loop
The single most valuable property of the twin is that it runs before publication. Draft a page, chunk it, embed it, inject the vectors into the index, and re-run the simulation:
def preflight(draft_html_or_text: str, url_stub: str,
chunks, chunk_vecs, head_queries):
new_chunks = extract_chunks_from_text(draft_html_or_text, url_stub)
new_vecs = embed_chunks(new_chunks)
merged_chunks = chunks + new_chunks
merged_vecs = np.vstack([chunk_vecs, new_vecs])
before = simulate(head_queries, chunks, chunk_vecs)
after = simulate(head_queries, merged_chunks, merged_vecs)
# diff: which sub-queries did the draft newly capture or improve?
return diff_coverage(before, after, offset=len(chunks))This converts GEO from a publish-and-pray discipline into an engineering loop with a test suite. Restructure a section, re-run, watch specific sub-query scores move. It’s unit testing for retrievability — and it costs pennies per run, because the only API calls are fan-out generation (cacheable) and everything else is local.
Calibration, caveats, and intellectual honesty
A diagnostic instrument you don’t understand the error bars on is worse than none, so:
The similarity floor is model-relative. The 0.80 threshold above is illustrative. Calibrate it empirically: take queries where you know you’re cited (from your presence tool) and queries where you know you’re not, run both through the twin, and find the score band that separates them. That calibration also validates the whole approach for your domain — if there’s no separation, the twin isn’t modeling your surface well and you should try a different encoder or chunking scheme.
Production retrieval is hybrid. Real systems blend dense retrieval with lexical signals (BM25 and descendants), rerankers, and — in Google’s case — the entire ranking and quality stack as a candidate filter. The twin models the dense core, which is the newest and least-understood component, but a passage from a site the quality systems distrust won’t be saved by great embeddings. The twin is necessary-condition analysis: it finds content that cannot win. Passing the twin doesn’t guarantee citation; failing it strongly predicts absence.
Fan-out sets are stochastic. The real engines generate different sub-queries per session, per user context, per day — one reason cited-source churn is so high. Run each head query’s fan-out several times and work with the union; the sub-intent structure is far more stable than the surface strings.
Google says you don’t need any of this. True, in the narrow sense: there is no “retrieval simulation” ranking factor, and the fundamentals — helpful content, crawlability, structure — remain the price of admission. But “the fundamentals still matter” and “the selection unit is now the passage” are both true, and only one of them is in the official guidance as an optimization target. The 68%-outside-top-ten finding is the empirical residue of that gap. The twin is how you optimize for the part Google described mechanically but declined to give you tooling for.
Why this matters more than another dashboard
The GEO market has organized itself around observation: track mentions, count citations, chart share-of-voice across engines. That’s necessary, and the funded players do it well. But observation without mechanism is astrology. Sources churn, answers vary per session, and the dashboards can only tell you that something changed, never what to do about it.
The retrieval twin inverts the stack. Instead of watching the black box’s outputs, you rebuild the box’s one legible component — dense passage retrieval under query fan-out — from open parts, and interrogate it directly. Every miss comes with a location (this chunk, this sub-query, this score deficit) and therefore an action.
It’s also, not incidentally, the same architectural move MeaningFlow made for content strategy: stop reasoning about content in the units humans see (pages, keywords, rankings) and start reasoning in the units machines actually process (vectors, passages, clusters, graphs). The answer layer thinks in embedding space. Your instrumentation should too.
The full pipeline in this article runs on a laptop with sentence-transformers, umap-learn, hdbscan, networkx, and one LLM API for fan-out generation. I'm folding the retrieval-simulation module into the MeaningFlow roadmap — if you build on it, break it, or calibrate it against your own citation data, I want to hear about it.
Brian Curry is the founder of Vector1 Research and the author of MeaningFlow, an open-source library for semantic content modeling and coverage gap analysis. He leads SEO/GEO and content strategy work at the intersection of traditional search and AI-driven discovery.
Code & prior work:
- MeaningFlow: https://github.com/Bodhi8/Meaning-Flow
- Vector1 Research: https://vector1.ai
- Prior article: The Answer Layer — on the AI layer between users and websites
References & further reading:
- Google Search Central — Google’s Guide to Optimizing for Generative AI Features on Google Search (May 2026): https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- Search Engine Journal — coverage of Google’s guide and the “AEO/GEO are still SEO” framing: https://www.searchenginejournal.com/googles-new-ai-search-guide-calls-aeo-and-geo-still-seo/575026/
- LLM Pulse — Generative Engine Optimization: The Complete Guide for 2026 (fan-out scale, passage-level selection, share of AI Mode citations outside top-10): https://llmpulse.ai/blog/geo-guide/
- Ekamoira — original research on query fan-out behavior, passage-length extraction findings, and fan-out volatility: https://www.ekamoira.com/blog/query-fan-out-original-research-on-how-ai-search-multiplies-every-query-and-why-most-brands-are-invisible
- Cirillo et al. — Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems (the retrieval-bottleneck argument): https://arxiv.org/pdf/2605.23618
- Tiger Data — reproducible open-source embedding evaluation methodology for RAG: https://www.tigerdata.com/blog/open-source-vs-openai-embeddings-for-rag
Brian Curry is a Kansas City–based cognitive systems architect and applied AI researcher working at the intersection of AI architecture, knowledge engineering, and intelligent-system design. He is the founder of Vector1 Research, an applied AI lab publishing on agent systems, AI evaluation, cognitive architecture, and the macroeconomics of AI-driven transformation. He has led applied AI and data initiatives across major enterprises including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, AT&T, and McClatchy, and founded KC AI Lab, a 1,500-member AI/ML community. His open-source work includes Papilon (causal inference and counterfactual simulation), PyCausalSim (causal discovery via agent-based simulation), MeaningFlow (semantic modeling and ontology), and Memory-Node Encapsulation (episodic memory for agentic systems). Contact: brian at vector1.ai
