Retrieval-Augmented Generation Done Right: Architecture, Pitfalls, and What Most Teams Get Wrong

The RAG demo works. You ask a question, the system pulls a relevant passage from your documents, the model writes an answer grounded in that passage. Everyone in the room is impressed. Someone says “ship it.”

Six months later, the system is quietly dying.

Users complain that it hallucinates. Answers cite documents that don’t exist. Retrieval returns the same five chunks no matter what you ask. When you add a new document type, everything gets worse. Nobody knows why.

This is the pattern. RAG systems that work in a demo and fail in production. The failure isn’t bad luck. It’s almost always the same mistake repeated across the industry: teams treat retrieval-augmented generation as a library call when it is actually an architecture problem.

This article is about how to build it as an architecture problem.

The library-call trap

Walk into most RAG projects and you will find roughly the same pipeline:

Some document loader reads PDFs into text.
A default chunker splits that text into 512-token windows with 50-token overlap.
An embedding API turns each chunk into a vector.
The vectors go into a vector database.
At query time, the user’s question is embedded, the top-k chunks are returned, and a large language model writes an answer.

Every step of that pipeline is defensible in isolation. Stacked together with defaults, they produce a system that breaks in specific, predictable ways:

Chunks contain half a sentence from one topic and half a sentence from another. The embedding model produces a vector that represents neither topic well. Retrieval returns chunks that are sort-of related to several questions and really related to none.
Exact-match queries fail silently. A user searches for “policy PR-442” and the system returns chunks about unrelated policies because dense vector search doesn’t privilege exact string matches.
Recently updated documents don’t surface. The metadata layer is an afterthought, so there’s no way to weight freshness or filter by date.
Generation hallucinates with confidence. The model was given a chunk that contains the phrase “revenue recognition” but not the actual policy update, so it reconstructs something plausible from its training data.

None of these failures show up in a demo. They all show up in production. They all come from treating the pipeline as “install framework, wire up defaults” rather than as a system where each layer needs to be designed against the properties of the next.

What RAG actually is

Underneath the marketing, RAG is three pipelines that happen to share data:

An ingestion pipeline that takes source documents and produces retrievable representations.
A retrieval pipeline that takes a query and returns relevant context.
A generation pipeline that takes context plus query and produces an answer.

The quality of the third pipeline is bounded by the first two. No prompting trick will recover an answer when the right chunk never made it into the top-k. Most teams over-invest in prompt engineering and under-invest in ingestion and retrieval, which is the exact opposite of where the leverage is.

Ingestion: the layer that decides whether any of this works

Chunking is the first and most consequential choice. It determines what the retrieval system is even capable of finding.

The default strategy — split on fixed token windows — is the worst option for almost every real corpus. It treats documents as undifferentiated streams of tokens. It doesn’t know that a section heading belongs with its body. It doesn’t know that a table row has meaning only in the context of the table. It will happily split the single sentence that contains the actual answer to a user’s question into two chunks, each of which independently represents the answer poorly.

What works better:

Structure-aware chunking. Respect the document’s native boundaries. Markdown has headers. PDFs have sections. HTML has semantic tags. Code has functions and classes. A structure-aware chunker produces chunks that look like units a human would recognize: a subsection, a function, a table, a bullet list. When a chunk is too large, it splits further — but never across a structural boundary unless forced.

Hierarchical chunking. Index at a fine granularity (paragraph or smaller) but store and return at a coarser granularity (section). Retrieval matches against the small chunk; generation sees the surrounding context. This solves the “the answer is in the chunk but the context needed to interpret it isn’t” problem without blowing up retrieval precision.

Semantic chunking. Use embedding similarity to detect topic boundaries. Adjacent sentences with high embedding similarity belong together; a sharp drop suggests a topic shift. This is computationally heavier and harder to tune, but it produces the cleanest chunks for prose-heavy corpora.

A sketch of a structure-aware markdown chunker that carries the heading path with every chunk:

from dataclasses import dataclass
from typing import Iterator

@dataclass
class Chunk:
    text: str
    source: str
    section_path: list[str]
    start_line: int
    end_line: int

def chunk_markdown(
    text: str,
    source: str,
    max_tokens: int = 400,
) -> Iterator[Chunk]:
    """
    Split markdown by heading structure. Within a section, split further
    only if the section exceeds max_tokens. Preserve the heading path on
    every chunk so retrieval has structural context.
    """
    lines = text.splitlines()
    section_path: list[str] = []
    buffer: list[str] = []
    buffer_start = 0

    def flush(end_line: int) -> Iterator[Chunk]:
        nonlocal buffer, buffer_start
        if not buffer:
            return
        chunk_text = "\n".join(buffer).strip()
        if chunk_text:
            yield Chunk(
                text=chunk_text,
                source=source,
                section_path=list(section_path),
                start_line=buffer_start,
                end_line=end_line,
            )
        buffer = []
        buffer_start = end_line + 1

    for i, line in enumerate(lines):
        if line.startswith("#"):
            yield from flush(i - 1)
            level = len(line) - len(line.lstrip("#"))
            title = line.lstrip("#").strip()
            section_path = section_path[: level - 1] + [title]
        else:
            buffer.append(line)
            if approx_tokens(buffer) >= max_tokens:
                yield from flush(i)

    yield from flush(len(lines) - 1)

def approx_tokens(lines: list[str]) -> int:
    return sum(len(l) // 4 for l in lines)

Notice what each chunk carries: the section path. When a chunk is retrieved for the query “what’s the policy on vendor onboarding,” the system knows the chunk came from ["Vendor Management", "Onboarding", "Approval Process"]. That structural context improves both re-ranking and generation.

Metadata matters. Every chunk should carry:

Source document identity (stable across re-ingestion)
Section or structural path
Timestamp of the source version
Access control tags if the corpus is multi-tenant
Content type (prose, table, code, list)

Metadata that gets added later is metadata that never gets added.

Embeddings: the choice that’s harder to undo

Changing your embedding model means re-embedding every chunk in your corpus. For a mid-sized corpus of a few million chunks, that’s hours of compute and a nontrivial bill. Teams that pick an embedding model casually tend to regret it.

Three properties matter:

Dimensionality. 384, 768, 1024, 1536, 3072 are all common. Higher dimensions tend to retrieve better but cost more in storage and ANN query time. The math is not subtle: 10M chunks at 1536 dimensions in float32 is 61 GB before any index overhead.

Domain fit. Public benchmarks like MTEB tell you how a model performs on average across a mix of tasks. They do not tell you how it performs on your corpus. A legal-domain corpus, a code corpus, a clinical corpus, and a marketing-content corpus will each have different winners. The only way to know is to measure.

Stability. Embedding providers update models. Hosted APIs sometimes change defaults silently. If retrieval quality regressing when a vendor ships an update would break your system, you need to version-pin or self-host.

The workflow that actually works:

Assemble a golden set of 100 to 300 query/passage pairs where you know the correct passage.
Embed your corpus with two or three candidate models.
Measure recall@k (usually k=10) on the golden set.
Pick the one that wins by the largest margin that justifies its cost.
Keep the golden set. Re-run it every time you consider a model change.

Most teams skip the golden set and pick the model their vendor recommends. Most teams have mediocre retrieval.

The vector store: pgvector vs dedicated systems

Here’s an opinion that will annoy some readers: pgvector is the right choice for most enterprise RAG systems, and the reflex to reach for a dedicated vector database is usually premature optimization.

What pgvector gives you:

A single operational surface (Postgres) that your team already knows how to back up, monitor, and replicate.
Transactional consistency between chunk text, metadata, and vectors (no “the vector updated but the metadata didn’t” class of bug).
Rich SQL filtering alongside vector search, which matters more than most teams realize.
Standard tooling, standard access control, standard everything.

When pgvector stops being enough:

Corpus above roughly 50 to 100 million vectors with sub-20ms latency requirements.
Highly selective pre-filters where the post-filtering approach used by pgvector’s HNSW implementation degrades recall.
Multi-tenant workloads where you need hard isolation at the storage layer.

The “pgvector is too slow” claim is usually wrong. It almost always traces back to no index, wrong index parameters, or running it on an undersized instance. With a properly tuned HNSW index, pgvector handles tens of millions of vectors comfortably.

A reasonable starting configuration:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id            bigserial PRIMARY KEY,
    source_id     text NOT NULL,
    section_path  text[] NOT NULL,
    content       text NOT NULL,
    content_tsv   tsvector GENERATED ALWAYS AS
                    (to_tsvector('english', content)) STORED,
    embedding     vector(768) NOT NULL,
    created_at    timestamptz NOT NULL DEFAULT now(),
    metadata      jsonb NOT NULL DEFAULT '{}'::jsonb
);

CREATE INDEX chunks_embedding_hnsw_idx
    ON chunks
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

CREATE INDEX chunks_content_tsv_idx ON chunks USING gin (content_tsv);
CREATE INDEX chunks_metadata_gin_idx ON chunks USING gin (metadata);
CREATE INDEX chunks_source_idx       ON chunks (source_id);

A few things to notice. The table carries a tsvector column for full-text search, generated from the content. That’s because you need both dense and sparse retrieval, and doing sparse retrieval in the same store keeps the join cheap. The HNSW parameters m=16, ef_construction=64 are reasonable defaults; tune ef_search at query time to trade recall against latency.

If you outgrow this — genuinely outgrow it, not imagine that you will — move to a dedicated vector store. Until then, one database is simpler than two.

Hybrid retrieval: the single biggest quality win

If you do one thing to improve a RAG system beyond the default, add sparse retrieval alongside dense.

Dense vectors encode semantic similarity. They excel when the query and the passage use different words for the same concept. They fail when the signal is exact-match: product codes, legal citations, proper nouns the embedding model has never seen, rare technical terms, error codes, ticket IDs.

BM25 (sparse) does the opposite. It matches on exact tokens, weighted by term frequency and inverse document frequency. It nails the exact-match cases and misses the paraphrase cases.

Running both and fusing the results recovers the best of each.

The simplest fusion method is Reciprocal Rank Fusion:

def reciprocal_rank_fusion(
    rankings: list[list[str]],
    k: int = 60,
) -> list[tuple[str, float]]:
    """
    Combine multiple ranked lists into a single ranking.
    `rankings` is a list of ranked id lists, one per retriever.
    Returns (id, score) pairs sorted by fused score.
    """
    scores: dict[str, float] = {}
    for ranked_ids in rankings:
        for rank, doc_id in enumerate(ranked_ids):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

Retrieve, say, the top 50 from each retriever. Fuse. Take the top 20 or 30 fused. Pass those into re-ranking. Simple, effective, and it removes an entire class of failure mode with maybe fifty lines of code.

At the SQL level, a hybrid retrieval query against the schema above looks like:

WITH dense AS (
    SELECT id, 1 - (embedding <=> :query_vec) AS score
    FROM chunks
    WHERE metadata @> :access_filter
    ORDER BY embedding <=> :query_vec
    LIMIT 50
),
sparse AS (
    SELECT id,
           ts_rank_cd(content_tsv, plainto_tsquery('english', :q)) AS score
    FROM chunks
    WHERE content_tsv @@ plainto_tsquery('english', :q)
      AND metadata @> :access_filter
    ORDER BY score DESC
    LIMIT 50
)
SELECT id, score, 'dense'  AS retriever FROM dense
UNION ALL
SELECT id, score, 'sparse' AS retriever FROM sparse;

The application layer does the RRF pass. You can also fuse in SQL if you prefer, but keeping it in application code makes it easier to add more retrievers later (a structured retriever over metadata, a graph retriever over relationships, and so on).

Re-ranking: turning top-50 recall into top-5 precision

Retrieval aims for recall — get the right passages into the top 50. Generation needs precision — have the right passages in the top 3 to 5, because that’s all that fits in the context budget without degrading the answer.

Cross-encoders bridge that gap. Where a bi-encoder (the thing producing your vectors) encodes query and passage independently, a cross-encoder takes the pair together and outputs a relevance score. That joint encoding captures interactions the bi-encoder cannot: negations, specific entity matches, subtle phrasing cues. It’s also much slower. You can’t run it over millions of passages, but you can run it over the 50 that made it through hybrid retrieval.

The pattern:

Hybrid retrieve top-50.
Cross-encode (query, passage) for each of the 50.
Sort by cross-encoder score.
Keep top-5 to top-10 for generation.

The measured quality jump from adding a cross-encoder re-ranker is usually larger than any other single change except adding hybrid retrieval. It’s also relatively cheap to integrate because it’s a single additional model call over a bounded candidate set.

Knowledge graphs: where retrieval meets meaning

Everything so far treats documents as bags of chunks with embedded semantic similarity. That representation works until you need to answer questions that depend on relationships between entities rather than similarity between texts.

“Which vendor contracts reference regulation 3.4.2 and expire in the next 90 days?”

No amount of hybrid retrieval answers that cleanly. The relationships — contract mentions regulation, contract has expiration date — are structural. They belong in a graph.

This is where the knowledge store developed earlier in this series earns its keep. The MeaningFlow pipeline from Articles 1 and 2 extracts entities and relationships during ingestion. Those land in Neo4j alongside the chunks that land in Postgres. The two stores reference each other: chunks carry the IDs of entities they mention; entities carry the IDs of chunks they appear in.

Three patterns show up in production:

Entity-scoped retrieval. Extract entities from the query. Look them up in the graph. Expand to immediately related entities. Scope vector search to chunks that mention any of those entities. This dramatically tightens retrieval when the query is about specific named things.

Graph-based re-ranking. Do hybrid retrieval first. Then use the graph to boost chunks that sit on shorter paths between the query’s entities. A chunk that connects two entities the query cares about is more relevant than a chunk that mentions one of them incidentally.

Path-traversal answering. For relationship questions (“who reports to whom,” “what depends on what”), the graph query is the primary retrieval and the chunks are the evidence. Run the Cypher query, return the matching entities, fetch the chunks that document each relationship, pass chunks plus graph result to generation.

A minimal integration looks like this. During ingestion, after extracting entities from a chunk, write both stores:

# Postgres: the chunk, its embedding, and its metadata
cur.execute(
    """
    INSERT INTO chunks
        (id, source_id, section_path, content, embedding, metadata)
    VALUES (%s, %s, %s, %s, %s, %s)
    """,
    (chunk_id, source_id, section_path, text, embedding, meta_json),
)

# Neo4j: the entities and their relationship to the chunk
session.run(
    """
    UNWIND $entities AS ent
    MERGE (e:Entity {id: ent.id})
      ON CREATE SET e.canonical_name = ent.name, e.type = ent.type
    MERGE (c:Chunk {id: $chunk_id})
    MERGE (e)-[:MENTIONED_IN {salience: ent.salience}]->(c)
    """,
    chunk_id=chunk_id,
    entities=entities,
)

At query time, entity-scoped retrieval looks roughly like:

def entity_scoped_retrieve(query_text: str, top_k: int = 30) -> list[str]:
    entities = extract_entities(query_text)
    if not entities:
        return hybrid_retrieve(query_text, top_k)

    entity_ids = [e.id for e in entities]
    related_chunk_ids = neo4j_session.run(
        """
        MATCH (e:Entity)-[:MENTIONED_IN]->(c:Chunk)
        WHERE e.id IN $entity_ids
        RETURN DISTINCT c.id AS chunk_id
        """,
        entity_ids=entity_ids,
    ).value()

    if not related_chunk_ids:
        return hybrid_retrieve(query_text, top_k)

    return hybrid_retrieve(query_text, top_k, candidate_ids=related_chunk_ids)

The fallback matters. If entity extraction produces nothing useful, or the graph has no matching entities, the system degrades to normal hybrid retrieval instead of returning empty.

This is what it means for the knowledge store to be the backbone of the retrieval system. The graph is not a side-channel decoration. It changes what the system can find.

Grounding and citations

Every answer the system produces should be traceable back to specific source passages. This is not a nice-to-have. It is the difference between a system you can debug, audit, and improve, and a system that works until one day it quietly doesn’t.

Practically:

Every chunk carries stable metadata about its source.
Retrieval returns that metadata alongside the chunk text.
The generation prompt instructs the model to reference chunks by an identifier (for example, [doc:42]).
The application parses those identifiers out of the response and renders them as citations linked to the source.

When a user reports a bad answer, you can look at exactly which chunks were retrieved and which the model chose to use. When an answer mentions a policy, a user can click through and read the policy. When compliance asks “how did the system produce this recommendation,” you have an answer.

Systems without a citation trace become systems that nobody fully trusts and nobody can fix.

Evaluation: the layer that keeps the rest honest

A RAG system that isn’t measured will degrade. Models change, corpora grow, query patterns shift, and the failure modes are quiet — bad answers look like good answers at a glance.

Three layers of measurement, from cheapest to most expensive:

Retrieval quality. Maintain a golden set of query/expected-chunk pairs. Measure recall@k and MRR on every change to chunking, embeddings, or retrieval. Run it in CI. When a pull request degrades recall@10 by five points, the pull request doesn’t merge without a reason.

Answer faithfulness. Given a generated answer and the retrieved chunks, does the answer only assert things supported by the chunks? This can be measured with a second model acting as a judge, with decent agreement against human eval on most domains. Not perfect, but cheap enough to run at scale.

End-to-end human review. A rotating sample of real queries, graded by people who know the domain. Small N, high signal. This is the layer that catches the problems the first two miss.

None of this is glamorous. All of it is what separates systems that stay good from systems that silently rot.

What to take away

RAG is not a library call. It is an architecture with five loosely coupled layers — ingestion, storage, retrieval, ranking, grounding — each of which interacts with the others and each of which fails in its own way. Teams that ship a library call succeed in demos and fail in production. Teams that build the layers deliberately, measure each one, and tune against their actual corpus ship systems that keep working.

The knowledge store matters more than the model. Good retrieval over a well-structured corpus, fed into a modest model, beats weak retrieval over a mush of chunks fed into the best model on the market. The leverage is in the layers most teams skip.

What’s next

The next article in this series takes the next step. Retrieval so far has been a one-shot call: question in, chunks out, answer back. Real agentic systems don’t work that way. They loop. They carry state across sessions. They decide when to retrieve, when to act, and when to ask. Article 4 introduces MNE as an architecture for agentic systems — how to keep an agent grounded when its context window runs out, and what memory, tools, and orchestration actually need to look like in production.