Beyond the Warehouse: A Reference Architecture for Production AI

Your team trained the model. It works in the notebook. Now you have to feed it in production, monitor it, retrain it on fresh data, and serve features in milliseconds without breaking the rest of the company. Your data warehouse cannot do that. This is what does.

1. The warehouse stops being enough

The data warehouse was designed for one thing: turn historical, structured data into dashboards. It batches once a day or once an hour, stores tables in columnar formats, exposes SQL, and serves a BI tool. For analytics, this is fine.

For AI, almost every assumption breaks:

Latency. A fraud model needs feature lookups in under 50 ms; the warehouse responds in seconds.
Modality. Text, images, audio, and embeddings are not tables. The warehouse cannot index them usefully.
Throughput. A recommender does a million inference calls a minute; the warehouse is sized for tens of analysts.
Freshness. A retrieval-augmented chatbot needs documents indexed minutes after they land, not tomorrow morning.
Reproducibility. Training a model six months from now on “the same data” requires point-in-time snapshots — something warehouses make hard.
Observability. Schema drift, distribution drift, prediction drift, and prompt regressions don’t show up in a query log.

The fix is not “a bigger warehouse.” It’s a layered platform where the warehouse becomes one consumer among many. The rest of this article walks through what those layers are, what tools fit each one, and the process that ties them together. I’ll be vendor-neutral by default and call out where a specific stack changes the choice.

2. The four data modalities, and what each demands

Before tools, agree on what data you actually have. Modern AI platforms move four kinds of data, each with different storage and access patterns.

Structured data — transactions, user profiles, events — lives in object storage under a table format and is read by scans and point lookups. Semi-structured data — clickstream JSON, application logs — uses the same substrate but is read mostly by scan. Unstructured data — documents, images, audio — sits as raw files in object storage and is read sequentially. Embeddings — dense vectors of the unstructured content — live in a vector index and are accessed by approximate nearest neighbor search.

Two observations matter. First, everything starts in object storage — the cheap, durable substrate. Second, embeddings are a derivative of unstructured data but treated as a first-class modality because the access pattern (approximate nearest neighbor search) demands its own infrastructure. Your platform must handle all four, not just the first.

3. A reference architecture: what sits where

The architecture below is the one most production AI shops converge on by year three, regardless of cloud or vendor. Named tools are representative; substitute equivalents freely.

Sources fan in through ingestion — Debezium plus Kafka for change data capture from OLTP databases, Airbyte or Fivetran for SaaS sources, Kafka or Kinesis for event streams, direct writes for documents and files — into the lakehouse: object storage (S3, GCS, or ADLS) plus an open table format (Iceberg, primarily).

Transformation happens in place: dbt or SQLMesh for SQL-shaped work, Spark or Trino for the heavier joins and the unstructured cases.

From the lakehouse, data fans out to four serving substrates: a feature store (Feast or Tecton) for ML features, a vector store (pgvector or Qdrant) for embeddings, a model registry plus serving runtime (MLflow with BentoML or KServe) for models, and the warehouse (Snowflake or BigQuery) for BI and analytics — now just one consumer among several.

Three cross-cutting layers wrap everything: a catalog and lineage system (Unity Catalog, Apache Polaris, DataHub, Atlan, or OpenMetadata) defining what exists and where it came from; orchestration (Dagster, Airflow, or Prefect) coordinating the pipelines; and observability (Great Expectations, Monte Carlo, Arize, Langfuse) instrumenting data, models, and prompts.

Two things separate this from a traditional analytics stack. First, the lakehouse — object storage plus an open table format — is the single source of truth, and the warehouse is one consumer, not the destination. Second, three new serving substrates branch off it: the feature store, the vector store, and the model registry/server. Each exists because a specific AI workload doesn’t fit a SQL query.

4. The storage substrate: object storage + open table format

Every modern data platform now puts everything cold in object storage and reads it through an open table format. This is the most important architectural decision and the one that buys you the most optionality.

Object storage (S3, GCS, ADLS) is essentially infinite, cheap (cents per GB-month), durable, and decoupled from compute. You can scale to petabytes without sizing a cluster.

Open table formats put ACID semantics on top of object storage so multiple engines can write and read consistently:

Apache Iceberg — broadest engine support (Spark, Trino, Flink, Snowflake, BigQuery, DuckDB), strongest schema-evolution story, vendor-neutral. The default choice today.
Delta Lake — excellent if you live on Databricks; the open spec has improved but the ecosystem is still Databricks-anchored.
Apache Hudi — strongest CDC/upsert story, narrower adoption.

Pick Iceberg unless you have a specific reason not to. Why this matters for AI: training jobs need to scan large columnar partitions efficiently; inference services need point lookups; both want the same physical files without ETL between them. Iceberg’s combination of column statistics, hidden partitioning, and time travel makes this work.

The data layout that pairs with the lakehouse is the medallion pattern:

Bronze — raw, append-only, ingested as-is from sources.
Silver — cleaned, deduplicated, joined into business entities.
Gold — aggregated and modeled for one specific consumer (a dashboard, a feature set, a training dataset).

Bronze lets you reprocess history when business logic changes. Silver is the canonical business model. Gold is purpose-built per consumer. AI pipelines mostly consume silver (for feature engineering) and gold (for training datasets).

5. Ingestion: batch, streaming, and change data capture

You need three ingestion patterns, not one.

Batch for SaaS sources where source-side change feeds don’t exist. Airbyte (open source, broadest connector library) and Fivetran (managed, more reliable) cover the long tail of CRM, billing, ads, and support tools. Don’t build these yourself.

Streaming for product events and clickstreams. Apache Kafka remains the standard; Redpanda is a Kafka-compatible alternative with lower operational overhead; AWS Kinesis or GCP Pub/Sub if you’re already deep in one cloud.

Change data capture (CDC) for operational databases. Debezium (Kafka Connect-based) streams row-level changes from Postgres, MySQL, MongoDB, and others as events. This is what lets you keep a near-real-time mirror of your OLTP state in the lakehouse without weekly full dumps.

A representative pipeline landing CDC events into Iceberg via Spark Structured Streaming:

from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json

spark = (SparkSession.builder
    .config("spark.sql.catalog.lakehouse", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.lakehouse.type", "rest")
    .config("spark.sql.catalog.lakehouse.uri", "http://catalog.internal:8181")
    .getOrCreate())

events = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "kafka:9092")
    .option("subscribe", "app.orders.cdc")
    .option("startingOffsets", "latest")
    .load())

parsed = (events
    .selectExpr("CAST(value AS STRING) AS payload")
    .select(from_json("payload", order_schema).alias("o"))
    .select("o.*"))

(parsed.writeStream
    .format("iceberg")
    .outputMode("append")
    .option("checkpointLocation", "s3://lakehouse/_checkpoints/orders_bronze")
    .toTable("lakehouse.bronze.orders"))

This is the bronze tier. Silver and gold are downstream transformations, not separate ingestion paths.

The two biggest mistakes in ingestion: trying to do everything via batch when one source clearly needs streaming, and building bespoke connectors for SaaS sources Airbyte and Fivetran already cover. Save engineering for sources nobody else has solved.

6. Transformation: the medallion pattern with dbt or SQLMesh

Bronze → silver → gold is the data-side architecture. The tools that build it:

dbt is the de facto standard for SQL-first transformation. Models declare their dependencies via ref(), the framework compiles a DAG, and the engine runs it. Tests, docs, and lineage come for free.
SQLMesh is the rising alternative: state-aware, with virtual environments, column-level lineage, and proper handling of incremental and slowly-changing models. Worth adopting on a green field; harder to migrate to from dbt.
Spark / Databricks notebooks for transformations that don’t fit in SQL — typically unstructured data, ML feature derivation, or expensive joins on very wide tables.

A representative silver model in dbt:

-- models/silver/orders_enriched.sql
{{ config(
    materialized='incremental',
    unique_key='order_id',
    on_schema_change='append_new_columns'
) }}

with orders as (
    select * from {{ ref('bronze_orders') }}
    {% if is_incremental() %}
      where ingested_at > (select max(ingested_at) from {{ this }})
    {% endif %}
),
customers as ( select * from {{ ref('silver_customers') }} ),
products  as ( select * from {{ ref('silver_products')  }} )

select
    o.order_id,
    o.placed_at,
    o.ingested_at,
    c.customer_id,
    c.segment,
    p.category,
    o.quantity * p.unit_price as gross_revenue
from orders o
join customers c on o.customer_id = c.customer_id
join products  p on o.product_id  = p.product_id

A few principles survive most arguments:

Idempotent and incremental wherever possible. Reruns must be safe; full refreshes must work.
Test the contract, not the data. Test that customer_id is unique and not null. Do not test "average order value is between $20 and $200" — that fires during a real shift and someone will silence it.
Don’t aggregate into silver. Silver is the modeled business entity. Aggregations are gold.
One model, one purpose. Long SQL files that do five things hide bugs.

For AI specifically, silver tables become feature sources and gold tables become training datasets. Both need to be reproducible: a model trained on “silver as of last Tuesday” must be retrainable a year from now on the same data. Iceberg’s time travel makes this trivial; in a warehouse without snapshotting, you’ll be rebuilding it yourself.

7. The feature store: when you actually need one

A feature store solves three problems most teams discover the hard way:

Online/offline parity. The feature computed in training is the feature computed at inference. Not “the same SQL run against a different table” — the same value.
Point-in-time correctness. When training a model that predicts churn at week T from features observed before T, the join must respect time. Naive joins leak the future.
Reuse across models. “Customer’s 30-day spend” should be one definition used by ten models, not ten slightly different ones.

The main tools:

Feast — open-source, lightweight, declarative. Right starting point for most teams.
Tecton — managed, opinionated, strong on streaming and real-time features.
Databricks Feature Store — tightly integrated if you’re already there.
Vertex AI Feature Store / SageMaker Feature Store — fine if you’ve committed to one cloud’s ML stack.

A Feast feature view declared once and shared across models:

from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64

customer = Entity(name="customer", join_keys=["customer_id"])

source_30d = FileSource(
    name="customer_spend_30d_source",
    path="s3://lakehouse/gold/customer_spend_30d/",
    timestamp_field="event_timestamp",
)

customer_spend_30d = FeatureView(
    name="customer_spend_30d",
    entities=[customer],
    ttl=timedelta(days=2),
    schema=[
        Field(name="spend_30d",     dtype=Float32),
        Field(name="orders_30d",    dtype=Int64),
        Field(name="last_category", dtype=Int64),
    ],
    source=source_30d,
    online=True,
)

Retrieval at inference time:

features = store.get_online_features(
    features=["customer_spend_30d:spend_30d",
              "customer_spend_30d:orders_30d"],
    entity_rows=[{"customer_id": 4711}],
).to_dict()

When you don’t need a feature store: a team with two or three batch models, predictions written to a table, no real-time inference. A well-designed dbt project is enough; a feature store will just be infrastructure you don’t use.

When you do need one: real-time inference, multiple teams sharing features, anything where point-in-time joins are non-trivial. The cost of training-serving skew compounds fast.

8. Vectors and RAG: a parallel pipeline for unstructured data

Retrieval-augmented generation has made vector infrastructure a first-class part of the platform. The pipeline mirrors the medallion pattern but operates on unstructured data:

Bronze — raw documents land in object storage (PDFs, HTML, markdown, transcripts).
Silver — parsed, normalized, chunked text with metadata.
Gold — embeddings indexed in a vector store.

Vector store choices:

pgvector (Postgres extension) — start here. Most teams’ “vector search problem” fits comfortably below 10M vectors and benefits from being co-located with the relational data the chunks reference.
Qdrant, Weaviate, Milvus — open-source, dedicated, scale to hundreds of millions of vectors. Pick when pgvector’s latency or recall degrades.
Pinecone — managed, fast, expensive at scale. Pick when you don’t want to run anything yourself.

A canonical chunk-embed-upsert pipeline:

import psycopg
from langchain_text_splitters import RecursiveCharacterTextSplitter
from openai import OpenAI

client = OpenAI()
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)

def index_document(doc_id: str, text: str, metadata: dict, conn):
    chunks = splitter.split_text(text)
    resp = client.embeddings.create(
        model="text-embedding-3-large",
        input=chunks,
    )
    vectors = [e.embedding for e in resp.data]

    with conn.cursor() as cur:
        cur.execute("DELETE FROM doc_chunks WHERE doc_id = %s", (doc_id,))
        cur.executemany(
            "INSERT INTO doc_chunks "
            "(doc_id, chunk_ix, text, metadata, embedding) "
            "VALUES (%s, %s, %s, %s, %s)",
            [(doc_id, i, c, metadata, v)
             for i, (c, v) in enumerate(zip(chunks, vectors))],
        )
    conn.commit()

The non-obvious parts of doing this well:

Chunking strategy matters more than embedding model choice. Recursive splitting on natural boundaries beats fixed-size in almost every case. Tune chunk size to your retrieval window, not to a default.
Hybrid search wins. Combine vector similarity with BM25 keyword search; you’ll catch both semantic and exact-match queries. Postgres can do both natively via tsvector plus pgvector.
Embeddings are versioned. Changing the embedding model invalidates every vector in your store. Treat the embedding model as part of your release artifact and plan for re-embedding when it changes.
Evaluate retrieval separately from generation. A bad RAG answer is usually a bad retrieval. Hold out a labeled set of (query, relevant-chunk) pairs and track recall@k as the pipeline changes.

9. Agentic systems: tools, memory, and trajectories

The shift from RAG to agentic systems is moving from one-shot retrieval to multi-step execution. A RAG app retrieves, generates, returns. An agent plans, calls tools, observes results, and decides what to do next — sometimes for dozens of steps before returning anything to the user. Three architectural primitives get added to the platform.

Tool registry. Agents act through tools — typed function calls into your systems. You need a place to define, version, schema, and permission them. The Model Context Protocol (MCP) has emerged as the de facto interface, so any agent framework can call the same tool servers. Whether you use MCP, native function calling in LangGraph, the OpenAI Agents SDK, the Claude SDK, or your own internal RPC, the discipline is the same: tools are versioned artifacts with schemas, owners, and access policies, just like models and features.

The hard part is what you expose. A query_orders(customer_id) tool against a curated read-only view is safe by default. A generic execute_sql(query) tool against your production warehouse is a foot-gun — prompt injection in any input the agent retrieves can turn into a dropped table. Narrow tool surface, narrow blast radius.

Memory. Agents need two kinds. Short-term memory is the conversation transcript — a few KB, usually held in process. Long-term memory survives across sessions: preferences, learned facts, prior outcomes. Long-term is typically a hybrid: a vector store for semantic recall (the same pgvector or Qdrant from the RAG section) plus a relational table for structured facts. The pattern is to retrieve the relevant slice of memory at the start of the turn, then write back whatever updates the turn produced at the end.

Trajectories. A RAG call has one prompt and one completion. An agent run has a trajectory: planner thoughts, N tool calls with arguments and results, intermediate model calls, the final answer. Observability has to capture all of it. LangSmith, Langfuse, and Arize Phoenix model this as nested spans; without it, debugging an agent failure is impossible — you can’t tell whether the model planned poorly, the tool returned junk, or the final synthesis hallucinated.

A minimal tool-using loop, framework-agnostic, with tracing as a first-class concern:

def run_agent(user_msg, tools, model, max_steps=10):
    messages = [{"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user",   "content": user_msg}]

    with tracer.start_as_current_span("agent.run"):
        for step in range(max_steps):
            with tracer.start_as_current_span(f"step.{step}"):
                resp = model.call(messages, tools=tool_schemas(tools))

                if not resp.tool_calls:
                    return resp.content

                for tc in resp.tool_calls:
                    with tracer.start_as_current_span(f"tool.{tc.name}"):
                        result = tools[tc.name].handler(**tc.arguments)
                    messages.append({"role": "tool",
                                     "tool_call_id": tc.id,
                                     "content": json.dumps(result)})

    raise StepLimitExceeded(max_steps)

Four design points worth internalizing:

Every step is a span. Tracing is not optional; it is the substrate. Without it you cannot debug, evaluate, or cost-attribute an agent run.
A step limit is mandatory. Unbounded loops are how agents burn through six-figure model bills overnight.
Writes are different from reads. Read tools can run on every step. Write tools need explicit confirmation, idempotency keys, a rollback story, and human approval for high-stakes operations.
Treat retrieved content as untrusted input. A document the agent pulled in via RAG can contain instructions targeting the agent. The only defense is least-privilege at the tool layer — assume any input the agent ingests can be hostile.

Evaluation gets harder. A model eval scores one answer. An agent eval scores a trajectory: did it pick the right tools, with valid arguments, in a sensible order, and arrive at the right answer at acceptable cost? Three dimensions, tracked separately — task success (did the final answer match ground truth), tool-use correctness (were the right tools called with valid arguments), and efficiency (how many steps, tokens, and dollars). A bad agent often gets the answer right but takes 40 steps to do it; optimizing only on task success rewards exactly that pathology.

The platform implications are concrete: add a tool registry alongside the model and prompt registries; reuse the vector store for long-term memory; upgrade observability from prompt-completion logging to trace-based span logging; treat tool permissions and rate limits as policy the agent runtime enforces, not as an afterthought.

10. Training and serving: closing the online/offline gap

The biggest production AI failure is the same one repeated everywhere: the model that worked in training does not work in production because the features it sees are different. The platform’s job is to make that impossible.

Model registry. Every trained model goes through a registry — MLflow is the open-source standard; Weights & Biases, SageMaker Model Registry, and Vertex Model Registry are managed equivalents. The registry holds the artifact, the training run, the dependencies, the metrics, and the lineage of the data the model saw.

import mlflow
import mlflow.sklearn

with mlflow.start_run(run_name="churn_xgb_v3"):
    mlflow.log_params(params)
    mlflow.log_metric("auc_holdout", auc)
    mlflow.log_metric("auc_oot", auc_oot)
    mlflow.set_tag("training_data_iceberg_snapshot", snapshot_id)
    mlflow.sklearn.log_model(
        model,
        artifact_path="model",
        registered_model_name="churn",
    )

The snapshot tag is the line that often gets skipped and the one that saves you a year later. With Iceberg, that ID resolves back to exactly the rows the model trained on.

Serving. Three patterns:

Batch scoring — write predictions to a table on a schedule. Cheapest; appropriate when you have hours of latency budget.
Online inference — a service (often BentoML, KServe, Ray Serve, or NVIDIA Triton) loads the model, pulls features from the feature store, and returns a prediction.
Edge / embedded — model runs in the client; rare and specialized.

A minimal BentoML service (using the 1.2+ class-based API), with the MLflow-registered model imported into BentoML’s model store:

import bentoml
import numpy as np
from bentoml.models import BentoModel

@bentoml.service(
    resources={"cpu": "2"},
    traffic={"timeout": 10},
)
class ChurnService:
    bento_model = BentoModel("churn:latest")

    def __init__(self):
        self.model = bentoml.mlflow.load_model(self.bento_model)
        # feature_store is initialised from the same registry / config
        # that the training job used
        from feast import FeatureStore
        self.fs = FeatureStore(repo_path="feature_repo")

    @bentoml.api
    def predict(self, customer_id: int) -> dict:
        feats = self.fs.get_online_features(
            features=["customer_spend_30d:spend_30d",
                      "customer_spend_30d:orders_30d"],
            entity_rows=[{"customer_id": customer_id}],
        ).to_df()
        score = float(self.model.predict(feats)[0])
        return {"customer_id": customer_id, "score": score}

The point isn’t this exact code. The point is: features come from the same feature store that produced the training data, the model came from the registry, and both are versioned. That’s how the gap closes.

For LLM workloads, the equivalent of a model registry is a prompt registry — same idea, applied to prompt templates, system messages, and the model + parameters they target. Humanloop, PromptLayer, and the prompt-management features inside LangSmith and Langfuse all play this role. The discipline is the same: version the artifact, log the version with every call, and you can debug a regression by diffing prompts the way you’d diff model weights.

11. Observability: data, models, and prompts

You need three observability layers because three different things can break.

Data observability — schema, freshness, volume, distribution. The pipeline is silent until a column is renamed upstream and your model starts predicting nonsense.

dbt tests, Great Expectations, Soda — declarative, run inside the pipeline.
Monte Carlo, Bigeye, Acceldata — managed, anomaly detection across the warehouse and lakehouse.

A minimal Great Expectations check inline with your transformation:

import great_expectations as ge

ds = ge.from_pandas(silver_orders_df)
ds.expect_column_values_to_not_be_null("customer_id")
ds.expect_column_values_to_be_unique("order_id")
ds.expect_column_values_to_be_between("gross_revenue", 0, 1_000_000)
result = ds.validate()
assert result["success"], result

Model observability — prediction drift, accuracy decay, latency, fairness. Arize, WhyLabs, Fiddler, Evidently all do this. Log predictions, log eventual outcomes when they arrive (labels often come days or weeks later), compare.

LLM observability — every prompt, completion, retrieved chunk, token count, latency, and eval score. LangSmith, Langfuse, Arize Phoenix, Helicone all serve this role. The non-obvious part is that LLM observability is also an eval pipeline: production traces feed a labeled set, the labeled set feeds offline evals, offline evals gate the next prompt release. A simple loop:

def eval_rag(test_set, retriever, generator):
    rows = []
    for q, ref_answer, ref_chunk_id in test_set:
        chunks = retriever(q, k=5)
        answer = generator(q, chunks)
        rows.append({
            "q": q,
            "recall_at_5": any(c["id"] == ref_chunk_id for c in chunks),
            "answer_score": llm_judge(answer, ref_answer),
        })
    return pd.DataFrame(rows).agg(["mean"])

llm_judge is itself an LLM call returning a scalar — fragile in isolation, useful in aggregate. Track recall@k separately from generation quality; a bad RAG answer is almost always a bad retrieval, and you want to know which side is failing.

12. Governance: catalog, lineage, and access

The last layer is the one that gets neglected until something breaks and nobody can answer “what data did this model see?”

Catalog. Where every table, dataset, model, feature, and prompt lives, with descriptions, owners, and tags. Unity Catalog, Apache Polaris (the open Iceberg catalog), DataHub, Atlan, OpenMetadata, Collibra all play this role with different opinions. Pick one early; migrating is expensive.

Lineage. Column-level lineage matters when a model drifts and you need to trace the feature back to a transformation back to a source table back to an upstream system. dbt and SQLMesh emit lineage natively; OpenLineage is the open standard that lets multiple tools share it.

Access control. Row-level and column-level controls in the lakehouse — Iceberg plus the catalog enforces this; vendor implementations like Unity Catalog or AWS Lake Formation make it operable. The constraint that bites AI specifically: training data with PII may be legal for analytics but illegal for an external model. Build the masking into the silver layer, not into every downstream consumer.

A subtle but important point: governance is not a separate workflow you do after the pipeline runs. It’s metadata emitted by the pipeline. dbt models declare ownership and tags. Dagster and Airflow emit OpenLineage events. MLflow logs the data snapshot. Done right, the catalog and lineage assemble themselves from artifacts the pipeline produces anyway.

13. A workflow you can reuse

For a green-field platform, in order:

Pick the storage substrate first. Object storage in your primary cloud, Iceberg as the table format, a single catalog implementation (Polaris, Unity, or the cloud’s native option). Everything downstream is a consumer.
Establish the bronze tier with one ingestion path you control end-to-end — usually CDC from your operational database. Don’t try to ingest everything in week one.
Layer silver and gold with dbt or SQLMesh. Resist the urge to skip silver; the cleaned business-entity layer is what every downstream consumer reuses.
Wrap it in orchestration. Dagster if you want type-safe assets and great local DX; Airflow if you have existing ops familiarity. Either works.
Add the AI-specific layers only when a workload demands them. A feature store before any real-time model exists is infrastructure looking for a problem. A vector store before any RAG pipeline is the same. An agent runtime before you have a multi-step use case is the third version of the same mistake. Build to demand.
Bake observability in from day one. dbt tests on every silver/gold model. Great Expectations on every bronze landing. Arize or Langfuse on the first model that goes live. Adding observability later means reconstructing baselines after the platform has already drifted.
Treat governance as metadata, not process. If your catalog is something humans update separately from the pipeline, it will be wrong within a quarter.
Pick one cloud, one warehouse, one orchestrator, one transformation tool. The platform’s complexity grows faster than the number of tools.

The mistake that kills more AI platforms than any other isn’t choosing the wrong tool — it’s choosing too many. A team running Airflow and Dagster and Prefect, with three feature stores and four vector databases, isn’t a more flexible platform; it’s a more broken one. Decide, document, consolidate.

The platform’s job, at the end, is one thing: make it cheap to go from a question to a model to a feature to a decision, repeatedly, reproducibly, and without breaking the things already running. Everything in this architecture is in service of that single throughput goal.

The reference stack from this article: object storage (S3/GCS/ADLS) + Apache Iceberg for the lakehouse, Kafka + Debezium + Airbyte for ingestion, dbt or SQLMesh + Spark for transformation, Feast for features, pgvector or Qdrant for vectors, MLflow + BentoML for the model lifecycle, an MCP-shaped tool registry with LangGraph or the OpenAI / Claude agent SDKs for agentic systems, Dagster for orchestration, Great Expectations + Arize + Langfuse for observability, Unity Catalog or Apache Polaris for governance. Substitute equivalents freely — the architecture survives the swap.

About the Author

Brian Curry is a Kansas City-based Data Scientist, AI expert, and founder of Vector1 Research, where he explores the frontier of Artificial Intelligence, Causal Inference, Agentic Systems, and Economic Modeling.

His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics. He is the creator of PyCausalSim (causal discovery through simulation), Papilon (complex system optimization), Daedalus (causal and economic analysis), and Memory-Node Encapsulation (data structures for artificial episodic memory).