Foundation Models in Production: Selection, Fine-Tuning, and the Build-vs-Buy Decision

The question gets asked at every technical review I’ve sat through in the last two years. We need to add an AI capability to this product. Do we call a frontier model API, fine-tune an open model, or train something from scratch?

The answer is almost never “it depends,” even though that’s the answer most people give. The decision has structure. It’s three axes, and they trade off against each other in predictable ways. Once you can name the axes, the answer to any specific instance falls out of where the use case lands on each one.

This article is the framework. It’s the same one I keep returning to because it actually works on production decisions, not just whiteboard exercises. The three axes are capability fit, cost profile, and control requirements. The cascade pattern that shows up across most of the systems in this series — rules first, fine-tuned encoder second, frontier model fallback third — is the worked example that makes the framework concrete. And the multimodal question, which is reshaping what’s possible at the boundary between text and everything else, gets its own treatment because it deserves one.

The three axes

Every model selection decision answers three questions. Most teams answer one of them well, one of them poorly, and forget the third entirely. The third is usually the one that bites.

Axis 1: Capability fit

Does the candidate model do the thing you need, on the data distribution you actually have, well enough to ship?

This is the axis everyone starts with, and it’s also the one teams overestimate the most. A model that scores 85% on a public benchmark is not a model that will score 85% on your corpus. The benchmark is a distribution. Your corpus is a different distribution. The gap between them is the gap between “looks promising in the demo” and “fails quietly in production.”

The only honest way to assess capability fit is to evaluate on your data. Build a small golden set — a few hundred examples is enough to start — labeled the way you’d want the model to label them. Run candidates against the golden set. Measure. The result is rarely what the public benchmark predicted, and the surprises are almost always in the same direction: worse than the benchmark, especially at the tails of the distribution where your edge cases live.

Capability fit also has a temporal dimension that gets ignored. Frontier models change. The model you evaluated in March is not the model serving requests in September, even if the version string is the same — the providers update underlying weights, change defaults, adjust safety filters. Self-hosted models don’t have this problem, which is a real consideration that doesn’t show up on cost comparisons.

Axis 2: Cost profile

What does this model cost to serve, at the volume and latency profile your use case actually has?

Frontier APIs price by token. Self-hosted models price by GPU hour, plus the engineering cost of operating them. The crossover point between the two — where it becomes cheaper to host your own model than to keep paying per token — depends on volume, latency requirements, and how much engineering capacity you have to spend on infrastructure.

A useful approximation: at low volume (say, under 100,000 requests per day, with relaxed latency), frontier APIs are almost always cheaper, because you don’t pay for idle capacity. At high volume (several million requests per day, with tight latency), self-hosted models are almost always cheaper, because the per-token economics of APIs at that scale are punishing. The middle is where the math gets interesting and the answer depends heavily on the specifics.

Cost is not just compute. Cost is the full envelope: compute, inference latency (which affects user experience and downstream system design), engineering time to operate, data transfer, observability, and the opportunity cost of the team building infrastructure instead of building features. A team that picks self-hosting because the spreadsheet shows it’s cheaper, then spends six months building serving infrastructure that doesn’t quite work, has chosen the more expensive option even though the spreadsheet said otherwise.

Axis 3: Control requirements

What level of control do you need over the model’s behavior, its data handling, and its evolution?

This is the axis most teams forget until an incident or an audit forces it. Control covers a cluster of related concerns:

Data handling. Can you send the model your data? For some data classes — regulated content, customer PII, internal financials — calling a third-party API is a contractual or regulatory non-starter, regardless of whether the API is technically capable.
Behavior stability. Can the model change underneath you? Frontier APIs evolve; downstream systems built against them inherit that evolution as silent behavioral drift.
Customization depth. Can you make the model do things the base model doesn’t do well? Fine-tuning a frontier model is constrained; fine-tuning an open model is unconstrained.
Failure transparency. When something goes wrong, can you investigate? With an API, you have observability into your own calls. With self-hosted, you have observability into the model itself.

Control requirements are the axis that flips otherwise-obvious decisions. A use case that looks like a clean fit for a frontier API turns into a self-hosted decision the moment “we can’t send this data outside our network” enters the conversation. A use case that looks expensive to self-host turns into a self-hosted decision the moment “this regulator wants reproducibility” enters.

How the axes compose

The three axes don’t combine through a formula. They combine through a decision pattern that I find myself drawing on whiteboards over and over:

Establish the floor on capability. What’s the minimum performance the use case needs to be viable? If no candidate model — frontier API, fine-tuned open, or anything else — clears that floor on your evaluation set, the answer is “not yet.” Don’t build around an inadequate model. Wait or scope down.
Eliminate by control requirements. If certain options are off the table for data handling or compliance reasons, eliminate them. This usually narrows the field faster than the cost analysis.
Choose by cost profile within the remaining options. Among the candidates that meet capability and control requirements, pick the one with the best cost profile at your projected volume.

The order matters. Capability and control are pass/fail filters; cost is the optimization. Teams that start with cost — “what’s the cheapest option that might work” — end up with systems that are cheap and don’t work. Teams that start with capability and forget control end up with systems that work and aren’t deployable.

The classification cascade

The most common application of this framework in the systems from earlier in this series is the classification cascade. It shows up in MeaningFlow’s content classification, in the input checks from Article 5, in the routing logic of agentic systems, and anywhere the question is “what kind of thing is this.”

The cascade is three layers:

Rules. Deterministic, fast, free. The first pass on every input.
Fine-tuned encoder. Cheap, fast, predictable. Handles the bulk of cases that rules can’t.
Frontier model fallback. Expensive, slow, capable. Handles the cases the encoder can’t.

The reason this pattern works is that real input distributions are deeply non-uniform. A small fraction of inputs are unambiguous and can be classified by a regex. A large fraction are typical and well-handled by a fine-tuned encoder model trained on your data. A small fraction are genuinely hard — novel, ambiguous, requiring reasoning — and benefit from a frontier model. Routing each input to the cheapest layer that can handle it is dramatically more economical than running every input through the most capable layer.

A sketch of the pattern:

def classify(content: str) -> Classification:
    # Layer 1: rules. If a deterministic rule matches, return immediately.
    rule_match = match_rules(content, RULE_REGISTRY)
    if rule_match is not None:
        return Classification(
            label=rule_match.label,
            confidence=1.0,
            decided_by="rules",
            cost_units=0.0,
        )

    # Layer 2: fine-tuned encoder. Fast, cheap, handles the bulk of cases.
    encoder_result = encoder_model.classify(content)
    if encoder_result.confidence >= ENCODER_CONFIDENCE_FLOOR:
        return Classification(
            label=encoder_result.label,
            confidence=encoder_result.confidence,
            decided_by="encoder",
            cost_units=encoder_result.cost_units,
        )

    # Layer 3: frontier model fallback. Expensive, but only invoked
    # when the cheaper layers couldn't decide with confidence.
    frontier_result = frontier_model.classify(
        content,
        candidate_labels=encoder_result.top_labels,
    )
    return Classification(
        label=frontier_result.label,
        confidence=frontier_result.confidence,
        decided_by="frontier",
        cost_units=frontier_result.cost_units,
    )

A few things about this pattern earn the space they take.

The frontier model gets the encoder’s top candidate labels as a hint. This is a small detail with a large effect. When the cascade falls through to the frontier, the encoder has usually narrowed the choice to two or three plausible labels — it just couldn’t decide between them with enough confidence. Passing those candidates to the frontier model focuses its reasoning on the actual ambiguity rather than asking it to consider every label in the taxonomy. The result is a faster, cheaper, more accurate frontier call.

The decision layer is recorded. Every classification carries which layer made the decision. This is operationally critical: when you investigate a misclassification, knowing whether the wrong call came from rules, the encoder, or the frontier model tells you where to fix it. Rules-layer mistakes get fixed in regex. Encoder-layer mistakes get fixed in training data. Frontier-layer mistakes get fixed in prompts or fall back to human review. Without the decision-layer attribution, every misclassification investigation starts from scratch.

Confidence floors are tunable per use case. A high-stakes classification (routing a customer complaint that triggers regulatory obligations) has a higher encoder confidence floor than a low-stakes one (tagging a piece of content for analytics). The same cascade pattern, with different thresholds, serves both.

The economics of this pattern are dramatic in practice. In production systems I’ve worked on, rules handle 30–50% of inputs at zero marginal cost. The encoder handles another 40–60% at near-zero cost. The frontier model handles the remaining 5–15% at meaningful cost — but because it’s only handling that fraction, the total cost of the cascade is a small fraction of the cost of running every input through the frontier model. The latency profile improves correspondingly: most inputs get answered in milliseconds, and only the genuinely hard ones pay the frontier round-trip.

This is the pattern that lets a knowledge engineering system scale to the volumes real enterprises produce. It’s also the pattern that makes the framework above concrete: each layer of the cascade represents a different point on the cost-capability-control space, and the cascade itself is the production-grade composition of those points.

Fine-tuning, in practice

The middle layer of the cascade — the fine-tuned encoder — deserves its own treatment, because fine-tuning is the part of the build-vs-buy decision that gets the most romanticized and the least understood.

Fine-tuning works well when three conditions hold:

You have enough labeled data. “Enough” depends on the task and the base model, but the floor is usually a few thousand high-quality labeled examples. Below that, you’re better off using the base model with prompt engineering or retrieval augmentation.
The task has a stable definition. If the labels you want to predict change every quarter — because the taxonomy is evolving, or business needs are shifting — fine-tuning becomes a treadmill. You retrain, redeploy, and the world has moved on by the time the new model hits production.
The base model is the right starting point. Fine-tuning amplifies what a model already does reasonably; it doesn’t conjure capabilities the base model lacks. Picking the right base model matters as much as the fine-tuning itself.

When those conditions hold, fine-tuning produces models that are smaller, faster, cheaper, and more accurate on your specific task than any frontier model. The economics are excellent. The control story is excellent — the model lives in your infrastructure, its weights are stable, and its behavior is reproducible.

When those conditions don’t hold, fine-tuning is a months-long detour that produces a model that’s worse than what you started with. The most common failure modes:

Insufficient training data. A few hundred examples doesn’t fine-tune a modern model; it overfits one. The fine-tuned model performs spectacularly on the training data and falls apart on production traffic.
Label noise. Fine-tuning trusts your labels. If 15% of them are wrong, you’re teaching the model to be 15% wrong. Label quality matters more than label quantity past a certain point.
Wrong base model. Fine-tuning a base model that’s marginally suited to the task produces a fine-tuned model that’s marginally suited to the task. The right base model is the one whose pre-training distribution most closely matches your domain.

The right way to approach fine-tuning is to treat it as the second move, not the first. First, use a frontier model with retrieval augmentation to establish what’s possible. Get the use case working at all. Then, once you understand the task and have collected production traffic with labels, fine-tune to drive cost down and latency down without sacrificing quality. Going straight to fine-tuning before you understand the task tends to produce a model that’s optimized for the wrong objective.

When to train from scratch (almost never)

Training a foundation model from scratch is not a build-vs-buy decision. It’s a decision to start a research lab. The capital, talent, data, and infrastructure required are not in the same category as fine-tuning, and approaching it as if it were is a way to spend tens of millions of dollars on a model that’s worse than the open-source alternatives that already exist.

The conditions under which training from scratch is justifiable are narrow:

The use case is genuinely outside the distribution that existing models cover (a specialized scientific domain with no public corpus, a non-text modality without strong existing models).
The data is uniquely yours and uniquely valuable, in volumes that existing pre-training pipelines couldn’t have ingested.
The economics work at the volumes you’re operating at, even after accounting for the engineering and research cost of the training infrastructure itself.

For almost all enterprise use cases, none of these conditions hold. The right move is to use an existing pre-trained model — open or closed — as the foundation, and invest the engineering capacity in the layers above it: retrieval, fine-tuning, evaluation, serving, safety. Those layers are where the differentiation actually comes from in production AI systems, not from owning the underlying weights.

Vision-language models and the multimodal turn

Until recently, the build-vs-buy framework above applied cleanly to text, with images and other modalities treated as separate problems handled by separate models. That’s no longer accurate. Vision-language models — models that natively reason about images and text together — have crossed the threshold of practical usability for enterprise content understanding, and they change the framework in specific ways.

What VLMs add: the ability to reason about content as it actually exists, rather than as a pre-extracted representation. A product page is a layout, not a blob of text. A financial document is a structured artifact with tables, signatures, stamps, and handwritten annotations. A support ticket includes screenshots that contain critical information the user didn’t transcribe. A VLM can take any of these as input and reason about them holistically, in ways that a pipeline of separate OCR + layout-detection + text-understanding models couldn’t.

This matters for the framework because it shifts where capability fit lives. A use case that previously required a custom pipeline — extract text from images, detect tables, classify regions, then run a text model — now has a single-model option that often outperforms the pipeline at lower engineering cost. The cost profile shifts too: VLM inference is more expensive per call than text-only inference, but the alternative is no longer a cheap text model — it’s an entire pipeline you’d otherwise have to build and maintain.

Two patterns are emerging as production-viable:

VLM as ingestion accelerator. During document ingestion, a VLM extracts structured data from complex documents in a single pass. The output goes into the same downstream systems as text extracted any other way — the chunks land in pgvector, the entities land in the knowledge graph, the metadata lands in the audit trail. The VLM is invisible to the rest of the architecture; it just produces better structured data than the alternatives.

VLM as runtime answerer for visual queries. When the user’s question involves an image — “what’s wrong with this dashboard,” “is this invoice in the right format,” “what does this error screen show” — the request goes to a VLM rather than a text-only model. The grounding and citation patterns from Article 3 still apply, with the addition that citations may reference image regions rather than text spans.

The same selection framework applies. Capability fit (does the VLM actually do the visual reasoning your use case requires, on documents that look like yours?). Cost profile (at your volumes, is per-call pricing or self-hosted GPU more economical?). Control requirements (can you send these documents to an external API, or do they have to stay on your infrastructure?). The answers may be different than for text-only use cases, but the questions are the same.

What hasn’t worked in my experience is treating VLMs as drop-in replacements for the text-only stack. The retrieval architecture, the embedding model, the chunking strategy — all of these need to be reconsidered when documents are being processed as visual artifacts rather than as text streams. The temptation to swap a VLM into an existing pipeline and expect everything to keep working is strong and should be resisted. VLMs change the architecture; they don’t slot into it.

What to take away

The model selection question has a structure: capability fit, cost profile, control requirements. Filter by capability and control, optimize on cost. The cascade pattern — rules, fine-tuned encoder, frontier API — is the production composition of those axes, with each layer occupying a different point on the cost-capability tradeoff and inputs routed to the cheapest layer that can handle them.

Fine-tuning is the second move, not the first. Train from scratch almost never. Use VLMs where the visual reasoning is the actual capability gap, not as a fashionable replacement for working text pipelines. And don’t let the model selection question crowd out the architectural questions that surround it — the model is one component in a system, and a great model in a poor system loses to a modest model in a good one.

That last point is the through-line for this entire series. The articles before this one — RAG architecture, agentic systems, safety and governance, serving infrastructure — are about the system the model lives in. This article is about the model itself. The combination is what makes a knowledge engineering system real.

What’s next

The final article in the series steps from architecture into measurement. The hardest question to answer about any knowledge engineering investment is whether it actually moved the needle — whether the taxonomy work, the retrieval improvements, the agent capabilities translated into measurable business outcomes. Article 8, Causal Attribution for Content Systems: Proving Knowledge Engineering Moves the Needle, brings the marketing-science lane into the series. It uses Papilon and the causal inference toolkit to show how knowledge engineering work can be evaluated with the same rigor a marketing analytics team would apply to a campaign — with confounders controlled for, counterfactuals estimated, and lift attributed to its actual source.

About the author

Brian Curry is a data scientist and AI researcher who builds systems that connect data to decisions — across marketing measurement, knowledge engineering, and applied ML infrastructure. He is the founder of Vector1 Research, an independent lab advancing knowledge engineering, marketing economics, and cognitive AI systems, and the creator of original research including Memory-Node Encapsulation (MNE) — a novel data structure for artificial episodic memory — and the accompanying NeoCortex-M architecture for memory-driven AI.

His open-source work focuses on the knowledge layer: MeaningFlow for semantic content modeling and gap analysis, and Papilon for causal inference and complex systems simulation. Both ship the patterns he applies in production — the taxonomies, classification systems, entity resolution pipelines, and semantic models that make search and content platforms intelligent.

Over twenty years he has led data and analytics initiatives at enterprises ranging from $4B to $125B — including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, AT&T, and McClatchy — spanning marketing analytics, SEO and search strategy, ML engineering, causal inference, and the editorial-engineering collaboration that turns unstructured content into structured understanding. He previously founded KC AI Lab, growing it to a 1,500-member AI/ML education community in Kansas City.

He writes about knowledge engineering, semantic systems, cognitive AI, and applied data science at medium.com/@brian-curry-research.

Contact: brian at vector1.ai · LinkedIn