The Hard Truth About Putting AI Agents Into Production
Why your brilliant multi-agent prototype will fail in production — and how to fix it before it does
You’ve built something amazing. Your multi-agent system demos beautifully. The CEO is impressed. The investors are excited. Time to ship it.
And then reality hits.
After months of working with production-grade multi-agent systems, I’ve learned that the gap between a compelling demo and a reliable production system isn’t a gap at all — it’s a chasm. The techniques that make demos work are often the exact techniques that cause production systems to fail.
Here’s what actually matters when you’re ready to get serious.
The Context Problem Nobody Talks About
Let’s start with the elephant in the room: context management.
Most agent tutorials teach you the “naive pattern” — append everything into one giant prompt. It works for demos. It’s intuitive. It’s also a ticking time bomb.
In production, this approach collapses under three simultaneous pressures:
Cost and latency spirals. Model cost and time-to-first-token grow with context size. When you’re shoveling raw conversation history and verbose tool payloads into the context window, your agents become prohibitively slow and expensive. That snappy demo response? It becomes a 30-second wait in production.
Signal degradation. Researchers call this “lost in the middle” — when your context window is flooded with irrelevant logs, stale tool outputs, or deprecated state, the model gets distracted. Your agent starts making decisions based on information from three conversations ago instead of what the user just said.
Unpredictable failures. A large context window will help context-related problems, but it won’t solve them. When your agent fails because of context pollution, good luck debugging it.
The solution isn’t a bigger context window. It’s context engineering — treating context as a first-class system with its own architecture, lifecycle, and constraints.
The Three Principles That Actually Work
After extensive work scaling complex single- and multi-agent systems, I’ve found that successful production architectures share three design principles:
1. Separate Storage From Presentation
This is counterintuitive but crucial: what you store should not be what the model sees.
Think of it like a database and a view. Your “Session” is your ground truth — the complete record of everything that’s happened. Your “working context” is a computed projection optimized for the current model call.
This separation gives you flexibility. You can evolve your storage schemas without breaking your prompts. You can optimize what the model sees without losing historical data. You can experiment with different context representations without rewriting your entire system.
2. Make Transformations Explicit
Context should be built through named, ordered processors — not ad-hoc string concatenation.
When something goes wrong (and it will), you need to know exactly how the context was constructed. Which events were included? What got summarized? What was filtered out?
If you’re building context through a series of unclear transformations scattered across your codebase, debugging becomes archaeology. Make the “compilation” step observable and testable.
3. Scope By Default
Every model call and sub-agent should see the minimum context required. Agents should reach for more information explicitly via tools rather than being flooded by default.
This is a philosophical shift. Instead of “give the agent everything and let it figure out what’s relevant,” you’re saying “give the agent nothing and make it ask for what it needs.”
It’s more work upfront. It’s also the difference between systems that scale and systems that don’t.
Context Compaction: Your Secret Weapon
Here’s a technique that solves one of the most common production problems: what do you do when conversations get really long?
If you keep appending raw events indefinitely, latency and token usage spiral out of control. But you can’t just truncate — you’d lose critical context.
The answer is context compaction. When a configurable threshold is reached (like a certain number of conversation turns), trigger an asynchronous process that:
- Uses an LLM to summarize older events over a sliding window
- Writes the summary back as a new event
- Prunes or de-prioritizes the raw events that were summarized
The key insight: compaction operates on your storage layer, not your presentation layer. The benefits cascade downstream. Sessions remain manageable even for extremely long-running conversations. Your context stays relevant. Your costs stay predictable.
Cache-Friendly Architecture
Modern models support prefix caching, which lets the inference engine reuse attention computation across calls. This can dramatically reduce latency and cost — but only if you design for it.
The architecture divides your context window into two zones:
Stable prefixes: System instructions, agent identity, and long-lived summaries. These rarely change between calls.
Variable suffixes: The latest user turn, new tool outputs, and small incremental updates. These change on every call.
Keep frequently reused segments stable at the front of the context window. Push highly dynamic content toward the end. Simple in theory, but it requires disciplined architecture.
For critical system prompts that absolutely must not change, enforce immutability. A static instruction primitive guarantees your cache prefix remains valid across invocations.
Multi-Agent Architecture: Lessons Learned
If you’re building anything complex, you’re probably thinking about multi-agent systems. Here’s what I’ve learned:
Hierarchy matters. The foundation for structuring multi-agent systems is the parent-child relationship. You create a tree structure where child agents are passed to their parent. This makes delegation, specialization, and coordination natural.
Workflow agents are underrated. Not every agent needs an LLM. Workflow agents (Sequential, Parallel, Loop) control execution patterns without model calls. They’re faster, cheaper, and more predictable.
AgentTool changes everything. Instead of treating specialized agents as destinations you transfer control to, treat them as tools in your root agent’s toolbox. The root agent can then reason about complex queries and decide which specialists to use — and in what combination.
Parallel execution isn’t just about speed. When you need flight details AND hotel information, fetch them simultaneously. A ParallelAgent lets you “fan out” to multiple specialists, then “gather” all results. Your users don’t wait for sequential API calls.
Self-review closes the loop. Add review agents that evaluate outputs against quality guidelines. A TripSummaryReviewer that outputs “pass” or “fail” lets your system catch its own mistakes before users do.
The Production Checklist
Before you deploy, ask yourself:
Context management:
- Is storage separated from presentation?
- Are context transformations explicit and observable?
- Does each agent call see minimum required context?
- Is there a compaction strategy for long conversations?
- Is the architecture cache-friendly?
Multi-agent design:
- Is the agent hierarchy clear?
- Are workflow agents used where LLMs aren’t needed?
- Can specialists be used as tools rather than transfer destinations?
- Can independent operations run in parallel?
- Is there a self-review mechanism?
Operational concerns:
- How will you debug context-related failures?
- What happens when context compaction fails?
- How do you handle partial agent failures?
- What are your latency and cost budgets?
The Mindset Shift
Here’s the real insight: building production AI agents is software engineering, not prompt engineering.
The teams that succeed treat context as a system. They design explicit architectures. They build observable pipelines. They think about failure modes before they happen.
The teams that struggle keep iterating on prompts, wondering why their brilliant demo doesn’t work at scale.
The techniques that impress in demos — massive context windows, everything-everywhere approaches, clever prompt hacks — are technical debt in production. The techniques that feel like overkill during prototyping — tiered storage, explicit transformations, minimum-scope design — are what let you sleep at night when real users depend on your system.
Production isn’t a phase. It’s a discipline.
Part 2: The Practical How-To Guide
Now let’s get concrete. Here’s how to actually deploy AI agents to production across major cloud providers and frameworks.
Choosing Your Stack: A Decision Framework
Before diving into specific platforms, understand what you’re optimizing for:
Priority Best Fit Speed to production + managed infrastructure AWS AgentCore, Azure Foundry Agent Service, Google Vertex AI Agent Engine Maximum control + open-source flexibility LangGraph + self-hosted, CrewAI + custom infrastructure Enterprise security + compliance AWS AgentCore, Azure Foundry (Microsoft ecosystem integration) Multi-agent orchestration complexity Google ADK, LangGraph No-code/low-code deployment CrewAI Studio, Azure Foundry visual tools
Google Cloud: ADK + Vertex AI Agent Engine
Google’s Agent Development Kit (ADK) is the open-source framework powering Google’s internal agent systems. It’s designed for multi-agent architectures from day one.
High-Level Architecture
┌─────────────────────────────────────────────────────────────┐
│ Your Application │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Google Agent Development Kit │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ LLM Agent │ │ Workflow │ │ Custom │ │
│ │ (Gemini) │ │ Agents │ │ Agents │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Session Manager │ │
│ │ (Context compaction, memory, state management) │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Vertex AI Agent Engine (Managed Runtime) │
│ • Auto-scaling • Enterprise security │
│ • Session persistence • Monitoring & observability │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Tool Integrations │
│ MCP Servers │ 100+ Connectors │ BigQuery │ AlloyDB │
└─────────────────────────────────────────────────────────────┘Key Concepts
Agent Hierarchy: Build trees of specialized agents. Parent agents delegate to child agents through sub_agents. Each agent can have its own model, tools, and instructions.
Workflow Agents: Use SequentialAgent, ParallelAgent, and LoopAgent for predictable execution patterns without LLM calls. Faster, cheaper, more deterministic.
AgentTool Pattern: Instead of transferring control to sub-agents (losing context), wrap agents as tools. The root agent decides which specialists to invoke and in what combination.
Production Deployment Path
- Develop locally with ADK’s debugging tools
- Containerize your agent application
- Deploy to Vertex AI Agent Engine for managed scaling
- Connect tools via MCP or pre-built connectors
- Monitor with built-in observability through Cloud Trace
When to Choose Google ADK
- You need sophisticated multi-agent orchestration
- You’re already in the Google Cloud ecosystem
- You want to use Gemini models with optimized integration
- Context engineering (compaction, caching) is critical for your use case
AWS: Bedrock AgentCore
AWS AgentCore is a comprehensive platform that went GA in late 2025. It’s designed to eliminate the infrastructure complexity of deploying agents at scale.
High-Level Architecture
┌─────────────────────────────────────────────────────────────┐
│ Your Agent Code │
│ (CrewAI, LangGraph, Strands, OpenAI SDK, etc.) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ AgentCore Runtime │
│ • MicroVM isolation per session │
│ • Auto-scaling (0 to thousands) │
│ • Support for 8-hour async workloads │
│ • Container or code-zip deployment │
└─────────────────────────────────────────────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ AgentCore │ │ AgentCore │ │ AgentCore │ │ AgentCore │
│ Memory │ │ Gateway │ │ Identity │ │ Observability│
│ │ │ │ │ │ │ │
│ • Session │ │ • API→MCP │ │ • OAuth │ │ • CloudWatch │
│ • Episodic │ │ • Lambda │ │ • Cognito │ │ • Tracing │
│ • Long-term │ │ • MCP servers│ │ • Entra ID │ │ • Metrics │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Built-in Tools │
│ Code Interpreter │ Browser Tool │ Web Search │
└─────────────────────────────────────────────────────────────┘Key Components
AgentCore Runtime: The deployment engine. Supports both quick code-zip uploads for rapid iteration and container-based deployment for complex configurations. Each session runs in an isolated microVM.
AgentCore Memory: Solves the “goldfish agent problem” with three memory types:
- Session memory: Within-conversation state
- Episodic memory: Learns from past interactions
- Long-term memory: Persistent knowledge across sessions
AgentCore Gateway: Transforms your existing APIs and Lambda functions into MCP-compatible tools. Also connects to existing MCP servers and third-party services.
AgentCore Identity: Integrates with Cognito, Entra ID, Okta for user authentication. Enables agents to securely access AWS resources and third-party tools on behalf of users.
Production Deployment Path
- Build your agent with any framework (LangGraph, CrewAI, etc.)
- Configure AgentCore services (Memory, Identity, Gateway)
- Deploy via console, CLI, or SDK (code-zip or container)
- Connect tools through Gateway
- Set policies to control agent actions (new in Dec 2025)
- Evaluate with AgentCore Evaluations for quality monitoring
When to Choose AWS AgentCore
- Framework-agnostic approach (use any agent library)
- Enterprise security requirements (VPC, PrivateLink, microVM isolation)
- You need sophisticated memory management out of the box
- Already invested in AWS infrastructure and IAM
Azure: Foundry Agent Service
Microsoft’s Foundry Agent Service integrates deeply with the Microsoft ecosystem, from Azure to Microsoft 365 to Teams.
High-Level Architecture
┌─────────────────────────────────────────────────────────────┐
│ Microsoft Foundry Platform │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Foundry Agent Service │ │
│ │ • Hosted Agents (managed runtime) │ │
│ │ • Model Router (dynamic model selection) │ │
│ │ • Built-in memory & state management │ │
│ └───────────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Foundry │ │ Foundry │ │ Control │ │
│ │ Tools │ │ IQ │ │ Plane │ │
│ │ │ │ │ │ │ │
│ │ • 1000+ tools│ │ • AI Search │ │ • Identity │ │
│ │ • MCP/A2A │ │ • RAG │ │ • Policy │ │
│ │ • OpenAPI │ │ • Knowledge │ │ • Observ. │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Microsoft │ │ Azure │ │ Agent 365 │
│ 365 │ │ Services │ │ │
│ │ │ │ │ • Lifecycle │
│ • Teams │ │ • Functions │ │ • Governance │
│ • Copilot │ │ • Logic Apps │ │ • Identity │
│ • SharePoint │ │ • Cosmos DB │ │ │
└──────────────┘ └──────────────┘ └──────────────┘Key Concepts
Hosted Agents: Deploy custom-code agents built with Microsoft Agent Framework, LangGraph, or other frameworks directly into a managed runtime. No containers or Kubernetes required.
Foundry Tools: Over 1,000 curated tools from Microsoft and partners. Supports MCP, A2A, and OpenAPI. Create private organizational catalogs for internal APIs.
Model Router: Dynamically selects the best model for each prompt, balancing cost, performance, and quality. Particularly useful for multi-agent systems.
Agent 365: The control plane for AI agents across Microsoft 365, Azure, and Windows. Applies the same identity, security, and compliance infrastructure used for human users.
Production Deployment Path
- Create a Foundry project in Azure
- Build agents using Microsoft Agent Framework or your preferred SDK
- Configure model deployment and tools
- Deploy to Hosted Agents runtime
- Publish to Microsoft 365 channels (Teams, Copilot) with one click
- Govern through Agent 365 and Entra Agent ID
When to Choose Azure Foundry
- Deep Microsoft 365 integration is required
- You want agents accessible through Teams and Copilot
- Enterprise governance (Entra ID) is non-negotiable
- You need access to both OpenAI and Anthropic models on one platform
LangChain: LangGraph + LangSmith Deployment
LangGraph is the open-source framework for building stateful, orchestrated workflows. LangSmith Deployment (formerly LangGraph Platform) handles production infrastructure.
High-Level Architecture
┌─────────────────────────────────────────────────────────────┐
│ LangGraph │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ State Graph Definition │ │
│ │ Nodes (agents, tools) + Edges (transitions) │ │
│ │ Checkpointing + Memory + Human-in-the-loop │ │
│ └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ LangSmith Deployment Options │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Cloud │ │ Hybrid │ │ Self-Hosted │ │
│ │ (SaaS) │ │ │ │ │ │
│ │ │ │ SaaS control │ │ Full VPC │ │
│ │ • Fastest │ │ Self-hosted │ │ deployment │ │
│ │ • Managed │ │ data plane │ │ • Helm charts│ │
│ │ • 1-click │ │ │ │ • EKS/GKE │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ LangSmith │
│ Observability │ Tracing │ Evals │ Monitoring │
└─────────────────────────────────────────────────────────────┘Key Concepts
State Graphs: Define agent workflows as graphs with nodes (processing steps) and edges (transitions). Built-in support for cycles, conditionals, and branching.
Checkpointing: Automatic state persistence. Enables time-travel debugging, human-in-the-loop workflows, and recovery from failures.
LangGraph Studio: Visual debugging environment. Inspect state at each step, test edge cases, rewind and replay failures.
Assistants: Templatize your cognitive architecture. Tools, prompts, and models become configurable without code changes.
Production Deployment Path
- Define your workflow as a LangGraph state machine
- Debug locally with LangGraph Studio
- Test with LangSmith evaluations
- Deploy with 1-click to LangSmith Cloud, Hybrid, or Self-Hosted
- Monitor with LangSmith tracing and observability
- Iterate using production traces to improve prompts and workflows
When to Choose LangGraph
- You need fine-grained control over workflow orchestration
- Complex state management and checkpointing are required
- Time-travel debugging is valuable for your use case
- You want the flexibility to self-host everything
CrewAI: Multi-Agent Orchestration
CrewAI focuses on role-based agent collaboration, where specialized agents work together like a team.
High-Level Architecture
┌─────────────────────────────────────────────────────────────┐
│ CrewAI Framework │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Crews │ │
│ │ Agent Teams with Roles, Goals, and Backstories │ │
│ │ Sequential │ Hierarchical │ Consensus │ │
│ └───────────────────────────────────────────────────────┘ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Flows │ │
│ │ Event-driven workflows with precise control │ │
│ │ Conditional logic │ State management │ │
│ └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ CrewAI AOP (Enterprise Suite) │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Control │ │ Tracing & │ │ Deploy │ │
│ │ Plane │ │ Observability│ │ Options │ │
│ │ │ │ │ │ │ │
│ │ • Manage │ │ • Logs │ │ • Cloud │ │
│ │ • Monitor │ │ • Metrics │ │ • On-premise │ │
│ │ • Scale │ │ • Traces │ │ • Self-host │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘Key Concepts
Crews: Teams of AI agents with autonomous decision-making. Each agent has a role, goal, backstory, and tools. Agents collaborate through natural language.
Flows: Production-ready event-driven workflows. Precise control over execution paths, conditional logic, and state management.
Crews + Flows: Combine autonomous agent teams with deterministic workflow orchestration. Best of both worlds.
Production Deployment Path
- Build crews in Python or use CrewAI Studio (no-code)
- Test locally with integrated debugging
- Push to GitHub (integrates with CrewAI platform)
- Deploy with one click to CrewAI Enterprise
- Monitor with built-in tracing and observability
- Train agents to improve over time
When to Choose CrewAI
- Role-based agent collaboration fits your mental model
- You want both autonomous crews and controlled flows
- Python-first development is preferred
- You need a simpler path from prototype to production
Cross-Platform Considerations
MCP and A2A: The Interoperability Standards
Regardless of which platform you choose, two protocols are becoming standard:
Model Context Protocol (MCP): Standardizes how agents connect to tools and data sources. Build an MCP server once, use it everywhere.
Agent-to-Agent (A2A): Enables agents from different frameworks and vendors to collaborate. Your LangGraph agent can talk to your CrewAI agent.
All major platforms (AWS, Azure, Google, LangChain, CrewAI) now support these protocols. Design your tools and agents with interoperability in mind.
The Hybrid Reality
In practice, most production systems are hybrid:
- Framework: LangGraph or CrewAI for orchestration logic
- Runtime: AWS AgentCore or Azure Foundry for deployment
- Models: Mix of providers based on task requirements
- Tools: MCP servers for standardized integrations
Don’t lock yourself into a single vendor. Build modular systems that can evolve.
Quick Reference: Platform Comparison
Feature Google ADK AWS AgentCore Azure Foundry LangGraph CrewAI Primary Strength Multi-agent architecture Framework-agnostic runtime Microsoft integration Workflow control Role-based teams Managed Runtime Vertex AI Agent Engine AgentCore Runtime Hosted Agents LangSmith Deployment CrewAI Enterprise Context Management Compaction, caching Memory service Built-in memory Checkpointing State management Tool Integration MCP, 100+ connectors Gateway, MCP 1000+ tools, MCP LangChain ecosystem Custom + integrations Deployment Options Cloud, containers Code-zip, containers Managed, containers Cloud, hybrid, self-host Cloud, on-premise Best For Sophisticated orchestration Any-framework production Enterprise Microsoft Maximum control Team-based collaboration
Final Thoughts
The tooling for production AI agents has matured dramatically. You no longer need to build everything from scratch. But you do need to make informed decisions about:
- Where your complexity lives: In orchestration logic? In tool integrations? In memory management?
- What you want to own: Full control with self-hosting? Or managed infrastructure with less customization?
- Where you’re going: Start with what works today, but design for interoperability tomorrow.
The platforms are converging on standards (MCP, A2A, OpenTelemetry). The frameworks are getting more capable. The managed services are getting more sophisticated.
The hard part isn’t choosing a platform anymore. It’s designing agent systems that actually work reliably. That’s where context engineering, multi-agent architecture, and production discipline come in.
Build for production from day one. Your future self will thank you.
If you’re building production agent systems, I’d love to hear what’s working for you — and what’s not. The field is evolving fast, and we’re all learning together.
About the Author
Brian Curry is a Kansas City-based Data Scientist, AI expert, and founder of Vector1 Research, where he explores the frontier of Artificial Intelligence, Causal Inference, Agentic Systems, and Economic Modeling.
His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics. He is the creator of PyCausalSim (causal discovery through simulation), Papilon (complex system optimization), Daedalus (causal and economic analysis), and Memory-Node Encapsulation (data structures for artificial episodic memory).
Contact: brian@vector1.ai
