Ollama + AI Agents: How to Use, Deploy, and Orchestrate Local LLMs in 2026
Run your own AI agent stack — privately, locally, and for free.
If you’ve ever wanted to build AI agents without sending your data to the cloud or paying per-token API fees, you’re in the right place. Ollama has quietly become one of the most popular tools for running large language models on your own hardware, and the ecosystem around it has matured to the point where you can orchestrate entire multi-agent workflows locally.
In this guide, I’ll walk you through the full journey: installing Ollama, wiring it into agent frameworks, deploying with Docker, and orchestrating multi-agent systems that actually work.
What Is Ollama (and Why Should You Care)?
Ollama is a lightweight runtime for running open-source LLMs on your local machine. Think of it as Docker, but for language models. One command pulls a model, another serves it over a local API — no Python environments to wrangle, no CUDA drivers to install manually.
Why it matters for agents:
- Privacy by default. Your data never leaves your machine. No API keys, no telemetry, no third-party data processing.
- Zero cost at inference. Once you’ve downloaded a model, every request is free.
- OpenAI-compatible API. Ollama exposes endpoints that slot directly into most agent frameworks without custom adapters.
- Model flexibility. Swap between Llama 3, Mistral, CodeLlama, Phi, Gemma, and dozens of other models with a single command.
Getting Started: Install Ollama and Pull Your First Model
Installation is dead simple across platforms:
# macOS / Linux curl -fsSL https://ollama.com/install.sh | sh # Or on macOS via Homebrew brew install ollama
Once installed, pull a model and start serving:
# Pull a model (Llama 3.1 8B is a great starting point) ollama pull llama3.1 # Start the server ollama serve # Test it ollama run llama3.1 "Explain what an AI agent is in two sentences."
Ollama now listens on http://localhost:11434 — this is the endpoint your agent frameworks will connect to.
Choosing the Right Model
Not every model is suited for every task. Here’s a practical breakdown based on real-world usage:
Model Best For Size Notes Llama 3.1 8B General tasks, reasoning ~5 GB Great all-rounder, fast on most hardware Mistral 7B Chat, summarization ~4 GB Efficient and snappy for conversational agents CodeLlama 7B Code generation, debugging ~4 GB Purpose-built for development workflows Phi-3 Mini Lightweight tasks ~2 GB Runs well on modest hardware Llama 3.1 70B Complex reasoning ~40 GB Needs 64 GB+ RAM, but very capable
A good rule of thumb: start with the smallest model that gets the job done. You can always scale up.
Building Your First Agent with Ollama
An agent is more than a chatbot. It has a role, a goal, tools it can use, and the ability to reason through multi-step tasks. Let’s build one.
Option 1: CrewAI (Simplest Path)
CrewAI is arguably the easiest agent framework to get working with local models. You define agents with roles and goals, group them into a “crew,” and let them collaborate.
Install:
pip install crewai langchain-ollama
Define an agent and task:
from crewai import Agent, Task, Crew
from langchain_ollama import OllamaLLM
# Point to your local Ollama instance
llm = OllamaLLM(model="llama3.1")
# Create a researcher agent
researcher = Agent(
role="Research Analyst",
goal="Find and summarize key trends in AI agents",
backstory="You are an expert analyst who specializes in AI technology trends.",
llm=llm,
verbose=True
)
# Define what the agent should do
research_task = Task(
description="Research the current state of AI agent frameworks in 2025. "
"Focus on local-first approaches and open-source tools.",
expected_output="A concise summary of the top 3 trends in AI agents.",
agent=researcher
)
# Assemble the crew and run
crew = Crew(agents=[researcher], tasks=[research_task])
result = crew.kickoff()
print(result)That’s it. No API keys, no cloud calls. Everything runs on your machine.
Option 2: LangGraph (More Control)
If you need stateful, graph-based workflows with branching logic, LangGraph gives you finer control. It models agent behavior as a directed graph where nodes are actions and edges are decisions.
from langgraph.graph import StateGraph, END
from langchain_ollama import OllamaLLM
llm = OllamaLLM(model="mistral")
def research_node(state):
response = llm.invoke(f"Research this topic: {state['topic']}")
return {**state, "research": response}
def write_node(state):
response = llm.invoke(
f"Write a brief report based on this research: {state['research']}"
)
return {**state, "report": response}
# Build the graph
workflow = StateGraph(dict)
workflow.add_node("research", research_node)
workflow.add_node("write", write_node)
workflow.add_edge("research", "write")
workflow.add_edge("write", END)
workflow.set_entry_point("research")
app = workflow.compile()
result = app.invoke({"topic": "Local AI deployment trends"})
print(result["report"])LangGraph shines when you need loops, conditional branching, or persistent memory across agent steps.
Multi-Agent Orchestration: Making Agents Collaborate
The real power of agents isn’t a single LLM answering questions — it’s multiple specialized agents working together. Here’s a practical pattern using CrewAI:
from crewai import Agent, Task, Crew, Process
from langchain_ollama import OllamaLLM
# You can even assign different models to different agents
researcher_llm = OllamaLLM(model="llama3.1")
writer_llm = OllamaLLM(model="mistral")
editor_llm = OllamaLLM(model="llama3.1")
researcher = Agent(
role="Senior Researcher",
goal="Conduct thorough research on the given topic",
backstory="An experienced researcher with a knack for finding key insights.",
llm=researcher_llm
)
writer = Agent(
role="Content Writer",
goal="Transform research into an engaging, well-structured article",
backstory="A skilled writer who makes complex topics accessible.",
llm=writer_llm
)
editor = Agent(
role="Editor",
goal="Review and polish the article for clarity and accuracy",
backstory="A meticulous editor with high standards for quality.",
llm=editor_llm
)
# Tasks form a pipeline — each builds on the previous
research_task = Task(
description="Research the topic: '{topic}'. Identify key points and data.",
expected_output="Detailed research notes with sources and key findings.",
agent=researcher
)
writing_task = Task(
description="Write an article based on the research notes provided.",
expected_output="A complete, engaging article draft.",
agent=writer
)
editing_task = Task(
description="Edit the article for grammar, clarity, and flow.",
expected_output="A polished, publication-ready article.",
agent=editor
)
# Run sequentially — each agent hands off to the next
crew = Crew(
agents=[researcher, writer, editor],
tasks=[research_task, writing_task, editing_task],
process=Process.sequential,
verbose=True
)
result = crew.kickoff(inputs={"topic": "The rise of local AI agents"})Each agent picks up where the last one left off. The researcher gathers information, the writer shapes it into prose, and the editor refines it. All running locally.
Deploying Ollama with Docker
Running Ollama on your laptop is great for development. For team use or production workloads, Docker is the way to go.
Basic Docker Setup
docker run -d \ -v ollama:/root/.ollama \ -p 11434:11434 \ --name ollama \ ollama/ollama
Then pull models into the container:
docker exec ollama ollama pull llama3.1
Production-Ready Docker Compose
For anything beyond local experiments, you’ll want a proper stack:
# docker-compose.yml
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
ports:
- "11434:11434"
volumes:
- ollama-data:/root/.ollama
environment:
- OLLAMA_HOST=0.0.0.0
- OLLAMA_NUM_PARALLEL=4
- OLLAMA_MAX_LOADED_MODELS=3
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
interval: 30s
timeout: 10s
retries: 3
webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- "3000:8080"
depends_on:
- ollama
environment:
- OLLAMA_BASE_URL=http://ollama:11434
restart: unless-stopped
volumes:
ollama-data:Key things to note:
- Volume mounts are critical. Without
ollama-data:/root/.ollama, you'll re-download multi-gigabyte models every time the container restarts. - Pin your image tags in production. Using
:latestmeans a randomdocker compose pullcould introduce breaking changes. - GPU passthrough requires the NVIDIA Container Toolkit installed on the host. Remove the
deploy.resourcesblock if running CPU-only. OLLAMA_NUM_PARALLELcontrols concurrent request handling. Tune this based on your hardware.
Spin it up:
docker compose up -d
You now have Ollama serving models with a web UI at http://localhost:3000.
Pre-Baking Models into Images
For faster cold starts, bake models directly into your Docker image:
FROM ollama/ollama RUN ollama serve & sleep 5 && ollama pull llama3.1 && pkill ollama CMD ["ollama", "serve"]
New containers start with the model ready to go — no download wait.
Architecture Patterns for Production Agents
Once you move beyond single-machine setups, a few patterns become essential:
1. Model Router Pattern
Use different models for different agent tasks. Lightweight models handle simple classification or routing, while heavier models tackle complex reasoning:
User Request → Router (Phi-3 Mini) → Simple Query → Mistral 7B
→ Complex Query → Llama 3.1 70B
→ Code Task → CodeLlama2. Caching Layer
Identical prompts hitting Ollama trigger full inference every time. Adding a Redis cache in front of Ollama eliminates redundant computation:
Agent Request → Redis Cache → HIT → Return cached response
→ MISS → Ollama → Cache + Return3. RAG-Augmented Agents
Combine agents with a local vector database for retrieval-augmented generation:
from chromadb import Client
from sentence_transformers import SentenceTransformer
# Set up local vector store
client = Client(path="./chroma_db")
collection = client.create_collection(name="knowledge_base")
embedder = SentenceTransformer("all-MiniLM-L6-v2")
# Embed your documents
docs = ["Your company docs...", "Product specs...", "Meeting notes..."]
embeddings = embedder.encode(docs).tolist()
collection.add(documents=docs, embeddings=embeddings, ids=["d1", "d2", "d3"])
# Retrieval function your agent can use as a tool
def search_knowledge(query, top_k=3):
q_emb = embedder.encode([query]).tolist()
results = collection.query(query_embeddings=q_emb, n_results=top_k)
return results["documents"][0]Wire this into your agents as a tool, and they can answer questions grounded in your specific data — entirely offline.
Hardware Considerations
Running local LLMs requires real resources. Here’s what to plan for:
- RAM is the bottleneck. 8B parameter models need ~8 GB of RAM. 70B models need 64 GB+. The model stays loaded in memory during use.
- GPU dramatically improves speed. CPU inference works but is 5–10x slower. An NVIDIA GPU with CUDA support (or Apple Silicon with Metal) makes the difference between 30-second and 3-second responses.
- Disk space adds up. Each model is 2–40 GB. If you’re running multiple models, plan for 50–100 GB of free space.
- For teams, a dedicated GPU server (even a single RTX 4090) can serve multiple agents concurrently and handle a small team’s workload.
When Ollama Isn’t Enough
Ollama is optimized for simplicity, not maximum throughput. If you’re hitting performance ceilings, consider these alternatives for the serving layer while keeping your agent orchestration intact:
- vLLM — implements PagedAttention and continuous batching for significantly higher tokens-per-second on the same hardware.
- Text Generation Inference (TGI) from Hugging Face — another high-performance option with production-grade features.
Both require more operational investment but can be worth it for high-concurrency workloads.
Wrapping Up
The local AI agent stack has reached a tipping point. With Ollama handling model serving, frameworks like CrewAI and LangGraph handling orchestration, and Docker handling deployment, you can build surprisingly capable multi-agent systems without ever touching a cloud API.
The key takeaways:
- Start with Ollama + CrewAI for the fastest path to working agents.
- Use Docker Compose once you need reproducibility or team access.
- Assign different models to different agents based on task complexity.
- Add RAG with ChromaDB when your agents need domain-specific knowledge.
- Monitor your resources — local LLMs are hungry for RAM and GPU.
The beauty of this approach is that it’s entirely under your control. No vendor lock-in, no surprise bills, no data leaving your infrastructure. Just your hardware, your models, and your agents doing the work.
If you found this useful, follow me for more practical guides on building with local AI. I write about the tools and patterns that actually work — no hype, no fluff.
