Agentic AI Systems: From Memory to Autonomy
In the previous article, retrieval was a one-shot transaction. A user asked a question, the system pulled context, the model wrote an answer, the conversation ended. Even an excellent RAG pipeline is built around that shape: stateless, request-response, no continuity.
Real agentic systems do not work that way. They loop. They carry state across turns and across sessions. They decide when to retrieve, when to act, when to ask, and when to stop. They make a decision on Tuesday and have to remember why they made it on Friday. They use tools that fail, recover, and try something different. They run for hours, sometimes days.
Stripping away the marketing, an agentic system is three things layered on top of a foundation model:
- A loop that decides what to do next.
- A memory that survives between iterations of the loop.
- A set of tools the agent can use to affect the world.
Most production failures of agentic systems trace back to the second item. The loop and the tools are tractable engineering problems. The memory is where systems quietly fall apart.
This article is about how to build the memory layer right, what the orchestration patterns around it actually look like in production, and how to keep an agent grounded when its context window runs out.
The context window is not memory
The first mistake most teams make is treating the model’s context window as the agent’s memory. They append the conversation, append the tool outputs, append the retrieved chunks, and watch the token count climb. When it gets uncomfortable, they truncate the oldest items and hope nothing important was in there.
This works for short tasks. It fails for anything that runs longer than a single sitting.
The context window is working memory. It is what the model can attend to in a single forward pass. It is fast, expensive per token, and lost the moment the request returns. Treating it as long-term storage produces a specific cluster of failures:
- The agent forgets a constraint the user gave it three turns ago because the constraint scrolled out of the window.
- The agent contradicts a decision it made earlier because the rationale for that decision is no longer visible.
- The agent re-runs a tool call it already ran, with the same arguments, because the previous result was truncated.
- The agent appears to learn something about a user during a session and reverts to defaults the moment a new session starts.
What the agent actually needs is a separation between working memory (the context window, transient) and durable memory (something that survives the request, the session, and the day).
The hard question is what shape that durable memory should take. Most teams reach for the obvious answer — a vector store of past turns — and discover within a few weeks that it is not enough. A vector store of conversation snippets produces semantic search over things the user said. It does not produce remembering.
This is where I want to introduce the architecture I’ve been working on.
Memory-Node Encapsulation as the memory substrate
The full specification is in the foundational paper, Memory-Node Encapsulation (MNE): An Advanced Data Structure for Artificial Episodic Memory and Cognitive AI Reasoning. What follows is how to use it as the memory substrate of an agentic system, not a summary of the paper.
The short version: an MNE is a memory object that bundles, in a single record, everything an agent should remember about a single experience. Not just the text, not just the embedding. The text plus the embedding plus the emotional or significance weighting plus the temporal context plus typed graph connections to other memories plus distilled insights extracted over time. Each MNE moves through consolidation states — fresh, consolidating, consolidated, archived — that mirror how biological memory actually works.
The reason this matters for agent design is that an MNE-based memory layer changes what the retrieval function returns. A vector store returns chunks ranked by cosine similarity. An MNE store returns experiences ranked by relevance, where relevance combines semantic match with significance, recency, access frequency, and graph-connected context. The retrieval function in the MNE specification looks roughly like this:
def retrieval_score(query_embedding, mne, now, decay_seconds=86400):
semantic = cosine(query_embedding, mne.embedding)
# Significance amplifies retrieval the way emotional weight
# amplifies biological memory recall.
significance_boost = 1.0 + abs(mne.emotional_weight) * 0.5
# Recency, but not as a hard filter — old memories that
# were significant should still surface.
age = (now - mne.timestamp).total_seconds()
recency = math.exp(-age / decay_seconds)
# Frequency boost models the long-term potentiation effect:
# memories accessed often become more accessible.
frequency_boost = math.log1p(mne.access_count)
# Memories that have been integrated with broader knowledge
# get a small bonus for being well-grounded.
integration_bonus = mne.integration_score * 0.2
return (
semantic
* significance_boost
* (0.3 * recency + 0.7)
* (1.0 + 0.1 * frequency_boost)
* (1.0 + integration_bonus)
)That single function changes everything downstream. The agent isn’t getting back “the five chunks most semantically similar to the current query.” It’s getting back “the five experiences most likely to be relevant given semantic match, what mattered, what’s recent, what’s been used before, and what’s well-integrated.” Those are different sets in almost every realistic case.
This is also where MNE diverges from the RAG architecture in the previous article in a way worth being explicit about. RAG retrieves documents. MNE retrieves experiences. A production agentic system needs both layers — RAG for the corpus of documents the agent has access to, MNE for the agent’s own accumulated history. They live in different stores, serve different queries, and are surfaced into the agent’s context at different points in the loop.
The orchestration loop
With those two memory layers in place, the agentic loop has a definite shape. Most teams skip the explicit orchestration layer and put all of this logic inline in a single prompt, then wonder why their agent’s behavior is impossible to reason about. The loop is the architecture. Make it explicit.
def agent_step(agent_state, user_input):
# 1. Encode the new input as a fresh memory.
new_mne = encode_experience(
raw_input=user_input,
memory_type="episodic",
session_id=agent_state.session_id,
)
# 2. Retrieve relevant context from both memory layers.
document_chunks = rag_retrieve(user_input, top_k=10) # Article 3 layer
relevant_mnes = mne_retrieve(
query=user_input,
agent_id=agent_state.agent_id,
top_k=10,
)
# 3. Compose working context. This is the only thing the
# foundation model will see in this turn.
working_context = compose_context(
system_prompt=agent_state.system_prompt,
recent_turns=agent_state.recent_turns, # last N turns verbatim
episodic_memories=relevant_mnes, # surfaced from MNE store
document_context=document_chunks, # surfaced from RAG
available_tools=agent_state.tool_registry,
)
# 4. Plan or act.
decision = foundation_model.decide(working_context)
if decision.kind == "tool_call":
result = execute_tool(decision.tool, decision.arguments)
# The tool call AND its result both become memories.
encode_experience(
raw_input={"tool": decision.tool,
"args": decision.arguments,
"result": result},
memory_type="procedural",
session_id=agent_state.session_id,
)
return AgentResponse(kind="continue", observation=result)
if decision.kind == "ask_human":
return AgentResponse(kind="ask", question=decision.question)
if decision.kind == "respond":
# The response itself is encoded as a memory of what the
# agent told the user — which matters for consistency later.
encode_experience(
raw_input={"agent_response": decision.text},
memory_type="episodic",
session_id=agent_state.session_id,
)
return AgentResponse(kind="final", text=decision.text)A few things about this loop are worth pulling out, because they are the parts most teams get wrong.
Every input and every output becomes a memory. Not just user messages. Tool calls become memories of type procedural. The agent's own responses become memories. This is what makes the agent capable of reasoning about its own past behavior. An agent that can't see what it said last week can't be consistent with what it said last week.
The compose_context step is its own thing, and it is doing real work. It is not just concatenation. It is making decisions about how much budget to give to verbatim recent turns versus surfaced episodic memories versus document context versus tool definitions. When the budget is tight, it has to drop something, and the drop should be principled. Verbatim recent turns are usually the most expendable, because the important information from them has already been encoded as MNEs.
The foundation model never sees the full memory store. It sees a window onto it, mediated by retrieval. This is the only design that works at scale. Anyone telling you their agent “has perfect memory” because it stuffs everything into a million-token context window has not deployed that agent against a real workload.
Tool calls are first-class state transitions. The agent decided to call a tool; the tool returned something; both halves are recorded. When the agent asks itself “did I already check the inventory system today,” the answer is in the procedural memory, not in working context.
Context window exhaustion: the failure mode that defines production
Eventually, every long-running agent runs out of context window. This is not a bug. It is a structural property of foundation models. An agent that is going to operate for hours has to be designed around the certainty that its working memory will be exhausted, and what happens at that exhaustion point is where most production agents fall apart.
The naive answer is summarization. The agent’s loop keeps a running summary of the conversation, and when the context fills, the summary replaces the verbatim turns. This is better than truncation, but it has well-known failure modes:
- Summaries lose specificity. The agent remembers it discussed pricing but not what numbers were quoted.
- Summaries compound errors. Each round of summarization is a chance to drop or distort something, and those errors don’t get corrected.
- Summaries treat all content as equally important. A throwaway pleasantry and a critical constraint get the same compression.
The MNE-based answer is better, and it’s worth being precise about why. When working context fills, the agent doesn’t summarize and replace. It checkpoints the verbatim turns into the MNE store, where each turn becomes a memory with full detail preserved, then loads back only what’s relevant via the retrieval function.
def handle_context_pressure(agent_state, working_context, threshold=0.85):
if working_context.token_usage_ratio < threshold:
return working_context
# Move verbatim turns out of working memory and into durable
# memory. They aren't gone — they're searchable.
for turn in agent_state.recent_turns[:-3]: # keep the last 3 verbatim
encode_experience(
raw_input=turn,
memory_type="episodic",
session_id=agent_state.session_id,
# Mark turns that contain commitments or constraints
# with higher significance so they surface easily later.
emotional_weight=detect_significance(turn),
)
agent_state.recent_turns = agent_state.recent_turns[-3:]
# Re-retrieve from MNE store with the *current* state as the query,
# which surfaces whatever is most relevant given where we are now,
# not just what was recent.
relevant_mnes = mne_retrieve(
query=agent_state.current_objective,
agent_id=agent_state.agent_id,
top_k=15,
)
return compose_context(
system_prompt=agent_state.system_prompt,
recent_turns=agent_state.recent_turns,
episodic_memories=relevant_mnes,
document_context=working_context.document_context,
available_tools=agent_state.tool_registry,
)The crucial property is that nothing is lost. When working memory got tight, the verbatim turns were displaced into durable memory, where they live with all their original detail. If a constraint mentioned twenty turns ago becomes relevant at turn fifty, the retrieval function will surface it back, because the constraint was encoded as an MNE with significance weight, and that significance weight is part of the retrieval score.
This is what it means to keep an agent grounded when the context window runs out. The grounding is not in working memory. The grounding is in the durable memory layer, surfaced back into working memory through retrieval, every turn.
Tool integration: the part nobody wants to design
Tools are where agentic systems meet the world, which means they are where the agent stops being a clean reasoning machine and starts dealing with timeouts, partial failures, rate limits, schema drift, and authentication errors. Most agent frameworks paper over this with optimistic abstractions. Production teams pay for that optimism.
Three patterns matter for tool integration in an MNE-based system.
Tools are typed contracts, not loose function calls. Every tool needs a strict schema for its inputs and outputs, and a defined error taxonomy. The agent should never be reasoning about whether a string return value indicates success or failure. The tool returns a structured result with a status field. The orchestrator distinguishes transient errors from terminal errors and decides whether to retry.
@dataclass
class ToolResult:
status: Literal["success", "transient_error", "terminal_error", "auth_error"]
payload: Optional[dict]
error_message: Optional[str]
retry_after: Optional[float]
cost_units: float # for budget trackingTool results become memories with the call that produced them. The pairing matters. A memory that says “the inventory API returned 47 units of SKU 4421” is useful. A memory that says “I called inventory_check(sku=’4421', warehouse=’KC1') and got 47 units” is useful and replayable. When the agent later wonders whether to re-check the same warehouse, it can find the prior call, see when it was made, and decide based on freshness rather than re-running blindly.
Tool budgets are enforced at the orchestration layer, not by the model. A foundation model with access to tools will, under the wrong conditions, call them in a tight loop. The orchestrator caps the number of calls per session, per user, per tool, with explicit limits visible to the agent through the system prompt. When a budget is exhausted, the agent gets a clear signal — not silence, not a failure, but an explicit “you have used your budget for this tool; here are the alternatives.”
Human-in-the-loop is an architecture decision, not a feature
The phrase “human-in-the-loop” gets used loosely. In practice, it covers at least three distinct architectures, and the differences matter.
Approval-gated. The agent proposes an action; a human approves or denies before the action executes. This is the right pattern for any action with material real-world consequence — sending external communications, writing to systems of record, spending money. The agent’s loop pauses on the approval, the proposal becomes an MNE so the agent remembers what it proposed, and the human’s response becomes an MNE so the agent learns from the approval pattern.
Clarification-driven. The agent runs to completion when it can, but pauses to ask the human when its confidence drops below threshold. The threshold is a real engineering decision — too high and the agent stalls constantly, too low and it confidently does the wrong thing. The right place to set it is empirically, by running the agent against a benchmark and measuring how often clarification requests align with cases where the agent would actually have been wrong.
Review-after. The agent runs to completion without intervention; humans review the trace afterward, often in batch. This is the right pattern for high-volume, low-stakes work where individual errors are tolerable but systemic errors are not. The MNE store is the trace. Reviewers are reading the agent’s episodic memory, ideally with annotations they can write back to specific MNEs to flag issues.
These are not exclusive. A serious production agent uses all three: approval-gated for the riskiest actions, clarification-driven for ambiguous decisions, review-after for everything else. The orchestration layer routes between them based on action type and confidence.
What MNE adds to all three is a memory of the human’s decisions. When the human approves, denies, clarifies, or flags, that interaction is encoded with high significance weight. Over time, the agent’s behavior shifts toward what humans have approved and away from what they have denied — not through retraining, but through retrieval. The agent’s distilled insights, accumulated across many human interactions, surface during the next planning step and shape the proposal.
This is the proto-agentic behavior the original MNE paper describes: the agent isn’t following hardcoded rules about what to ask. It’s drawing on its accumulated experience of how this human (or this organization) tends to make decisions, and adjusting accordingly. That capability is impossible without persistent, retrievable memory of the human interactions themselves.
Consolidation: the part that runs while the agent sleeps
One of the properties of MNE that doesn’t have an obvious analogue in standard agent designs is consolidation. The original paper describes it as a periodic offline process — every six hours in the prototype — that does several things:
- Replays recent memories and increments their replay counts.
- Promotes well-integrated memories from
freshtoconsolidatedstate. - Identifies redundant memories and abstracts them into summary memories, archiving the originals.
- Prunes weak graph connections.
- Extracts new distilled insights from accumulated patterns.
In an agentic system, this is the layer that turns experience into wisdom. Without it, the memory store grows unboundedly and the retrieval function gets slower and noisier over time. With it, the agent’s effective behavior improves between sessions, even when the underlying foundation model is unchanged.
Operationally, consolidation runs as a scheduled batch job. It does not run inline with the agent’s loop. The agent’s hot path stays fast; consolidation happens in the background.
The bridge to the next article in this series is sitting right here. Consolidation has obvious safety implications — it is the layer that decides what gets compressed, what gets pruned, what gets promoted into “long-term beliefs” the agent will act on. An adversary who could influence consolidation could shape the agent’s future behavior in ways that would be invisible at the per-turn level. Article 5, on AI safety and governance, picks up exactly this thread: how to build guardrails, monitoring, drift detection, and audit trails for systems that, by design, learn from experience.
What to take away
An agentic system is a loop, a memory, and a set of tools, all sitting on top of a foundation model. The loop and the tools are well-understood engineering problems. The memory is the part that determines whether the agent can actually be agentic, and most production systems get it wrong.
The wrong answer is “stuff everything into the context window.” The half-right answer is “store conversation history in a vector database.” The right answer is to recognize that agent memory is fundamentally an episodic problem — what happened, what mattered, when, in what context, connected to what — and to design the memory layer around that shape. That is what MNE provides, and what NeoCortex-M wraps in a usable architecture.
Build the loop explicitly. Encode every experience. Separate working memory from durable memory. Retrieve based on relevance, not just similarity. Treat tools as typed contracts. Decide deliberately where humans belong in the loop. Run consolidation in the background.
Do those things and you have an agent that gets better the longer it runs, instead of worse.
What’s next
The next article in the series turns to the question that follows naturally from everything above. An agent with persistent, evolving memory is an agent whose behavior changes over time — and any system whose behavior changes over time is a system that has to be governed, observed, and held to account. Article 5, AI Safety and Governance in the Enterprise: A Practitioner’s Framework, is the engineering side of that problem. Not the theoretical ethics paper. The actual implementation of guardrails, monitoring, drift detection, audit trails, and lifecycle controls in a production AI system.
Brian James Curry is an Enterprise AI Strategist, Data and Marketing Scientist, and Applied Researcher. He is the founder of Vector1 Research, an independent research lab focused on marketing economics, causal systems, cognitive AI architecture, and the economic implications of intelligent systems. He also founded KC AI Lab, a data science community and incubator for applied machine learning education.
