How Much Memory Does a Long-Horizon Agent Actually Need?

Long context, truncation, or retrieval? A first-principles model of the recall-versus-cost tradeoff shows that the “just use a bigger window” instinct is wrong almost everywhere — and exactly where it goes wrong is predictable.

An agent that runs for more than a few minutes has a memory problem. Every tool call, every observation, every intermediate result is something it might need later — and the pile grows without bound. The field has three standard answers, and a surprising amount of architecture hinges on which one you pick.

The first answer is just use a bigger context window: append everything to the prompt and let a long-context model sort it out. The second is truncation: keep the last W items and let the rest fall off the back. The third is retrieval: store everything outside the prompt and fetch only the few items that look relevant to the current step — the pattern behind retrieval-augmented generation (Lewis et al., 2020) and agent memory systems like MemGPT (Packer et al., 2023).

The usual way this gets decided is vibes, vendor defaults, and whatever the context limit happens to be this quarter. This article tries to decide it on purpose. We build a small, first-principles model of what each strategy actually buys you — how reliably the needed fact reaches the model’s working attention, and what it costs to put it there — and we let the model tell us which strategy wins, as a function of two things you can actually measure: how long the agent has been running, and how good your retriever is.

The result is a phase diagram. It has three regions, and the boundaries between them are not where intuition puts them. In particular: at a decent retriever, full context stops being optimal after roughly a dozen accumulated items, and past a few thousand items it becomes a net negative — the agent would be better off with an empty memory than with everything crammed into the prompt.

All code is available for a fee and runs in under a minute on NumPy and SciPy. No model calls, no network — the point is the structure, and the structure is analytic.

The question, stated precisely

An agent has accumulated N items in its history. The current task requires K of them — the relevant facts — which sit somewhere among the N. A memory strategy decides which items actually make it into the model’s prompt. We score a strategy on two axes:

Recall — the probability that a needed fact both reaches the prompt and is actually used by the model.
Cost — how many items the strategy stuffs into the context, which drives tokens, latency, and (because attention is quadratic in sequence length; Vaswani et al., 2017; Dao et al., 2022) compute.

Everything below is built from two empirically grounded ingredients.

Ingredient 1: being in the prompt is not the same as being used

The single most important fact about long context is that a model does not reliably use what you put in front of it. Liu et al. (2024) showed that the probability a model correctly uses a relevant passage follows a U-shaped curve in position — high at the very start and very end of the prompt, sagging badly in the middle — and that the whole curve droops as the prompt gets longer, even for models explicitly built for long context. The RULER benchmark (Hsieh et al., 2024) found much the same: a model’s effective context is far shorter than its advertised window.

We capture this with a single decay function: the probability that a fact already present in a prompt of m items is actually used.

Here g falls from near 1 in a short prompt toward a floor as m grows. This is the term that punishes “just make the window bigger”: adding items dilutes the model’s use of every item, including the one that matters.

def g(m):  # P(use a fact | it is present in a prompt of m items)
    return G_MIN + (G_MAX - G_MIN) / (1 + (m / N_HALF)**ETA)

Ingredient 2: a retriever is a noisy classifier

A retriever assigns a relevance score to every stored item and returns the top k. Relevant items tend to score higher than distractors, but the distributions overlap. Borrowing straight from signal-detection theory (Green & Swets, 1966), let relevant items score with mean μ_rel and distractors with mean μ_dist, both with spread σ. The retriever’s quality is one number — its separability:

A d′ of 0 is a coin flip; a strong dense retriever (Karpukhin et al., 2020) sits well above 1. This single parameter is what determines whether retrieval is worth doing at all.

The three strategies, as recall functions

Now we can write down what each strategy recovers per needed fact. Let L be the context window cap (in items) and assume relevant facts are uniformly distributed in age.

Full context puts everything in, up to the cap. A fact survives the cap with probability min(1, L/N), and if present is used with probability g evaluated at the (large) prompt size:

Truncation keeps the last W items. Recall is the chance the fact is recent enough to still be in the window, times the (high, because W is small) usage probability:

[ FIGURE: insert eq3_trunc.png ]

Retrieval fetches the top k. A relevant item lands in the top k exactly when its score beats the k-th-highest score among the rest; for a pool of N items that threshold is Φ⁻¹(1 − k/N), giving a clean closed form — which we then multiply by g evaluated at the small retrieved set:

from scipy.stats import norm
def recall_ret(N, dprime, k):
    tau = norm.ppf(1 - k / N)          # score threshold to make the top-k
    return norm.cdf(dprime - tau) * g(k)

That closed form is not a hand-wave; a Monte-Carlo check matches it tightly (e.g. at N=1000, d′=2, k=20: closed-form 0.479 vs. simulated 0.482). The script runs the comparison so you can see it.

Notice the asymmetry that drives everything: full context pays the g-penalty at the full prompt length, while retrieval pays it at only k items. Retrieval doesn’t just cost less — it keeps the model in the regime where it actually uses what it’s given.

Cost, success, and the agent’s choice

A task needing K facts succeeds only if all K are recalled, so success is recall raised to the K. Net utility is the value of solving the task minus the cost of the items placed in context:

Each strategy gets to pick its own free parameter optimally — full context has none, truncation tunes W, retrieval tunes k — and then the agent picks the best strategy for its situation:

Everything that follows is just evaluating this argmax over a grid of horizons N and retriever qualities d′.

Results

Parameters are illustrative and calibrated qualitatively to the long-context literature, not fit to it; the conclusions are structural, not tuned. We use a context cap of L = 1000 items and report a single relevant fact (K = 1) except where noted.

Recall holds for retrieval and decays for everyone else. As the horizon grows, full-context recall sags (dilution plus the window cap) and truncation collapses (old facts fall off the back). Retrieval stays roughly flat: a fixed-quality retriever pulling a small k is largely indifferent to how much junk it’s sifting through.

Per-fact recall vs. horizon. Retrieval is evaluated at d′ = 2 with k chosen optimally.

On net utility, full context doesn’t just lose — it goes negative. Once cost is counted, full context is best only at a tiny horizon. At a strong retriever (d′ = 2) the crossover lands at N ≈ 12 items; beyond a few thousand items, the cost of cramming the window exceeds the value of the task, so full context scores below zero — worse than answering with no memory at all. Retrieval degrades gracefully across four orders of magnitude.

Net utility vs. horizon (d′ = 2). The crossover where retrieval overtakes full context is marked.

The headline: a three-region phase diagram. Putting horizon on one axis and retriever quality on the other, the optimal strategy carves the plane into three regions. Full context owns the small-horizon corner. Retrieval owns essentially everything else — and the better your retriever, the earlier (smaller N) it takes over. The interesting part is the thin sliver in the bottom-right: at a huge horizon with a near-useless retriever (d′ ≈ 0), a cheap sliding window beats retrieval, because a bad retriever is worse than simply keeping the most recent items.

Optimal memory strategy as a function of horizon and retriever quality.

Two practical readings of that picture. First, retriever quality buys you horizon: improving d′ moves the full-to-retrieval boundary left, so a better retriever lets you commit to retrieval sooner. Second, retrieval is not free lunch when your retriever is bad — below a separability floor, you are better off with a dumb recency window than with confident-but-wrong fetches.

Multi-hop tasks widen the gap. When a task needs several remembered facts chained together (K > 1), success is a product of per-fact recalls, so small recall differences compound. At long horizons, full-context success on two or three facts collapses toward zero while retrieval retains meaningful success.

Task success for K = 1, 2, 3 facts. The recall advantage of retrieval compounds with hops.

Why this matters for cognitive-systems design

The prevailing narrative says context windows are growing so fast that memory architecture is a temporary problem — wait a generation or two and you can just paste everything in. The model here says the opposite, for a reason that doesn’t go away with scale: a larger window raises the cap, but it does not repair the usage decay or the cost curve. Even with an effectively infinite window, dumping an agent’s entire history into every prompt means paying the g-penalty at maximum length and the cost at maximum length, on every single step. Bigger windows move the cliff; they don’t remove it.

That reframes the engineering priority. The lever that actually shifts the phase boundary is not window size — it’s retriever quality. Every increment of d′ you can buy (better embeddings, reranking, hybrid lexical-plus-dense retrieval, smarter chunking) pushes the retrieval region earlier and widens the horizon over which your agent stays effective. For long-running agents, the retriever is not a peripheral RAG component; it is the thing that determines how long the agent can think before it starts forgetting what matters.

It also says something about what “memory” should be. A memory system’s job is not to store more — storage is cheap and N only ever grows. Its job is to keep the per-step working set small and relevant, which is exactly the regime where the usage decay is mildest and the cost is lowest. Selection, not accumulation, is the whole game.

What to do about it

Stop equating context length with memory. They live on different axes of the diagram. A longer window raises L; it does nothing for g or cost.
Measure your retriever’s d′. It is one number, it is estimable from labeled relevant/irrelevant examples, and it tells you which region of the phase diagram you are actually in.
Spend on retrieval quality before context length. Improving d′ moves the boundary; buying tokens does not.
Keep the working set small. Retrieval’s structural advantage is that it pays the usage penalty at k items, not N. Large k throws that away — there is an interior optimum, and it is smaller than people expect.
Default to recency only when your retriever is genuinely weak. That bottom-right sliver is real, but it is small, and it is an admission that the retriever needs work.

Limitations

This is a model, deliberately spare. The usage-decay and retriever-recall forms are stylized — grounded in the direction and rough shape of the long-context and retrieval literature, not fit to a specific model’s numbers — so the exact crossover values will move with the parameters. We treat relevant facts as uniformly aged, hold task value and per-item cost fixed, and ignore the dynamics of writing and forgetting memories over time. What is robust is the structure: full context pays its penalties at maximum length on every step, retrieval pays them at the working-set size, and that asymmetry is what produces the phase diagram. Calibrating g and d′ to a particular stack would sharpen the numbers without changing the shape.

Reproduce it

The full script — decay model, all three recall functions, the Monte-Carlo validation of the retrieval closed form, the utility curves, and the phase diagram — are available for a fee.

References

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.
Hsieh, C.-P., et al. (2024). RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560.
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022.
Green, D. M., & Swets, J. A. (1966). Signal Detection Theory and Psychophysics. Wiley.

Written at Vector1, an independent research lab working on cognitive systems, search, and applied AI research.

About the Author

Brian Curry is a Kansas City-based data scientist and applied AI researcher with two decades of enterprise experience across retail, agriculture, energy, hospitality, telecommunications, and media. He is the founder of Vector1 Research, an independent lab focused on applied AI, causal inference, agentic systems, and the economics of intelligent organizations. His work centers on building systems that surface the causal and structural drivers of performance — bridging advanced analytics, machine learning, and executive decision-making. He writes regularly on Medium about applied AI, knowledge engineering, decision science, and the engineering disciplines that make machine learning actually work in production.