AI Memory Types

May 2026

A comprehensive guide to in-context, external, in-weights, and in-cache memory

 

Introduction

AI systems use four distinct memory types, each with different persistence, speed, and capacity. Understanding when and how to use each type is critical for building reliable, efficient AI solutions.

 

Think of AI memory like human memory:

•        In-context memory is what you are actively thinking about right now

•        External memory is your notebook or smartphone

•        In-weights memory is everything you learned in school and cannot un-know

•        In-cache memory is muscle memory — repeating a familiar task faster without re-thinking it from scratch

 

1. In-context Memory

 

Type: Short-term | Nature: Volatile, Fast

The active working memory — everything currently visible to the model inside the context window.

 

Analogy

📝 Your desk while working Everything you can see and touch right now. Just like a messy desk, once you clear it (end the session), everything is gone. No desk, no memory.

 

Strengths

•        Instant recall — no retrieval step needed

•        Precise and faithful to what was said

•        No infrastructure required

Limitations

•        Resets every session — no persistence

•        Limited by context window size (100K–1M tokens)

•        Costly at scale — every token is processed

Use Cases

•        Customer support chatbot: The full conversation history is in context, so the agent knows what was said earlier in the same ticket.

•        Document Q&A: Paste an entire document into context and ask questions. The model reads it fresh each time.

•        Multi-step reasoning: Chain-of-thought prompting keeps intermediate reasoning steps visible while the model works through a problem.

•        Code assistants: The current file and recent edits live in context, so suggestions stay coherent with what was just typed.

 

 

2. External Memory

 

Type: Long-term | Nature: Persistent, Queryable

Databases, vector stores, knowledge bases, or files the AI reads and writes to persist information across sessions.

 

Analogy

📚 Your bookshelf, filing cabinet, or phone contacts You do not hold all of it in your head, but you know how to look it up. The information outlives any single conversation, just like books outlive a reading session.

 

Strengths

•        Survives session resets — truly persistent

•        Unlimited scale — limited only by storage

•        Updateable anytime without retraining

Limitations

•        Retrieval latency — adds a round-trip to every query

•        Retrieval errors — if the wrong document is fetched, the model may hallucinate confidently

•        Infrastructure overhead — requires databases, embedding pipelines, and retrieval logic

Use Cases

•        RAG (Retrieval-Augmented Generation): AI searches a vector database of company documents to answer employee questions accurately with up-to-date information.

•        Personal AI assistant: Stores preferences, past decisions, and notes so the next session feels continuous.

•        CRM integration: A sales agent reads and writes a customer database so every rep gets an up-to-date history of interactions.

•        Research assistant: Ingests papers and summaries into a vector store; retrieves the most relevant chunks when a new question is asked.

 

 

3. In-weights Memory

 

Type: Permanent |  Nature: Baked-in, Implicit

Knowledge encoded into the model's parameters during training or fine-tuning. It is the model's worldview — always available, never updatable at runtime.

 

Analogy

🎓 Everything you learned in school Language, math, history, common sense — you cannot un-learn that Paris is in France. You did not look it up; it is wired in. But your knowledge has a cutoff: you do not know what happened after you graduated.

 

Strengths

•        Zero retrieval cost — knowledge is always available

•        Broad general knowledge across many domains

•        No infrastructure required at runtime

Limitations

•        Knowledge cutoff date — does not know recent events

•        Cannot be updated at runtime — requires full retraining

•        Retraining is expensive in time and compute

•        Can hallucinate confidently on topics at the edges of training data

Use Cases

•        Language understanding: Grammar, idioms, and reasoning patterns are in-weights; every prompt benefit without any extra memory system.

•        Domain fine-tuning: Fine-tune on medical literature so the model knows clinical terminology deeply, not just via retrieval.

•        Coding assistants: Knowledge of programming languages, APIs, and patterns is in-weights; no lookup needed for standard library calls.

•        Style and tone: Fine-tuning on brand voice bakes that tone in-weights so every response feels on-brand without explicit prompting.

 

 

4. In-cache Memory (KV-cache)

 

Type: Ephemeral |  Nature: Fast, Compute-level

Saved intermediate computation states (key-value attention matrices). Avoids re-processing tokens the model has already seen, cutting latency and cost dramatically.

 

Analogy

🎹 Muscle memory for a pianist After playing a piece many times, fingers move without consciously re-thinking each note. The AI does not recompute the system prompt tokens on every reply — those computations are cached and reused.

 

Strengths

•        Reduces latency dramatically on repeated context

•        Cuts API cost — prompt caching charges less for cached tokens

•        Largely transparent to developers — works automatically

Limitations

•        Cleared between sessions — not persistent storage

•        Cache is invalidated if the prefix changes

•        Not real knowledge storage — purely a performance optimisation

Use Cases

•        Long system prompts: A 10,000-token system prompt is cached so every user turn does not re-pay the full processing cost.

•        Agentic loops: An agent that calls the model dozens of times in a task reuses cached context of prior steps, keeping latency low.

•        High-volume APIs: Apps with thousands of users sharing the same base prompt benefit from shared KV-cache at the infrastructure level.

•        Multi-turn chat: Conversation history is incrementally cached; each new user message only pays to process the new tokens.

 

 

Side-by-side Comparison

The table below summarizes the key dimensions across all four memory types.

 

Dimension	In-context	External	In-weights	In-cache
Persistence	Session only	Permanent (DB)	Until retrained	Session / server life
Capacity	Context window limit	Unlimited (DB scale)	Model parameter space	GPU memory bound
Speed	Instant	Retrieval latency	Zero (it is the model)	Near-instant
Update	Auto as chat grows	Explicit write ops	Requires retraining	Automatic, prefix-based
Cost	Per-token processing	Storage + retrieval	High upfront training	GPU memory usage

 

 

Combining Memory Types

The most powerful AI systems combine all four types strategically:

 

•        RAG system: In-weights for language understanding + external for fresh facts + in-context for the query + in-cache for a repeated system prompt.

•        Enterprise chatbot: Fine-tune for tone (in-weights) + company knowledge base (external) + conversation history (in-context).

•        Personal AI assistant: User preferences stored externally, general knowledge from weights, active task in context, long system prompt cached.

 

 

Key Takeaways

•        No single memory type does everything — they are complementary, not competing.

•        In-context memory is your default; add external memory when sessions must persist.

•        In-weights memory is your floor of capability; fine-tune to raise it for a specific domain.

•        In-cache memory is a performance multiplier — use it aggressively for long, repeated prompts.

•        Always consider the cost tradeoff: context tokens are processed every request; external retrieval adds latency; retraining is expensive; caching saves both.