When you use an AI coding agent, it seems to "remember" what you said three messages ago, track which file it is editing, recall your project conventions, and search through documentation for relevant examples. But that memory is not one system — it is five distinct mechanisms working together.
This tutorial explains each type of agent memory, how they differ, when to use each, and the privacy risks that come with storing agent knowledge.
The Five Types of Agent Memory
┌─────────────┐ ┌─────────────┐
│ 💬 History │ │ 📝 Working │
│ (ring buffer)│ │ (scratchpad)│
└──────┬──────┘ └──────┬──────┘
│ │
└────────┬─────────┘
│
▼
┌────────┐
│ 🧠 │
│ LLM │
└────────┘
│
┌───────┼───────┐
│ │ │
┌───▼───┐ ┌──▼──┐ ┌──▼─────┐
│💾 Pers.│ │🔍 Vec│ │⚙️ State│
│(store) │ │(search)│ │(tasks) │
└───────┘ └─────┘ └────────┘
Type 1: Conversation History
💬 What It Does
Stores the sequence of user and assistant messages. This is what gives an AI chatbot the feeling of "remembering" what you said earlier in the conversation.
Conversation history is typically implemented as a ring buffer — a fixed-size list that drops the oldest messages when full:
| Property | Value |
|---|---|
| Scope | Current session only |
| Persists | No — cleared when session ends |
| Search | FIFO (first in, first out) |
| Size limit | Context window (4K–1M tokens) |
| Use case | Multi-turn conversation, follow-up questions |
| Privacy risk | Low — stays in session, not stored |
When it fails: If the conversation is long, early context is dropped. The agent "forgets" what you said 20 messages ago.
Type 2: Working Memory
📝 What It Does
A per-task scratchpad that tracks current progress: which file is being edited, what bugs were found, what the current plan is. Cleared when the task finishes.
| Property | Value |
|---|---|
| Scope | Current task only |
| Persists | No — cleared between tasks |
| Search | Key lookup (fast) |
| Size limit | Small (few hundred tokens) |
| Use case | Tracking current debugging state, file being edited, errors encountered |
| Privacy risk | Low — ephemeral |
When it fails: If the task is long, the scratchpad can overflow. The agent may forget which step it was on.
Type 3: Persistent Memory
💾 What It Does
Stores information across sessions: user preferences, project conventions, known patterns, frequently used configurations. Survives between conversations.
| Property | Value |
|---|---|
| Scope | Cross-session, long-term |
| Persists | Yes — stored in database or file |
| Search | Key/category lookup |
| Size limit | Moderate (megabytes) |
| Use case | User preferences, project style, past decisions |
| Privacy risk | Medium — stored data may contain sensitive information |
Type 4: Vector Memory
🔍 What It Does
Enables semantic search over past knowledge. Instead of exact keyword matching, it finds conceptually similar content using embedding vectors. This is the foundation of RAG (Retrieval-Augmented Generation).
Vector memory works by converting text into numerical vectors and finding the closest matches:
| Property | Value |
|---|---|
| Scope | Cross-session, large corpus |
| Persists | Yes — stored in vector database |
| Search | Semantic (embedding similarity) |
| Size limit | Large (millions of entries) |
| Use case | RAG, documentation search, knowledge retrieval |
| Privacy risk | High — stored embeddings can leak information |
Type 5: Structured State
⚙️ What It Does
Tracks multi-step task progress with typed fields: task ID, status, steps completed, results, errors. This is what allows an agent to manage complex workflows across many iterations.
| Property | Value |
|---|---|
| Scope | Current task, typed |
| Persists | During task execution |
| Search | By task ID or status |
| Size limit | Depends on task complexity |
| Use case | Multi-step workflows, debugging, refactoring |
| Privacy risk | Low — task-scoped |
How the Five Types Work Together
In a real agent loop, all five memory types cooperate:
Memory and the Context Window
All memory types must ultimately fit into the LLM's context window. The challenge is choosing what to include:
| Strategy | What Goes in Context | Token Cost | Quality |
|---|---|---|---|
| Everything | Full history + working + persistent | High | Complete but expensive |
| Recent only | Last N messages + working | Low | May miss important context |
| RAG-retrieved | Relevant past docs + working + recent | Medium | Balanced, relevant |
| Summarized | Summary of history + working + relevant | Medium | May lose details |
Privacy Risks by Memory Type
| Memory Type | Data Stored | Privacy Risk | Mitigation |
|---|---|---|---|
| History | Messages, code snippets | Low-Medium | Clear on session end, encrypt in transit |
| Working | Task state, file paths | Low | Ephemeral, clear between tasks |
| Persistent | Preferences, patterns | Medium | Encrypt at rest, access control |
| Vector | Embeddings, document chunks | High | Encrypt, restrict access, audit |
| Structured | Task progress, results | Low | Task-scoped, clear after completion |
Memory vs Context vs State
These terms are often confused. Here is the precise distinction:
| Term | Meaning | Lifetime | Example |
|---|---|---|---|
| Context | What the LLM sees right now | Per request | The prompt sent to the API |
| State | Accumulated information about progress | Per task/session | Steps completed, current file |
| Memory | Persisted knowledge across sessions | Cross-session | User preferences, past documents |
Context is a snapshot. State is a record. Memory is a store.
Implementing Agent Memory: Practical Patterns
Pattern 1: Layered Context Assembly
Pattern 2: Memory-Aware Agent Loop
Common Memory Mistakes
| Mistake | Problem | Fix |
|---|---|---|
| Storing everything in history | Context overflow, high token cost | Use ring buffer, summarize old messages |
| Ignoring working memory | Agent repeats work, loses progress | Track current file, plan, and errors |
| Storing secrets in persistent memory | Security breach if store is compromised | Never store credentials; use secret managers |
| Vector memory without deduplication | Redundant search results | Hash and deduplicate before inserting |
| No structured state | Agent cannot resume interrupted tasks | Track task status and progress |
| Over-relying on context window | Expensive, slow, may exceed limits | Use RAG to retrieve relevant context |
Memory and Security
Agent memory creates specific security considerations:
- Injection via memory: If an attacker can write to persistent or vector memory, they can inject instructions that affect future agent behavior
- Data leakage: Vector embeddings of sensitive documents can sometimes be reconstructed
- Scope creep: Information stored for one purpose may be accessed for another
- Retention: Keeping memory too long increases exposure; not keeping it long enough reduces capability
- Access control: Multiple users sharing an agent should not see each other's memory
✅ Agent Memory Security Checklist
- ☐ Never store API keys, passwords, or tokens in agent memory
- ☐ Encrypt persistent memory at rest
- ☐ Encrypt vector embeddings at rest
- ☐ Implement access control per user/workspace
- ☐ Clear working memory between tasks
- ☐ Set retention limits on persistent memory
- ☐ Audit memory writes and reads
- ☐ Sanitize inputs before storing in vector memory
- ☐ Isolate user memory in multi-tenant systems
- ☐ Provide memory deletion capabilities (right to be forgotten)
Practical Exercises
Exercise 1: Trace the Memory
Read through the demo output. For each output, identify which memory type it demonstrates. What data is stored in each? What would be lost if that memory type were removed?
Exercise 2: Design Memory for a Coding Agent
Design the memory architecture for an AI coding agent. Which of the 5 types would you use? What would you store in each? What size limits would you set?
Exercise 3: Vector Memory Trade-offs
The demo uses TF-IDF for semantic search. What are the limitations compared to real embedding models? When would TF-IDF be insufficient? What privacy risks do real embedding models introduce?
Exercise 4: Privacy Audit
Review the security checklist. For each item, describe a realistic scenario where ignoring that item could lead to a security incident. Which items are most critical for a production system?
FAQ
Q: Which memory type is most important?
A: None is inherently more important — they serve different purposes. A well-designed agent needs at least conversation history (for dialogue), working memory (for current task), and some form of persistent or vector memory (for knowledge).
Q: Can I use all five types in one agent?
A: Yes, and production agents typically do. The key is assembling the right subset into the context window for each LLM call.
Q: How much context should I include?
A: Enough to be relevant, not so much that it overwhelms. A good rule: working memory + last 3-5 conversation turns + top 3-5 vector search results.
Q: Is vector memory the same as RAG?
A: Vector memory is a component of RAG. RAG adds chunking, retrieval, and context assembly on top of vector search.
Q: How do I handle memory across multiple users?
A: Isolate memory per user or workspace. Each user should have their own conversation history, working memory, and persistent memory. Vector stores should be partitioned by access control.
Q: Can agent memory be deleted?
A: It should be deletable for privacy compliance (e.g., GDPR right to erasure). Design memory stores with deletion capabilities from the start.
Further Reading
- Context Engineering Explained: The Next Skill After Prompt Engineering
- RAG Explained: A Complete Beginner-to-Advanced Guide
- Embeddings Explained: How AI Converts Meaning into Numbers
- How AI Agent Loops Work: Plan, Act, Observe and Repeat
- RAG Architecture Explained
- AI Tokens and Context Windows Explained
- AI Privacy by Design
Continue Learning: Understand context engineering, explore RAG from scratch, learn how agent loops work, and protect data with privacy by design.
Discuss this topic on BestWordz Community.