Agent Memory Systems: Design and Implementation

Hello! Welcome to your fourth lesson on Agentic AI Systems.

In our last session, you masterfully implemented the ReAct framework, giving your agent an "inner monologue" to reason through multi-step tasks. The agent we built used a simple list of messages as its memory. This is effective for a single conversation but has two major limitations:

1. It's ephemeral: The memory is lost once the session ends.
2. It's finite: As the conversation grows, it will eventually exceed the LLM's context window.

Today, we will address these limitations head-on. Your learning outcome is to design and implement basic memory systems for agents. We will explore how to give our agents both a short-term working memory and a persistent long-term memory, enabling them to learn and retain information across multiple interactions.

1. A Conceptual Framework for Agent Memory

Before we write any code, it's crucial to have a solid mental model for what "memory" means for an agent. Drawing inspiration from cognitive psychology can be incredibly helpful. An agent's memory can be broadly categorized into several types, each serving a different purpose.

As you've just seen, we can think of an agent's memory system as having several layers:

• Working Memory (Short-Term): The context of the current interaction. This is typically the history of messages in the current conversation. It's volatile and fast.
• Long-Term Memory: A persistent store of information that survives across sessions. We can further divide this into:
  • Episodic Memory: Recalls past conversations and interactions. Answers "What happened between us before?"
  • Semantic Memory: Stores general knowledge and facts. Answers "What is true about the world or the user?"

For this lesson, we will focus on implementing a robust Working Memory and a foundational Semantic Memory that can be updated, laying the groundwork for more complex episodic systems later.

2. From Simple History to a True Memory System

The simplest form of memory, which you've already implemented, is just keeping a running list of user and assistant messages.

The challenge, as mentioned, is the limited context window. A simple but effective technique to manage this is Context Compaction. Before the history becomes too large, you can use an LLM to summarize the older parts of the conversation, replacing many messages with a single summary message.

# Conceptual example of context compaction
def compact_context(conversation_history: list, max_tokens: int) -> list:
    # (Implementation would use a tokenizer to count tokens)
    if num_tokens(conversation_history) < max_tokens:
        return conversation_history

    # Keep the most recent turns intact
    recent_messages = conversation_history[-6:]
    old_messages = conversation_history[:-6]

    # Use an LLM to summarize the old messages
    summary_prompt = f"Summarize the following conversation concisely: {old_messages}"
    summary = call_llm(summary_prompt)

    # Replace the old messages with the summary
    return [{"role": "system", "content": f"Summary of prior conversation: {summary}"}] + recent_messages

This is a powerful pattern for managing working memory in long-running conversations.

3. Implementing a Persistent, Updatable Long-Term Memory

Now for the main event: creating a memory that persists. We'll use a vector database to build a semantic memory store, but with a crucial addition—the ability to update and delete memories to prevent contradictions.

This process involves four key challenges:

1. Extraction: Deciding what information is worth remembering.
2. Indexing: Storing that information for efficient retrieval.
3. Retrieval: Fetching relevant memories when needed.
4. Updating: Modifying or deleting stored memories when new information supersedes them.

We will follow a practical, from-scratch implementation inspired by the Mem0 paper. The video below uses the DSPy framework, but the underlying logic is pure Python and can be implemented with standard libraries and an LLM client. Your experience with Python and APIs will allow you to easily adapt these concepts.

Step 1: Extraction and Indexing

First, we need to extract atomic "memories" from a conversation and store them in a vector database for semantic search.

At this stage, you have a RAG pipeline: text is converted into memories, embedded, and stored. When a new query arrives, you embed the query, search the vector DB for similar memories, and add them to the agent's context.

Step 2: The Critical Update Mechanism

This is what transforms a simple RAG store into a dynamic memory system. What happens when information changes?

To implement this, we'll give our agent a new set of tools specifically for memory management. This perfectly builds on the ReAct and function-calling skills you developed in the last two lessons.

Step 3: Integrating Memory into the Agent Loop

Now, we integrate this memory system into our main agent's operational cycle.

By completing this, you've designed and implemented a full-fledged memory system that can extract, index, retrieve, and intelligently update information over time.

Conclusion

In this lesson, you've moved beyond the simple, ephemeral memory of a conversation list and designed a truly dynamic memory system for your agent. This is a fundamental component for building sophisticated, long-running AI assistants that learn and adapt.

Key Takeaways:

• Agent memory is layered: We can model it with a Working Memory (current context) and a Long-Term Memory (semantic and episodic).
• Working Memory can be managed: Techniques like context compaction (summarization) help overcome context window limitations.
• Long-Term Memory needs to be dynamic: A simple vector store is not enough. A robust system must handle updates, preventing the accumulation of stale or contradictory information.
• LLMs can manage their own memory: By using a ReAct agent with tools like add, update, and delete, you can create a self-managing memory system that intelligently maintains its own knowledge base.

Preview of the Next Lesson:

With a robust memory system in place, your agent can now remember goals, sub-tasks, and the results of its actions over extended periods. This capability is the bedrock of advanced planning. In the next lesson, we will apply task decomposition and planning for multi-step agent workflows, enabling your agent to tackle highly complex problems by breaking them down into smaller, manageable steps and tracking its progress using its newly acquired memory.