Short-Term and Long-Term Memory for LLM Agents

A language model is stateless. Each call is a function of its inputs, and anything the call should know about prior turns has to be in the prompt. Agents that appear to remember a conversation do so because their harness puts the previous turns back into the prompt. That is a form of memory, but it is only one kind. Production agents need several.

This article covers the four distinctions that make agent memory tractable: short-term versus long-term, and procedural versus episodic. The terms are borrowed from cognitive psychology but serve a concrete engineering purpose: they let a team name which store to write to, which to read from, and which to compact when the window fills. CrewAI's production memory system, Anthropic's effective-agents guide, and the generative-agents memory architecture from Stanford all use some form of this split (Park et al., 2023). The separations are not academic; they change which components the agent needs and how the data flows between them.

Four kinds of memory

flowchart TD
    subgraph Memory["Memory architecture"]
        direction TB
        ST[Short-term: within one session]
        LT[Long-term: across sessions]
        PR[Procedural: how to do things]
        EP[Episodic: what happened]
    end
    ST -.includes.-> MSG[Message history]
    ST -.includes.-> WS[Working state]
    LT -.includes.-> PREF[User preferences]
    LT -.includes.-> FACTS[Learned facts]
    PR -.includes.-> SP[System prompt]
    PR -.includes.-> TS[Tool schemas]
    EP -.includes.-> HIST[Past conversations]
    EP -.includes.-> TASKS[Task outcomes]

Short-term memory holds everything relevant to the current session. The running message history is short-term. The scratchpad where the agent holds intermediate computations is short-term. Working state that tracks where the agent is in a multi-step task is short-term. When the session ends, short-term memory can be discarded or compacted into long-term memory, but it does not need to persist by default.

Long-term memory holds what should survive across sessions. A user's preference for concise answers is long-term. Facts the agent has learned about the domain are long-term. Task outcomes, patterns of success and failure, and durable entity records are long-term. Long-term memory is usually a database, not a message list.

Procedural memory holds knowledge of how to do things. The system prompt is procedural. Tool schemas are procedural. Skill files, covered in a later article, are procedural. Procedural memory changes infrequently and is typically read at session start rather than queried during operation.

Episodic memory holds what happened. Past conversation summaries are episodic. Traces of completed tasks are episodic. Reflexion-style reflections, covered in the Reasoning stage, are episodic. Episodic memory is queried when the current task resembles a past one and the past outcome is informative.

The four are not exclusive. A specific piece of data can be both long-term and episodic (the summary of a past user complaint), or both short-term and working state (a running count of tool calls in the current turn). Naming which kind a memory belongs to makes the data flow explicit.

The composite score

Long-term memory usually has more content than fits in a context window. Retrieval picks a subset. The simplest retrieval is semantic similarity: embed the query, find the top-k nearest stored memories. In practice, this is not enough. Semantic similarity ignores how recent a memory is and how important it was when stored.

CrewAI's production system and the Stanford generative-agents architecture both use a composite score to address this (Park et al., 2023).

score = w_sem × similarity(query, memory)
      + w_rec × recency_decay(memory.last_accessed)
      + w_imp × importance(memory)

Similarity is standard embedding distance. Recency is an exponential decay based on how long ago the memory was last accessed; memories recently used float to the top. Importance is a score from 1 to 10 that an LLM assigns at creation time: mundane observations score low, significant events score high. Weights w_sem, w_rec, and w_imp are tuned empirically; Stanford reports roughly equal weighting as a reasonable default.

The composite score is why agent memory is more than RAG. A vector store alone finds similar memories; a composite-scored memory finds the memories that matter for the current turn.

Two versions in code

The excerpt below shows a minimal short-term plus long-term setup without a framework. Short-term is a message list; long-term is a dict of embedded entries with timestamps and importance scores.

from openai import OpenAI
from datetime import datetime
import math

client = OpenAI()

def embed(text: str) -> list[float]:
    return client.embeddings.create(model="text-embedding-3-small",
                                    input=text).data[0].embedding

def cosine(a: list[float], b: list[float]) -> float:
    s = sum(x*y for x, y in zip(a, b))
    na = sum(x*x for x in a) ** 0.5
    nb = sum(y*y for y in b) ** 0.5
    return s / (na * nb + 1e-9)

class Memory:
    def __init__(self):
        self.short_term: list[dict] = []
        self.long_term: list[dict] = []  # {text, embedding, importance, last_access}

    def remember(self, text: str, importance: float) -> None:
        self.long_term.append({
            "text": text, "embedding": embed(text),
            "importance": importance, "last_access": datetime.now(),
        })

    def recall(self, query: str, top_k: int = 3) -> list[str]:
        q = embed(query)
        now = datetime.now()
        scored = []
        for m in self.long_term:
            sim = cosine(q, m["embedding"])
            hours = (now - m["last_access"]).total_seconds() / 3600
            recency = math.exp(-hours / 72)  # half-life around 50 hours
            score = 0.5 * sim + 0.25 * recency + 0.25 * (m["importance"] / 10)
            scored.append((score, m))
        scored.sort(reverse=True, key=lambda x: x[0])
        for _, m in scored[:top_k]:
            m["last_access"] = now
        return [m["text"] for _, m in scored[:top_k]]

The LangGraph version uses the built-in checkpointer for short-term memory (covered in the next article) and the built-in Store for cross-thread long-term memory. The short-term portion is handled automatically; the long-term store exposes put and search methods.

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore
from langchain.chat_models import init_chat_model
from langgraph.prebuilt import create_react_agent
from langchain_core.tools import tool

store = InMemoryStore()

@tool
def remember(fact: str, user_id: str = "default") -> str:
    """Save a fact to long-term memory for the user."""
    store.put(("user", user_id, "facts"), fact, {"text": fact})
    return "saved"

@tool
def recall(query: str, user_id: str = "default") -> str:
    """Recall facts about the user matching the query."""
    results = store.search(("user", user_id, "facts"), query=query, limit=3)
    return "\n".join(r.value["text"] for r in results) or "none"

agent = create_react_agent(
    model=init_chat_model("gpt-4o-mini"),
    tools=[remember, recall],
    checkpointer=InMemorySaver(),  # short-term
)

Full runnable versions will live at github.com/subodhjena/agentic-patterns under examples/21_memory.py as that lesson lands.

Where each kind earns its keep

Each of the four kinds addresses a specific failure.

Short-term memory. Without it, the agent treats every turn as fresh. Users who say "what I just asked" get confused responses. Fix: include the recent message history on every call, up to a bounded number of turns.

Long-term memory. Without it, the agent forgets across sessions. A user who has expressed a preference in one session restates it in the next. Fix: a durable store the agent can write to and read from by key (user id, project id) across sessions.

Procedural memory. Without it, the agent does not know what tools it has, what policies apply, or how to format its responses. Fix: a system prompt, tool schemas, and optionally a skills folder that are loaded at session start.

Episodic memory. Without it, the agent cannot learn from past outcomes. A task the agent has solved before is solved from scratch; a mistake it made before is made again. Fix: summaries of completed tasks and reflections on failures, stored durably and retrieved when similar tasks arise.

Teams sometimes try to solve all four with a single undifferentiated "memory" store. The symptom is a store that is both too small (missing what matters) and too polluted (carrying what does not).

Where memory goes wrong

Common failures cluster around three issues: hygiene, retrieval calibration, and leakage.

Hygiene: unbounded short-term growth. The message list grows each turn; without compaction, it fills the window. Apply compaction or sliding windows on a schedule, not on a crisis.

Hygiene: long-term bloat. Memories written to long-term storage without an importance filter accumulate low-signal entries. Retrieval becomes noisy. Only write memories the agent judges important; delete or decay unimportant ones on a schedule.

Retrieval: weights miscalibrated. A composite score with wrong weights retrieves irrelevant memories more often than relevant ones. Measure retrieval quality on a labeled set before relying on it in production.

Retrieval: missing reflections. Raw observations are often too noisy to drive good decisions. Higher-level reflections (covered in the next article) compress the observations into more useful entries. Without reflections, retrieval surfaces the noise.

Leakage: cross-user contamination. Memories scoped wrongly leak across users. A preference saved for one user surfaces for another. Make the key hierarchy explicit: (scope, user_id, category) is the minimum; store boundaries should be enforced at the retrieval layer, not trusted to prompt discipline.

Leakage: cross-session contamination. A session-local working state written to long-term by mistake pollutes future sessions. Keep short-term and long-term stores physically separate and promote deliberately.

Trade against a bigger context window

A reasonable question is whether long context eliminates the need for memory. It does not. A one-million-token window is enough for a day of work but not for a year. A memory system is the compression that turns a year of context into a few thousand tokens of relevant retrieval per turn. The table below compares the approaches.

Axis	Long context only	Memory with retrieval
Token cost per turn	Proportional to full context	Proportional to top-k retrieved
Persistence across sessions	None	Yes, durable store
Retrieval cost	None	Embedding lookup plus scoring
Freshness control	Implicit (everything stays)	Explicit (recency decay)
Importance control	Implicit	Explicit (importance score)
Leakage control	Per-window	Per-store

Long context helps inside a session. Memory systems are what let an agent have a year of history without paying for it on every turn.

Neighbors in the series

The augmented LLM article, in Foundations, introduces retrieval, tools, and memory as the three primitives. Context engineering, also in Foundations, covers the compaction strategies that apply to short-term memory specifically. Generative agents memory, the next article, describes the Stanford architecture in depth, including the reflection mechanism. Persistence and checkpointing, two articles later, covers the concrete infrastructure (thread state, cross-thread store, backends) that implements the abstractions here. Skills as contextual memory, the article after that, covers procedural memory at scale.

References

1. Park, Joon Sung, et al. Generative Agents: Interactive Simulacra of Human Behavior. UIST 2023.
2. Anthropic. Building effective agents. December 2024.
3. CrewAI. Memory systems documentation. 2024.
4. LangChain. LangGraph checkpointers and Store. 2024.
5. Shinn, Noah, et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023.