Reflexion: Verbal Self-Critique After Failure for LLM Agents

Reinforcement learning trains an agent by updating its parameters based on a reward signal. For a language model driving an agent, that traditional formulation is expensive and rigid: it requires gradient access, a carefully shaped reward, and substantial compute. The result is also opaque. The agent improves, but nothing explains what changed or why.

Reflexion, introduced by Shinn and collaborators in 2023, proposes a lightweight alternative. After a task fails, a separate LLM call writes a natural-language critique of the attempt. The critique is stored in an episodic memory. On the next attempt, the agent sees the critique in its prompt. The model's parameters are not updated; what changes is the conditioning text (Shinn et al., 2023). The authors call this verbal reinforcement learning, and the approach works because modern LLMs can effectively condition on detailed descriptions of their own past failures.

The results are strong. HumanEval code generation accuracy reaches 91 percent pass-at-1, compared with a GPT-4 baseline of 80 percent. ALFWorld success rates reach 97 percent after twelve iterative trials, substantially above the ReAct baseline of 75 percent. This article describes the three components of the pattern, the training-free mechanism that makes it work, and the conditions under which the pattern earns its cost.

Three components

Reflexion decomposes into three roles, each an LLM call with a specific mandate.

The actor is the base agent. It takes actions in the environment, typically in a ReAct-style tool-calling loop. The actor is prompted with the task, any prior reflections, and the conversation history for the current attempt. The actor is the only component that interacts with the outside world.

The evaluator judges whether the actor's trajectory succeeded. For tasks with a ground truth (code tests pass, a puzzle is solved, an answer is correct), the evaluator can be a deterministic checker. For open-ended tasks, the evaluator is an LLM with a scoring prompt. It returns a score and a verdict; the loop terminates when the verdict is success.

The self-reflector is the new component. After a failed attempt, the self-reflector reads the task, the full trajectory, and the evaluator's verdict. It writes a natural-language critique that names what went wrong, what should have been tried instead, and what to remember on the next attempt. The critique is appended to an episodic memory that persists across attempts.

The separation of roles matters for the same reason the evaluator in evaluator-optimizer must be separate from the generator: a model primed to produce a trajectory is not calibrated to find flaws in that trajectory. A distinct self-reflector can afford to be blunt about what failed.

The loop

flowchart LR
    T([Task]) --> A[Actor agent]
    MEM[(Reflection memory)] -.reads.-> A
    A --> TR[Trajectory]
    TR --> EV[Evaluator]
    EV -->|success| OUT([Return result])
    EV -->|failure| SR[Self-reflector]
    SR --> R[Verbal critique]
    R -.appends.-> MEM
    R --> A

Each iteration of the outer loop runs a full attempt of the task. The inner structure, the actor's tool-calling loop, is unchanged from the tool-calling agent loop covered earlier in this series. Reflexion wraps that inner loop with an evaluation and a critique on failure.

Two versions in code

The excerpt below sketches the pattern without a framework. The actor, evaluator, and self-reflector are three LLM calls; reflections accumulate in a list that the actor reads at the start of each attempt.

from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Literal

client = OpenAI()

class Verdict(BaseModel):
    score: float = Field(ge=0.0, le=1.0)
    outcome: Literal["success", "failure"]

class Reflection(BaseModel):
    what_went_wrong: str
    what_to_try_next: str

def actor(task: str, reflections: list[Reflection]) -> str:
    notes = "\n".join(f"- {r.what_to_try_next}" for r in reflections)
    return client.responses.create(
        model="gpt-4o-mini",
        instructions=(f"Complete the task. Apply lessons from past attempts.\n\n"
                      f"Past lessons:\n{notes}" if notes else "Complete the task."),
        input=task,
    ).output_text

def evaluate(task: str, trajectory: str) -> Verdict:
    r = client.responses.parse(
        model="gpt-4o-mini",
        instructions="Was the task completed correctly? Be strict.",
        input=f"Task: {task}\nAttempt:\n{trajectory}",
        text_format=Verdict,
    )
    return r.output[0].content[0].parsed

def reflect(task: str, trajectory: str, verdict: Verdict) -> Reflection:
    r = client.responses.parse(
        model="gpt-4o-mini",
        instructions=("Diagnose the failure. Name what went wrong and what to "
                      "try differently on the next attempt."),
        input=(f"Task: {task}\nTrajectory:\n{trajectory}\n"
               f"Verdict: {verdict.outcome}, score {verdict.score}"),
        text_format=Reflection,
    )
    return r.output[0].content[0].parsed

def run(task: str, max_attempts: int = 4) -> str:
    memory: list[Reflection] = []
    best = ""
    for _ in range(max_attempts):
        attempt = actor(task, memory)
        v = evaluate(task, attempt)
        best = attempt
        if v.outcome == "success":
            return attempt
        memory.append(reflect(task, attempt, v))
    return best

The LangGraph version expresses the three components as nodes. The state carries accumulated reflections; a conditional edge routes back to the actor on failure.

from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langchain.chat_models import init_chat_model

model = init_chat_model("gpt-4o-mini")

class State(TypedDict):
    task: str
    attempts: int
    trajectory: str
    verdict: Verdict
    memory: list[Reflection]
    result: str

def act_node(s: State) -> State:
    notes = "\n".join(f"- {r.what_to_try_next}" for r in s["memory"])
    out = model.invoke(f"Task: {s['task']}\nLessons:\n{notes}").content
    return {**s, "trajectory": out, "attempts": s["attempts"] + 1}

def eval_node(s: State) -> State:
    v = model.with_structured_output(Verdict).invoke(
        f"Task: {s['task']}\nAttempt: {s['trajectory']}")
    return {**s, "verdict": v, "result": s["trajectory"]}

def reflect_node(s: State) -> State:
    r = model.with_structured_output(Reflection).invoke(
        f"Task: {s['task']}\nTrajectory: {s['trajectory']}")
    return {**s, "memory": s["memory"] + [r]}

def route(s: State) -> str:
    if s["verdict"].outcome == "success": return "done"
    if s["attempts"] >= 4: return "done"
    return "reflect"

graph = (StateGraph(State)
         .add_node("act", act_node).add_node("evaluate", eval_node)
         .add_node("reflect", reflect_node)
         .add_edge(START, "act").add_edge("act", "evaluate")
         .add_conditional_edges("evaluate", route,
                                {"reflect": "reflect", "done": END})
         .add_edge("reflect", "act")
         .compile())

Full runnable versions live at github.com/subodhjena/agentic-patterns under examples/14_reflexion.py and examples/14_reflexion_langgraph.py.

Why verbal reinforcement learning works

Traditional reinforcement learning updates model parameters. Reflexion updates text. The model is unchanged across attempts; what changes is the prompt it reads. The technique works because large language models condition effectively on detailed natural-language descriptions of past failures. The reflection is not a training signal in the gradient sense; it is context that changes the distribution of next-token predictions on the following attempt.

The practical implications are substantial. No gradient access is required. No curated reward shaping is needed beyond the binary success-or-failure evaluator. The reflections are human-readable, which makes debugging and auditing tractable. The technique works on closed-weight models and on open-weight models equally.

Shinn et al. frame this as verbal reinforcement learning to emphasize the analogy; the pattern is not reinforcement learning in the strict technical sense, but the functional role of the reflections in the outer loop is analogous to the role of gradient updates in traditional RL.

When Reflexion helps

The pattern has a narrow and defensible sweet spot.

Tasks with a binary success signal. Code with passing tests, a puzzle that is solved, a math problem with a correct answer. The evaluator is cheap and reliable.

Tasks where failures have describable causes. A bug in generated code usually has a root cause that can be named in one or two sentences. That sentence is a good reflection. Tasks where failures are opaque ("the model just feels off") benefit less.

Tasks where multiple attempts are acceptable. Reflexion takes N attempts to converge. If the user cannot wait for N attempts, the pattern is not the right choice.

Tasks with a large gap between first-attempt accuracy and upper-bound accuracy. Reflexion closes a gap. If the first attempt is already at 95 percent, the upside is small; if first-attempt accuracy is at 70 percent and the ceiling is 95 percent, the pattern has room to work.

HumanEval and ALFWorld both fit all four conditions. They are the benchmarks the paper uses because they fit; other benchmarks may not.

Where Reflexion goes wrong

The failure modes cluster around memory hygiene and evaluator quality.

Unbounded memory. Reflections accumulate over attempts and pollute the window on long runs. Cap memory size and keep the most recent or highest-signal reflections.

Vague reflections. A reflection that says "think more carefully" does not help the next attempt. Prompt the self-reflector to name a specific action or decision to change.

Contradictory reflections. Two reflections from different failures advise opposite approaches. The actor oscillates. Prompt the self-reflector to consult prior reflections and resolve conflicts when writing a new one.

Evaluator drift. An evaluator that becomes more lenient across attempts terminates the loop on a mediocre outcome. Fix evaluator prompts once and do not tune them across attempts.

Successful-but-brittle attempts. An attempt that technically succeeds under a narrow test may fail under a broader one. Evaluators should test as strictly as the production context.

Agent memorizes and overfits. A reflection tailored to one task gets stored and retrieved on a different task where it does not apply. Keep reflections scoped to the task instance when possible.

Trade against evaluator-optimizer and ReAct

Reflexion is a cousin of evaluator-optimizer (Workflows stage) and a superset of ReAct (Agents stage). The axes below name the choice.

Axis	ReAct	Evaluator-optimizer	Reflexion
What iterates	Nothing	A single artifact (draft)	The whole task attempt
Critique target	N/A	Current draft	Past trajectory
Memory across iterations	None	None (or single feedback)	Explicit episodic memory
Typical cost	1 attempt	2 to 4 draft calls	N full attempts
Best-fit tasks	Exploratory, tool-using	Writing, code review cycles	Task reattempt scenarios

Reflexion is what evaluator-optimizer becomes when the artifact being refined is a full trajectory rather than a single draft.

Neighbors in the series

Evaluator-optimizer, in the Workflows stage, is the non-trajectory cousin that runs the generate-evaluate loop on a single artifact. ReAct, earlier in the Agents stage, is the actor Reflexion wraps. LATS, next in the Reasoning stage, combines reflection-style reasoning with Monte Carlo tree search across a single attempt. The Short-term and Long-term Memory article, in the Memory stage, covers the persistence side of Reflexion's episodic memory more broadly.

References

1. Shinn, Noah, et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023.
2. Yao, Shunyu, et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.
3. Madaan, Aman, et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023.
4. LangChain. Reflexion-style agents in LangGraph. 2024.
5. Anthropic. Agents that iterate. 2024.