Hierarchical Teams: Supervisors of Supervisors in Multi-Agent Systems

A single supervisor works well when it coordinates a small number of specialists. Beyond that, the supervisor's prompt fills with delegation policy, its context fills with intermediate results, and its decisions begin to degrade in the same way a single overloaded agent does. The next obvious step is a second layer of delegation: supervisors that manage supervisors.

Hierarchical teams generalize the supervisor-and-router pattern to multiple layers. A top-level supervisor coordinates team leads; each team lead coordinates its own specialists. The structure mirrors how human organizations scale, and it is the dominant pattern when the number of specialists crosses roughly ten. Microsoft Research's MagenticOne is the most studied production reference (Fourney et al., 2024): an Orchestrator agent with a dual-ledger system directs four specialists (WebSurfer, FileSurfer, Coder, ComputerTerminal) and achieved first-place accuracy on the GAIA benchmark across all difficulty levels. The dual-ledger idea, covered below, is what makes hierarchical coordination tractable at scale.

Two layers

flowchart TD
    TOP[Top Supervisor] --> LEAD1[Research Lead]
    TOP --> LEAD2[Dev Lead]
    TOP --> LEAD3[QA Lead]
    LEAD1 --> R1[Web specialist]
    LEAD1 --> R2[DB specialist]
    LEAD1 --> R3[API specialist]
    LEAD2 --> D1[Frontend specialist]
    LEAD2 --> D2[Backend specialist]
    LEAD2 --> D3[Infra specialist]
    LEAD3 --> Q1[Unit tests]
    LEAD3 --> Q2[E2E tests]
    LEAD3 --> Q3[Performance tests]

The top supervisor sees only the team leads. Each team lead sees only its own specialists. No agent in the tree is responsible for more handlers than a single agent can steward well. The specialists' scratchpads remain isolated; the team leads' outputs flow to the top supervisor; the top supervisor synthesizes a response.

The dual ledger

A flat supervisor holds its state in a single message list: the request, the tool calls it has made, and the responses it has received. This works for a handful of turns. At scale, the supervisor loses track of what the original plan was and how far it has progressed. MagenticOne's innovation is to separate these two concerns into two ledgers.

Task Ledger holds the plan. It names the goal, the sub-tasks that have been decomposed from it, and the constraints the plan must honor. The task ledger is written once when the supervisor receives the request and rewritten only when the plan materially changes (typically after a surprise that forces replanning).

Progress Ledger holds the state of execution. It names which sub-tasks have been completed, which are in progress, which are blocked, and which remain. The progress ledger is updated after every specialist response.

Keeping the two separate gives the supervisor an explicit answer to two questions that would otherwise be buried in the message history: what are we trying to do, and where are we? Without the separation, those questions are answered by the supervisor rereading the full conversation every turn, which is both expensive and error-prone.

Two versions in code

The excerpt below shows a two-level hierarchy without a framework. The top supervisor delegates to team leads; each team lead delegates to specialists. The dual ledger lives on the top supervisor.

from openai import OpenAI
from pydantic import BaseModel
import json

client = OpenAI()

class TaskLedger(BaseModel):
    goal: str
    subtasks: list[str]

class ProgressLedger(BaseModel):
    completed: list[str] = []
    in_progress: list[str] = []
    blocked: list[str] = []

def specialist(prompt: str, instructions: str) -> str:
    return client.responses.create(
        model="gpt-4o-mini", instructions=instructions, input=prompt,
    ).output_text

def team_lead(task: str, role: str, specialists: dict) -> str:
    # A team lead dispatches to its own specialists. Minimal one-level dispatch.
    # In production, each team lead is a full tool-calling agent.
    instructions = f"You lead the {role} team. Call specialists as needed."
    # ... a full implementation mirrors the supervisor pattern at a smaller scope ...
    return specialist(task, instructions)

def top_supervisor(goal: str) -> str:
    task_ledger = TaskLedger(
        goal=goal,
        subtasks=["research the topic", "implement the solution", "run QA checks"])
    progress = ProgressLedger()

    leads = {
        "research": ("Research Lead", {"web": None, "db": None}),
        "dev": ("Dev Lead", {"frontend": None, "backend": None}),
        "qa": ("QA Lead", {"unit": None, "e2e": None}),
    }

    outputs = []
    for subtask in task_ledger.subtasks:
        progress.in_progress.append(subtask)
        # Simple map from subtask to team lead; real systems use an LLM.
        if "research" in subtask: role, specs = leads["research"]
        elif "QA" in subtask:     role, specs = leads["qa"]
        else:                     role, specs = leads["dev"]
        out = team_lead(subtask, role, specs)
        outputs.append(f"{subtask}: {out}")
        progress.in_progress.remove(subtask)
        progress.completed.append(subtask)

    return client.responses.create(
        model="gpt-4o-mini",
        instructions="Synthesize team outputs into a final response.",
        input=f"Goal: {goal}\n\nResults:\n" + "\n".join(outputs),
    ).output_text

The LangGraph version uses nested create_react_agent instances. Each team lead is itself a supervisor with its own tool-delegation surface; the top supervisor's tools are the team leads.

from langchain_core.tools import tool
from langchain.chat_models import init_chat_model
from langgraph.prebuilt import create_react_agent

model = init_chat_model("gpt-4o-mini")

research_lead = create_react_agent(model=model, tools=[web_tool, db_tool])
dev_lead = create_react_agent(model=model, tools=[frontend_tool, backend_tool, infra_tool])
qa_lead = create_react_agent(model=model, tools=[unit_tool, e2e_tool, perf_tool])

@tool
def delegate_research(task: str) -> str:
    """Delegate to the research team. Returns a research summary."""
    return research_lead.invoke({"messages": [("user", task)]})["messages"][-1].content

@tool
def delegate_dev(task: str) -> str:
    """Delegate to the dev team. Returns implementation plus notes."""
    return dev_lead.invoke({"messages": [("user", task)]})["messages"][-1].content

@tool
def delegate_qa(task: str) -> str:
    """Delegate to the QA team. Returns QA report."""
    return qa_lead.invoke({"messages": [("user", task)]})["messages"][-1].content

top_supervisor = create_react_agent(
    model=model,
    tools=[delegate_research, delegate_dev, delegate_qa],
)

Full runnable versions will live at github.com/subodhjena/agentic-patterns under examples/17_hierarchical_teams.py as that lesson lands in the repo.

Where the hierarchy helps

Hierarchical teams earn their cost in a specific operational regime.

Team counts in the dozens. Once the number of specialists exceeds roughly ten, a single supervisor's prompt and context window no longer handle delegation cleanly. Adding a layer cuts the fan-out at every level.

Natural domain splits. Research, development, and QA are meaningfully different domains with different tools, different vocabularies, and different review criteria. The hierarchy follows those splits; forcing a hierarchy onto a homogeneous workload produces arbitrary divisions.

Long tasks with shifting focus. A task that takes minutes and crosses domains needs a progress-tracking mechanism at the top. The dual ledger is designed for exactly this case.

Auditability requirements. Each layer of the hierarchy produces its own trace; debugging which team produced a problematic output is straightforward. Single-agent systems are harder to audit.

Where the hierarchy hurts

The failure modes follow predictably from any organizational hierarchy, human or artificial.

Message dilution. Each delegation summarizes context. A request passed through two layers of delegation loses detail the specialist at the bottom might need. Either widen the delegation payload or flatten the hierarchy.

Latency stacking. Each layer adds round-trips. A task that would be one call in a flat ReAct agent becomes three or four calls across two layers of delegation. Pay this only when the organizational clarity is worth the latency.

Synchronization overhead. Two team leads may need to coordinate, but in a strict hierarchy they cannot. The shared scratchpad pattern, covered in two articles, addresses this case directly. Mixing hierarchy with shared workspaces is common in practice.

Over-layered trees. Three layers of delegation for a small task is bureaucracy. Start with one supervisor and add layers only when the turn budget at each layer starts to overflow.

Ledger rot. A task ledger that is not updated when the plan actually changes leaves the system coordinating against a plan that no longer matches reality. The ledger is load-bearing; treat it as code, not documentation.

Leaks across scratchpads. Information that a specialist should not see sometimes flows up to the supervisor and then back down to a different specialist. Keep delegation payloads minimal and redact information the next layer does not need.

Trade against a flat supervisor

A flat supervisor-and-router pattern is a simpler version of the same idea. The table describes when to add the second layer.

Axis	Flat supervisor	Hierarchical teams
Number of specialists	Up to roughly ten	Beyond ten, naturally grouped
Context pressure on supervisor	Manageable	Distributes across leads
Latency overhead	One layer	Two or more layers
State tracking	Message list	Dual ledger becomes necessary
Implementation complexity	Medium	High
Debug surface	Two levels	Three or more levels

Adopting a hierarchy is a one-way door of sorts: it solves the scaling problem at the cost of permanent latency and complexity overhead. Reach for it when the flat pattern is actually failing.

Neighbors in the series

Supervisor and router, the previous article, is the single-layer pattern hierarchical teams extend. Orchestrator-workers, in the Workflows stage, is the non-agent cousin of the specialist layer: workers are LLM calls rather than full agents. Handoffs and swarms, the next article, is the peer-to-peer alternative that abandons the hierarchy entirely. Harness design, in the Production stage, describes the planner-generator-evaluator architecture that often sits at the top of a hierarchical system.

References

1. Fourney, Adam, et al. Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks. Microsoft Research, 2024.
2. Anthropic. Building effective agents. December 2024.
3. Wu, Qingyun, et al. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. 2023.
4. LangChain. Hierarchical agent teams in LangGraph. 2024.
5. Mialon, Gregoire, et al. GAIA: A Benchmark for General AI Assistants. 2023.