Parallelizing LLM Calls: Sectioning and Voting

When a team decides to run multiple LLM calls in parallel, the first question is not whether to parallelize but why. Parallelization solves two entirely different problems depending on how the work is split. Running three different analyses on the same text, then merging the results, is one thing. Running the same analysis three times, then voting, is something else. The patterns look similar from a distance and diverge sharply under inspection.

Anthropic names both in its taxonomy and treats them as distinct sub-patterns of the same workflow family: sectioning, where each parallel call handles a different aspect of the input, and voting, where the same task runs multiple times and the results are aggregated (Anthropic, 2024). Choosing between them is a design decision, not a cost-saving shortcut. This article separates the two shapes and notes where each applies.

Sectioning: different aspects of the same input

flowchart LR
    IN([Input]) --> SPLIT{Fan out}
    SPLIT --> T1[Extract entities]
    SPLIT --> T2[Analyze sentiment]
    SPLIT --> T3[Generate summary]
    T1 --> MERGE[Merge]
    T2 --> MERGE
    T3 --> MERGE
    MERGE --> OUT([Combined result])

Sectioning is parallelism for breadth. The input is analyzed along several independent dimensions at once, and the dimensions are known in advance. Each branch receives the same input and returns its own typed output; a merge step combines the results into a single object for downstream use.

The shape rewards tasks where the dimensions are genuinely independent. Entity extraction does not depend on sentiment analysis; neither depends on a summary. Running them in parallel uses wall-clock time well, and each subtask gets a focused prompt that would have been compromised by any composite request.

Sectioning also has a safety use. Running the main task on one model while a second model, in parallel, screens the input or the output for policy violations turns a guardrail into a zero-added-latency check. That variant reappears later in this series under guardrails.

The excerpt below shows sectioning without a framework. Three async coroutines run concurrently under asyncio.gather.

import asyncio
from openai import AsyncOpenAI
from pydantic import BaseModel, Field
from typing import Literal

client = AsyncOpenAI(timeout=60.0, max_retries=2)

class Entities(BaseModel):
    people: list[str]; orgs: list[str]; locations: list[str]

class Sentiment(BaseModel):
    label: Literal["positive", "negative", "neutral", "mixed"]
    confidence: float = Field(ge=0.0, le=1.0)

class Summary(BaseModel):
    one_line: str; key_points: list[str]

async def analyze(text: str) -> dict:
    async def extract():
        r = await client.responses.parse(model="gpt-4o-mini",
            instructions="Extract named entities.", input=text, text_format=Entities)
        return r.output[0].content[0].parsed

    async def sentiment():
        r = await client.responses.parse(model="gpt-4o-mini",
            instructions="Classify the overall sentiment.", input=text, text_format=Sentiment)
        return r.output[0].content[0].parsed

    async def summarize():
        r = await client.responses.parse(model="gpt-4o-mini",
            instructions="Summarize in one line plus 2-3 key points.",
            input=text, text_format=Summary)
        return r.output[0].content[0].parsed

    ents, sent, summ = await asyncio.gather(extract(), sentiment(), summarize())
    return {"entities": ents, "sentiment": sent, "summary": summ}

Voting: the same task, several times

flowchart LR
    IN([Input]) --> FAN{Fan out}
    FAN --> A1[Attempt 1]
    FAN --> A2[Attempt 2]
    FAN --> A3[Attempt 3]
    A1 --> VOTE[Majority or weighted vote]
    A2 --> VOTE
    A3 --> VOTE
    VOTE --> OUT([Aggregate result])

Voting is parallelism for confidence. The same prompt runs N times under different sampling, and the results are compared. For classification, the majority wins. For scoring, the mean or median. For free-text, a final model can rank or merge the candidates.

Voting trades cost for reliability. The technique is closely related to self-consistency in reasoning research, where sampling multiple chains of thought and taking the majority answer improves accuracy on arithmetic and commonsense benchmarks (Wang et al., 2022). It is most useful when individual calls are correct often enough to have a signal but wrong often enough that a single sample is not trustworthy.

The LangChain version below runs three parallel classifications of the same ticket and returns the majority vote.

import asyncio
from collections import Counter
from langchain.chat_models import init_chat_model
from pydantic import BaseModel
from typing import Literal

class Verdict(BaseModel):
    category: Literal["billing", "technical", "general", "urgent"]

model = init_chat_model("gpt-4o-mini", temperature=0.7).with_structured_output(Verdict)

async def vote(query: str, n: int = 5) -> str:
    results = await asyncio.gather(*[model.ainvoke(query) for _ in range(n)])
    counts = Counter(r.category for r in results)
    return counts.most_common(1)[0][0]

Full runnable versions of both shapes live at github.com/subodhjena/agentic-patterns under examples/06_parallelization.py and examples/06_parallelization_langgraph.py.

When each shape fits

Sectioning and voting are not interchangeable. The conditions below decide which one applies.

Reach for sectioning when the task naturally splits into independent aspects. Analysis pipelines (entities plus sentiment plus summary), multi-view generation (for different audiences, for different lengths), and parallel guardrails all fit. The aspects should not depend on each other; if they do, prompt chaining is the right shape.

Reach for voting when a single call is right often enough to have a signal but wrong often enough to need redundancy. Classification on ambiguous inputs, numeric scoring, and code solutions on multi-choice benchmarks benefit. Voting adds nothing when the task is deterministic; a single call is cheaper and equally accurate.

Reach for neither when the parallel calls would depend on each other. That is a signal to use orchestrator-workers, covered in the next article, where the set of parallel tasks is decomposed dynamically and the results are synthesized rather than merged.

Where voting fools you

Voting appears to increase confidence. Sometimes it does. Other times it amplifies a shared bias, because every sample inherits the same model's priors.

• Agreement is not accuracy. N calls that all land on the same wrong answer vote confidently for the wrong answer. Voting helps when errors are uncorrelated; it hurts when they are correlated. Correlated errors are common on hallucination-prone tasks.
• Temperature matters. Voting at temperature zero is nearly identical calls. Temperature must be high enough that samples diverge, low enough that they remain coherent. Calibrate empirically.
• Ties need a plan. With three samples and three different answers, the majority has one vote. A tiebreaker model or a "no majority" branch is mandatory.
• Cost scales linearly. Five-way voting costs five times as much as one call. The reliability gain needs to justify the bill.

Where sectioning wastes calls

Sectioning is safer than voting but has its own failure modes.

• Fake independence. When two subtasks overlap (entity extraction that implicitly summarizes, summary that implicitly classifies), running them in parallel recomputes shared work. Consolidate overlapping subtasks into one call.
• Merge step as bottleneck. If the merge is done by an LLM that has to read every parallel output, latency savings evaporate. Prefer a deterministic merge when possible.
• Overfanned pipelines. Ten parallel subtasks with marginal value each are a sign the task should have been one sharper prompt with structured output.

The trade against serial calls

The cost-benefit shape of parallelization depends on which variant is in play.

Axis	Serial calls	Sectioning	Voting
Wall-clock latency	Highest	Lowest, bound by slowest branch	Same as single call
Cost	One call per task	One call per branch	N times one call
Reliability	Baseline	Same per branch, narrower per prompt	Higher on tasks with uncorrelated errors
Complexity	Low	Medium, async plus merge	Medium, aggregation and ties
Typical use	Simple or dependent tasks	Multi-dimensional analysis	Ambiguous classification, numeric scoring

Sectioning trades cost for latency. Voting trades cost for reliability. Both are cheap to introduce and worth measuring rather than assuming.

Neighbors in the series

Routing, the previous article, picks one handler from a fixed list. Parallelization runs several branches at once. The patterns are often mistaken for each other; the distinction is whether the work is split or routed. Orchestrator-workers, the next article, generalizes the "fan out" side of parallelization to dynamic task decomposition: the set of parallel tasks is decided by a planning LLM at runtime rather than fixed in code. Self-consistency chain-of-thought, covered in the Reasoning stage, is voting applied specifically to reasoning traces.

References

1. Anthropic. Building effective agents. December 2024.
2. Wang, Xuezhi, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. 2022.
3. OpenAI. Parallel function calls and batched evaluation. 2024.
4. LangChain. LangGraph parallel branches. 2024.
5. Kojima, Takeshi, et al. Large Language Models are Zero-Shot Reasoners. 2022.