Chain-of-Thought Prompting for LLM Reasoning

Before 2022, the default way to extract an answer from a language model on a multi-step problem was to ask for the answer directly. The model produced the final token first, and any reasoning that happened was implicit and compressed into that single emission. Wei and collaborators at Google Brain showed that an almost trivial change, prompting the model to show its work, produced substantial gains on arithmetic, commonsense, and symbolic reasoning benchmarks (Wei et al., 2022).

The technique is called chain-of-thought prompting. The paper demonstrated three variants that now anchor the reasoning literature: few-shot CoT, in which the prompt contains examples of reasoning traces; zero-shot CoT, in which the phrase "let's think step by step" alone triggers the behavior (Kojima et al., 2022); and self-consistency, in which multiple reasoning paths are sampled and the majority answer is taken (Wang et al., 2022). This article covers all three, the one scale-dependent catch that makes the technique work, and how to wire it into agentic patterns.

Why show the work

A language model predicts the next token given the tokens that came before. When the answer is the first token emitted, every bit of reasoning that should have preceded it is compressed into the hidden state. On multi-step problems, this compression is lossy; the model often produces a confident wrong answer. When the model emits its reasoning first, each step becomes a token sequence that the next step can condition on. The chain is explicit. The final answer is produced after the reasoning rather than instead of it.

The technique scales with problem depth. A one-step problem barely benefits from chain-of-thought; a five-step arithmetic word problem benefits substantially. This matches the mechanistic story: the longer the chain needed to reach the answer, the more a model benefits from producing the intermediate steps as tokens rather than holding them in its hidden state.

There is an important caveat. Chain-of-thought is an emergent ability of scale. Smaller models prompted to show their work produce incoherent chains that lead to worse answers, not better ones. The original paper reported that benefits appear around roughly 100 billion parameters and grow from there. Teams sometimes reach for CoT on small models and observe regressions; the fix is not a better prompt but a larger model.

Three variants

flowchart LR
    subgraph "Few-shot CoT"
        A1[Question + examples with reasoning] --> A2[Answer with visible chain]
    end

    subgraph "Zero-shot CoT"
        B1[Question + 'Let us think step by step'] --> B2[Answer with visible chain]
    end

    subgraph "Self-consistency"
        C1[Question] --> C2[Sample N chains]
        C2 --> C3[Take majority final answer]
    end

Few-shot CoT. The prompt contains two or three worked examples, each with the question, the reasoning, and the final answer. The model extends the pattern to the new question. This variant is the original and tends to produce the highest-quality chains when the few-shot examples are well chosen.

Zero-shot CoT. The phrase "let's think step by step" (or a close paraphrase) is appended to the question. No examples are needed. This variant is weaker on raw accuracy than few-shot CoT but trivial to deploy. It is the default most practitioners reach for first.

Self-consistency. Rather than sampling a single chain, the harness samples N chains at a temperature above zero and takes the majority final answer. The technique closes much of the gap between greedy decoding and careful reasoning. On GSM8K, self-consistency raised a CoT baseline from 74.4 percent to 82.4 percent with 40 samples. The cost is N times the tokens of a single call, which is a meaningful bill for real workloads.

Three versions in code

The excerpt below shows all three variants without a framework. Few-shot requires an examples block; zero-shot appends a phrase; self-consistency samples N times and votes.

from openai import OpenAI
from pydantic import BaseModel
from collections import Counter

client = OpenAI()

class Answer(BaseModel):
    reasoning: str
    final_answer: str

FEW_SHOT = """
Q: Roger has 5 tennis balls. He buys 2 more cans. Each can has 3 balls. How many does he have?
A: Roger started with 5. He bought 2 cans * 3 = 6 more. Total: 5 + 6 = 11. Answer: 11.

Q: The cafeteria had 23 apples. They used 20 for lunch and bought 6 more. How many now?
A: Start with 23. 23 - 20 = 3. Then 3 + 6 = 9. Answer: 9.
"""

def few_shot_cot(question: str) -> Answer:
    r = client.responses.parse(
        model="gpt-4o-mini",
        instructions=f"Answer the question. Show your reasoning.\n\n{FEW_SHOT}",
        input=f"Q: {question}\nA:", text_format=Answer,
    )
    return r.output[0].content[0].parsed

def zero_shot_cot(question: str) -> Answer:
    r = client.responses.parse(
        model="gpt-4o-mini",
        instructions="Answer the question. Let's think step by step.",
        input=question, text_format=Answer,
    )
    return r.output[0].content[0].parsed

def self_consistency(question: str, n: int = 5) -> str:
    answers = [zero_shot_cot(question).final_answer for _ in range(n)]
    return Counter(answers).most_common(1)[0][0]

The LangChain version wires the same idea through with_structured_output. Self-consistency uses asyncio.gather to run samples concurrently.

import asyncio
from collections import Counter
from langchain.chat_models import init_chat_model
from pydantic import BaseModel

class Answer(BaseModel):
    reasoning: str
    final_answer: str

model = init_chat_model("gpt-4o-mini", temperature=0.7).with_structured_output(Answer)

async def zero_shot_cot(question: str) -> Answer:
    return await model.ainvoke([
        ("system", "Answer the question. Let's think step by step."),
        ("user", question),
    ])

async def self_consistency(question: str, n: int = 5) -> str:
    answers = await asyncio.gather(*[zero_shot_cot(question) for _ in range(n)])
    return Counter(a.final_answer for a in answers).most_common(1)[0][0]

Full runnable versions live at github.com/subodhjena/agentic-patterns under examples/12_chain_of_thought.py and examples/12_chain_of_thought_langgraph.py.

When chain-of-thought helps and when it does not

The technique has a narrow and well-documented sweet spot.

It helps on multi-step reasoning. Arithmetic word problems, symbolic manipulation, commonsense chains, and multi-hop question answering all benefit. The deeper the chain needed, the larger the gain.

It helps on problems where intermediate steps have discrete values. Math problems are the canonical case; the model can carry a running total. Problems with continuous or fuzzy intermediates benefit less.

It does not help on one-step problems. Asking a model to show its work for a fact lookup adds tokens without improving the answer.

It does not help on small models. Below roughly 100 billion parameters, reasoning chains are incoherent and accuracy drops.

It does not substitute for tool use. A chain that requires a fact the model does not know will fabricate the fact. ReAct, covered earlier in this series, pairs CoT-style reasoning with tool calls to avoid this.

Where chain-of-thought fails

Failures are less dramatic than on stronger reasoning patterns but still worth naming.

Plausible-but-wrong chains. The model produces a confident, well-structured trace that arrives at a wrong answer. CoT reduces the rate of this failure but does not eliminate it. On factual problems, grounding is needed; on math problems, a calculator tool is needed.

Over-long chains. For simple problems, prompted CoT can produce verbose traces that add latency without accuracy. Cap expected chain length in the prompt or use structured output with a length-limited reasoning field.

Format drift. A chain without structured output is a chain that might be followed by "Final answer:" or "The answer is:" or simply a paragraph. Parse failures follow. Structured output, covered earlier in this series, fixes the shape.

Self-consistency at low diversity. If temperature is too low, all samples produce the same chain, and the vote is meaningless. If temperature is too high, chains are incoherent. Calibrate empirically; 0.7 is a common starting point.

Self-consistency on open-ended tasks. Voting requires discrete final answers. For a question like "summarize this article," voting is ill-defined; a ranked or merged aggregation is more appropriate.

Trade against a direct answer

Chain-of-thought is cheap to introduce and expensive to over-use. The table describes the decision.

Axis	Direct answer	Zero-shot CoT	Few-shot CoT	Self-consistency
Prompt length	Short	Short	Longer	Short per sample
Output length	Short	Longer	Longer	Longer times N
Accuracy on multi-step	Baseline	Higher	Highest of the three	Higher still, with N samples
Latency	Low	Low	Low	N times baseline
Cost	Low	Low	Medium	High
Best for	Fact lookup, classification	Default for reasoning	When examples exist	When correctness must be defended

Zero-shot CoT is the default for reasoning tasks because it is essentially free. Few-shot and self-consistency are investments made when the task's accuracy pays for the extra cost.

Neighbors in the series

Structured output, earlier in this series, is what makes CoT output parseable: a reasoning field plus a final_answer field in a schema. ReAct, earlier in the Agents stage, combines CoT-style reasoning with tool calls. Self-consistency is the CoT-specific version of the voting variant of parallelization, covered earlier. Tree-of-thought, covered next, generalizes CoT from a single chain to a search over branching chains. Reflexion, covered later in this stage, adds a self-critique step on top of a CoT-style agent.

References

1. Wei, Jason, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
2. Kojima, Takeshi, et al. Large Language Models are Zero-Shot Reasoners. NeurIPS 2022.
3. Wang, Xuezhi, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. 2022.
4. Anthropic. Claude's extended thinking. 2025.
5. OpenAI. Reasoning with o1 and its successors. 2024.