AI / LLM
Thin SDK, Fat Runtime: Owning the Agent Loop in Pure Python
A team starting an agent project today faces a layered choice. There is the model. There is a provider SDK. Above the SDK there is, optionally, a framework: LangChain, LangGraph, CrewAI, AutoGen, the OpenAI Agents SDK, the Claude Agent SDK, DSPy, LlamaIndex, half a dozen others. Each layer claims a slice of responsibility. The question this article answers is: where, exactly, should that boundary stop?
The answer this article defends is narrow. A provider SDK should be a transport client. It owns six concerns that are genuinely provider-specific and tedious to reimplement: authentication, request shape, HTTP transport, retries on transient failures, streaming event reassembly, and response parsing. Everything above the transport layer, the agent loop, planning, routing, tool dispatch, parallelism, memory, guardrails, and workflow state, is application architecture. Application architecture belongs in pure Python: asyncio, typed dataclasses, normal classes, and an orchestration layer the team owns and can read end to end.
This is a defensible position with two soft spots that the rest of the article will name honestly. The first is that the "transport boundary" is wider than just HTTP, because tool-call schemas and streaming event types differ across providers. The second is that some adjacent concerns, durable workflow execution, prompt optimization, and large-team standardization, do reward a thin abstraction layer. The thesis survives both, but only if it is held with eyes open.
The boundary question
flowchart LR
APP[Your application code] --> RUN[Agent runtime: loop, planning, tools, memory]
RUN --> SDK[Provider SDK: auth, transport, retries, streaming, parsing]
SDK --> API[Provider HTTP API]
The diagram is the entire thesis. The SDK is a transport client. The runtime is yours. Everything that has to do with the wire stays in the SDK. Everything that has to do with the application stays in your code.
The opposite arrangement, where a framework owns both the transport and the runtime, is the default for many teams in 2026. It is also the arrangement that produces the failure modes documented in the rest of this article: hidden control flow, debugging through three abstraction layers, and architectural decisions that turn into framework lock-in.
What the SDK is irreplaceable for
The case for keeping the SDK begins with what it actually does. The OpenAI and Anthropic Python SDKs each handle six concerns automatically, with reasonable defaults that a hand-written client would have to recreate.
Authentication and request shape. The SDKs read API keys from environment variables, set the correct authentication header (Authorization: Bearer ... for OpenAI, x-api-key for Anthropic), and serialize Python objects into the JSON shape the API expects. Anthropic's messages.create requires a model, max_tokens, and a messages list with strict user/assistant alternation; the SDK enforces this client-side.
HTTP transport. Both SDKs use httpx under the hood, with synchronous and asynchronous clients sharing the same surface (OpenAI Python README, Anthropic Python SDK). Connection pooling, HTTP/2, TLS, and proxy handling are inherited from httpx.
Retries with exponential backoff. This is the concern most often misattributed to application code. By default, both SDKs retry up to two times on connection errors, 408 Request Timeout, 409 Conflict, 429 Rate Limit, and any 5xx response, with exponential backoff between attempts (OpenAI Python, Anthropic Python). The behavior is configurable per request via with_options(max_retries=...) or globally at client construction. Re-implementing this correctly, with jitter, retry-after honoring, and idempotency awareness, is a non-trivial amount of code.
Streaming reassembly. This is where the SDK earns its keep most visibly. OpenAI and Anthropic each send Server-Sent Events with provider-specific event types. OpenAI's Responses API streams response.function_call_arguments.delta events whose delta field accumulates partial JSON for tool arguments. Anthropic streams content_block_start, input_json_delta, and content_block_stop events, where the input_json_delta chunks must be concatenated into the final tool input (OpenAI streaming, LangChain.js streaming bug report). The SDKs reassemble these chunks into typed Python objects so application code can iterate over completed messages or partial deltas without parsing SSE itself.
Response parsing. The SDKs return typed objects, not dictionaries. response.output[0].content[0] for OpenAI's Responses API and response.content[0] for Anthropic's Messages API are statically typed, so editor tooling and Pyright catch field misuse before runtime.
API version pinning. The SDKs are versioned to specific API versions. When OpenAI introduces a new field or Anthropic adds a new tool category, upgrading the SDK is the cheapest way to absorb the change. Hand-rolled clients accumulate compatibility cruft.
These six concerns are not application architecture. They are protocol mechanics. The SDK does them well, and the cost of reimplementing them, even for one provider, is several hundred lines of code that exists only to be wrong sometimes.
Where the boundary leaks
The thesis as stated, "the SDK is just a transport client," is a useful mental model but not literally accurate. The boundary leaks in two places worth naming.
Tool-call schemas differ. OpenAI's function-calling format and Anthropic's tool-use format converge on the same idea (the model returns a structured request to invoke a function) but disagree on the shape:
- OpenAI returns a
tool_callsarray on the assistant message, with each entry holdingid,type: "function",function.name, and a stringifiedfunction.arguments. The application echoes results back as messages withrole: "tool"and a matchingtool_call_id. - Anthropic returns content blocks with
type: "tool_use", each holdingid,name, and a structuredinputobject. The application echoes results back inside a user message as a content block withtype: "tool_result",tool_use_id, andcontent(Anthropic tool use docs).
Re-shaping these for cross-provider portability is application code, not SDK work. A team that targets one provider can ignore the difference. A team that needs both writes a small adapter, typically 50 to 150 lines, that normalizes both shapes into one internal representation. That adapter belongs in your codebase, not behind a heavy framework.
Streaming event types differ. As noted above, input_json_delta (Anthropic) and response.function_call_arguments.delta (OpenAI) carry the same information in different envelopes. The SDK reassembles the bytes into typed objects, but the shape of those typed objects is provider-specific. This is the same leak as tool-call schemas, expressed in the streaming axis.
These leaks are not arguments for adopting LangChain. They are arguments for being honest that "transport client" is shorthand for "transport plus the message-shape layer." Once a team picks a provider, the leak narrows to nothing. Once a team decides to support multiple providers, the right response is a small in-house adapter, not a framework with a hundred dependencies.
What is genuinely application architecture
The list below is the set of concerns the user thesis names as "everything above transport." Every item on the list is implementable in pure Python, in tens to low hundreds of lines, against the provider SDK directly. None of them require a framework.
| Concern | What it is | Pure-Python primitive |
|---|---|---|
| Agent loop | Model call, parse tool calls, dispatch, feed observations, repeat | A while loop with a turn counter |
| Planning | Decompose a task into ordered subtasks | One LLM call returning a Pydantic list |
| Routing | Classify input, dispatch to a specialist | One LLM call returning a Literal, a dict of handlers |
| Tool calling | Match name, validate arguments, invoke, return result | A dict[str, Callable] registry |
| Parallel execution | Run independent tool calls simultaneously | asyncio.TaskGroup (Python 3.11+) |
| Memory (short-term) | Trim, summarize, or window the conversation | List slicing and an optional summary call |
| Memory (long-term) | Embed, store, retrieve | A vector-store client (FAISS, pgvector, Chroma) plus retrieval logic |
| Workflow state | Track where you are between calls | A typed dataclass, optionally persisted to disk or a database |
| Guardrails | Validate input or output before/after the model | A function that returns a verdict |
| Evaluator-optimizer | Generator call, evaluator call, retry on fail | Two LLM calls and a comparator |
| Retries (semantic) | Retry the agent step when the output is wrong, not when the network is | A bounded loop with feedback in context |
| Queues | Process multiple agent runs concurrently | asyncio.Queue plus worker tasks |
The pattern is consistent. Each concern is a small piece of orchestration code that calls into the SDK at exactly one place: the model call. Everything around the model call is normal Python. Loops, dictionaries, dataclasses, async tasks. The SDK does not need to know about any of it.
This is the same conclusion Anthropic reaches in their public guidance. From "Building Effective Agents" (Anthropic, 2024):
Start by using LLM APIs directly: many patterns can be implemented in a few lines of code. If you do use a framework, ensure you understand the underlying code. Incorrect assumptions about what's under the hood are a common source of customer error.
And from the same article:
Frameworks often create extra layers of abstraction that can obscure the underlying prompts and responses, making them harder to debug. They can also make it tempting to add complexity when a simpler setup would suffice.
The position is corroborated by Dex Horthy's "12-Factor Agents," which codifies it as principles 2, 3, and 8: own your prompts, own your context window, own your control flow (Horthy, 2025). Horthy's practical claim is that "if you can't draw a reasonably accurate flowchart of how your agent makes decisions and moves from one step to the next, you don't own the system." A framework that hides the loop hides the system.
A pure-Python agent loop, end to end
The example below is a complete tool-calling agent. It uses the OpenAI Python SDK as a transport client. The loop, the tool registry, the parallelism, and the typed schemas are all application code. It runs as written.
import asyncio
import json
from typing import Any, Awaitable, Callable
from openai import AsyncOpenAI
from pydantic import BaseModel
client = AsyncOpenAI() # SDK handles auth, retries, transport, parsing
class Tool(BaseModel):
name: str
description: str
parameters: dict[str, Any]
run: Callable[..., Awaitable[str]]
class Config:
arbitrary_types_allowed = True
async def get_weather(city: str) -> str:
return f"Weather in {city}: 22C, clear."
async def search_docs(query: str) -> str:
return f"Top result for {query!r}: docs.example.com/intro"
TOOLS: dict[str, Tool] = {
"get_weather": Tool(
name="get_weather",
description="Return current weather for a city.",
parameters={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
run=get_weather,
),
"search_docs": Tool(
name="search_docs",
description="Search internal documentation.",
parameters={"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]},
run=search_docs,
),
}
def tool_specs() -> list[dict]:
return [
{"type": "function", "function": {"name": t.name, "description": t.description, "parameters": t.parameters}}
for t in TOOLS.values()
]
async def dispatch(call) -> dict:
name = call.function.name
args = json.loads(call.function.arguments or "{}")
if name not in TOOLS:
return {"role": "tool", "tool_call_id": call.id, "content": f"unknown tool: {name}"}
try:
result = await TOOLS[name].run(**args)
except Exception as e:
result = f"tool error: {e!r}"
return {"role": "tool", "tool_call_id": call.id, "content": result}
async def run_agent(user_prompt: str, max_turns: int = 6) -> str:
messages = [
{"role": "system", "content": "You are a helpful assistant. Use tools when relevant."},
{"role": "user", "content": user_prompt},
]
for _ in range(max_turns):
resp = await client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tool_specs(),
)
msg = resp.choices[0].message
messages.append(msg.model_dump(exclude_none=True))
if not msg.tool_calls:
return msg.content or ""
async with asyncio.TaskGroup() as tg: # parallel tool execution
tasks = [tg.create_task(dispatch(c)) for c in msg.tool_calls]
for t in tasks:
messages.append(t.result())
return "max turns reached"
if __name__ == "__main__":
print(asyncio.run(run_agent("What's the weather in Paris and find me the intro docs.")))Forty-eight lines. The SDK does authentication, retries on rate limits, request signing, JSON parsing, and typed response objects. The application owns the loop, the tool registry, the parallel dispatch via asyncio.TaskGroup, the message history, and the turn cap. Every decision the agent makes is visible in this file.
A few details are worth pointing out:
asyncio.TaskGroupoverasyncio.gather. Python 3.11 introducedTaskGroupas a structured-concurrency primitive that gives every concurrent task an explicit lifetime (Pan, 2026). When the agent loop is cancelled mid-run, every running tool call is cancelled with it.asyncio.gatherdoes not give that guarantee and can leak tasks.- The tool registry is a dict. Adding a tool means adding an entry. Removing one means removing an entry. There is no class hierarchy to inherit from and no registration decorator to import.
- The turn cap is enforced in application code. No framework default sets it for you; no framework hides the fact that it exists. A team that wants to log per-turn metrics, persist intermediate state, or stream partial results edits this loop directly.
The same loop, rewritten in LangChain
The contrast is illustrative. The example below is a roughly equivalent agent built with langchain and langgraph.
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent
@tool
async def get_weather(city: str) -> str:
"""Return current weather for a city."""
return f"Weather in {city}: 22C, clear."
@tool
async def search_docs(query: str) -> str:
"""Search internal documentation."""
return f"Top result for {query!r}: docs.example.com/intro"
agent = create_react_agent(
model=ChatOpenAI(model="gpt-4o-mini"),
tools=[get_weather, search_docs],
prompt="You are a helpful assistant. Use tools when relevant.",
)
async def run(user_prompt: str) -> str:
out = await agent.ainvoke({"messages": [{"role": "user", "content": user_prompt}]})
return out["messages"][-1].contentTwenty lines, fewer than half. The framework wins on line count. It loses on three other axes that matter more in production.
Visibility. The loop is gone. There is no for _ in range(max_turns): to read; the loop runs inside create_react_agent and is configured by recursion_limit on the graph object. A new engineer reading this file cannot tell how many times the model is called, in what order, or under what condition the loop terminates.
Debugging. When something goes wrong, the trace path goes through create_react_agent, into a prebuilt LangGraph state machine, into the LangChain runnable layer, and finally into the LangChain OpenAI adapter that wraps the OpenAI SDK. Octomind, after twelve months in production, removed LangChain for exactly this reason: "abstractions on top of other abstractions" forced their team to debug framework internals rather than ship features (Octomind, 2024).
Extensibility. Adding per-turn logging, custom retry semantics on tool failures, or a guardrail that rewrites the model output requires either subclassing internal LangGraph nodes or building a parallel implementation. In the pure-Python version, it is a line edit.
These are not arguments that frameworks are bad. They are arguments about where the abstraction tax is paid. For a one-off prototype, twenty lines is genuinely faster. For a system that runs in production for two years, the forty-eight-line version is the cheaper one to maintain.
When the thesis cracks
The thesis survives most everyday agent work, but three concerns reward a thin abstraction layer above the SDK. Naming them honestly is part of holding the position.
Durable execution. Long-running, resumable, human-in-the-loop workflows need persistence between turns. A naive pure-Python loop loses state when the process restarts. Building durable execution from scratch means a state schema, a checkpointer, an event log, and a resume protocol. LangGraph's checkpointing primitives, Temporal's workflow runtime, and AWS Step Functions all solve this with infrastructure investment a typical team will not match. For agents that take hours, span machine restarts, or pause for human approval, importing a checkpointer is the right call.
Prompt and pipeline optimization. DSPy compiles prompts and few-shot examples against a metric, raising published benchmarks substantially in some configurations (Khattab et al., 2023). A team that wants this capability cannot get it by writing more pure Python. They adopt the framework. The trade is a real one.
Team scale. A five-person team that already knows LangGraph, has shared mental models for its primitives, and ships into a codebase others maintain is genuinely faster with LangGraph than learning to roll their own. The SDK-only argument is strongest for new projects, small teams, and codebases where a single engineer can read the whole runtime in an afternoon. It is weaker, though not wrong, in larger organizations with established framework standards.
Multi-provider routing at scale. A single in-house adapter is fine for two or three providers. A platform team supporting fifteen models across five providers, each with its own tool-call shape and streaming protocol, is essentially building LiteLLM. At that scale, importing LiteLLM is the right call. The thesis still holds at the runtime layer; the transport layer simply gets fatter.
These caveats narrow the thesis. They do not refute it. The default for an application team building one or two agents on one or two providers is direct SDK use plus a runtime they own. The exceptions are real, and a team that hits them should adopt the smallest abstraction that solves the actual problem, not the most popular one.
A trade-off table
| Axis | Direct SDK plus pure Python | Framework on top |
|---|---|---|
| Time to first prototype | Hours | Minutes |
| Lines of code (basic agent) | ~40 to 80 | ~10 to 30 |
| Visibility of control flow | Full | Partial |
| Debugging surface | One file plus SDK | Application plus framework plus SDK |
| Cost of changing loop semantics | Edit one file | Subclass or replace framework primitives |
| Cost of supporting a new provider | Add ~50 to 150 lines of adapter | Wait for framework support |
| Onboarding cost for new engineer | Read one file | Learn framework primitives |
| Durable execution | Build it (or import a checkpointer) | Often included |
| Prompt optimization | Build it (or import DSPy) | Sometimes included |
| Production runway | Long, stays simple | Long, but tax compounds |
The right column is faster for the first week. The left column is cheaper for the next two years. Which timeline matters depends on the project; for production systems, the second timeline almost always wins.
The shape of the recommendation
A reader asking "what should I actually do" should leave with three concrete defaults.
First, default to the SDK plus pure Python. Use openai or anthropic directly. Build the agent loop as a function. Use asyncio.TaskGroup for parallelism, Pydantic for schemas, a dict for the tool registry, and a typed dataclass for state. The agent loop in the previous section is a starting template that ports cleanly to either provider with about a dozen line changes.
Second, when an adjacent concern is genuinely hard, import the smallest tool that solves it. Vector store: pick chromadb or pgvector, not LangChain's retrieval stack. Durable execution: import LangGraph's checkpointer, not the whole graph runtime. Prompt optimization: adopt DSPy for the modules that need it; let the rest stay in pure Python. The principle is the same Anthropic gives: "consider adding complexity only when it demonstrably improves outcomes" (Anthropic, 2024).
Third, audit the boundary annually. Every component above the SDK encodes an assumption about what the model and the language do not do well. Models improve. Python improves: asyncio.TaskGroup did not exist before 3.11; match statements did not exist before 3.10. A runtime that was expensive to maintain in 2024 may be free to maintain in 2026 because the language and the models both got better. Re-measure.
The thesis is not anti-framework. It is anti-default-framework. Frameworks are good when their abstractions match the problem; they are expensive when they do not. The SDK boundary is the cheapest defensible default because it is the layer below which the work is genuinely provider-specific and above which the work is genuinely application-specific. Putting the boundary anywhere else costs more, in either lock-in or rebuild, than the convenience earns back.
Neighbors in the series
Three articles from the agentic-patterns series are direct neighbors. "Workflows Versus Agents in LLM Systems" makes the case that most production tasks are workflows, not agents, and benefit from the simpler structure that pure Python gives them. "The Tool-Calling Agent Loop" walks through the same agent loop pattern shown above with more attention to the per-turn mechanics. "Choosing the Right Agentic Pattern: A Decision Framework" closes the series with an explicit recommendation that direct API calls plus asyncio is a reasonable default for a team's first production agent.
References
- Anthropic. Building Effective Agents. December 2024.
- Anthropic. Tool Use with Claude. API documentation, 2026.
- Anthropic. Anthropic Python SDK. GitHub repository, 2024 to present.
- Horthy, Dex. 12-Factor Agents: Patterns of Reliable LLM Applications. HumanLayer, 2025.
- Khattab, Omar, et al. DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines. 2023.
- OpenAI. OpenAI Python SDK. GitHub repository, 2023 to present.
- OpenAI. Streaming Responses API. API documentation, 2026.
- OpenAI. OpenAI Agents SDK for Python. 2025.
- Octomind. Why We No Longer Use LangChain for Building Our AI Agents. Engineering blog, 2024.
- Pan, Tian. Structured Concurrency for AI Pipelines: Why asyncio.gather() Isn't Enough. 2026.
- Willison, Simon. Notes on Anthropic's "Building Effective Agents". December 2024.