Start with the problem: why a bare LLM is not yet an agent
Dig to the bottom. A bare LLM is essentially a stateless function: give it text, it returns text, done. It doesn't remember your previous sentence (unless you paste the history back), can't touch the outside world, can't get today's data, can't do a multiplication it didn't memorize. Its knowledge is frozen in the weights at the moment of training, so when you ask about something it doesn't know, it won't say "let me look it up" — it fluently makes something up. That's the root of hallucination.
But real tasks don't look like that. "Look up the part-number spec for this maintenance ticket, decide whether it needs derating, and if so open a follow-up tracking ticket" — this requires perception (read the ticket), reasoning (judge the derating), action (query the database, open a ticket), observation (get the query result before deciding the next step), and memory (remember this production line had similar issues before). A bare LLM can do none of these.
An AI agent is the system that connects the LLM — that "only predicts the next token" brain — to these bolt-on capabilities, and wires them together with a closed loop. That sentence is the whole article's foundation. Below I'll stack it up layer by layer: first the innermost loop, then, following the historical evolution, bolt on tools, memory, planning, and reflection one block at a time, and finally discuss how to design. Every layer asks the same question — what does this block mechanically do, and what flaw of the previous layer does it fix?
Layer 0 (the bottom): an agent's core is a loop, not a single call
The bottom of every agent architecture is the same thing: a loop. This is the most important sentence, and the one most often buried under fancy frameworks.
A bare LLM is "call once, get one output." An agent is "call → look at result → call again → look again … until the task is done":
┌──────────────────────────────────────────────────┐
│ │
▼ │
LLM reads current context ──▶ produces output │
│ │
┌──────────────┴───────────────┐ │
▼ ▼ │
"this is the final answer" "I want to use tool X"
│ │ │
▼ execute tool X │
done │ │
paste result back into context ─┘
See the loop's essence: every round the LLM does only the one thing it can do — predict, from the current context, what to produce next. What gets smart is not the LLM but the structure of "its output is intercepted, executed, and the real result fed back." The context keeps growing inside the loop, accumulating "what I did, what I saw," so the next round's prediction is built on real feedback rather than imagination.
This loop is the agent's heart. Every "layer" below essentially answers: inside this loop, what format should the LLM output, what goes into the context, how do tools connect, how does memory move in and out.
Layer 1: ReAct — interleave "thinking" and "doing"
We have a loop; the first question is: what exactly should the LLM output each round? The evolution starts with reasoning.
Wei et al.'s (2022) Chain-of-Thought (CoT)[1] found that simply having the model write out reasoning steps before answering greatly improves complex reasoning. Why it works mechanically: writing "thinking" as tokens laid out in the context makes each reasoning step an input to the next prediction, breaking a multi-step problem into a chain of single-step predictions it's better at. But CoT has a ceiling — all its reasoning happens "in the head," out of contact with the world. If the first reasoning step is built on a hallucinated fact, everything after it, however elegant, is wrong.
Yao et al.'s (2022) ReAct (Reason + Act)[2] closes that gap, and it's the skeleton of most agents today. It has the model, each round of the loop, interleave reasoning and action:
Thought: I need to know RC0402's rated power first ← reasoning (think)
Action: search["RC0402 rated power"] ← action (call a tool)
Observation: rated power 63 mW @ 70°C ← observation (tool return, pasted back externally)
Thought: ticket environment is 85°C, above 70°C, check derating curve ← reason on real observation
Action: search["RC0402 derating curve"]
Observation: above 70°C derate 1.67% per degree...
Thought: at 85°C rated power drops to ~..., derating is needed
Action: finish["needs derating, recommend..."]
The underlying mechanism is plain and crucial: these Thought:/Action:/Observation: are not special instructions — they're plain text. The prompt first demonstrates this format, so the model continues in it; an external parser watches the model's output, and the moment it sees Action: search[...] it intercepts, actually runs the search, and pastes the result back into the context as Observation:, then lets the model keep writing. ReAct's key leap over CoT: reasoning is now grounded in real feedback — each Thought builds on the last real Observation, and hallucination has no room to snowball. That is the mechanistic meaning of "synergizing Reason and Act."
Layer 2: how a tool actually gets "called" — the internals of function calling
ReAct lets the model "decide to use a tool," but a deeper question is: the LLM only generates text, so how does it "call" a function?
The answer punctures many intuitions: the LLM never executes anything. It merely generates tokens that "look like a tool call"; the thing that actually executes is the harness (host program) outside. The whole mechanism is a carefully designed text relay.
There are two evolutionary lines. Schick et al.'s (2023) Toolformer[3] takes "let the model learn to insert API calls at the right spots in text itself" — using self-supervision to have the model annotate, in the pretraining corpus, "call a calculator/search here," learning the when and how. Today's mainstream function calling / tool use takes a more engineered line:
① Developer describes each tool with a JSON schema:
{
"name": "query_oracle",
"description": "query MES part-number spec",
"parameters": { "part_no": {"type": "string"}, ... }
}
│ this schema is placed into the context (system prompt or a dedicated field)
▼
② The model is trained so that, when a tool is needed, it produces a structured
output matching that schema (usually a JSON):
{ "name": "query_oracle", "arguments": {"part_no": "RC0402"} }
│ the model's "output" ends here — it just generated compliant JSON tokens
▼
③ The harness parses this JSON → actually calls query_oracle("RC0402") → gets the result
│
▼
④ The harness pastes the result back into context as a new message → model continues
Key understanding: the tool's definition, execution, and result-return all happen outside the LLM; the LLM's role is only "produce the correct call per the schema" and "decide the next step from the returned result." Whether a model can use tools depends on whether it's been trained to reliably produce compliant structured output — which is why function calling is a capability requiring dedicated alignment training.
One layer up is standardization. When every tool and external system needs its own bespoke integration, cost explodes. Anthropic's Model Context Protocol (MCP)[13] exists for this: like "USB-C for AI," it describes tools and data sources with one standard protocol, decoupling agent from external systems — a tool provider writes one MCP server and any MCP-supporting agent can plug in. This turns "what tools an agent can use" from hardcoded integration into a pluggable ecosystem.
Layer 3: memory — short-term context and long-term memory
The loop runs and tools work; the next bottleneck is memory. An agent's "memory" is actually two kinds, mechanically completely different.
Short-term memory = the context window (working memory). As said, the context grows inside the loop — that's the agent's short-term memory, holding all Thought/Action/Observation of this task so far. But it has two hard limits: one, it's capped (overflow forces truncation or summarization); two — tying back to the rerank article — Liu et al.'s "Lost in the Middle" phenomenon: stuff too much in and the model ignores what's in the middle. So "what to put in the context, how much, in what order" is itself core agent engineering — more is not better.
Long-term memory = external storage + retrieval. To remember across tasks and sessions, you can't rely on context; you write experience into external storage and fetch it back when needed. This connects straight to your familiar RAG series: store past conversations, cases, and knowledge as embeddings in a vector store, and when needed retrieve + rerank the most relevant few into the context. Long-term memory is mechanically one RAG.
Park et al.'s (2023) Generative Agents (the Stanford experiment with 25 AI townsfolk)[7] gave long-term memory a classic design worth dissecting: it maintains a memory stream, and each memory's retrieval weight is set by three scores —
retrieval_score = α·recency + β·importance + γ·relevance
recency : more recent memories weigh higher (exponential decay)
importance: at storage time, the LLM scores "how important is this"
relevance : similarity of this memory's embedding to the current-situation query
Term by term: relevance alone (pure RAG) isn't enough, because "a small thing that just happened yesterday" may deserve recall more than "a big relevant thing half a year ago"; importance lets the model distinguish "had a meal" from "decided to quit the job." Further, Generative Agents periodically reflect: feed a pile of scattered memories to the LLM to synthesize higher-level conclusions ("I notice this line's resistors often overheat in summer"), then store the conclusions back into the stream. This is the key step upgrading "memory" from a logbook into "structured experience," foreshadowing the next layer.
Layer 4: planning — decompose a big task into steps
Once a task gets complex, "one step at a time" ReAct looks shortsighted. Layer 4 is planning: figure out the overall decomposition first, then act.
The most basic is task decomposition — have the LLM first break "open a maintenance tracking ticket" into subtasks like "read ticket → look up spec → judge → create ticket → notify." Two schools here, with different mechanistic trade-offs:
- Interleaved (ReAct-style): think a step, do a step, see the result, think the next. The upside is each step can adapt to real observations and handle surprises; the downside is each step needs one LLM call and re-reads the full history, so token cost is high.
- Plan-then-execute: list the full plan at once, then execute in order. Xu et al.'s (2023) ReWOO[10] is the representative: it decouples reasoning (planning) from observation (tool results) — the model produces a blueprint of all steps at once (using variables as placeholders for tool results), then executes uniformly and aggregates at the end. Because it need not stuff observations back to re-reason at every step, token efficiency is several times ReAct's[10]. The cost is a fixed plan, less adaptive to mid-course surprises than the interleaved style.
More advanced is exploratory planning. Yao et al.'s (2023) Tree of Thoughts (ToT)[8] no longer runs reasoning in a straight line but, at each decision point, expands multiple candidate thoughts, pushes each forward, self-evaluates, and backtracks to switch when a path fails — like exploring several moves in your head while playing chess. On tasks needing search and trial-and-error (like the Game of 24), it lifted GPT-4's success rate from 4% to 74%[8]. The mechanistic cost is blunt: exploring many paths = several times more LLM calls.
Planning can also mean orchestrating other models. Shen et al.'s (2023) HuggingGPT[4] lets an LLM be the "brain/controller" that, after decomposing a task, dispatches a bunch of specialized Hugging Face models (image, speech, translation) to do the parts and then aggregates — expanding the notion of "tool" from an API to "an entire model ecosystem."
Layer 5: reflection and self-correction — learn from failure
The topmost layer lets the agent learn from its own failures, the key to going from "can do" to "gets better at doing."
Madaan et al.'s (2023) Self-Refine[5] gives the most basic loop: the model produces → critiques itself → improves per the critique, over several rounds. Why it works mechanically? Because "judging whether an answer is good" is usually easier than "writing a perfect answer in one shot" — like a human revising a draft. Splitting generation and critique into two roles (even if the same model plays both) lets the model's critique ability correct its generation, yielding about a 20% average task improvement[5].
Shinn et al.'s (2023) Reflexion[6] goes further, joining reflection with memory, proposing verbal reinforcement: after an agent fails a task, instead of updating model weights (too expensive), it writes down in natural language "why I failed and how to fix it next time" and stores it in an episodic memory buffer, then reads that reflection back into context on the next attempt at a similar task. The key insight: this replaces "gradients" with "language" to do reinforcement learning — experience isn't carved into weights but written as text in memory and read back in-context to influence behavior. This is the same spirit as Generative Agents' reflection and your familiar CRAG-style self-correcting retrieval.
The design layer: workflow or agent? Distinguish this first
With the machinery covered, we reach your "how to design." The most important judgment here, from Anthropic's Building Effective Agents[12]: first distinguish whether you want a workflow or an agent.
workflow: LLMs and tools orchestrated through "predefined code paths"
— fixed steps, predictable, easy to debug, cheap
e.g. every ticket runs the fixed pipeline "extract→classify→lookup→reply"
agent: the LLM "dynamically decides" the next step and which tool to use
— non-fixed path, adaptive, but expensive, unpredictable, hard to debug
e.g. open-ended problems where even how many steps is decided on the fly
This distinction matters enormously in practice: most "agent projects" actually need only a workflow. What truly needs an agent (letting the model autonomously decide the path) are open-ended tasks whose steps can't be enumerated in advance. The core design principle is one line: start from the simplest thing that works, and add complexity only when the simple solution can't hold — don't jump to a multi-agent framework on day one.
Anthropic's simple-to-complex common patterns (all worth considering before a "fully autonomous agent"):
① Prompt Chaining: break the task into fixed steps, feed one step's output to the
next, with programmatic checkpoints between. Simplest, most controllable.
② Routing: classify the input, then direct to a specialized flow (e.g. support triage).
③ Parallelization: multiple LLMs run at once, dividing work or voting.
④ Orchestrator-Workers: a main LLM dynamically breaks down tasks, delegates to worker LLMs,
then aggregates (nearly an agent, but still has a controlling structure).
⑤ Evaluator-Optimizer: one LLM generates, another critiques, in an improving loop
(the two-role version of Self-Refine).
Only when these structured patterns aren't enough and the task truly needs on-the-spot autonomous planning and adaptation do you upgrade to a full autonomous agent (a pure ReAct loop) or a multi-agent system (multiple specialized agents collaborating).
Assembling all layers: how a complete agent operates
Now stack the six layers into a full architecture diagram. An agent system usually comprises five components — brain (model), tools, memory, planning, orchestration — and operates like this:
user goal: "for this batch of RC0402 tickets, judge which need derating and open tracking tickets"
│
▼
[planning] LLM decomposes the goal into a subtask blueprint
│
▼
┌──────────────────── AGENT LOOP ─────────────────────────────┐
│ │
│ [memory] fetch relevant long-term memory (past cases) + read short-term context │
│ │ │
│ [brain] LLM reasons (Thought): what to do this step │
│ │ │
│ [brain] produces tool call (Action, compliant JSON) │
│ │ │
│ [tools] harness parses and executes (query Oracle / RAG retrieval / compute) │
│ │ └─ this RAG step = vector retrieval + rerank (your familiar stack) │
│ │ │
│ [memory] paste Observation back into context; write key conclusions to long-term memory │
│ │ │
│ [reflection] if a step failed → verbal reflection back to memory, adjust plan │
│ └──────────────── if unfinished, back to loop top ───────┘
│
▼ (task judged complete)
final output (with the action trace of every step, auditable)
See clearly: there is no magic in this diagram; every cell is machinery dissected above. The brain is that next-token-predicting LLM; a tool call is the text relay of "produce compliant JSON → harness executes → paste result back"; the long-term half of memory is one RAG (embedding + retrieval + rerank); planning and reflection are LLM calls in specific formats. The agent's "intelligence" emerges from these parts cooperating in the loop — no single component holds magic.
Evolution summary and design decisions
Laying the evolution line and design choices straight:
Layer Representative work What it fixed Mechanistic key
────────────────────────────────────────────────────────────────────────────
loop (ReAct skeleton) single call → closed loop output intercepted, executed, fed back
L1 CoT→ReAct[1,2] in-head reasoning → grounded Thought/Action/Observation interleave
L2 Toolformer/FC[3] no world contact → uses tools generate compliant call, harness executes
L3 Gen.Agents[7] no cross-task memory → long-term memory stream + retrieval + reflection
L4 ReWOO/ToT[8,10] shortsighted → planning decouple reason/observe / tree explore+backtrack
L5 Reflexion[6] can't learn → learn from failure verbal RL: reflection to memory, not weights
design Anthropic[12] over-engineering → level clarity workflow (fixed) vs agent (autonomous)
A one-line design decision map: steps enumerable → use a workflow (chaining/routing/parallel); steps not enumerable in advance, needing on-the-spot adaptation → then an autonomous agent; collaboration too complex for a single agent → then multi-agent. Always start from the simplest, observable, debuggable end, and treat "autonomy" as a cost paid only when forced, not a default.
Closing: one logic chain tying the whole article together
A bare LLM is a stateless function, only predicts the next token, can't touch the world, has frozen knowledge and hallucinates → real tasks need perception/reasoning/action/observation/memory, so you bolt capabilities onto the LLM and wire them with a closed loop, the agent's heart → in the loop the LLM only predicts each round; what gets smart is the structure of "output intercepted, executed, result fed back" → CoT lays reasoning out as tokens but traps it in the head, ReAct interleaves Thought/Action/Observation to ground reasoning in real feedback → the truth of tool calling is the LLM only generates compliant JSON and the harness executes; MCP standardizes this into a pluggable ecosystem → memory splits into short-term (context, limited by Lost-in-the-Middle) and long-term (external storage + RAG retrieval; Generative Agents weight by recency+importance+relevance) → planning decomposes big tasks; interleaved adapts but is costly, ReWOO decouples to save tokens, ToT explores a tree, HuggingGPT orchestrates a model ecosystem → reflection lets the agent learn from failure: Self-Refine's generate-critique loop, Reflexion's verbal reinforcement writing experience into memory rather than weights → in design first distinguish workflow (fixed path, cheap, controllable) from agent (autonomous, expensive, hard to test), start simple and add autonomy only when necessary → assembled, the five components (brain/tools/memory/planning/orchestration) cooperate in the loop, and intelligence emerges with no single-point magic.
One line for the whole article: agent = LLM (a next-token-predicting brain) + a closed loop wiring "produce instruction → external execution → result feedback," with tools, memory, planning, and reflection stacked on as needed; none of its blocks is mysterious, and the essence of design is always choosing, between "autonomy" and "control/cost," the simplest end that solves the problem.
Extension hooks
To go deeper, pick from: multi-agent communication protocols and the "who should be the orchestrator" design; agent observability and evaluation (how to measure whether an agent is good, how to trace its every decision); collecting an agent's action traces to PEFT/LoRA-fine-tune a specialized model that's better at using your tools (tie-back to the PEFT article); and agent safety — tool permission boundaries, prompt-injection defense, and human review gates before critical actions. Any one could be its own article.
References
[1] Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. — https://arxiv.org/abs/2201.11903
[2] Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. — https://arxiv.org/abs/2210.03629
[3] Schick, T. et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. — https://arxiv.org/abs/2302.04761
[4] Shen, Y. et al. (2023). HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. — https://arxiv.org/abs/2303.17580
[5] Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. — https://arxiv.org/abs/2303.17651
[6] Shinn, N. et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. — https://arxiv.org/abs/2303.11366
[7] Park, J. S. et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior. — https://arxiv.org/abs/2304.03442
[8] Yao, S. et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. — https://arxiv.org/abs/2305.10601
[9] Wang, G. et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. — https://arxiv.org/abs/2305.16291
[10] Xu, B. et al. (2023). ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. — https://arxiv.org/abs/2305.18323
[11] Xi, Z. et al. (2023). The Rise and Potential of Large Language Model Based Agents: A Survey. — https://arxiv.org/abs/2309.07864
[12] Anthropic (2024). Building Effective Agents. — https://www.anthropic.com/engineering/building-effective-agents
[13] Anthropic (2024). Model Context Protocol (MCP). — https://modelcontextprotocol.io/introduction