The concepts: how an agent actually works

Every chapter after this one uses a vocabulary: loops, tools, event logs, queues, quotas, microVMs, principals, protocols. This chapter builds that vocabulary from zero, one concept at a time, each connected to the one before it. Nothing here is AWS-specific yet; these are the ideas, and the next chapter shows where the book teaches each one in depth, while the chapter after that maps them onto actual AWS services.

To keep things concrete, we carry one running example the whole way: your checkout page got slow last night, and you want a software agent to find out why. By the end of the chapter you will be able to describe that agent, and everything it needs around it, in precise terms.

Start with one model call

Strip everything away and the foundation is a single API call to a large language model. You send text, you get text back. Three properties of that call shape every system in this book.

First: the model reads and writes in tokens. A token is the unit the model actually processes: a word fragment, roughly three-quarters of an English word on average. "Checkout latency spiked" is about five tokens. This matters because everything is priced and limited in tokens: you pay per million tokens in and per million tokens out (output tokens typically cost around five times more, because generating text is more work than reading it), and the model can only consider a bounded amount at once.

Second: that bound is the context window. Think of it as the model's working memory: the total tokens (your instructions, the conversation so far, any documents you pasted in) it can attend to in one call. Modern models hold hundreds of thousands to a million tokens, which sounds infinite until an agent starts pasting log files into it.

Third, and least obvious: the model is stateless. It remembers nothing between calls. There is no session on the model's side, no memory of what you asked five seconds ago. Every call must carry the entire conversation history in its request. When a conversation reaches turn forty, turn forty's request contains turns one through thirty-nine, re-sent and re-read in full.

That third property is the quiet engine behind half this book's economics and design. Re-sending everything every turn is why long agent sessions get expensive, why prompt caching exists (we will get there), and why "the agent's state" has to live somewhere you control, because it certainly does not live in the model.

The loop: from one call to an agent

A single call can answer a question. It cannot investigate one, because investigation requires acting on the world: run this query, read that log, then decide what to do next based on what came back. Models cannot act. They can only emit text.

The trick that turns a text generator into an agent is almost embarrassingly simple. You tell the model, in its instructions: "here are some tools you may use; if you want one, say so in a specific format instead of answering." Then your code watches for that format:

        +-------------------------------------------------------+
        |                                                       |
        v                                                       |
  [ model call ] --"use tool: query_metrics(service=checkout)"--+
        |                                                       |
        |                                              your code runs the
   "final answer"                                      tool, appends the
        |                                              result to the
        v                                              conversation, and
      done                                             calls the model again

The model says query_metrics(service="checkout", window="24h"). Your code (not the model; this distinction matters and we will return to it) actually executes that query, takes the result, appends it to the conversation as a new message, and calls the model again. Now the model sees the metrics and decides its next move: maybe search_logs("checkout timeout"), maybe a final answer. The cycle repeats until the model stops asking for tools.

That cycle is the agent loop, and this is the book's working definition: an agent is a model calling tools in a loop, with the model choosing each next step. The three components are the model (judgment), the tools (hands), and the loop code (the harness that ferries messages back and forth and decides when to stop).

Don't be confused: workflow vs agent. If you know the steps in advance ("always: fetch metrics, then fetch logs, then summarize"), you do not need an agent; you need a workflow, a flowchart with a model call in some boxes. Workflows are cheaper, faster, and easier to debug, because the control flow is yours. An agent is for when the next step depends on what the last step revealed, so the flowchart cannot be drawn in advance. Our latency investigation is genuinely agent-shaped: whether to look at the database, the cache, or last night's deploy depends entirely on what the metrics show. This book will say "use a workflow" often, on purpose.

One more piece of loop mechanics you will meet constantly: the stop reason. Every model response ends with a machine-readable reason: "I am done," "I want a tool," "I ran out of output room." Loop code is really a small state machine keyed on stop reasons, and most beginner agent bugs are mishandled stop reasons. Part 1 builds this loop in about a hundred lines and treats each stop reason as the contract it is.

Tools are the contract, and the security boundary

Look again at who does what in the loop. The model proposes: it emits the text search_logs("checkout timeout"). Your code disposes: it decides whether and how that proposal becomes a real action. The model never touches a database, a file, or a network socket. It writes text; your harness makes text into effects.

Two consequences follow, and they anchor entire parts of this book.

Tools are described by schemas. A tool is registered with the model as a name, a description, and a typed parameter list (this file format is called a schema). The description is not documentation for humans; it is the steering input the model uses to decide when to call the tool, which makes tool descriptions one of the highest-leverage lines of text in the whole system. A vague description produces an agent that calls the wrong tool at the wrong time, and the fix is editing prose, not code.

The tool list is the blast radius. Since the model can only act through tools, the complete list of what the agent could ever do, including under manipulation by a hostile input, is exactly the tool list and the permissions behind it. An agent whose only tools read metrics can be tricked into reading weird metrics. An agent with a "run any shell command" tool can be tricked into anything the shell can reach. This is why our latency investigator will get query_metrics and search_logs but never restart_service: security in agent systems is mostly tool design, done before any attack exists.

The transcript is a ledger: event sourcing

Where does the agent's state live? The model is stateless, so the conversation (instructions, tool calls, tool results, answers) must be stored by the harness. The naive answer is a variable in memory, which works until the process crashes forty minutes into an investigation and takes the whole conversation with it.

The durable answer this book commits to is event sourcing, and the best way to understand it is your bank account. Your bank does not store a balance that gets overwritten; it stores a ledger, an append-only list of deposits and withdrawals, and computes the balance by summing the ledger. The ledger is the truth; the balance is derived.

An event-sourced agent works the same way. Every occurrence (run started, model responded, tool called, tool returned, checkpoint written) is appended to an event log and never edited. "The current state of the run" is not stored anywhere; it is computed by replaying the log from the start, the way a balance is computed from transactions. Three superpowers fall out of this one storage decision:

• Resumption. A crash loses nothing. Replay the log, land exactly where the run was, continue. A checkpoint is just a bookmark ("replaying up to event 214 is known-good") so resumption does not reread everything.
• Audit. "Why did the agent restart the cache node?" has an exact answer: here is the event where the model proposed it, here is the tool result that preceded it, here is the approval that allowed it.
• Replay debugging. Because model responses are recorded as events, you can re-run yesterday's strange session against the recorded responses, deterministically, on your laptop. Change one tool result mid-log and watch how the run diverges. Debugging becomes an experiment instead of archaeology.

Part 1 builds this event store in plain Python, and it becomes the spine of every later lab. When we reach production databases, the design carries over almost unchanged; only the storage engine grows up.

Prompt caching: paying for re-reading

Now connect statelessness to money. Turn forty re-sends turns one through thirty-nine, so the model re-reads them, and you are billed for that re-reading, every single turn. An agent whose conversation has grown to 50,000 tokens pays to process those 50,000 tokens again on every loop iteration. Multiply by a fleet of five hundred agents and the bill is the architecture.

Prompt caching is the providers' answer. If the beginning of your request (the prefix) is byte-for-byte identical to a recent request's beginning, the provider can reuse its internal processing of that prefix instead of redoing it, and charges you around a tenth of the normal price for those tokens (with a small premium the first time, when the cache is written). For a loop, this is nearly perfect: each turn's request is the previous turn's request plus a little more at the end, so almost everything is a cache hit, if and only if you keep the prefix stable. One "helpful" timestamp inserted at the top of the instructions changes the bytes, invalidates everything after it, and silently multiplies your bill.

That "if and only if" makes caching a design discipline rather than a checkbox, and it shows up in this book's fleet arithmetic: in the lab catalog you will watch caching cut a worked fleet's bill by 1.8x, and also see exactly why it was not 10x (caching discounts what the model re-reads; it does nothing for what the model writes).

Queues, backpressure, and the delivery promise

One agent is a program. A production system is many runs arriving at once, and the moment you have "arriving at once" you need the oldest tool in distributed systems: the queue. Work goes in; workers take work out; the queue absorbs the mismatch between how fast work arrives and how fast it can be done. When arrivals outrun workers, the queue grows, and that growth pushing back on the system ("stop accepting, we are full") is called backpressure. Healthy systems slow down at the front door; unhealthy ones collapse in the middle.

Queues come with a subtle promise that shapes agent design. Most real queues guarantee at-least-once delivery: a message will be delivered, but possibly more than once. Why not exactly once? Because when a worker takes a message and then goes silent, the queue cannot know whether the work finished (and the crash ate the acknowledgment) or never happened. It must redeliver to be safe. Your agent's work can therefore run twice, and the standard defense is the idempotency key: every unit of work carries a unique id, and every effect ("file ticket #123 for run 7f3a") checks that id before executing, so a duplicate delivery becomes a harmless no-op instead of a duplicate ticket.

And when a message fails repeatedly (a malformed document, a task that crashes every worker that touches it), you do not want it redelivered forever, burning money each time. Such poison work gets moved, after N failures, to a dead-letter queue: a parking lot for broken work where a human can inspect it. Boring machinery, and it is precisely the machinery that separates "the fleet ran all night" from "one bad PDF ate the budget."

Concurrency, throughput, and Little's law

How big should the fleet be? There is a sixty-year-old, one-line answer from queueing theory called Little's law, and a supermarket makes it intuitive. If customers arrive at 2 per minute and each spends 10 minutes inside, the store holds 2 x 10 = 20 customers on average. In symbols:

$$L = \lambda , W$$

where $L$ is how many are in the system at once (concurrency), $\lambda$ is the arrival or completion rate (throughput), and $W$ is the time each one spends (latency). The law is merciless in both directions: given any two, the third is decided. If a work unit takes 4 minutes of wall clock and you want 50 units finished per minute, you will need 200 units in flight; no cleverness changes that.

For agent fleets there is always a second, sneakier ceiling. Your model provider limits you to a quota: so many requests and so many tokens per minute, account-wide. If a unit consumes 54,000 tokens and your quota is 2,000,000 tokens per minute, the whole account cannot finish more than 37 units per minute no matter how many workers you run. Fleet sizing is the discipline of computing both ceilings (Little's law for workers, quota arithmetic for tokens) and building for the lower one. The lab catalog turns this into a ten-line calculator you will reuse constantly.

Don't be confused: quota vs budget. A quota is a rate imposed on you by the provider: tokens per minute, a speed limit. A budget is a total you impose on yourself: tokens or dollars per run, a fuel tank. The mechanisms are different too. Quotas are managed by a governor (spread the fleet's calls so the account never slams into the limit); budgets are managed by a ledger (admit no work you cannot pay for, stop runs that exhaust their tank). A healthy fleet cruises at the speed limit with fuel to spare.

Isolation: the ladder from process to microVM

Our investigator so far only reads. Real agents also run code: the model writes a Python snippet to parse a log file, and something has to execute it. Executing code written by a model, possibly influenced by hostile input the model read along the way, is executing untrusted code, and untrusted code needs a wall around it. Computing offers a ladder of walls:

• A process with restricted permissions: one shared operating system, walls made of kernel bookkeeping. Cheap, and escapable through any of the kernel's enormous surface.
• A container: processes plus namespace tricks that hide the rest of the machine. This is packaging technology that got drafted into security duty; every container on a host still talks to one shared kernel, and kernel bugs are found monthly.
• A virtual machine: the wall moves below the operating system. Each VM runs its own kernel on virtual hardware, and the host exposes only a tiny hardware-shaped interface. Escapes are rare events, not monthly ones. The historical cost: VMs took seconds to boot and megabytes upon megabytes of overhead, because they emulated decades of PC hardware.
• A microVM: a VM with the nostalgia deleted. Emulate almost no devices (a disk, a network card, little else), and a VM boots in about a tenth of a second in a few megabytes. You get VM-strength walls at almost-container prices.

That last rung is the load-bearing one for this book, because the microVM implementation everyone actually uses is an open-source project called Firecracker, built and published by AWS, and it turns out to sit under nearly every AWS service we will touch. The next chapter but one tells that story properly; Part 5 runs Firecracker with our own hands.

The rule the book will repeat: agent harness code (yours, trusted) can share walls; agent output (code it wrote, content it fetched) executes behind a microVM-grade wall with its network egress controlled. The reference architecture hardens this into a structural rule about planes.

Identity: who is the agent?

When the agent's query_metrics tool hits your metrics API, the API asks the oldest question in computing: who is asking? Identity systems answer it with three ideas worth naming precisely, because AWS's identity service (IAM, which the decoder ring chapter unpacks) is built from them.

A principal is anything that can be authenticated: a person, a server, an agent. A role is a wearable identity: a bundle of permissions that a principal can temporarily assume, receiving short-lived credentials for it. Least privilege is the design rule that each principal gets the minimum permissions its current task needs, so the blast radius of a compromise is the task, not the company.

For agents, the sharp edge is a classic called the confused deputy. Give the agent a powerful identity of its own ("the agent role can read all customer records, so it can serve any user") and a hostile input will eventually trick it into using that power on the attacker's behalf: the deputy is honest but confused about whose errand it is running. The fix is structural, not clever: the agent acting for Alice holds credentials scoped to Alice's task, so even fully fooled, it can only do what Alice could do. Part 8 builds this with short-lived, task-scoped credentials, and it is gate 1 of the production bar.

Protocols: MCP and A2A

Last concept. Every tool so far was wired directly into our harness, which is fine for three tools and absurd for three hundred spread across teams who have never met. The industry's answer is the same as hardware's answer to peripherals: agree on a port.

MCP, the Model Context Protocol, is the port for tools. It is an open protocol (started by Anthropic in late 2024, since adopted broadly) in which a server exposes tools with their schemas, and any client (your agent, someone's IDE, a managed runtime) can discover and call them over a standard wire format. Write your internal API's MCP server once and every MCP-speaking agent can use it, the way one USB-C device works with every laptop. AWS leans on this hard: AgentCore's Gateway component exists to turn your existing APIs and functions into MCP servers without you writing them.

A2A, the Agent2Agent protocol, is the port one level up: not "agent uses tool" but "agent hires agent." An A2A agent publishes a card describing what it can do; other agents send it tasks and track their lifecycle, without either side knowing what framework or vendor the other is built on. It is younger than MCP and governed by the Linux Foundation, and Part 10 gives an honest assessment of how much of it is load-bearing today versus aspiration.

Don't be confused: MCP vs A2A. MCP connects an agent to capabilities (tools with typed inputs; the callee has no judgment of its own). A2A connects an agent to counterparties (other agents that plan, act, and report back). If the thing on the other end just executes and returns, that is MCP. If it thinks, that is A2A.

The story, retold in the vocabulary

Now the payoff. Here is the checkout investigation, described the way the rest of this book will talk:

An alarm event lands on a queue. The scheduler checks the run's budget, admits it, and appends run.created to the event log. A worker leases the run and starts the agent loop under a task-scoped role whose tools are read-only. Each turn is a stateless model call carrying the full transcript, cheap because the prefix is cached; each tool call and result is appended to the log. When the model writes a log-parsing script, it executes in a Firecracker microVM with no network egress. Twenty minutes in, the worker dies; the lease expires, another worker replays the log from the last checkpoint, and the run continues, idempotency keys ensuring the one ticket it already filed is not filed twice. The final report cites its evidence by event id, and the proposed fix parks behind an approval gate for a human. Throughput held steady at the token quota all along, because the governor was shaping calls, and the whole run cost $0.31 against its $2 budget.

Every noun in that paragraph now has a definition and, from the map, an address: a part and chapter where it becomes code you run.

👉 Next: the map, the complete curriculum, which you can now read with every term carrying its full weight.