Event-driven agents: work that arrives
Every fleet run so far was submitted: someone called the API with a task. But many of the best agent use cases are reactive, the incident responder that wakes when an alarm fires, the document swarm that starts when a file lands, the pipeline babysitter that triages a failed job. For these, work does not get handed to the agent; it arrives, and the plumbing that turns "something happened" into "an agent ran" is the event-driven skeleton. This chapter is that plumbing, and it carries the part's runnable lab, because the queue underneath it makes a promise that shapes every effectful agent.
From event to agent
Two AWS pieces, both decoded in Part 3, do the reactive plumbing:
- EventBridge is the event bus: services and SaaS apps publish events (an alarm fired, a deploy finished, a ticket was created), rules pattern-match them, and a match triggers a target. It is how "an alarm fired" becomes "invoke the incident agent" without anything polling. The agent becomes a subscriber to the events it cares about.
- SQS is the work queue: it sits between the event and the fleet, absorbing bursts (a hundred documents land at once) so the workers see a steady queue rather than a stampede. It is the scheduler's queue, now a managed service with a specific, honest delivery contract.
The shape is: event happens → EventBridge rule matches → work lands on SQS → a worker (Lambda for short agents, Fargate for long) picks it up → the agent runs. The backpressure is free: a burst grows the queue, not the failure rate, and the fleet drains it at its sustainable pace.
The contract that shapes everything
SQS makes one promise that is not what people want but is the only honest one a distributed queue can make: at-least-once delivery. A message will be delivered, possibly more than once. The reason is unavoidable, and the concepts chapter named it: when a worker takes a message and goes silent, the queue cannot distinguish "finished, but the acknowledgment was lost" from "crashed before doing anything." To guarantee the work happens, it must redeliver, and redelivery means your agent's effect can run twice.
Two mechanisms tame the contract, and both were designed into the book already:
- Visibility timeout. When a worker receives a message, the queue hides it for a configured window; if the worker deletes it (success) within the window, done, and if not (crash, or just slow), the message reappears for redelivery. Tuning this for agent latencies matters: agent tasks run minutes, so a visibility timeout tuned for millisecond web handlers will redeliver a healthy worker's message mid-run, spawning a duplicate. Set it above your tasks' real duration.
- Idempotency keys. The tool-design chapter's
operation_id, earning its keep exactly here: an effect keyed by a unique id checks whether that id already ran and no-ops the duplicate, turning at-least-once delivery into exactly-once effect. - Dead-letter queues. A message that fails repeatedly (poison work) is moved, after N receives, to a separate queue for human inspection, instead of retrying forever and burning budget. The failure-semantics taxonomy's poison case, as a queue configuration.
Lab 4.1: the contract, made concrete
The lab models one alert flowing through the queue three ways, so the contract and its two mechanisms are visible rather than asserted:
python3 queue_sim.py
--- at-least-once, no idempotency ---
deliver 1: effect done, worker crashed before delete -> will redeliver
deliver 2: effect done, message deleted
result: 2 ticket(s) filed for 1 real alert -> DUPLICATE BUG
--- at-least-once, with an idempotency key ---
deliver 1: effect done, worker crashed before delete -> will redeliver
deliver 2: no-op (idempotency key already done); message deleted
result: 1 ticket(s) filed for 1 real alert -> CORRECT
--- poison message: fails every attempt ---
attempt 1: worker crashed on malformed input (receive 1/3)
attempt 2: worker crashed on malformed input (receive 2/3)
attempt 3: worker crashed on malformed input (receive 3/3)
attempt 4: dead-lettered (exceeded 3 receives)
result: 0 tickets, 1 in the dead-letter queue
Three lessons, each a production bug avoided:
- Without idempotency, at-least-once means at-least-once effects. The worker filed the ticket, crashed before deleting the message, and the redelivery filed it again: one alert, two tickets. Nothing was "wrong" with the code; the queue kept its honest promise and the effect was not built to expect it. This is the single most common agent-fleet bug that reaches production, because it only manifests on the crash-then-redeliver path that testing rarely hits.
- The idempotency key fixes it structurally. Same crash, same redelivery, but the second attempt sees the key already done and no-ops: one alert, one ticket. Exactly-once effect on top of at-least-once delivery, which is the strongest guarantee a distributed system offers and the one every effectful agent tool needs.
- The dead-letter queue bounds the poison. The malformed message failed three times and then stopped being retried, parking in the DLQ for a human instead of looping forever. Without it, one bad input is an unbounded spend; with it, it is a bounded incident and a queue to inspect.
Idempotency is a tool property, not a plumbing add-on
The lab's deepest point is where responsibility lives. The queue cannot
make your effect idempotent; only the tool can, by checking its
operation id before acting. This is why the tool-design
chapter put operation_id in the schema of every
effectful tool long before this chapter existed: the queue's contract
and the tool's contract were designed to meet here. An agent platform
whose tools are idempotent can run on at-least-once infrastructure
safely; one whose tools are not will double-file, double-charge, and
double-send the first time a worker crashes at the wrong moment, which
on a long-running fleet is not "if" but "which night."
Don't be confused: at-least-once vs exactly-once. People ask for exactly-once delivery and cannot have it: a distributed queue that guaranteed a message arrives exactly once would have to distinguish "done but unacknowledged" from "never started" with perfect certainty, which the network does not permit. What you can build is exactly-once effect: accept that delivery is at-least-once, and make the effect idempotent so duplicates are harmless. The industry shorthand "exactly-once" almost always means this, at-least- once delivery plus idempotent processing, and a design that waits for true exactly-once delivery waits forever.
Full source
"""Lab 4.1: the queue contract. At-least-once, idempotency, dead-letter.
The queue under an agent fleet (SQS in production) makes one honest,
awkward promise: at-least-once delivery. A message will arrive, maybe
more than once, because when a worker takes a message and goes silent
the queue cannot tell "finished but the ack was lost" from "crashed
before starting," so it must redeliver to be safe. This lab makes that
promise, and the two mechanisms that tame it, concrete:
1. without an idempotency key, a redelivered message runs its effect
twice (a duplicate ticket),
2. with one, the redelivery is a harmless no-op (exactly-once effect
on top of at-least-once delivery),
3. a poison message that fails every time lands in a dead-letter
queue after N attempts instead of retrying forever.
Standard library only. Deterministic.
"""
from __future__ import annotations
from dataclasses import dataclass, field
MAX_RECEIVES = 3 # then the message is dead-lettered
@dataclass
class Message:
id: str
idempotency_key: str
receives: int = 0
@dataclass
class World:
tickets: list = field(default_factory=list) # the effect (side effects!)
done_keys: set = field(default_factory=set) # the idempotency store
dlq: list = field(default_factory=list)
def deliver(msg: Message, world: World, use_idempotency: bool,
worker_crashes: bool) -> str:
"""One delivery attempt. Returns what happened."""
msg.receives += 1
if msg.receives > MAX_RECEIVES:
world.dlq.append(msg.id)
return "dead-lettered (max receives exceeded)"
# idempotency check BEFORE the effect
if use_idempotency and msg.idempotency_key in world.done_keys:
return "no-op (idempotency key already done); message deleted"
# the effect: file a ticket
world.tickets.append(msg.id)
if use_idempotency:
world.done_keys.add(msg.idempotency_key)
if worker_crashes:
# crashed after the effect, before deleting -> queue will redeliver
return "effect done, worker crashed before delete -> will redeliver"
return "effect done, message deleted"
def scenario(title: str, use_idempotency: bool) -> None:
print(f"--- {title} ---")
world = World()
msg = Message("msg-A", idempotency_key="ticket-for-alert-99")
# first delivery: worker crashes after the effect
print(f" deliver 1: {deliver(msg, world, use_idempotency, True)}")
# visibility timeout expires; the queue redelivers
print(f" deliver 2: {deliver(msg, world, use_idempotency, False)}")
print(f" result: {len(world.tickets)} ticket(s) filed "
f"for 1 real alert -> {'CORRECT' if len(world.tickets) == 1 else 'DUPLICATE BUG'}\n")
if __name__ == "__main__":
scenario("at-least-once, no idempotency", use_idempotency=False)
scenario("at-least-once, with an idempotency key", use_idempotency=True)
print("--- poison message: fails every attempt ---")
world = World()
poison = Message("msg-POISON", idempotency_key="never-completes")
for attempt in range(1, MAX_RECEIVES + 2):
# worker always crashes BEFORE the effect (malformed input)
poison.receives += 1
if poison.receives > MAX_RECEIVES:
world.dlq.append(poison.id)
print(f" attempt {attempt}: dead-lettered "
f"(exceeded {MAX_RECEIVES} receives)")
break
print(f" attempt {attempt}: worker crashed on malformed input "
f"(receive {poison.receives}/{MAX_RECEIVES})")
print(f" result: {len(world.tickets)} tickets, "
f"{len(world.dlq)} in the dead-letter queue")
print(" the poison message stopped burning retries after "
f"{MAX_RECEIVES} attempts; a human inspects the DLQ, the fleet "
"moves on.")
👉 Next: the scheduling layer in practice, where the queue, the fleet, and the governor become one operable service with priorities, tenant fairness, backpressure, and a clean drain during a deploy.