Distributed map and fan-out

Part 6's fan-out chapter named distributed map as one of three ways to run a swarm and moved on. This chapter opens it up, because it is the managed embodiment of the swarm scheduler you built by hand, and understanding its knobs is understanding what you are buying versus building when a fleet runs on it. The one-line version: distributed map takes a dataset and runs the same work on every item, thousands in parallel, with the queue, concurrency governor, and failure handling that Part 6's simulator implemented in Python now supplied as configuration.

The state, from the inside

A distributed map state has four moving parts, and each maps onto a piece of the simulator:

• The item source. Where the work units come from: an S3 object (one big file the map shards by line or by object), an S3 prefix (one item per object), or an inline array. This is the orchestrator's decomposition output, the work-unit list, read from durable storage rather than held in memory.
• Max concurrency. How many children run at once, up to 10,000. This is the simulator's workers parameter, and the Little's law crossover still governs it: setting concurrency above what the token quota can feed just makes children queue, so the right value is N*, not the maximum. The managed service will happily let you set 10,000 workers against a quota that feeds 119, and the map will run correctly and no faster.
• The child workflow. What runs per item: a Lambda, a Fargate task, or a nested Step Functions execution. For agent work this is the worker running the loop for one unit. Inline for short items, a nested execution for units that need their own multi-state flow.
• The result writer and tolerated failure. Where results aggregate (back to S3) and how many child failures the run tolerates before it fails overall, a percentage or count. This is the failure-threshold knob: a fleet where 2% of units are expected to be poison should tolerate 2%, not abort the night on the first bad shard.

Batching, the hidden cost lever

One knob deserves its own note because it silently drives cost: item batching. By default the map runs one child per item, but a child has fixed overhead (a Lambda invocation, a state transition), and for small items that overhead can dwarf the work. Batching groups several items into one child invocation, amortizing the overhead. For an agent fleet this is a real dial: batching several tiny units into one worker invocation cuts per-item infrastructure cost, at the price of coarser failure granularity (a batch fails as a unit) and coarser checkpointing. It is the fan-out cousin of the cost gate's fixed-overhead lesson, small units pay disproportionately for fixed costs, so amortize them.

Redrive: resume, as a button

The failure-semantics lab built kill-and-resume by hand and found the payoff: re-run only the failed slice, not the whole night. Distributed map ships that as redrive: after a map run partially fails, one call re-runs only the failed child executions, reusing the successful ones. It is Part 6's resume, purchased, and it is genuinely good, one of the strongest reasons to run a batch fleet on distributed map rather than a hand-rolled fleet.

The one thing redrive does not know about is your budget ledger. The resume lab's sharp lesson was that a crash spends budget: the tokens the failed children already burned are gone, and redrive re-runs the failed items again, spending more. So redrive resumes the work correctly while your ledger must still account for the doubled spend on the redriven slice, exactly the crash-reserve the lab argued for. Buying the resume mechanism does not buy the budget accounting; that stays yours, on top, in every bundle.

What you buy, and what stays yours

Read against the simulator, distributed map supplies the mechanical half of the swarm scheduler and leaves the agentic half to you:

The pattern generalizes across every managed piece in this book: AWS supplies the distributed-systems mechanics (fan-out, concurrency, retry, resume) extremely well, and the two things it cannot supply, because they are specific to agents spending tokens, the governor and the ledger, are the things Part 6 built and Hive owns. Buy the fan-out; keep the economics.

Don't be confused: distributed map vs the Parallel state. Step Functions has both and they solve opposite problems, as Chapter 24 noted and this chapter can now sharpen. Parallel runs a fixed, small set of named branches written into the workflow: "do these four different things at once," a pipeline with simultaneous stages. Distributed map runs one branch over a large dataset: "do this same thing ten thousand times," fleet fan-out. Named-and-few versus same-and-many. A workflow can nest a map inside a Parallel branch (fan out one stage of a multi-stage pipeline), which is exactly how a capstone that fans finders out and then runs a separate merge stage is shaped.

👉 Next: event-driven agents, where work stops being handed to the fleet and starts arriving, and the queue contract under all of it, at-least-once delivery, idempotency, and dead-letter queues, gets its runnable lab.