Build your own sandbox service

The previous chapter explained why an agent's code runs behind a microVM and why snapshot-restore beats a warm pool. This chapter turns that into a service: an API an agent calls to say "run this code, in a fresh isolated box, and give me the output," with Firecracker underneath. We build it in the round (the pieces, the request path, the pool policy the lab already sized) and mark the parts that need real bare metal as the T2 follow-along they are. The goal is not that most readers will run this in production; it is that after building it once, the managed equivalent in Chapter 31 is a thing you can price rather than a black box you must trust.

The pieces

A code-execution sandbox service is five components, and only one of them is exotic:

  1. A control API. Ordinary web service: POST /execute with code and a language, returns stdout, stderr, exit code, and any artifacts. This is the surface the agent's tool calls, and it looks like every other tool from Chapter 8.
  2. A microVM manager. The exotic piece: it asks Firecracker to restore a microVM from a snapshot, hands the code in, collects the result, and destroys the microVM. One microVM, one execution, then gone.
  3. A rootfs and kernel image. What the microVM boots into: a Linux kernel plus a root filesystem containing the interpreter and whatever libraries the agent's code may need. Building this is a pipeline, below.
  4. A snapshot. Produced once from a booted, initialized rootfs; restored per request. This is what makes restores fast enough to be interactive.
  5. A host fleet. Bare-metal instances (Firecracker needs KVM, which nested virtualization on ordinary instances does not reliably provide), each running many microVMs, fronted by the control API.

Notice how little of this is Firecracker-specific. Four of the five components are the same web-service plumbing any team already knows; the microVM manager is the only piece demanding new knowledge, which is precisely the shape of "operations, not secrets" from the decoder ring.

The rootfs pipeline

The root filesystem is the microVM's whole world, so building it is a security decision as much as a packaging one. The pipeline, sketched:

  • Start from a minimal base (a small distro rootfs, or a from-scratch busybox image). Minimal is safer: every binary you do not include is a binary the attacker cannot use.
  • Install exactly the runtime the sandbox offers (say Python plus a fixed, audited set of data libraries) and nothing else. No compilers you did not intend, no network tools, no shells beyond what execution needs.
  • Bake in the execution harness: a tiny init that reads code from a known channel (a virtio-served file, or the serial console), runs it under a timeout, and writes results back.
  • Make the filesystem read-only where possible, with a small writable scratch area that dies with the microVM.

This is a follow-along on any machine (building a rootfs image is just assembling a directory tree and packing it), and becomes a live sandbox only on a KVM-capable host in the next section.

The request path

Put the pieces in motion and one POST /execute flows like this:

agent tool call
   |
   v
POST /execute {code, lang}     <- control API
   |
   v
manager: restore microVM from snapshot   (~tens of ms, per Ch 28)
   |
   v
hand code in over virtio ; init runs it under a timeout
   |
   v
collect stdout / stderr / exit / artifacts
   |
   v
DESTROY the microVM            <- fresh box guaranteed for next request
   |
   v
return results to the agent

Two properties of that path are the reason the service exists. Freshness by destruction: the microVM is destroyed after one execution, so no state, no file, no leaked secret survives into the next tenant's box. Reuse would be faster and is the wrong trade; isolation between requests is the product. Restore, not boot: the manager restores the pre-initialized snapshot rather than cold-booting, which is what keeps a per-request-destroyed microVM economically and latency-wise viable, exactly the snapshot row from the lab.

The pool policy, decided by the lab

The sandbox-pool lab already made the policy call, so here it just gets stated as the service's configuration. A big warm pool of booted microVMs is the intuitive design and the wrong one: warm(20) bought instant latency by reserving ten gigabytes of memory around the clock, and warm(5)'s small pool posted a respectable-looking 71% hit rate whose p99 was still the cold cliff the moment a burst arrived. Snapshot-restore dominated both: near-warm latency, zero standing reservation. So the service keeps a snapshot on disk, restores per request, and holds at most a tiny warm buffer (single digits) to absorb microsecond-scale restore latency, not to serve the load. The pool size stops being a capacity lever and becomes a smoothing one.

The bare-metal reality (T2)

Standing this up for real is where the tier jumps to T2, and the honesty tax comes due. You need bare-metal instances (a *.metal shape, so KVM is genuinely available); you install Firecracker and the jailer; you build and host the rootfs, kernel, and snapshot; you write the manager against Firecracker's local API (a REST interface over a Unix socket that configures the boot source, drives, and network, then triggers InstanceStart or LoadSnapshot); you handle networking (a tap device per microVM, or a shared bridge) and, critically, egress policy (the next chapter's entire subject); and you now operate a fleet with capacity planning, host failures, patching, and monitoring. None of it is research; all of it is work, and the sum is a real service with an on-call rotation.

The point of walking through it is not to talk most teams into building it. It is that the cost is now legible: a fleet of metal hosts, a rootfs pipeline, a manager service, and the operational load of all three, weighed in Chapter 31 against a per-second managed price. You cannot make that trade honestly without knowing what "build" actually contains, and now you do.

Don't be confused: the sandbox vs the agent. The agent loop and its harness run on trusted infrastructure (the execution plane); the sandbox runs the agent's output (the sandbox plane). This service is the second thing, and it must never share credentials, network, or filesystem with the first, a separation the reference architecture drew as two planes for exactly this reason. If your sandbox can read the worker's IAM role, you built an expensive way to hand an attacker your keys.

👉 Next: locking the box, where the sandbox stops being merely isolated and becomes controlled: egress policy, secrets kept outside, quotas, and the failure modes that isolation alone does not stop.