Beyond text: computer-use and voice agents

The book has assumed a text agent: a prompt in, tokens out, tools in between. Two modalities break that assumption and have matured into real categories by 2026, agents that operate graphical interfaces and agents that hold a voice conversation. This chapter covers both, not as novelties but as workloads that stress the platform's existing planes in specific ways, computer-use hits the sandbox and trust planes hardest, voice hits latency hardest. Treat product specifics as of mid-2026; the architectural stresses are stable.

Computer-use agents

A computer-use agent perceives a screen (a screenshot) and acts on it (click, type, scroll, navigate), which lets it drive software that has no API, the browser-automation use case generalized to any GUI. The category is now populated: Anthropic's computer use (a generic screenshot-in, actions-out tool, still customer-sandboxed), OpenAI's Operator folded into ChatGPT's agent mode (built on a computer-using model), Google's Project Mariner for browser automation, and Amazon's Nova Act SDK for web agents. The shape is convergent: a model that plans and a driver (often Playwright or the browser's debug protocol) that executes.

The architectural point is that computer-use is the maximal stress on two planes this book already built:

  • The sandbox plane (Part 5). A computer-use agent runs a real browser or desktop, and that browser fetches arbitrary web content, the untrusted-input firehose. So it belongs in the strongest isolation you have (a microVM session, which AgentCore Browser provides by default), with egress controlled to the domains the task needs. A computer-use agent without a sandbox is the container-for-hostile-code mistake with a GUI attached.
  • The trust plane (Part 8). Every page the agent views is untrusted content, and the agent's actions are exactly the egress the lethal trifecta worries about, a computer-use agent that can log into your bank and read attacker-controlled web pages has all three trifecta legs in one session. The defenses are the ones you already have: scope what it can reach, keep secrets out of the sandbox and inject them at the boundary, and gate consequential actions (Chapter 38). Computer-use raises the stakes; it does not require new defenses.

Two more planes get exercised. Observability matters more, because a computer-use trace should include screenshots at each step (the visual audit trail), which AgentCore Browser's live-view and recording provide. And evals are harder, because success is "did the GUI end in the right state," a check that itself may need a vision model, so the golden set for a computer-use agent is a set of start-states and target-states, scored by inspection.

Voice and realtime agents

A voice agent holds a spoken conversation, and its defining constraint is latency: a human notices a pause past roughly a few hundred milliseconds, so the whole loop, hear, think, speak, must close fast enough to feel like conversation. The 2026 pattern is a single persistent connection carrying audio in and out through a speech-to-speech model (OpenAI's Realtime API is the reference endpoint), rather than the old chain of separate speech-to-text, then text model, then text-to-speech, which stacked too much latency. The open-source frameworks for building these, Pipecat and LiveKit Agents, handle the real-time media plumbing (WebRTC, telephony/SIP, turn detection, interruption) so you build the agent, not the audio pipeline.

Voice stresses the platform differently, and the differences are worth naming because they invert some of the book's defaults:

  • Latency dominates over cost. The cost-versus-latency tension resolves hard toward latency: a voice agent cannot wait for a batch tier or a slow verification panel mid-conversation, so the model portfolio skews to the fastest capable model and heavy verification moves off the conversational path (check after, or in a parallel track), not into it.
  • Streaming is mandatory, not a UX nicety. The streaming the book treated as good practice becomes the entire interaction model, and turn-taking (knowing when the human stopped talking, and handling interruptions) is a real problem the voice frameworks solve and a text platform never had.
  • The session is long and stateful. A voice call is a long session with continuous state, exactly the shape AgentCore Runtime's session model and its bidirectional-streaming support target, and where the CPU-active-only pricing helps, because a call spends much of its wall clock waiting for the human to speak.

Everything else carries over: a voice agent still calls tools, still needs scoped identity when it acts, still benefits from the ledger (the transcript is the event log). The modality changes the transport and the latency budget; the platform underneath is the same.

What the modalities teach

The lesson of both, and the reason they close this survey rather than opening a new book: new modalities are new I/O on the same platform. Computer-use changes what a tool is (a click instead of an API call) and voice changes the transport (audio instead of text), but neither changes the reference architecture's planes. The sandbox still isolates untrusted execution, the trust plane still bounds blast radius, the ledger still records the trajectory, the scheduler still governs the fleet. A platform built for text agents extends to computer-use and voice by adding an I/O adapter and re-weighting a few dials (isolation up for computer-use, latency up for voice), not by rearchitecting. That extensibility is the payoff of having built planes instead of a text-shaped monolith.

Don't be confused: multimodal model vs multimodal agent. A multimodal model accepts images or audio as input (Claude reading a screenshot, a model transcribing speech), which the book already used when a tool returned an image block. A multimodal agent additionally acts in that modality, clicking a GUI, speaking a reply, which is a loop-and-tooling concern, not just a model capability. Computer-use and voice agents need the multimodal model and the action side, and it is the action side, the driver, the audio transport, the turn-taking, that this chapter is about, because the model half was solved in the model plane.

👉 Next: the coding-agent ecosystem, the most mature applied agentic systems in the world, and what a platform builder learns from studying how Claude Code, Devin, and their peers are actually built.