Watching and governing: Observability, Policy, Evaluations

Three chapters of Part 3 gave the managed agent a home, a memory, and hands. This one covers the supervision: how you see what sessions did, how you bound what they may do, and how you judge whether they did it well. The three concerns are general to every agentic system (a LangGraph fleet or a hand-rolled loop needs all three just as much), and AgentCore's post-launch additions made its answers concrete enough to decode: Observability at GA, then Policy and Evaluations, the two components AWS added once real customers ran real agents and discovered what was missing.

Seeing: traces are the unit of truth

Agent observability differs from service observability in one structural way: the interesting unit is not the request but the trajectory, the whole multi-turn arc of a session with its model calls, tool calls, token spends, and decisions. A latency histogram cannot tell you an agent looped uselessly for nine turns; a trace can. The industry converged on representing trajectories as distributed traces (spans for turns, child spans for model and tool calls, token counts as attributes), and, importantly, on carrying them in OpenTelemetry, the open standard, with emerging generative-AI semantic conventions for what to call things.

AgentCore Observability is that convergence, productized: sessions emit OTEL traces (via AWS's supported distribution, ADOT), CloudWatch grows agent-shaped dashboards (sessions, traces, token usage, error taxonomies), and, because the format is open, the same stream feeds whatever else you run: the Langfuse-style open trace stores, an existing Datadog, or Part 9's own ledger jobs. The event store relationship is worth pinning: traces and event ledgers overlap but serve different masters. Traces are for operators (sampled, dashboarded, retention-limited); the ledger is for replay and audit (complete, byte-faithful, yours). Hive emits both from the same hooks, and neither substitutes for the other.

Bounding: policy outside the loop

The control ladder ended on rung four, forbid, and promised a managed example. AgentCore Policy is it: you write rules about agent behavior in natural language ("this agent may read any ticket but may only file tickets tagged with its own run id," "never call payment tools during business-hours incidents"), the service compiles them to Cedar, the open-source, formally verified policy language from the decoder ring, and enforcement happens in the platform, on every tool call, outside the model's reasoning loop.

That placement is the entire point, and it is worth saying with precision: a rule in the system prompt is input to a process an attacker also gets to feed; a Cedar rule evaluated by the platform is not part of the conversation at all. No jailbreak reaches it, because there is nothing linguistic to reach. The natural-language authoring is ergonomics; the compiled artifact is what runs, it is inspectable, versionable, and testable like any policy-as-code, and it is portable in the way this stack keeps choosing (Cedar is Apache-licensed and runs anywhere, so the rules you write are not hostage to the service that compiled them).

The general lesson travels beyond AWS: every agent platform needs a place where rules are enforced that the model cannot argue with. If your stack lacks a policy engine, that place is IAM plus your tool gateway, which is coarser but real; what it must never be is the prompt alone.

Judging: graders over trajectories

The third leg answers "was it any good?", and the managed version arrived for the same reason Policy did: teams shipped agents, then discovered they had no way to know whether Tuesday's prompt change made Wednesday's agents worse. AgentCore Evaluations runs graders over your captured session traces: built-in evaluators for the qualities agent work actually has (goal success, correctness, helpfulness, tool-selection accuracy, safety), scored continuously or on demand, surfaced next to the traces they judged.

Decode the mechanism and it is familiar from the companion books' LLM-judge chapters: a grader is a model with a rubric reading a trajectory, and everything known about judges applies. They are probabilistic; they need calibration against human-labeled examples before their absolute numbers mean anything; and their real power is relative, catching regressions between versions rather than certifying quality in the void. Which frames what buying does and does not get you, one last time in this part:

Gate 8 of the production bar said evals must be able to fail a pull request. Managed graders shorten the build; the bar, the suite, and the enforcement remain architecture, which is why Part 9 exists regardless of what is bought here.

Don't be confused: Policy vs Evaluations. Both "check the agent," on opposite sides of the action. Policy is preventive and binary: evaluated before the tool call, allow or deny, no judgment involved, and a denial is enforcement, not feedback. Evaluations is retrospective and graded: it reads what already happened and scores it, changing nothing about the run it scored. Policy stops the forbidden; Evaluations notices the mediocre. Confuse them and you get either a grader asked to block production traffic in real time (too slow, too probabilistic) or a policy engine asked to assess quality (wrong tool entirely).

👉 Next: the verdict, where Part 3's six components meet the build-versus-buy matrix, the production-bar scorecard, and three honest team profiles.