The supporting-platform layer

Part 9 built observability, evals, and cost as planes you own, and built them from scratch to demystify them. In practice you rarely build all of it: a mature ecosystem of platforms supplies the machinery, and the decision is buy-versus-build per plane against what Part 9 taught you the plane must do. This chapter maps that ecosystem, the observability platforms, the eval frameworks, and the model-routing gateways, so you can place each against the planes you already understand and know what stays yours even after you buy. Structural roles are stable; treat specific product claims as of mid-2026 and verify current status.

Observability platforms

Chapter 40's requirement was trajectory traces plus a token ledger, in OpenTelemetry so you are not locked in. Six platforms supply this, and they sort by two axes, open-source versus SaaS, and observability-first versus eval-first:

  • LangSmith (SaaS, LangChain) has the deepest LangGraph/LangChain integration: tracing, evals, and datasets in one place, the default if your worker is a LangChain-family agent.
  • Langfuse (open-source, self-hostable) is the open-source leader: full tracing and eval tooling with no per-seat pricing, the pick when you want to own the deployment.
  • Arize Phoenix (open-source, OTEL-native) leans on the open telemetry standard and ML-grade eval metrics, the natural fit for a team already emitting OTEL (which, per Chapter 40, you should be).
  • Braintrust (SaaS, eval-first) centers on eval-driven development and CI gating, the strongest fit for the regression-gate discipline of Part 9.
  • W&B Weave (SaaS) brings LLM/agent tracing and eval into the Weights & Biases world, natural if your ML org already lives there.
  • Helicone (open-source, proxy-based) is the lightest install: a drop-in gateway that captures traces and cost with a one-line change, the fastest way to start seeing token spend.

The buy-versus-build call, against Part 9: because you should emit OTEL regardless, adopting an OTEL-native platform (Phoenix, or CloudWatch's GenAI dashboard) is addition, not replacement, you point the exporter at it and keep your event ledger for replay and audit. What stays yours no matter which you buy: the ledger (traces are sampled, the ledger is complete), and the decision of which traces are interesting enough to keep, which is your tail-sampling policy, not the vendor's default.

Eval frameworks

Chapter 41's requirement was a golden set, a grader, and a gate. The eval frameworks supply the grader and gate machinery; the golden set and the judge calibration stay yours:

  • Ragas is the research-driven set of RAG metrics, faithfulness, answer relevancy, context precision and recall, mostly reference-free, the pick when the agent is retrieval-heavy and you want to score the retrieval, not just the answer.
  • DeepEval is "pytest for LLMs": assertion-style tests, a broad metric library (including agent and RAG metrics), and CI gating, the pick when you want evals that look like tests and fail a build the way tests do, exactly gate 8.
  • promptfoo is a CLI and library for prompt testing, provider comparison, and security red-teaming (injection, jailbreaks, PII), which makes it double as a Part 8 tool: it is where you run the injection drill at scale against your real prompts.
  • OpenAI Evals is the open-source eval framework and registry for defining and running model evals, the pick inside an OpenAI-centric stack.

The through-line from Chapter 41: these frameworks shorten the build of the grader and the gate, but they do not build your golden task set (the thing that encodes what "good" means for your domain) or calibrate the judge against human labels. A team that adopts DeepEval and skips the golden set has a testing framework with nothing worth testing; the framework is the easy 20%.

Model routing gateways

One layer Part 2 did not dwell on: a gateway in front of the model plane, so your workers talk to one endpoint and the gateway handles provider access, fallback, cost tracking, and rate limiting. Three shapes:

  • LiteLLM (open-source, self-hosted proxy) normalizes 100-plus providers behind one OpenAI-format endpoint, with virtual keys, per-key budgets, rate limits, and fallback in a versioned config, the pick when you want provider portability and to run the gateway yourself. It is, in effect, a self-hosted implementation of the governor-plus-budget concerns as a proxy.
  • OpenRouter (hosted) is a marketplace: one key, hundreds of models, automatic fallback, day-one access to new models, the pick when you want breadth without operating anything.
  • Portkey (gateway, guardrails-first) leads with advanced routing (conditional and metadata-based), load balancing, and circuit breakers, the pick when routing sophistication is the priority.

Where this sits relative to the book: a gateway is a convenience over the model plane, useful when you serve many providers or want provider portability. On a Claude-on-Bedrock platform committed to one provider, it is often unnecessary weight, the Mantle client plus your own governor and ledger already do the fallback, budgeting, and rate-shaping a gateway sells. The decision is the recurring one: buy the gateway when provider breadth is a real requirement; skip it when you are one-provider and the planes you built cover its job.

Don't be confused: observability vs evaluation platforms. They overlap in product (many tools do both) and differ in when they run, the same in-the-run vs about-the-system split from Part 9. Observability watches production traffic live (traces, cost, latency, what is happening now); evaluation scores candidates offline or on a canary (did this change make the agent better). A tool that does both is convenient, but do not let the overlap blur the disciplines: you alert on observability, you gate merges on evaluation, and wiring an offline eval into a live alert (or gating a merge on a production dashboard) breaks both.

👉 Next: beyond text, the modalities the book has mostly left implicit, computer-use agents that operate GUIs and voice/realtime agents, and how each stresses the platform's sandbox, trust, and latency assumptions.