Sidecar, ambassador and adapter

What it is

Three names for one mechanism: run a helper process alongside your application, sharing its network namespace and lifecycle, so it can add capability without the application knowing. The three names distinguish what the helper does.

SIDECAR     the general pattern: a co-located process sharing
            the pod's network and volumes. Adds capability to
            an application it cannot modify.

AMBASSADOR  a sidecar that proxies OUTBOUND connections. The
            app dials localhost; the ambassador does service
            discovery, mTLS, retries, timeouts, circuit
            breaking, load balancing.

ADAPTER     a sidecar that normalises OUTPUT. The app emits its
            own metrics or log format; the adapter translates
            it into the platform's standard one.

A service mesh is ambassadors everywhere plus a control plane that configures them: Istio with Envoy, Linkerd with linkerd2-proxy.

What this is confused with: a sidecar and any second container. The defining property is the shared network namespace, which is what lets the application talk to localhost and lets the proxy intercept traffic transparently via iptables or eBPF. A second container that does unrelated work is just a second container.

Also confused: service mesh and API gateway. A gateway sits at the edge, handles north-south traffic, and terminates external concerns (auth, rate limiting, public routing). A mesh handles east-west traffic between internal services. They overlap and they are not substitutes, and teams that deploy a mesh expecting it to replace the gateway discover this at the ingress.

The problem it solves

Cross-cutting concerns in a polyglot estate have to be implemented once per language, and then upgraded once per service.

mTLS, retries with jitter, circuit breaking, timeouts,
outlier detection, load balancing, distributed tracing
propagation, per-request authorisation.

As LIBRARIES, in a 4-language estate:
  4 implementations, each with its own bugs
  a CVE in one means coordinating 60 service deploys
  a policy change (new retry budget) is a code change,
    a release and a deploy, per service
  a service written in a fifth language cannot participate
  and the team that owns the library becomes a bottleneck
    for everyone else's release

As SIDECARS:
  one implementation, in one language
  a CVE is a sidecar image roll
  a policy change is a control-plane config push, in seconds
  any language participates, including a vendor binary you
    cannot modify

The library approach is not wrong; it is wrong at a specific scale. Netflix ran the library model (Ribbon, Hystrix, Eureka) successfully for years, in a predominantly JVM estate. The mesh argument becomes decisive when the estate is polyglot or when policy must change faster than code ships.

Mechanics

The sidecar's defining properties

SHARED NETWORK NAMESPACE
  same IP, same loopback. The app connects to 127.0.0.1:1234
  and the sidecar handles the rest. Or iptables/eBPF redirects
  the app's outbound traffic to the sidecar transparently, so
  the app needs no change at all.

SHARED LIFECYCLE
  scheduled together, scaled together, killed together. This
  is the source of the pattern's operational problems (below).

SHARED VOLUMES
  the adapter case: the app writes logs to a volume, the
  adapter reads and forwards.

INDEPENDENT FAILURE, SORT OF
  the sidecar can crash and restart without the app
  restarting, which is good, and while it is down the app's
  outbound traffic fails, which is not.

Ambassador: the outbound proxy

Without:   app -> DNS -> service B (app owns discovery,
                                    retries, TLS, LB)
With:      app -> 127.0.0.1 -> sidecar -> service B's sidecar
                                          -> service B

What the app no longer contains:
  - service discovery
  - client-side load balancing and outlier ejection
  - mTLS: certificate issuance, rotation, validation
  - retries, timeouts, hedging, circuit breaking
  - traffic splitting for canaries
  - per-request authorisation policy
  - the metrics and traces for all of the above

# Istio: a policy change that would otherwise be a code change
# in every calling service, applied by config push in seconds.
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: pricing }
spec:
  hosts: [pricing]
  http:
    - route:
        - destination: { host: pricing, subset: v1 }
          weight: 95
        - destination: { host: pricing, subset: v2 }
          weight: 5
      timeout: 2s
      retries:
        attempts: 2
        perTryTimeout: 800ms
        # Without this, a retry on a 5xx that is actually an
        # overload amplifies the overload. Retry only on
        # connection-level failures.
        retryOn: connect-failure,refused-stream

The retryOn line is the one to get right. A mesh makes retries trivially configurable, which makes retry amplification trivially configurable too, and a mesh-wide default of "retry on 5xx" is a documented way to turn a degradation into an outage.

Adapter: normalising output

The app emits Prometheus metrics on :9090 in its own naming
scheme. The platform wants OpenTelemetry semantic conventions.

Adapter sidecar: scrape :9090, rename, re-export as OTLP.

Or: the app writes unstructured logs to a file. The adapter
parses, adds pod/trace metadata, and forwards as structured
JSON.

This is the least glamorous of the three and often the most
immediately useful, because it lets a vendor binary or a
legacy service participate in the platform's observability
without being modified.

The costs, stated with numbers

LATENCY. Two extra hops per call, one at each end.
  Istio's published benchmarks put the added latency in the
  region of a fraction of a millisecond to ~1ms at p90 per
  proxy for a simple request, so ~1-2ms round trip with both
  sides meshed.
  Irrelevant for a 200ms request. Significant for a 3ms
  in-memory cache lookup, where you have added 50% to the
  latency of the fastest calls in the system.

MEMORY. A sidecar's memory scales with how much of the cluster
  it knows about, because the proxy holds configuration for
  every destination it might reach.
  Default (whole-cluster config): tens of megabytes per
  sidecar, and it grows with service count.
  Scoped (Istio's Sidecar resource, limiting visible
  namespaces): materially smaller.
  Multiply by pod count. At 2,000 pods, 60MB each is 120GB of
  cluster memory doing no application work. This is usually
  the number that starts the conversation.

CPU. Proxying costs CPU proportional to request rate,
  concentrated on the highest-throughput services.

COMPLEXITY. Every failure now has a "is it the app or the
  sidecar" step. Debugging requires reading Envoy config dumps,
  which is a skill nobody on the team has on day one.

The lifecycle problem, and how Kubernetes fixed it

The classic sidecar failure is ordering, in both directions.

STARTUP RACE
  the app container starts before the proxy is ready, makes an
  outbound call, and it fails. Job pods and short-lived
  workloads hit this constantly.

SHUTDOWN RACE
  the proxy exits before the app finishes in-flight work, so
  the last requests fail. Or worse, for a Job: the app
  completes, the proxy keeps running, and the pod never
  terminates.

The old workarounds were all bad: a startup script that polls
the proxy's health endpoint before exec'ing the app; a preStop
hook that curls the proxy's quit endpoint; holdApplicationUntil
ProxyStarts flags.

Kubernetes 1.28+ (stable in 1.29) fixed it properly: a sidecar
is an INIT CONTAINER with restartPolicy: Always.
  - it starts before app containers and must be ready first
  - it keeps running alongside them
  - it terminates AFTER the app containers
  - and pod completion is not blocked by it, which fixes Jobs

spec:
  initContainers:
    - name: proxy
      image: envoy:v1.31
      restartPolicy: Always      # <- makes it a native sidecar
      startupProbe:
        httpGet: { path: /ready, port: 15021 }
  containers:
    - name: app
      image: checkout:2026.03

Knowing that native sidecars are init containers with restartPolicy: Always is a concrete recency signal, because it replaced roughly five years of accumulated workarounds.

The alternatives, honestly

LIBRARY (Ribbon/Hystrix historically, Resilience4j, Polly)
  + no extra hop, no extra memory, in-process context
  - per-language, and upgrade coordination is the killer
  Right when: one or two languages, and you control all the
  services.

PROXYLESS MESH (gRPC's xDS support)
  The gRPC client speaks xDS to the control plane directly and
  implements the policy in-process. Mesh policy, no sidecar.
  + no extra hop, no per-pod memory
  - gRPC only, and the client library must support it
  Right when: an internal gRPC estate wanting mesh policy
  without the proxy tax.

eBPF (Cilium's mesh, Ambient-style architectures)
  Move L4 (mTLS, identity, basic routing) into the kernel and
  into a per-node component, keeping a per-workload proxy only
  for L7 features that need it.
  + one proxy per node rather than per pod, so the memory
    multiplier disappears
  - L7 features still need a proxy somewhere, and the
    operational model is newer
  Right when: the per-pod memory cost is the binding
  constraint, which at a few thousand pods it usually is.

DO NOTHING
  Genuinely correct for a small estate. mTLS between 6
  services can be a cert-manager configuration and a TLS
  client, and it is not worth a control plane.

A worked example: a mesh adopted for one reason and kept for another

A platform team, 140 services, four languages (Java, Go, Python, Node), around 2,600 pods. The driver was a compliance requirement: encryption in transit for all internal traffic, with an audit deadline.

The alternatives, costed:

LIBRARY TLS in each service:
  4 language implementations, 140 services to change and
  deploy, certificate rotation logic in each, and no way to
  prove coverage to an auditor other than reading 140
  codebases.
  Estimate: ~2 quarters across 9 teams.

MESH (Istio):
  one sidecar injection label per namespace, mTLS on by
  default, and a single control-plane assertion of coverage.
  Estimate: ~6 weeks for the platform team, plus a rollout.

Chosen: mesh. The deciding factor was the AUDIT ARTIFACT: a
single query proving 100% of traffic was mTLS, rather than an
argument assembled from 140 repositories.

"The deciding factor was the audit artifact" is worth recording, because it is a non-technical property that made the technical decision, and it is the kind of reason that does not appear in architecture comparisons.

The rollout, and what it cost:

Phase 1: PERMISSIVE mode namespace by namespace over 5 weeks.
  Sidecars injected, mTLS accepted but not required, so
  anything unmeshed still worked.
Phase 2: STRICT per namespace, once the mesh dashboard showed
  zero plaintext for that namespace for a week.

Measured after full rollout:
  p50 added latency per hop        ~0.4ms
  p99 added latency per hop        ~1.9ms
  memory: 2,600 sidecars x 68MB    ~177 GB
  control plane                    3 pods, 4 vCPU each
  cluster cost increase            ~14%

The 177GB is the number that caused the second phase of work, and it was not in the original estimate.

Cause: every sidecar received configuration for every service
in the cluster, because no Sidecar resource scoped its
visibility.

Fix: a default Sidecar resource per namespace listing only the
namespaces that namespace actually calls, generated from the
mesh's own observed traffic graph rather than from a guess.

  memory per sidecar   68MB -> 21MB
  total                177GB -> 55GB
  cluster cost increase 14% -> 4%

Generating the visibility scope from the observed traffic graph, rather than asking teams to declare their dependencies, is the detail that made it feasible, because 140 teams do not accurately declare their dependencies and the mesh already knew.

The latency problem, which was real and narrow:

Three services were materially affected:
  a session cache with a 2.1ms p50 -> 3.9ms
  a feature-flag service, 1.4ms -> 3.1ms
  an internal auth check, 3.0ms -> 4.8ms

All three were in the critical path of every request, so the
per-request cost was ~5ms across the three.

Options considered:
  - exclude them from the mesh (loses mTLS, fails the audit)
  - move them in-process as a library (a real answer for the
    flag service, which became an in-process client with a
    background refresh, and the mesh hop disappeared entirely)
  - accept it

Outcome: the flag service became in-process, the other two were
accepted. Net critical-path cost fell from ~5ms to ~3.4ms.

Moving the flag service in-process was better than either meshing or excluding it, which is the general lesson: the mesh made a latency cost visible that had always been an unnecessary network hop.

The reason the mesh was kept, which was not the reason it was adopted:

18 months later, the compliance requirement was one of the
least valuable things the mesh was doing.

What it was actually used for:
  - canary traffic splitting for 40 services, by config rather
    than by deploy (see the deployment strategy matrix)
  - a global retry and timeout policy, changed centrally when
    the retry-amplification incident happened, in one config
    push rather than 140 deploys
  - per-service authorisation policy (which services may call
    which), which was previously an unwritten convention
  - automatic mTLS identity for the workload-identity work,
    replacing long-lived service tokens
  - a traffic graph, which nobody had had before and which
    made the Sidecar-scoping fix possible

Any one of these individually would not have justified the
14% cost. Together, with the compliance driver paying for the
adoption, they did.

Two things went wrong:

1. THE DEFAULT RETRY POLICY.
   The initial mesh-wide policy retried on 5xx, twice.
   During a downstream degradation, this tripled load on an
   already-failing service and turned a 4-minute blip into a
   22-minute outage.
   Fixed: retryOn limited to connect-failure and
   refused-stream, i.e. failures where the request provably
   did not reach the application. Retries on application
   errors are now per-route and require a stated reason.

2. THE STARTUP RACE, on Jobs.
   Nightly batch Jobs failed intermittently for two months:
   the app container started before Envoy was ready and its
   first outbound call failed, and separately, completed Jobs
   never terminated because Envoy kept running.
   Worked around with a wrapper script and a preStop curl,
   then fixed properly by moving to native sidecars
   (initContainers with restartPolicy: Always) on the 1.29
   upgrade, which deleted both workarounds.

A mesh-wide "retry on 5xx" default is a documented way to build a retry storm generator, and it is the single most dangerous default in the pattern.

Production evidence

Istio with Envoy and Linkerd with linkerd2-proxy are the two production service meshes with the widest deployment; Istio publishes latency and resource benchmarks, and both document the sidecar injection, mTLS-by-default and control-plane configuration model described above.

Kubernetes native sidecars (KEP-753), init containers with restartPolicy: Always, reached beta in 1.28 and stable in 1.29, and the Kubernetes documentation states the ordering guarantees explicitly: start before app containers, terminate after them, and do not block pod completion. They exist because the workarounds accumulated for years.

Istio's Sidecar resource is the documented mechanism for limiting a proxy's configuration scope, and Istio's own performance guidance identifies whole-cluster configuration distribution as the primary driver of per-proxy memory.

gRPC's xDS support implements proxyless mesh: the gRPC client speaks the same configuration protocol as Envoy and applies policy in-process, which Google has documented as the model for internal gRPC estates that want mesh policy without the proxy hop.

Cilium's eBPF-based mesh and Istio's ambient mode both move L4 concerns out of per-pod proxies into per-node components, explicitly to remove the per-pod memory multiplier, keeping a proxy only where L7 processing is required.

Netflix's library-based stack (Ribbon, Hystrix, Eureka) is the reference implementation of the alternative, and the fact that it worked well in a predominantly JVM estate and was progressively replaced as the estate diversified is the clearest available evidence for where the trade-off point sits.

The sidecar, ambassador and adapter naming comes from Brendan Burns and David Oppenheimer's "Design Patterns for Container-based Distributed Systems" (HotCloud 2016), which is where the three were distinguished as separate patterns over a shared mechanism.

The debate

Is a service mesh worth it? For a polyglot estate needing mTLS, per-service authorisation and policy that changes faster than code ships, yes. For a homogeneous estate of a dozen services, no: the control plane, the per-pod memory and the debugging complexity are real, and cert-manager plus a TLS client covers the compliance case. The honest threshold is language count and policy change rate, not service count.

Sidecar or library? Library when you control every service and they share a language, because there is no extra hop and no per-pod memory. Sidecar when a CVE fix or a policy change would otherwise mean coordinating dozens of deploys, which is the cost that grows superlinearly with estate size and is consistently underestimated.

Is the latency acceptable? For a 200ms request, entirely. For a 2ms internal cache lookup you have added 50 to 100 percent, and the right response is usually neither to accept it nor to exclude the service from the mesh, but to ask whether that call should be a network hop at all. In the worked example the feature-flag service became an in-process client and the hop disappeared.

Should the mesh handle retries? It should be able to, and the default must be conservative. A mesh-wide "retry on 5xx" turns every downstream degradation into an amplified one, and because the mesh makes retries a one-line config it makes the mistake easy to deploy fleet-wide. Retry only on failures where the request provably did not reach the application, and require a stated reason for anything more.

Is per-node better than per-pod? For the memory multiplier, clearly: one proxy per node rather than per pod removes the dominant cost at a few thousand pods. The trade is that L7 features still need a proxy somewhere, so ambient and eBPF architectures are a two-tier model rather than a removal, and the operational tooling is younger.

Does a mesh replace an API gateway? No. The gateway owns north-south concerns (public routing, external auth, rate limiting, WAF) and the mesh owns east-west, and they overlap enough that teams adopt a mesh and then discover the ingress still needs solving. Many meshes ship a gateway component, which is a deployment of the same proxy in a different role rather than a different thing.

Follow-up Q&A

"What distinguishes a sidecar, an ambassador and an adapter?"

The mechanism is the same, a co-located process sharing the pod's network namespace and lifecycle, and the names describe the role. An ambassador proxies outbound connections, so the application dials localhost and the proxy handles discovery, mTLS, retries, timeouts and circuit breaking. An adapter normalises output, translating the application's own metrics or log format into the platform's standard one. Sidecar is the general term. A service mesh is ambassadors everywhere plus a control plane that configures them.

"When does a mesh beat libraries?"

When a policy change or a CVE fix would otherwise mean coordinating deploys across many services in several languages. With four languages and 140 services, a TLS library approach is four implementations, 140 deploys, and no way to prove coverage to an auditor except by reading 140 repositories; the mesh is one config assertion. The threshold is language count and policy change rate rather than service count, and Netflix ran the library model successfully for years in a predominantly JVM estate, which is the clearest evidence for where the line sits.

"What is the real cost of a mesh?"

Latency, memory and debugging. Roughly a fraction of a millisecond to a couple of milliseconds per proxy hop, which is nothing for a 200ms request and is 50 to 100 percent for a 2ms internal call. Memory scales with how much of the cluster each proxy knows about: in one estate 2,600 sidecars at 68MB each was 177GB and a 14 percent cluster cost increase, which fell to 55GB and 4 percent after scoping each proxy's configuration to the namespaces it actually calls. And every incident acquires an "is it the app or the sidecar" step that requires reading proxy configuration dumps.

"How do you reduce per-sidecar memory?"

Scope each proxy's configuration to the destinations it actually needs, using Istio's Sidecar resource or the equivalent, because the default is to distribute configuration for every service in the cluster. The detail that makes it feasible at scale is generating that scope from the mesh's own observed traffic graph rather than asking teams to declare their dependencies, since teams declare them inaccurately and the mesh already knows. In one case that took per-proxy memory from 68MB to 21MB.

"What is the most dangerous mesh default?"

Retry on 5xx. A mesh makes retries a one-line configuration, which makes retry amplification a one-line configuration too, and a fleet-wide default of two retries on any 5xx triples load on an already-failing downstream. In one incident it turned a four-minute blip into a twenty-two-minute outage. Retry only on failures where the request provably did not reach the application, connection failure and refused stream, and require a stated reason for anything broader.

"What was the sidecar lifecycle problem and how is it fixed?"

Ordering, in both directions. The app container could start before the proxy was ready and its first outbound call would fail, and the proxy could exit before the app finished in-flight work, or keep running after a Job completed so the pod never terminated. Years of workarounds followed: startup scripts polling the proxy's health endpoint, preStop hooks curling a quit endpoint, vendor-specific flags. Kubernetes fixed it with native sidecars, an init container with restartPolicy: Always, which starts first, runs alongside, terminates last, and does not block pod completion.

Common misconceptions

"A sidecar is any second container in the pod." The defining property is the shared network namespace, which is what allows localhost communication and transparent interception.

"A mesh replaces the API gateway." The gateway owns north-south traffic and external concerns; the mesh owns east-west. Teams discover this at the ingress.

"Mesh latency is negligible." It is negligible relative to a 200ms request and material relative to a 2ms one, and the right question is often whether that call should be a network hop at all.

"Sidecar memory is a fixed per-pod cost." It scales with how much of the cluster the proxy is configured to know about, which is why scoping visibility can cut it by two thirds.

"Retries are a free win from the mesh." A fleet-wide retry-on-5xx default is a retry storm generator, deployable in one line.

"You need a mesh for mTLS." For a small estate, cert-manager and a TLS client is sufficient and far cheaper. The mesh earns its cost on policy change rate and language diversity.

Interview delivery note

Say this verbatim: "Sidecar, ambassador and adapter are one mechanism and three roles: a co-located process sharing the network namespace, proxying outbound calls, or normalising output. The mesh argument is not about service count, it is about how many languages you have and how fast policy needs to change relative to how fast code ships." It defines the family and states the actual decision criterion.

The senior-versus-staff separator is pricing the memory multiplier and knowing why it exists. A senior engineer notes that sidecars use memory. A staff engineer says that per-proxy memory scales with how much of the cluster each proxy is configured to see, that 2,600 sidecars at 68MB was 177GB and a 14 percent cluster cost increase, and that scoping each proxy's visibility from the mesh's own observed traffic graph, rather than from team-declared dependencies, took it to 21MB and 4 percent. The generated-from-observed-traffic detail is what makes it work at 140 teams.

The second signal is treating the added latency as a question rather than a cost. Saying "the mesh made three internal calls 50 to 100 percent slower, and for the feature-flag service the right answer was neither to accept it nor to exclude it from the mesh but to make it an in-process client with a background refresh, which removed the hop entirely" shows the mesh surfaced a network call that should never have existed.

The Staff & Team Lead Interview Reference