NetworkPolicy and service mesh authorization

What it is

Two layers that both restrict which workloads may talk to which, operating at different points in the stack and answering different questions:

	NetworkPolicy	Mesh authorization
Layer	L3/L4: IP, port, protocol	L7: method, path, header, plus identity
Identity	Pod labels, resolved to IPs	Cryptographic (mTLS certificate / SPIFFE ID)
Enforced by	CNI (iptables, eBPF)	Sidecar or per-node proxy
Trusts	The network topology	The certificate
Answers	"Can this IP reach that IP:port?"	"Can this service call `POST /admin`?"

The distinction that matters: NetworkPolicy identity is an IP address. The policy is written in terms of labels, and the CNI resolves those labels to pod IPs and writes rules about IPs. Anything that can send from an allowed IP is allowed, so a compromised pod in an allowed namespace passes.

Mesh authorization identity is a certificate. The caller proves who it is with a key it holds, so spoofing requires stealing the key rather than occupying the right IP.

What they are confused with: alternatives. They are layers. NetworkPolicy is a coarse default-deny that costs nothing at runtime and applies to everything including traffic the mesh does not see; mesh authorization is fine-grained and identity-based and only covers traffic through the proxy. The common failure is deploying a mesh and skipping NetworkPolicy, which leaves everything the mesh does not intercept (and every path that bypasses the sidecar) unrestricted.

The problem it solves

A default Kubernetes cluster is a flat network. Every pod can reach every other pod, in every namespace, on every port. There is no boundary between the payments service and the marketing site's CMS.

The concrete consequences:

Lateral movement. An RCE in any pod gives an attacker network reach to every service, every database, and every internal API. The container security page's red-team chain depended on this: once inside one pod, the whole cluster was reachable.

No egress control. A compromised pod can exfiltrate to any address on the internet, and nothing in the default configuration observes or prevents it.

Blast radius from misconfiguration. A staging service pointed at a production database by a copy-paste error connects successfully, because nothing says it should not.

The measurable version, from a cluster audit:

340 services, no NetworkPolicy:
  reachable pairs:                340 x 339 = 115,260
  pairs with a legitimate reason:  ~1,100 (from service dependency graphs)
  unnecessary reachability:        99.05%

Ninety-nine percent of the reachable surface has no purpose, and that is the number that motivates the work.

Mechanics

NetworkPolicy: default-deny first

A namespace with no policy allows everything. A namespace with any policy selecting a pod denies everything to that pod except what a policy allows. That is the rule, and it means the first policy you write must be the default-deny, or partial policies give a false sense of coverage.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-all, namespace: payments}
spec:
  podSelector: {}                    # every pod in the namespace
  policyTypes: [Ingress, Egress]     # BOTH; omitting Egress leaves it open

Omitting Egress from policyTypes is the most common mistake, because ingress feels like the security direction and egress is where exfiltration happens.

Then allow what is needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-api-to-ledger, namespace: payments}
spec:
  podSelector: {matchLabels: {app: ledger}}
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector: {matchLabels: {app: payments-api}}
      # NOTE: podSelector alone means SAME namespace only.
    - namespaceSelector: {matchLabels: {name: reconciliation}}
      podSelector: {matchLabels: {app: recon-worker}}
      # This pair is an AND: that pod IN that namespace.
    ports:
    - {protocol: TCP, port: 8080}

The AND/OR distinction in from is a genuine trap:

# TWO SOURCES (OR): any pod in ns=foo, OR any pod labelled app=bar anywhere
#                    the policy's namespace
from:
- namespaceSelector: {matchLabels: {name: foo}}
- podSelector: {matchLabels: {app: bar}}

# ONE SOURCE (AND): pods labelled app=bar IN namespace foo
from:
- namespaceSelector: {matchLabels: {name: foo}}
  podSelector: {matchLabels: {app: bar}}

One - versus two changes the meaning entirely, and the permissive version looks correct in review. This is worth checking explicitly in any NetworkPolicy review.

The DNS exception, which breaks every first attempt

# Without this, default-deny egress breaks EVERYTHING, because nothing
# can resolve a name.
egress:
- to:
  - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
    podSelector: {matchLabels: {k8s-app: kube-dns}}
  ports:
  - {protocol: UDP, port: 53}
  - {protocol: TCP, port: 53}

Every default-deny egress rollout breaks on DNS in the first five minutes. It is the single most predictable failure and it belongs in the default-deny template rather than being rediscovered per namespace.

Egress to external addresses

egress:
- to:
  - ipBlock:
      cidr: 0.0.0.0/0
      except:
      - 10.0.0.0/8          # no internal ranges
      - 172.16.0.0/12
      - 192.168.0.0/16
      - 169.254.169.254/32  # THE CLOUD METADATA ENDPOINT
  ports:
  - {protocol: TCP, port: 443}

Blocking 169.254.169.254 is the highest-value single rule in an egress policy. The cloud instance metadata service returns IAM credentials for the node's role, and SSRF or an RCE that can reach it escalates from "compromised pod" to "the node's cloud permissions." IMDSv2 (requiring a PUT to obtain a token) mitigates the SSRF case and does not stop an RCE, so the network-level block still matters.

NetworkPolicy cannot express DNS names. ipBlock is CIDRs only, so "allow egress to api.stripe.com" is not expressible, because the address changes. That gap is why FQDN policies exist as CNI extensions (Cilium's toFQDNs, Calico's GlobalNetworkPolicy with domains), and using them means writing a CNI-specific resource rather than a portable one.

Mesh authorization: identity instead of topology

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: {name: ledger-authz, namespace: payments}
spec:
  selector: {matchLabels: {app: ledger}}
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/payments/sa/payments-api"]
        #            ^^^ the SPIFFE identity from the mTLS certificate
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/v1/entries", "/v1/entries/*"]
  - from:
    - source:
        principals: ["cluster.local/ns/reconciliation/sa/recon-worker"]
    to:
    - operation:
        methods: ["GET"]              # read only
        paths: ["/v1/entries/*"]

Two things NetworkPolicy cannot do are visible here. The reconciliation worker is allowed to read and not write, which is a method-level distinction; and the identity is the service account's certificate rather than a pod IP, so a different pod that happens to occupy an allowed IP does not pass.

# And the prerequisite that makes the identity meaningful:
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: {name: default, namespace: payments}
spec:
  mtls: {mode: STRICT}         # PERMISSIVE accepts plaintext: not a boundary

PERMISSIVE mode accepts both mTLS and plaintext, which is the correct migration setting and is not a security boundary. A policy in PERMISSIVE mode with principals rules denies plaintext requests (they have no principal), and traffic that should be authenticated can still arrive unauthenticated on paths the policy does not cover. Ending a mesh rollout in PERMISSIVE is the common half-finished state.

Why you need both

Traffic the mesh does NOT see:
  - pods without a sidecar (jobs, DaemonSets, anything excluded)
  - traffic to non-mesh services (a managed database, an external API)
  - traffic on ports excluded from interception
  - anything that bypasses the sidecar's iptables redirect

Traffic NetworkPolicy cannot restrict:
  - method, path, or header
  - identity that survives IP reuse
  - anything requiring cryptographic proof of the caller

A sidecar can be bypassed if the workload can manipulate its own netns. Istio's interception is iptables rules in the pod's network namespace, and a container with NET_ADMIN can remove them. NetworkPolicy is enforced outside the pod, by the CNI on the node, so it holds even when the sidecar does not. That is the strongest argument for the belt-and-braces position.

The division of labour that works:

NetworkPolicy:   coarse default-deny. Namespace isolation, egress control,
                 metadata endpoint blocking, database access. Cheap, always on,
                 covers everything.
Mesh authz:      fine-grained. Method and path restrictions, identity-based
                 rules, cross-cluster identity. Where the granularity is
                 worth the sidecar.

A worked example: 99 percent unnecessary reachability

A financial services platform, 340 services, 12 namespaces, no NetworkPolicy at all. The driver was a compliance requirement for network segmentation between card-data and non-card-data workloads.

Starting point:

NetworkPolicies:              0
reachable service pairs:      115,260
mesh:                         Istio, PERMISSIVE mTLS, no AuthorizationPolicies
egress:                       unrestricted, including 169.254.169.254

Step 1: measure the actual dependency graph before writing any policy.

Writing policies from architecture diagrams fails, because the diagrams are wrong. They used flow logs:

# Cilium's flow export (Hubble), or a mesh's telemetry, or VPC flow logs.
hubble observe --output json --last 0 | \
  jq -r '[.source.namespace, .source.pod_name, .destination.namespace,
          .destination.pod_name, .destination.port] | @csv' \
  > flows.csv

observation window:           14 days
distinct service pairs seen:  1,143
services with zero inbound:   38      <- dead services nobody had removed
services with zero outbound:  12
unexpected pairs:             41      <- the interesting finding

Forty-one pairs nobody expected, including:

marketing-cms      -> payments-db     (a debugging connection from 2023,
                                       still in a config file)
analytics-worker   -> ledger:8080     (undocumented, but load-bearing)
batch-reconcile    -> 169.254.169.254 (fetching node credentials to reach S3,
                                       instead of using IRSA)

The marketing-cms to payments-db connection is what a compliance auditor is looking for, and it had existed for over a year with nothing reporting it.

Step 2: generate policies from observed flows, then review.

# Generate a candidate policy per service from observed flows.
# CRITICAL: this is a starting point for review, not the final artifact,
# because it encodes existing mistakes as permissions.
for svc, flows in group_by_destination(observed):
    policy = build_ingress_policy(svc, sources=flows.sources, ports=flows.ports)
    emit(policy, needs_review=True)

generated policies:      340
reviewed and accepted:   299
rejected (the flow was a bug, not a dependency):   41

Rejecting 41 generated rules is the point of the review step. A generator run without review would have codified the marketing-to-payments-database connection as an approved permission, which is worse than having no policy, because now it looks deliberate.

Step 3: roll out in audit mode.

# Cilium supports a policy audit mode: log what WOULD be denied, deny nothing.
cilium config set policy-audit-mode true

week 1-2 (audit):
  would-be-denied flows:      2,841
  legitimate, missed by the
    14-day observation:       94     <- monthly jobs, quarterly reports,
                                        DR failover paths
  genuinely unwanted:         2,747

Ninety-four legitimate flows were missed by a fourteen-day window, because monthly and quarterly jobs had not run. Audit mode is not optional, and the window has to exceed your longest business cycle or you will break something at month end.

Step 4: enforce, namespace by namespace.

order: least critical first
  marketing (week 3), analytics (week 4), internal-tools (week 5),
  ... payments (week 9), card-data (week 10)

incidents during rollout:  2
  - a health-check probe from a monitoring namespace, not in the flow data
    because it used hostNetwork
  - a Helm hook job with a different service account than the deployment

Both incidents were things that do not appear in pod-to-pod flow data, which is the limitation of the flow-based generation approach: hostNetwork pods and short-lived jobs are easy to miss.

Step 5: egress and the metadata endpoint.

# Applied cluster-wide via a Cilium ClusterwideNetworkPolicy.
egress:
- toCIDRSet:
  - cidr: 0.0.0.0/0
    except: ["169.254.169.254/32", "10.0.0.0/8"]

pods that broke:  1  (batch-reconcile, which was using node credentials)
fix:              migrated to IRSA (a per-pod IAM role)

That one break was a finding rather than a regression: a workload using the node's IAM role had the union of every workload's permissions on that node, and moving it to a per-pod role reduced its access substantially.

Step 6: mesh authorization, for the card-data namespace only.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: {name: default, namespace: card-data}
spec:
  mtls: {mode: STRICT}      # was PERMISSIVE for 18 months

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: {name: card-vault-authz, namespace: card-data}
spec:
  selector: {matchLabels: {app: card-vault}}
  action: ALLOW
  rules:
  - from: [{source: {principals: ["cluster.local/ns/payments/sa/payments-api"]}}]
    to:   [{operation: {methods: ["POST"], paths: ["/v1/tokenize"]}}]
  - from: [{source: {principals: ["cluster.local/ns/payments/sa/settlement"]}}]
    to:   [{operation: {methods: ["POST"], paths: ["/v1/detokenize"]}}]

The method and path granularity is what NetworkPolicy could not express: payments-api may tokenise and may not detokenise, and both use the same port.

Moving from PERMISSIVE to STRICT broke three things:

- a legacy service without a sidecar, calling into card-data
  -> given a sidecar
- a Prometheus scrape on a port not in the mesh
  -> excluded via an exclusion annotation and covered by NetworkPolicy instead
- a health check from the cloud load balancer (no client certificate)
  -> a PERMISSIVE port exception for the health path only

Final:

                                  before      after
NetworkPolicies                   0           341
reachable service pairs           115,260     1,102       (-99.0%)
egress to metadata endpoint       allowed     blocked
undocumented dependencies         41          0 (removed or documented)
mTLS mode (card-data)             PERMISSIVE  STRICT
L7 authz policies                 0           14 (card-data only)
compliance finding                open        closed
incidents caused by the rollout   2           (both during audit-to-enforce)
p99 latency change                baseline    +0.4 ms (eBPF policy enforcement)

The 41 undocumented dependencies were worth more than the segmentation. The compliance requirement drove the project and the deliverable was a map of what actually talks to what, which nobody had, and which found a marketing CMS connected to a payments database.

The transferable practice: observe before you write. Fourteen days of flow data plus two weeks of audit mode found 94 legitimate flows the architecture diagrams did not contain and 41 illegitimate ones nobody knew about. Writing policies from the intended architecture produces an outage; writing them from observed flows without review codifies existing mistakes as permissions. You need both steps.

Production evidence

NetworkPolicy is a Kubernetes API with no built-in implementation, so it does nothing unless the CNI enforces it. Flannel (in its default configuration) does not, which is a recurring surprise: policies apply cleanly, appear in kubectl get netpol, and have no effect. Calico, Cilium, Weave and the cloud CNIs do enforce it.

Cilium's policy audit mode and Hubble flow observability exist because rolling out default-deny without observation breaks things, and the two-phase approach (observe, then audit, then enforce) is documented practice rather than an invention.

Istio's PERMISSIVE mTLS is explicitly a migration mode in their documentation, with STRICT as the target, and the number of clusters that stop at PERMISSIVE is a known pattern in the community.

SPIFFE/SPIRE standardises the workload identity model that mesh authorization depends on, and its adoption across Istio, Linkerd and several non-mesh systems is why "identity is a certificate, not an IP" is a portable idea rather than an Istio one.

The cloud metadata endpoint as an escalation path is well documented, and IMDSv2 exists specifically to mitigate the SSRF variant by requiring a token obtained via PUT. GKE Workload Identity, EKS IRSA and AKS Workload Identity all exist to remove the need for pods to use node credentials at all, which is the structural fix.

Kubernetes 1.31 added AdminNetworkPolicy (a cluster-scoped, priority-ordered policy that cannot be overridden by namespace-level policies), addressing the long-standing gap that NetworkPolicy is namespace-scoped and additive, so a namespace owner can always allow more than the platform team intended.

The debate

Do you need both NetworkPolicy and a mesh? If you have a mesh, yes, still write NetworkPolicy. The mesh does not see traffic from pods without sidecars, traffic to non-mesh endpoints, or ports excluded from interception, and a sidecar's iptables interception can be removed by a container with NET_ADMIN, while NetworkPolicy is enforced by the CNI outside the pod. NetworkPolicy is the cheap floor; the mesh is the fine-grained layer above it.

Is a mesh worth it for authorization alone? Usually not. A sidecar per pod costs memory, latency (typically 1 to 3 ms per hop), and a substantial operational surface, and if the only requirement is "service A may call service B" then NetworkPolicy expresses that for free. The mesh earns its place when you need method and path granularity, cryptographic identity that survives IP reuse, or cross-cluster identity, and it is usually adopted for mTLS, traffic management and observability with authorization as one of several benefits.

Ambient mode changes this calculation. Istio's ambient mode replaces per-pod sidecars with a per-node ztunnel for L4 and an optional waypoint proxy for L7, so you can have mTLS and L4 authorization without a sidecar per pod. That makes the "mesh for authorization" argument much stronger than it was, because the cost is per node rather than per pod.

Default-deny or default-allow? Default-deny, and the only question is the rollout path. Default-allow with specific denies is unmaintainable, because you must enumerate every bad thing rather than every good one, and new services are open by default. The rollout must be observe, audit, enforce, with an observation window longer than your longest business cycle, or you will break the month-end job.

Should you generate policies from observed traffic? As a starting point, and never as the final artifact. Generation encodes existing mistakes as approved permissions, and in the worked example 41 of 340 generated policies described connections that should not have existed. A generated policy that is not reviewed is worse than no policy, because it makes a mistake look deliberate.

What about FQDN-based egress? NetworkPolicy cannot express it, because ipBlock is CIDRs and external service addresses change. CNI extensions (Cilium toFQDNs, Calico domain policies) implement it by intercepting DNS and allowing the returned addresses, which works and is CNI-specific. The portability cost is real and for most teams it is worth paying, because "allow egress to api.stripe.com" is the policy people actually want to write.

Follow-up Q&A

"What is the difference between NetworkPolicy and mesh authorization?"

The layer and the identity. NetworkPolicy is L3/L4 (IP, port, protocol) enforced by the CNI, and its identity is ultimately an IP address: labels are resolved to pod IPs and rules are written about IPs, so anything sending from an allowed IP passes. Mesh authorization is L7 (method, path, header) enforced by a proxy, and its identity is a certificate, so the caller proves who it is cryptographically. They are layers rather than alternatives.

"If I have a mesh, do I need NetworkPolicy?"

Yes. The mesh only sees traffic through its proxies, so pods without sidecars, traffic to non-mesh endpoints like a managed database, and excluded ports are all unrestricted. More importantly, a sidecar's interception is iptables rules inside the pod's network namespace, so a container with NET_ADMIN can remove them, while NetworkPolicy is enforced by the CNI outside the pod and holds regardless. NetworkPolicy is the floor.

"How do you roll out default-deny without an outage?"

Three phases. Observe real traffic for long enough to cover your business cycles, using flow logs rather than architecture diagrams, because the diagrams are wrong. Then run in audit mode, logging what would be denied while denying nothing: in one case that surfaced 94 legitimate flows the fourteen-day observation had missed, because monthly and quarterly jobs had not run. Then enforce namespace by namespace, least critical first. And note that hostNetwork pods and short-lived jobs do not appear in pod-to-pod flow data, which is where the residual breakage comes from.

"What is the single highest-value egress rule?"

Blocking 169.254.169.254, the cloud metadata endpoint. It returns IAM credentials for the node's role, so SSRF or an RCE that reaches it escalates from a compromised pod to the node's cloud permissions, which is the union of every workload on that node. IMDSv2 mitigates the SSRF variant by requiring a token via PUT and does not stop an RCE. The structural fix is per-pod cloud identity (IRSA, Workload Identity) so pods never need node credentials.

"What is the AND/OR trap in NetworkPolicy?"

Two list entries under from are an OR; one entry with both a namespaceSelector and a podSelector is an AND. So a single - versus two changes "pods labelled X in namespace Y" into "anything in namespace Y, or anything labelled X in this namespace." The permissive version looks correct in review, which is what makes it worth checking explicitly.

"Should you generate policies from traffic?"

As a first draft only. Generation from observed flows encodes existing mistakes as approved permissions: in one case 41 of 340 generated policies described connections that should not have existed, including a marketing CMS talking to a payments database. Reviewing and rejecting those was the point of the exercise, and an unreviewed generated policy is worse than none because it makes the mistake look deliberate.

Common misconceptions

"NetworkPolicy is enabled by default." It is an API with no built-in implementation, so it does nothing unless the CNI enforces it. Flannel in its default configuration does not, and policies apply cleanly and have no effect.

"A policy on one pod protects the namespace." Policies are additive and pod-scoped: a pod selected by no policy allows everything. The default-deny must come first or partial policies give false coverage.

"Ingress rules are the security-relevant ones." Egress is where exfiltration and metadata access happen, and omitting Egress from policyTypes is the most common gap.

"A mesh replaces NetworkPolicy." It covers only traffic through its proxies, and its interception can be removed from inside the pod by a container with NET_ADMIN. NetworkPolicy is enforced outside the pod.

"PERMISSIVE mTLS means mTLS is on." It accepts both mTLS and plaintext and is explicitly a migration mode. Ending a rollout there is a common half-finished state that provides encryption for compliant clients and no boundary.

Interview delivery note

Say this verbatim: "NetworkPolicy's identity is ultimately an IP address, because labels get resolved to pod IPs, so anything sending from an allowed IP passes. Mesh authorization's identity is a certificate. They are layers, not alternatives, and I would still write NetworkPolicy with a mesh, because a sidecar's interception is iptables rules inside the pod and a container with NET_ADMIN can remove them, while NetworkPolicy is enforced by the CNI outside the pod." The distinction and the specific reason both are needed.

The senior-versus-staff separator is the rollout method. A senior engineer writes correct policies. A staff engineer knows that policies written from architecture diagrams cause outages because the diagrams are wrong, that policies generated from observed traffic codify existing mistakes as permissions (41 of 340 in one case, including a marketing CMS reaching a payments database), and that the observation window must exceed the longest business cycle or the month-end job breaks. Observe, audit, review, enforce, in that order.

The second signal is the metadata endpoint. Blocking 169.254.169.254 in egress is one line and it converts an RCE from "compromised pod" into "compromised pod" rather than "the node's cloud permissions," and pairing it with per-pod cloud identity is the structural version.

The Staff & Team Lead Interview Reference