By Riya Madaan
Kubernetes NetworkPolicy debugging: when pods can't talk
Kubernetes NetworkPolicy not working - or working too well? A test method for silent packet drops: default-deny surprises, selector typos, DNS egress.
NetworkPolicy failures are the quietest failures in Kubernetes. A pod that can't reach its database because of a policy doesn't get an event, a status, or a log line - anywhere. The connection just times out. No component says "denied," because the CNI plugin implementing the policy drops packets silently by design. So when someone says "Kubernetes NetworkPolicy not working," they mean one of two opposite things: the policy blocks traffic it should allow, or it allows traffic it should block. Both debug the same way.
First, the model. Everything confusing about NetworkPolicy comes from four rules:
- Policies select pods, not traffic. A policy applies to the pods matching its
podSelector, in its own namespace only. - Empty selector selects everything.
podSelector: {}means every pod in the namespace, not no pods. - Isolation is per-direction and switches on at the first policy. A pod with no
ingress policies accepts everything. The moment any policy with
policyTypes: [Ingress]selects it, the pod accepts only what some policy explicitly allows. Same for egress, independently. - Policies are additive allowlists. There is no deny rule and no ordering. Union of all allows, applied to isolated pods.
Rule 3 is the incident-maker: someone applies a default-deny to the namespace, and every pod that previously had no policy flips from allow-all to deny-all at once.
| Symptom | Likely cause | First command |
|---|---|---|
| Policies never seem to do anything | CNI doesn't enforce NetworkPolicy | kubectl get pods -n kube-system | grep -Ei 'calico|cilium|flannel' |
| Everything in a namespace broke at once | New default-deny isolated every pod | kubectl get networkpolicy -n <namespace> |
nc to pod IP hangs |
Packet drop by policy (or dead backend) | nc -zv -w 3 <pod-ip> <port> from a labeled debug pod |
| Pod IP works, names don't resolve | Egress policy missing UDP/TCP 53 to kube-dns | nslookup <service>.<namespace>.svc.cluster.local |
| Cross-namespace allow has no effect | namespaceSelector matching a label the namespace lacks |
kubectl get ns --show-labels |
| Traffic gets through that should be blocked | Pod not selected, or a broader policy | kubectl get pod <pod> --show-labels |
Step 0: does your CNI even enforce NetworkPolicy?
The API server accepts NetworkPolicy objects on every cluster. Enforcement is the CNI plugin's job - Calico, Cilium, and friends enforce; bare Flannel does not. A policy on a non-enforcing CNI is a YAML file with good intentions: no error, no effect. If policies have never visibly done anything on this cluster, check what's running before debugging selectors:
kubectl get pods -n kube-system | grep -Ei 'calico|cilium|flannel|weave|antrea'
A repeatable test method (stop guessing from app logs)
App-level errors add noise - retries, pools, timeouts with their own names. Test the network layer directly with a throwaway pod, labeled to impersonate the real client:
kubectl run nettest -n <client-namespace> --rm -it \
--image=nicolaka/netshoot --labels="app=frontend" -- bash
# Inside:
nc -zv -w 3 backend.backend-ns.svc.cluster.local 8080
nc -zv -w 3 <backend-pod-ip> 8080
nslookup backend.backend-ns.svc.cluster.local
The labels flag matters - policies match on labels, so an unlabeled debug pod tests a different policy path than your real workload. Read the three results together:
- Pod IP times out → packet-level block (or the backend is dead). NetworkPolicy territory.
- Pod IP works, service name fails to resolve → DNS, not policy - unless it's the DNS egress trap below. See Debugging Kubernetes DNS.
- Everything resolves and connects, app still fails → not the network. Check Service not reachable for the selector/endpoint/targetPort layer, then the app itself.
A timeout (nc hanging) smells like a drop - NetworkPolicy or routing. An immediate
connection refused means packets arrive and nothing listens - that's not a policy.
Now read the policies like the CNI does
kubectl get networkpolicy -A
kubectl describe networkpolicy <name> -n <ns>
For the target pod, answer in order: which policies select it (podSelector vs the pod's
actual labels - kubectl get pod <pod> --show-labels)? Is it therefore isolated for the
direction that's failing? Does any policy allow this specific peer, port, and protocol?
The recurring selector mistakes:
namespaceSelectormatches labels, not names.name: backend-nsmatches only if the namespace actually carries that label. Since v1.21 every namespace getskubernetes.io/metadata.name: <name>automatically - use that one.- The AND/OR trap.
- namespaceSelector: {...}and- podSelector: {...}as two list items means OR (either matches). One list item with both selectors means AND. That two-character indentation difference is a real production incident generator. - Forgetting the return path doesn't need a rule - connection tracking handles replies. If A can open a connection to B, responses flow without an egress rule on B.
Why can pods reach IPs but not service names?
Because the egress allowlist forgot DNS itself. Every strict egress policy eventually
reproduces this incident: pods can connect to IPs but not to names. Symptom: nc to a pod IP works,
nslookup times out. Fix - allow UDP and TCP 53 to kube-dns:
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
A NetworkPolicy that blocks egress DNS is one of the practice labs, and the clock makes the "check DNS egress first" habit stick faster than reading this will.
Why is traffic getting through that the policy should block?
Almost always one of these: the pod
isn't isolated (your default-deny's podSelector doesn't select it), the traffic enters via a
path policies don't see the way you expect (NodePort with externalTrafficPolicy: Cluster
SNATs traffic to a node IP, which your ipBlock may accidentally allow), or a second, broader
policy you forgot exists - remember, additive, no ordering. Audit with the same netshoot
method: try to make the connection that should fail, from a pod labeled like the untrusted
peer. Deliberately breaking your own policies like this is one of the highest-yield drills in
learning Kubernetes by breaking it.
If you're on Cilium, hubble observe --verdict DROPPED turns the silent drops into a log,
the single biggest quality-of-life upgrade in this entire problem space. Calico can do
similar with flow logs. Turn one of these on before the incident, not during.
FAQ
Why is there no "deny" log anywhere? Vanilla NetworkPolicy has no logging in the spec; drops happen in the CNI dataplane (iptables/eBPF) with logging off by default. Cilium Hubble or Calico flow logs add observability; without them, your evidence is timeouts plus policy-reading.
Do NetworkPolicies apply to traffic from the host or kubelet? Health-check probes come from the kubelet on the node and are generally not blocked by NetworkPolicy (CNIs allow them) - so a pod can be Ready yet unreachable from every other pod. Don't use probe success as evidence that the network path works.
Can I write a cluster-wide policy? Not with the core NetworkPolicy API - it's namespaced. CNI extensions (CiliumClusterwideNetworkPolicy, Calico GlobalNetworkPolicy) and the newer AdminNetworkPolicy API cover the cluster-wide case.
Why did applying one policy break unrelated traffic to the same pods? Because the first policy that selects a pod isolates it for the listed policyTypes. Traffic that used to flow by default now needs an explicit allow. Roll out default-deny namespace-by-namespace, with the allow rules landing in the same change.