skip to content
‹ All posts

By Riya Madaan · Updated

Debugging Kubernetes DNS: from nslookup to CoreDNS

Kubernetes DNS troubleshooting from symptom to CoreDNS: reproduce with a debug pod, decode the ndots trap, read CoreDNS logs, check upstream resolvers.

#kubernetes

A large fraction of "the app can't reach the database" pages are DNS. Kubernetes DNS troubleshooting has one rule that saves more time than any other: reproduce the failure from inside the cluster before you theorize. The app's error message ("no such host", "i/o timeout on lookup") tells you a lookup failed; it doesn't tell you which of the five layers between the pod and the answer is broken. A debug pod does.

kubectl run dnsutils --rm -it --restart=Never \
  --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- sh

(Or nicolaka/netshoot if you want the full toolbox.) Run it in the same namespace as the failing app - namespace matters for what resolves, and NetworkPolicy may treat namespaces differently. From inside:

nslookup kubernetes.default
nslookup <your-service>
nslookup google.com

Three lookups, four outcomes, and each one points somewhere different:

  • All three fail → CoreDNS itself, or the path to it. Go to "Is CoreDNS healthy."
  • Internal works, external fails → upstream forwarding. Go to "Why do internal names resolve but external ones don't?"
  • External works, one service fails → the service, not DNS. Check why the Service isn't working - an existing Service always gets a DNS record, even with zero endpoints.
  • Everything works from your debug pod but not the app → the app pod's own resolv.conf, its NetworkPolicy, or the name it's actually asking for. Attach to the real pod with kubectl debug and ephemeral containers and repeat the lookups there.
Symptom Likely cause First command
Every lookup fails, internal and external CoreDNS down or unreachable kubectl get pods -n kube-system -l k8s-app=kube-dns
Internal names resolve, external don't Upstream forwarding or node resolver kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
External lookups intermittently slow ndots:5 search-list expansion cat /etc/resolv.conf (inside the pod)
One namespace can't resolve anything NetworkPolicy blocking port 53 egress kubectl get networkpolicy -n <namespace>
Lookups time out, CoreDNS pods look fine Empty kube-dns endpoints or broken Service path kubectl get svc,endpoints kube-dns -n kube-system
One service name fails, others work Wrong name or namespace qualifier nslookup <service>.<namespace>

Know which names should resolve

Half of "DNS is broken" reports are a wrong name. The forms that work:

<service>                          # same namespace only
<service>.<namespace>              # cross-namespace
<service>.<namespace>.svc.cluster.local.   # fully qualified

Pods get records too (<pod-ip-with-dashes>.<namespace>.pod.cluster.local), but almost nothing should depend on them. Headless services (clusterIP: None) return the pod IPs directly instead of a single virtual IP - if your client caches one of those and the pod died, that's your bug, not the resolver's.

Why are external DNS lookups slow in Kubernetes?

Usually ndots:5, which turns one external lookup into several. Look at the pod's resolver config before blaming CoreDNS:

cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

ndots:5 means: any name with fewer than five dots is tried against every search domain first. So api.stripe.com becomes four queries - api.stripe.com.default.svc.cluster.local, then the next two suffixes, and only then the real name. Each of those extra queries is a chance to time out, which is why this presents as intermittent slowness rather than clean failure.

Two fixes: use a trailing dot for external names (api.stripe.com. skips the search list), or set dnsConfig.options with a lower ndots in the pod spec for workloads that mostly talk to the internet.

Is CoreDNS healthy?

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Crash-looping CoreDNS is a cluster-wide incident, not a DNS quirk - usually a Corefile that doesn't parse after an edit, or a loop with the node's resolver (the loop plugin will say so explicitly and exit). Broken CoreDNS resolution is one of the practice labs if you want to chase it on a live cluster instead of production.

If the pods are Running but you suspect queries aren't arriving, turn on query logging: add the log plugin to the Corefile.

kubectl -n kube-system edit configmap coredns
.:53 {
    log
    errors
    ...
}

CoreDNS reloads the config automatically within about half a minute. Now run your failing lookup and watch the logs. If the query never appears, the problem is between the pod and CoreDNS. If it appears with NXDOMAIN or SERVFAIL, CoreDNS is answering - read what it says. Remember to remove log afterward; it's noisy at cluster scale.

The kube-dns Service and the path to it

Queries reach CoreDNS through a Service (still named kube-dns for historical reasons):

kubectl get svc,endpoints kube-dns -n kube-system

Empty endpoints → no ready CoreDNS pods; fix that first. If endpoints exist, bypass the Service and query a CoreDNS pod directly from your debug pod:

dig @<coredns-pod-ip> kubernetes.default.svc.cluster.local

Direct query works but the Service IP doesn't → kube-proxy or the CNI is mangling the path. Also check for a NetworkPolicy that forgot DNS egress - a default-deny policy that allows traffic to the app's dependencies but not UDP/TCP 53 to kube-system is a classic, and it logs nothing. That failure mode has its own guide: NetworkPolicy debugging.

If your cluster runs NodeLocal DNSCache, the nameserver in resolv.conf is a link-local address on the node, not the kube-dns ClusterIP - check the node-local-dns DaemonSet before concluding CoreDNS is unreachable.

Why do internal names resolve but external ones don't?

Because CoreDNS answers cluster names itself but forwards everything else upstream, and that forwarding path is broken. It forwards per the forward plugin - by default to whatever /etc/resolv.conf on the node says.

forward . /etc/resolv.conf

So external resolution breaks when the node's resolver config is wrong, the upstream DNS server is down, or egress from the CoreDNS pods is blocked. Test the upstream from a node directly; if the node can't resolve either, you've left Kubernetes entirely - that's Linux DNS debugging territory.

FAQ

Why is DNS only slow/flaky instead of failing outright? Almost always ndots. A five-dot threshold multiplies every external lookup into several queries over UDP, and any one of them can drop. Conntrack races on UDP DNS packets cause the same intermittent 5-second stalls; NodeLocal DNSCache exists largely to fix that.

Why does DNS fail in only one namespace? A NetworkPolicy in that namespace blocking port 53 egress, or the service name being used without a namespace qualifier from the wrong namespace. Reproduce from a debug pod in the affected namespace before concluding anything.

Is it safe to restart CoreDNS? Yes - it's a Deployment with multiple replicas on any sane cluster, and clients retry. But restart it because the logs told you to, not as a first move; if a bad Corefile is the problem, the restart makes things worse.

Can I change the cluster.local domain? It's set at cluster creation (kubelet clusterDomain). Changing it on a live cluster breaks every existing FQDN reference; treat it as fixed.