By Riya Madaan
Linux DNS not resolving: resolv.conf and systemd-resolved
Linux DNS not resolving? Trace the real resolution path - nsswitch, /etc/hosts, resolv.conf, systemd-resolved's stub - and why dig works but curl fails.
DNS not resolving on a Linux box is rarely a DNS server problem. It's usually a path
problem: your application, dig, and ping can each take a different route to an answer, and
the box has at least three places that claim to know the nameserver - /etc/resolv.conf,
systemd-resolved, and whatever NetworkManager last wrote. The classic symptom of a Linux DNS
misconfiguration is exactly this split brain: dig example.com works while curl https://example.com says Could not resolve host. That's not a contradiction; the two tools
asked different oracles.
| Symptom | Likely cause | First command |
|---|---|---|
dig works, curl says Could not resolve host |
Local path: nsswitch, /etc/hosts, stub, search/ndots |
getent hosts <name> |
Hand edits to resolv.conf keep vanishing |
Another daemon owns the file | ls -l /etc/resolv.conf |
| Internal names fail only with a VPN up | Per-link DNS or a hijacked ~. routing domain in systemd-resolved |
resolvectl status |
Both getent and dig fail |
Upstream nameserver or network path to it | dig @<upstream> <name> |
| Every lookup slow but eventually works | First nameserver in the list is dead | cat /etc/resolv.conf |
| Works on the host, fails in the container | musl/Go resolver differences or missing nsswitch.conf in the image |
cat /etc/resolv.conf (inside the container) |
Why does dig work but curl can't resolve the host?
Because they take different roads to an answer.
Road 1 - the libc road (what real apps use): getaddrinfo() → /etc/nsswitch.conf →
usually hosts: files dns → checks /etc/hosts first, then queries the nameservers in
/etc/resolv.conf (or the resolved stub). Honors search domains and ndots.
Road 2 - the dig road: dig/host/nslookup read resolv.conf themselves and query
DNS directly. They skip nsswitch, skip /etc/hosts, and (for dig) skip search domains
unless you ask.
So test with the road your app actually uses:
getent hosts api.internal.example.com # libc road - what curl/your app sees
dig +short api.internal.example.com # DNS road - what the nameserver says
getentfails,digworks → the problem is local: nsswitch,/etc/hosts, stub resolver, search/ndots, or a broken NSS plugin.- Both fail → the problem is the nameserver or the network path to it.
getentworks, app still fails → the app isn't using libc resolution (static Go binaries, musl containers, apps with built-in resolvers) or it's caching a stale failure.
Read the local configuration honestly
cat /etc/nsswitch.conf | grep ^hosts
cat /etc/resolv.conf
ls -l /etc/resolv.conf # is it a symlink? to what?
That ls -l is the highest-value command in this post. On systemd-resolved systems,
/etc/resolv.conf is a symlink, and which target it points to changes everything:
../run/systemd/resolve/stub-resolv.conf→ nameserver is127.0.0.53, the local stub; resolved is in the loop.../run/systemd/resolve/resolv.conf→ the real upstream servers, resolved bypassed for DNS but still managing the list.- A plain file → something else (NetworkManager, dhclient, an admin, cloud-init) owns it, and edits by hand will be overwritten by whichever daemon owns it on the next lease renewal.
What does nameserver 127.0.0.53 mean?
127.0.0.53 means every query goes through systemd-resolved, which maintains per-link DNS
servers and search domains - the part everyone forgets. A VPN that installed its own DNS on
tun0 for *.corp while eth0 keeps the office resolver is invisible in resolv.conf and
fully visible here:
resolvectl status # per-link servers, domains, DNSSEC state
resolvectl query api.internal.example.com # resolve THROUGH resolved, shows which link answered
resolvectl statistics # cache hits, failures
journalctl -u systemd-resolved --since -1h # timeouts, DNSSEC failures, server switches
resolvectl query failing while dig @8.8.8.8 succeeds pins the fault inside resolved's
config - wrong per-link server, a ~. routing domain hijacked by a dead VPN link, or DNSSEC
validation rejecting your internal zone. Flush and retest after any change:
resolvectl flush-caches
If the upstream is the problem
Both roads fail? Test the servers directly, in order:
dig @127.0.0.53 example.com # the stub (if in use)
dig @10.0.0.2 example.com # the configured upstream, from resolv.conf/resolvectl
dig @9.9.9.9 example.com # a known-good public resolver
First failure in that chain is your boundary. Upstream unreachable is then a network problem,
not a DNS problem - port 53 UDP/TCP blocked by a firewall change, a security group, or a dead
resolver VM. ss -unap | grep 53 and a quick nc -vzu 10.0.0.2 53 settle it. And check
resolv.conf's options timeout:… attempts:… - the default 5-second timeout per server
explains "everything is slow but eventually works" when the first nameserver in the list is
dead and every lookup burns its timeout before trying the second.
The container twist
Containers get their own resolv.conf, written by the runtime. Docker's embedded resolver
lives at 127.0.0.11 inside bridge-network containers; Kubernetes points pods at CoreDNS and
sets ndots:5, which turns every short name into multiple search-domain lookups before trying
it as an FQDN. If the failure is inside a pod, that's its own diagnostic tree,
Kubernetes DNS troubleshooting - but remember the
node's DNS health is the floor the cluster stands on: CoreDNS forwards to the node's
upstreams, so a broken host resolver becomes "the cluster's DNS is flaky." When the node
itself is sick, start with the kubelet and node-level checks.
The cluster side is one place where a lab does exist: the practice labs include a
Kubernetes cluster with CoreDNS resolution broken, and the same layer-by-layer triage applies.
Also containers: Alpine/musl resolves differently than glibc (no ndots handling quirks are
identical), and static Go binaries choose between their pure-Go resolver and cgo at build
time. "Works on the host, fails in the container" usually lands in one of those gaps or in a
missing /etc/nsswitch.conf in the image.
A five-command triage, in order
ls -l /etc/resolv.conf && cat /etc/resolv.conf # who owns resolution?
getent hosts <name> # libc road
dig +short <name> # DNS road
resolvectl status # per-link truth (if resolved)
dig @<upstream> <name> # is upstream even alive?
Run these five before touching anything, and you'll know which of the four layers - nsswitch,
stub, upstream, network - is lying. That ordering discipline, not memorized trivia about
ndots, is what separates a five-minute fix from an afternoon; it's the same habit the
Linux troubleshooting interview scenarios are
built to observe.
FAQ
Why does dig work but curl fail?
dig queries DNS directly from resolv.conf; curl uses libc's getaddrinfo, which goes
through /etc/nsswitch.conf, /etc/hosts, search domains, and the resolved stub. The
mismatch means the fault is in that local path - compare getent hosts <name> (curl's view)
with dig <name>.
What is 127.0.0.53 in resolv.conf?
systemd-resolved's local stub listener. Apps send queries there; resolved forwards to the real
per-link upstream servers shown by resolvectl status. Don't replace it by hand-editing the
symlinked resolv.conf - configure resolved (or the network manager) instead.
How do I flush the DNS cache on Linux?
With systemd-resolved: resolvectl flush-caches. With dnsmasq: restart it or send SIGHUP.
Plain glibc doesn't cache at all - if you're seeing stale answers without a local caching
daemon, the caching is upstream or inside the application.
Why is DNS slow but not broken?
Usually a dead first nameserver: each lookup waits timeout seconds (default 5) before
falling back to the next server in resolv.conf. Fix the server order or the dead resolver,
or lower options timeout:1 attempts:2.