skip to content
‹ All posts

By Riya Madaan

Linux DNS not resolving: resolv.conf and systemd-resolved

Linux DNS not resolving? Trace the real resolution path - nsswitch, /etc/hosts, resolv.conf, systemd-resolved's stub - and why dig works but curl fails.

#linux

DNS not resolving on a Linux box is rarely a DNS server problem. It's usually a path problem: your application, dig, and ping can each take a different route to an answer, and the box has at least three places that claim to know the nameserver - /etc/resolv.conf, systemd-resolved, and whatever NetworkManager last wrote. The classic symptom of a Linux DNS misconfiguration is exactly this split brain: dig example.com works while curl https://example.com says Could not resolve host. That's not a contradiction; the two tools asked different oracles.

Symptom Likely cause First command
dig works, curl says Could not resolve host Local path: nsswitch, /etc/hosts, stub, search/ndots getent hosts <name>
Hand edits to resolv.conf keep vanishing Another daemon owns the file ls -l /etc/resolv.conf
Internal names fail only with a VPN up Per-link DNS or a hijacked ~. routing domain in systemd-resolved resolvectl status
Both getent and dig fail Upstream nameserver or network path to it dig @<upstream> <name>
Every lookup slow but eventually works First nameserver in the list is dead cat /etc/resolv.conf
Works on the host, fails in the container musl/Go resolver differences or missing nsswitch.conf in the image cat /etc/resolv.conf (inside the container)

Why does dig work but curl can't resolve the host?

Because they take different roads to an answer.

Road 1 - the libc road (what real apps use): getaddrinfo() → /etc/nsswitch.conf → usually hosts: files dns → checks /etc/hosts first, then queries the nameservers in /etc/resolv.conf (or the resolved stub). Honors search domains and ndots.

Road 2 - the dig road: dig/host/nslookup read resolv.conf themselves and query DNS directly. They skip nsswitch, skip /etc/hosts, and (for dig) skip search domains unless you ask.

So test with the road your app actually uses:

getent hosts api.internal.example.com   # libc road - what curl/your app sees
dig +short api.internal.example.com     # DNS road - what the nameserver says
  • getent fails, dig works → the problem is local: nsswitch, /etc/hosts, stub resolver, search/ndots, or a broken NSS plugin.
  • Both fail → the problem is the nameserver or the network path to it.
  • getent works, app still fails → the app isn't using libc resolution (static Go binaries, musl containers, apps with built-in resolvers) or it's caching a stale failure.

Read the local configuration honestly

cat /etc/nsswitch.conf | grep ^hosts
cat /etc/resolv.conf
ls -l /etc/resolv.conf     # is it a symlink? to what?

That ls -l is the highest-value command in this post. On systemd-resolved systems, /etc/resolv.conf is a symlink, and which target it points to changes everything:

  • ../run/systemd/resolve/stub-resolv.conf → nameserver is 127.0.0.53, the local stub; resolved is in the loop.
  • ../run/systemd/resolve/resolv.conf → the real upstream servers, resolved bypassed for DNS but still managing the list.
  • A plain file → something else (NetworkManager, dhclient, an admin, cloud-init) owns it, and edits by hand will be overwritten by whichever daemon owns it on the next lease renewal.

What does nameserver 127.0.0.53 mean?

127.0.0.53 means every query goes through systemd-resolved, which maintains per-link DNS servers and search domains - the part everyone forgets. A VPN that installed its own DNS on tun0 for *.corp while eth0 keeps the office resolver is invisible in resolv.conf and fully visible here:

resolvectl status            # per-link servers, domains, DNSSEC state
resolvectl query api.internal.example.com   # resolve THROUGH resolved, shows which link answered
resolvectl statistics        # cache hits, failures
journalctl -u systemd-resolved --since -1h  # timeouts, DNSSEC failures, server switches

resolvectl query failing while dig @8.8.8.8 succeeds pins the fault inside resolved's config - wrong per-link server, a ~. routing domain hijacked by a dead VPN link, or DNSSEC validation rejecting your internal zone. Flush and retest after any change:

resolvectl flush-caches

If the upstream is the problem

Both roads fail? Test the servers directly, in order:

dig @127.0.0.53 example.com          # the stub (if in use)
dig @10.0.0.2 example.com            # the configured upstream, from resolv.conf/resolvectl
dig @9.9.9.9 example.com             # a known-good public resolver

First failure in that chain is your boundary. Upstream unreachable is then a network problem, not a DNS problem - port 53 UDP/TCP blocked by a firewall change, a security group, or a dead resolver VM. ss -unap | grep 53 and a quick nc -vzu 10.0.0.2 53 settle it. And check resolv.conf's options timeout:… attempts:… - the default 5-second timeout per server explains "everything is slow but eventually works" when the first nameserver in the list is dead and every lookup burns its timeout before trying the second.

The container twist

Containers get their own resolv.conf, written by the runtime. Docker's embedded resolver lives at 127.0.0.11 inside bridge-network containers; Kubernetes points pods at CoreDNS and sets ndots:5, which turns every short name into multiple search-domain lookups before trying it as an FQDN. If the failure is inside a pod, that's its own diagnostic tree, Kubernetes DNS troubleshooting - but remember the node's DNS health is the floor the cluster stands on: CoreDNS forwards to the node's upstreams, so a broken host resolver becomes "the cluster's DNS is flaky." When the node itself is sick, start with the kubelet and node-level checks. The cluster side is one place where a lab does exist: the practice labs include a Kubernetes cluster with CoreDNS resolution broken, and the same layer-by-layer triage applies.

Also containers: Alpine/musl resolves differently than glibc (no ndots handling quirks are identical), and static Go binaries choose between their pure-Go resolver and cgo at build time. "Works on the host, fails in the container" usually lands in one of those gaps or in a missing /etc/nsswitch.conf in the image.

A five-command triage, in order

ls -l /etc/resolv.conf && cat /etc/resolv.conf     # who owns resolution?
getent hosts <name>                                # libc road
dig +short <name>                                  # DNS road
resolvectl status                                  # per-link truth (if resolved)
dig @<upstream> <name>                             # is upstream even alive?

Run these five before touching anything, and you'll know which of the four layers - nsswitch, stub, upstream, network - is lying. That ordering discipline, not memorized trivia about ndots, is what separates a five-minute fix from an afternoon; it's the same habit the Linux troubleshooting interview scenarios are built to observe.

FAQ

Why does dig work but curl fail? dig queries DNS directly from resolv.conf; curl uses libc's getaddrinfo, which goes through /etc/nsswitch.conf, /etc/hosts, search domains, and the resolved stub. The mismatch means the fault is in that local path - compare getent hosts <name> (curl's view) with dig <name>.

What is 127.0.0.53 in resolv.conf? systemd-resolved's local stub listener. Apps send queries there; resolved forwards to the real per-link upstream servers shown by resolvectl status. Don't replace it by hand-editing the symlinked resolv.conf - configure resolved (or the network manager) instead.

How do I flush the DNS cache on Linux? With systemd-resolved: resolvectl flush-caches. With dnsmasq: restart it or send SIGHUP. Plain glibc doesn't cache at all - if you're seeing stale answers without a local caching daemon, the caching is upstream or inside the application.

Why is DNS slow but not broken? Usually a dead first nameserver: each lookup waits timeout seconds (default 5) before falling back to the next server in resolv.conf. Fix the server order or the dead resolver, or lower options timeout:1 attempts:2.