By Riya Madaan
Ingress 502, 503, 504: which layer is lying to you
Kubernetes ingress 502, 503, and 504 errors decoded: which layer failed, which logs to read, and the fastest path from load balancer to pod to find it.
A Kubernetes ingress 502 tells you exactly one thing: some proxy was disappointed by something behind it. It doesn't tell you which proxy, and that's the whole game. A request that dies with a 5xx crossed up to five layers - load balancer → ingress controller → Service → Endpoints → container - and any of them can be the one generating the error page. Engineers who fix these fast don't debug the app first; they find out which layer answered.
The request path, and what each hop can do to you:
client → cloud LB → ingress controller pod → Service → Endpoints → app container
| Symptom | Likely cause | First command |
|---|---|---|
| 503 on every request | Service has no ready endpoints | kubectl get endpoints <service> |
Instant 502s, connection refused in controller logs |
Wrong targetPort, or app bound to 127.0.0.1 |
kubectl exec <pod> -- ss -tlnp |
| Intermittent 502s, RESTARTS climbing | App crashing mid-request | kubectl get pods -l app=<name> |
| 502s only during deploys | Pods closing listeners while still in Endpoints | kubectl get deploy <name> -o yaml (look for preStop) |
| 504 after exactly 60s | App slower than proxy-read-timeout |
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=100 |
| 504, curl to pod IP hangs | NetworkPolicy dropping packets | kubectl get networkpolicy -n <namespace> |
| Controller logs show 200, client sees 5xx | Cloud LB timeout or health check | Check the LB's own logs and idle timeout |
First: who actually generated the error?
Fastest check in the whole post. An error page generated by nginx-ingress looks different
from one generated by your app or your cloud LB (they name themselves - nginx, or the
cloud provider's distinctive plain-text error). Then confirm in the controller logs:
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=100 \
| grep -E ' (502|503|504) '
The nginx access-log line includes the upstream pod IP and what happened to the connection. Three patterns cover most incidents:
connect() failed (111: Connection refused) while connecting to upstream→ reached the pod, nothing listening on that port → 502no live upstreams/ no endpoints for the service → 503upstream timed out (110: Connection timed out)→ 504
If the controller logs show your request completing with a 200, the error came from a layer in front of it - LB timeout or health-check misconfiguration. Stop debugging the cluster.
What does a 503 from the ingress mean?
A 503 from the ingress controller almost always means the Service has no ready endpoints. One command:
kubectl get endpoints <service>
Empty ENDPOINTS? Two causes:
- Selector mismatch - the Service selects labels no pod has. Compare
kubectl get svc <service> -o yamlselectors against actual pod labels. The full checklist is in Service not reachable. - No pod is Ready - pods exist but their readiness probes fail, so they're excluded
from endpoints.
kubectl get pods -o wideand look at the READY column. A readiness probe pointed at the wrong port or path removes a perfectly healthy app from rotation; the probe-induced outage has its own field guide.
A 503 that appears only during deploys usually means a rollout is replacing pods faster than new ones become Ready - see rolling update stuck.
Why is my ingress returning 502?
A 502 means the proxy reached a pod IP and the conversation went badly - connection refused, reset, or a malformed response. In descending order of frequency:
Wrong targetPort. The Service's targetPort doesn't match the port the container
actually listens on. connection refused in the controller logs, instant 502s, 100%
failure rate.
kubectl get svc <service> -o jsonpath='{.spec.ports}'
kubectl exec <pod> -- ss -tlnp # what's actually listening
The app is bound to 127.0.0.1. It answers curl localhost from inside the container
and refuses everything else. ss -tlnp shows 127.0.0.1:8080 instead of 0.0.0.0:8080.
The app crashed mid-request. Intermittent 502s plus a climbing RESTARTS count. That's not an ingress problem - go read the CrashLoopBackOff field guide.
Protocol mismatch. The controller speaks HTTP to a port expecting TLS (or vice versa). nginx logs complain about malformed responses; the fix is the backend-protocol annotation.
Intermittent 502s only during deploys. Pods receive SIGTERM and close listeners while
still listed in Endpoints for a few hundred milliseconds. The standard fix is a preStop
sleep of a few seconds so the pod drains before it dies.
504: everyone is waiting for someone
A 504 means some layer's timeout expired while waiting for the layer behind it. Work out whose:
- The app is genuinely slow. Time it directly from inside the cluster (next section).
If a request takes 70 seconds and nginx's default
proxy-read-timeoutis 60, that's your 504. Raise the annotation or fix the query. - A NetworkPolicy is blackholing the connection. Dropped SYNs produce no error, no log, no reset - just a timeout. If curl to the pod IP hangs entirely, suspect policy before app.
- The cloud LB's idle timeout is shorter than the request. The LB returns 504 while the controller logs show the request still in flight, or completing after the client already got the error.
Bypass layers until it works
When logs are ambiguous, binary-search the path from the inside out:
kubectl run curl --rm -it --restart=Never --image=curlimages/curl -- sh
curl -sv http://<pod-ip>:8080/ # 1. pod directly
curl -sv http://<service>.<ns>.svc/ # 2. through the Service
curl -sv -H "Host: app.example.com" http://<ingress-controller-svc>/ # 3. through ingress
Each hop that succeeds is exonerated. Pod works, Service doesn't → endpoints/kube-proxy. Service works, ingress doesn't → controller config or Ingress resource. Everything inside works → the cloud LB or DNS in front of it. Five minutes, and you know which layer was lying. (The ingress backend port mismatch in the practice labs is this exact bisect, on a broken cluster with a clock running - it separates people who guess from people who trace.)
FAQ
What's the difference between 502 and 504? 502: the backend answered wrongly - refused, reset, or garbled the connection. 504: the backend didn't answer in time. 502 points at ports, crashes, and protocol mismatches; 504 points at slowness, timeouts, and silent packet drops.
Why do I get 502s only during deployments?
Terminating pods linger in Endpoints for a moment after they stop accepting connections,
and new pods may be marked Ready before they're truly warm. A preStop sleep plus an
honest readiness probe removes almost all of it.
The Ingress worked yesterday and nothing changed. Now 503. Why?
Something changed - usually readiness. Pods that drop out of endpoints (failing probe,
OOM-kill loop, node eviction) produce a 503 with zero changes to the Ingress itself. Start
with kubectl get endpoints, not with the Ingress YAML.
Should I raise the ingress timeout to stop 504s? Only after you know why the request is slow. Raising timeouts to cover a degrading backend converts a visible error into invisible latency, and the page you get later is worse.