By Riya Madaan
Requests and limits: the misconfigurations that page you
Kubernetes resource limits and requests, minus the folklore: CPU throttling, limit=request tradeoffs, missing requests, and the LimitRange safety net.
Most Kubernetes resource limits advice fits on a sticky note: "set requests and limits." Most Kubernetes resource incidents come from teams that followed it - with numbers copied from a blog post, a default Helm chart, or the service next door. Requests and limits are not hygiene settings. They are scheduling contracts and kill thresholds, and each misconfiguration has a specific, recognizable failure mode. Here are the ones that actually page you.
First, the mechanics in one paragraph, because every mistake below traces back to them. Requests are what the scheduler reserves: they determine placement, and they're your rank in the eviction queue. Limits are enforcement: a container over its memory limit is OOM-killed by the kernel; a container over its CPU limit is throttled, never killed. CPU is compressible (you wait), memory is not (you die). That asymmetry drives everything.
| Symptom | Likely cause | First command |
|---|---|---|
| Pods evicted first under memory pressure | No requests, so BestEffort QoS |
kubectl get pod <pod> -o jsonpath='{.status.qosClass}' |
HPA shows <unknown> targets |
Missing CPU requests | kubectl describe hpa <name> |
p99 latency spikes, kubectl top looks calm |
CPU limit throttling | kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat |
| Container restarts with exit code 137 | Memory limit too low, or a leak | kubectl describe pod <pod> |
| Pods Pending on a half-idle cluster | Requests far above real usage | kubectl describe node <node> (Allocated resources) |
| Deploy rejected at admission | ResourceQuota requires requests | kubectl describe resourcequota -n <namespace> |
Mistake 1: no requests at all
With no requests, the scheduler treats your pod as weightless and packs nodes on vibes. The consequences arrive in this order:
- Nodes oversubscribe - twenty "weightless" pods land on one node.
- Under memory pressure, your requestless pod is
BestEffort- first against the wall. See Evicted pods: node pressure and who gets killed first. - Your HPA silently stops working, because percentage targets divide by requests - the
missing request for cpufailure covered in HPA not scaling.
Check for offenders across a namespace:
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.resources.requests.cpu}{"/"}{.resources.requests.memory}{" "}{end}{"\n"}{end}'
Blank fields are the bug. Every container - sidecars included - needs requests.
Mistake 2: Why is my pod throttled when CPU usage looks low?
Because a CPU limit is a quota per 100ms period, not an average, and a bursty service spends it
early and then waits. CPU limits are enforced by CFS quota: your container gets limit × 100ms of CPU time per
100ms period, and when it's spent, every thread stops until the next period. A multi-threaded
service can burn its entire quota in the first 20ms of a period and then sit frozen for 80ms,
while serving a request. That's your p99.
The signature is invisible in kubectl top (which shows averages) and explicit in cgroup
stats:
kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat # cgroup v2
# nr_throttled and throttled_usec climbing = you're being throttled
Prometheus users: container_cpu_cfs_throttled_periods_total is the same story over time.
The pragmatic position most SRE teams converge on: set honest CPU requests (they guarantee share under contention) and skip CPU limits for latency-sensitive services, unless you're in a multi-tenant cluster with a reason to cap. Throttling a healthy service to protect an idle neighbor is a bad trade.
Memory is the opposite: always set memory limits. An unbounded leak on a limitless container doesn't kill the container - it kills the node, taking every neighbor with it. A memory limit converts "node down" into "one pod OOMKilled," which is a much better page. What exit code 137 looks like and how to chase the leak: OOMKilled and exit code 137.
Mistake 3: Should limits equal requests?
Only for the workloads that earn it. Setting limits == requests on every container gives you Guaranteed QoS: last to be
evicted, immune to neighbor noise, and (with integral CPUs and the static CPU manager policy)
eligible for pinned cores. For your database, your queue, your stateful core - good trade.
For everything else, it means paying for your worst case all day. A service that needs 2Gi
once an hour and 300Mi otherwise, run as Guaranteed at 2Gi, wastes ~1.7Gi × replicas of
schedulable capacity. Multiply across a cluster and this is why "we're out of nodes" and
"cluster utilization is 30%" are true at the same time - and why pods sit
stuck in Pending on a half-idle cluster: the scheduler
does arithmetic on requests, not on usage. A pod Pending on an oversized request is one of
the free practice labs, and it's a quick way to see that arithmetic in
describe output.
Burstable (requests at ~P50–P90 usage, memory limit above it, no CPU limit) is the right default for stateless services. Guaranteed is a deliberate upgrade, not a best practice.
Mistake 4: requests nobody ever re-measured
Requests copied from the chart's values.yaml in 2023 are fiction with a signature. Compare
fiction to reality:
kubectl top pods --containers -n <namespace>
kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'
Allocated resources on the node shows the gap: requests can total 95% of the node while
actual usage sits at 25% (over-requesting - wasted money), or requests total 40% while usage
sits at 90% (under-requesting - an eviction storm on schedule). VPA in recommendation mode
(updateMode: "Off") will do this measurement for you continuously and put its answer in
kubectl describe vpa - you don't have to let it act to benefit from its math.
Mistake 5: no LimitRange or ResourceQuota, then one bad manifest
Everything above is per-team discipline. LimitRange is the namespace-level backstop that
makes discipline survivable: it injects default requests/limits into containers that specify
none, and rejects the pathological ones.
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
spec:
limits:
- type: Container
defaultRequest: { cpu: 100m, memory: 128Mi }
default: { memory: 512Mi }
max: { memory: 4Gi }
Pair it with a ResourceQuota per namespace and a single runaway manifest stops being able
to eat the cluster. One warning label: a new LimitRange applies to new pods only, and a
quota'd namespace requires requests on every pod - adding a ResourceQuota to a namespace
full of requestless manifests turns the next deploy into a wall of admission errors. Roll it
out namespace by namespace, defaults first, quota second.
FAQ
Should I always set CPU limits in Kubernetes? No. Memory limits: essentially always. CPU limits: only with a concrete reason (hard multi-tenancy, chargeback, capping a known-runaway batch job). CFS throttling from well-intentioned CPU limits is one of the most common self-inflicted latency problems in production clusters.
What happens if a container exceeds its memory request but not its limit? Nothing - until the node comes under memory pressure. Then pods using more than they requested are first in the kubelet's eviction ranking. Requests are the promise the eviction logic holds you to.
What's the difference between OOMKilled and Evicted? OOMKilled: your container crossed its own memory limit; the kernel killed the process; it restarts in place with exit code 137. Evicted: the node ran short; the kubelet killed the whole pod; it gets rescheduled elsewhere. Limit problem vs node problem.
How do I pick initial numbers for a new service?
Deploy with generous requests in a non-prod cluster, put load through it, read
kubectl top pods --containers and throttling/OOM counters, then set CPU request ≈ typical
usage, memory request ≈ steady-state, memory limit ≈ peak + headroom. Then re-measure
quarterly - the service you deployed is not the service you're running.