Kubernetes requests and limits: throttling, OOM kills and QoS

Four numbers in a pod spec decide where it runs, how much CPU it gets under contention, whether it is throttled, and whether it is killed. They are usually copied from another service, set once and never revisited, and they cause a steady stream of incidents that look like something else: tail latency nobody can explain, pods restarting with exit code 137, nodes that are full while sitting half idle.

This article covers what CPU and memory requests and limits actually do, which mechanism enforces each one, why CPU limits throttle services that look idle, how OOM kills and node-pressure eviction choose their victims, the QoS classes the settings add up to, runtimes that misread their limits, and how to choose the numbers.

Requests and limits are different things

A request is a reservation. The scheduler places a pod only on a node whose allocatable capacity, minus the requests of the pods already there, can fit it. Requests say nothing about what the pod actually uses. A limit is a ceiling enforced by the Linux kernel through cgroups on the node, regardless of what the scheduler thought.

And CPU and memory behave differently when they run out. CPU is compressible: a process that can’t get CPU just runs slower. Memory is not: a process that can’t get memory must be killed or must free some. Those two distinctions give four settings with four different mechanisms behind them.

From these settings Kubernetes derives a QoS class: Guaranteed (requests = limits for CPU and memory in every container; evicted last), Burstable (at least one request or limit set; evicted by usage above request) and BestEffort (nothing set; evicted first).

CPU requests: placement and shares

Besides placement, the CPU request sets the container’s cgroup CPU weight. When the node’s CPUs are contended, each container gets CPU time in proportion to its weight, so a container that requested 2 CPUs gets twice the share of one that requested 1. When the node is not contended, weights do nothing: any container may use idle CPU, up to its limit if it has one.

That makes the CPU request the important CPU setting. Set too low, the pod is packed onto busy nodes and loses out to its neighbours exactly when load is high. It is also the unit the Horizontal Pod Autoscaler uses: a target of 65% CPU means 65% of the request, so a request that doesn’t reflect real use makes autoscaling misbehave.

CPU limits and CFS throttling

A CPU limit is enforced by the kernel’s CFS bandwidth control as a quota per period. The period is 100 ms, and a limit of 1 CPU means 100 ms of CPU time per period; a limit of 500m means 50 ms. The quota is shared by all of the container’s threads. Once they have used it, every thread in the container is stopped until the next period begins, even if the node has idle cores.

This interacts badly with multi-threaded servers. A request that keeps four threads busy for 40 ms uses 160 ms of CPU. With a limit of 1 CPU, the threads exhaust the 100 ms quota in 25 ms of wall-clock time, wait 75 ms for the next period, and finish 15 ms into it:

Because throttling acts on 100 ms windows and CPU metrics are averaged over minutes, a throttled service looks underused on every dashboard and shows the problem only in its tail latency. The evidence is in the cgroup counters: container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total is the fraction of periods in which the container was throttled. Anything more than a few percent on a latency-sensitive service is worth fixing.

Hence the common advice for latency-sensitive services: set an accurate CPU request and no CPU limit. The request protects the service’s share under contention, and unused CPU on the node absorbs bursts instead of being wasted. The trade-off is less predictability: a service that has grown used to spare CPU slows down when the node fills up. Limits still make sense for batch work, for untrusted tenants, and where a platform requires them. Where they are required, set them well above the request.

Memory: limits, OOM kills and eviction

A memory limit becomes the cgroup’s memory.max. When the container’s usage reaches it, the kernel first tries to reclaim page cache belonging to the cgroup; if that isn’t enough, the OOM killer kills a process in the container. Kubernetes reports the container as OOMKilled with exit code 137 (128 plus SIGKILL’s 9) and restarts it. There is no graceful shutdown and no warning, and in-flight requests fail.

A different mechanism handles a node that runs short of memory overall. The kubelet watches node memory, and when available memory falls below its eviction threshold it evicts pods: it terminates them so they can be rescheduled elsewhere. It ranks candidates first by whether their usage exceeds their memory request, then by pod priority, then by how far usage exceeds the request. A pod using less than it requested is close to safe; a pod with no request at all is first in line.

If memory runs out faster than the kubelet can react, the kernel’s node-level OOM killer acts instead, choosing by an oom_score_adj that Kubernetes sets from the pod’s QoS class.

QoS classes

Kubernetes assigns every pod one of three quality-of-service classes from its settings:

• Guaranteed: every container has CPU and memory limits equal to its requests. Evicted last, and given an oom_score_adj of −997 so the kernel avoids it. With whole-number CPU requests and the kubelet’s static CPU manager policy, it can also get exclusive cores.
• Burstable: at least one container sets a request or limit, but the pod isn’t Guaranteed. Most services land here. Eviction depends on usage relative to request.
• BestEffort: nothing set at all. Scheduled anywhere, evicted first, and given an oom_score_adj of 1000. A LimitRange with default requests stops pods ending up here by accident.

Note that a service with no CPU limit can’t be Guaranteed, which is usually fine: the memory request-equals-limit rule gives the eviction protection that matters, and the class mostly affects behaviour under node pressure.

Runtimes that can’t see the limits

A container sees the host’s CPU count and memory unless the runtime reads its cgroup. Several common mistakes follow:

• Go set GOMAXPROCS to the host’s core count before Go 1.25. On a 64-core node with a 2-CPU limit, 64 threads compete for 200 ms of quota per period and are throttled constantly. Go 1.25 reads the cgroup CPU limit; on older versions, use automaxprocs or set the variable.
• The JVM has been container-aware since Java 10 (and 8u191), but its default heap is a quarter of the container’s memory. -XX:MaxRAMPercentage sets it explicitly; leave room for metaspace, thread stacks, direct buffers and the collector, or the kernel kills a JVM whose heap is nowhere near full.
• Node.js sizes its default V8 heap from available memory too, and its native memory (buffers, native modules) sits outside the heap. Set --max-old-space-size below the limit.
• Worker and thread pool sizes derived from os.cpu_count() or similar (the usual Gunicorn recipe, for example) see the host’s cores and need the same treatment: derive them from the CPU request or limit, not from nproc on the host.

Choosing the numbers

Requests and limits should come from measurement, not from the last service’s manifest:

• Memory: watch the container’s working set (container_memory_working_set_bytes, the figure the kubelet uses) over a week including peak traffic and deploys. Set request and limit to the peak plus 15–30%.
• CPU request: around the 90th–95th percentile of measured use per pod, at the replica count you expect at peak. Lower means relying on neighbours being idle; higher means paying for idle capacity.
• CPU limit: none for latency-sensitive services where policy allows; otherwise several times the request, and check throttling after every change.
• Revisit them. The Vertical Pod Autoscaler in recommendation mode reports suggested requests from observed usage without changing anything, which is a cheap way to find the services that are badly off.

Over-requesting is the more common failure in practice: clusters where the sum of requests says the nodes are full while actual use is 20–30%. It is invisible in incidents and very visible in the bill. Under-requesting shows up as incidents instead. Both are fixed the same way, by measuring. (Rolling resource changes out safely is the subject of the article on progressive delivery, and deciding how much latency is acceptable is the subject of the article on SLOs.)

A deployment with sane resources

A latency-sensitive service with memory request equal to limit, a CPU request and no CPU limit, a JVM told how much of the container to use, an autoscaler that scales on CPU relative to the request, namespace defaults so nothing is BestEffort, and the query that shows throttling.

# A latency-sensitive HTTP service. Memory: request = limit, so the pod is never
# overcommitted and is among the last to be evicted. CPU: a request sized from
# measured p95 use, and no limit, so bursts use idle cores instead of being throttled.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout }
  template:
    metadata:
      labels: { app: checkout }
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:2026-10-29.1
          resources:
            requests:
              cpu: "750m"
              memory: "1Gi"
            limits:
              memory: "1Gi"
          env:
            # The JVM sizes its heap from the container limit; leave room for
            # metaspace, threads, direct buffers and the GC itself.
            - name: JAVA_TOOL_OPTIONS
              value: "-XX:MaxRAMPercentage=70 -XX:ActiveProcessorCount=2"
---
# Scale on CPU as a percentage of the request: the request is the unit, so it
# has to reflect real use for this target to mean anything.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: checkout }
  minReplicas: 6
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 65 }
---
# Defaults for containers that set nothing, so nothing in the namespace is BestEffort.
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: "100m", memory: "256Mi" }
      default: { memory: "256Mi" }

# Is anything being throttled? (PromQL, per container, over 5 minutes)
#   rate(container_cpu_cfs_throttled_periods_total[5m])
#     / rate(container_cpu_cfs_periods_total[5m])

References

1. Kubernetes Documentation. Resource Management for Pods and Containers. kubernetes.io
2. Kubernetes Documentation. Pod Quality of Service Classes and Node-pressure Eviction. kubernetes.io
3. Linux Kernel Documentation. CFS Bandwidth Control and Control Group v2 (cpu.max, cpu.weight, memory.max). docs.kernel.org/scheduler/sched-bwc.html
4. Chiluk, D. (2019). sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices. Linux kernel commit, merged for 5.4. github.com/torvalds/linux
5. Go Team (2025). Container-aware GOMAXPROCS. The Go Blog. go.dev/blog
6. Burns, B., Beda, J., Hightower, K. & Evenson, L. (2022). Kubernetes: Up and Running, 3rd ed. O’Reilly. oreilly.com