Why Your Kubernetes Resource Limits Are Wrong

If you have been running workloads on Kubernetes for more than a few months, you have almost certainly encountered this: a pod gets OOMKilled at 3 AM, your on-call engineer bumps the memory limit, and the problem disappears — until it happens again two weeks later.

The root cause is almost always the same: limits were set by intuition, not measurement.

Why limits matter more than you think

In Kubernetes, requests and limits do two fundamentally different things.

• Requests influence scheduling. The scheduler places a pod on a node that has at least the requested amount of CPU and memory available.
• Limits constrain runtime behaviour. A container that exceeds its memory limit is killed immediately. A container that exceeds its CPU limit is throttled — silently, with no pod restart.

CPU throttling is particularly insidious because it produces latency spikes that are extremely difficult to correlate with resource config without the right tooling.

The naive approach and why it fails

Most teams start by picking a round number — 500m CPU, 512Mi memory — and adjusting reactively when things break. This leads to one of two failure modes:

1. Over-provisioning: Nodes are half-empty. You are paying for capacity that never gets used.
2. Under-provisioning: Workloads are throttled or OOMKilled under peak load.

A systematic approach using VPA in recommendation mode

The Vertical Pod Autoscaler (VPA) can be run in Off mode, which means it only recommends without applying changes. This is the safest way to gather data.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-api
  updatePolicy:
    updateMode: "Off"

After a few days of traffic, inspect the recommendations:

kubectl describe vpa my-api-vpa

VPA will give you lowerBound, target, and upperBound estimates based on observed usage histograms. Use target as your new requests and upperBound as your limits.

Tracking CPU throttling in Prometheus

The single most useful metric for validating your limits post-change is container_cpu_cfs_throttled_seconds_total. A PromQL query to compute throttle ratio:

rate(container_cpu_cfs_throttled_seconds_total[5m])
  /
rate(container_cpu_cfs_periods_total[5m])

Anything consistently above 0.25 (25% of CPU scheduling periods are throttled) for a latency-sensitive service warrants increasing the CPU limit or the request.

Summary

• Run VPA in Off mode for at least 72 hours of representative traffic.
• Use the target recommendation for requests, upperBound for limits.
• Instrument CPU throttling with Prometheus and alert on a sustained throttle ratio above your SLO threshold.
• Revisit limits after any significant traffic pattern change — seasonal peaks, new features, major deploys.

Getting resource configs right is a one-time investment that pays back in stability and significantly lower infrastructure bills.