Kubernetes Pod CPU Throttling but CPU Usage Low: Fix the Quota

Kubernetes · Intermediate · 7 min read · published 2026-10-28

This article was written by Claude (Anthropic) and published automatically. Published by Osama Fayyaz.

What this solves: Your pod's CPU graph shows 30% of its limit, yet p99 latency spikes and throttling metrics climb. Here's why CFS quotas do this and how to stop it.

The Problem

Your dashboard says the pod is fine. CPU usage sits at 0.3 cores against a limit of 1, yet you're seeing Kubernetes pod CPU throttling while CPU usage looks low. p99 latency jumps from 20ms to 120ms in bursts, health checks occasionally time out, and container_cpu_cfs_throttled_periods_total climbs steadily.

The numbers seem to contradict each other. The pod uses 30% of what it's allowed, so how can it be throttled? The answer is that the average you're looking at hides what happens inside each 100 millisecond window.

Typical symptoms:

Latency spikes cluster around multiples of ~100ms (for example 100ms, 200ms).
Spikes get worse under bursty traffic, during GC, or at startup, even when the average load is low.
Single-threaded load tests look perfect, but production concurrency falls apart.
The node is mostly idle the whole time.

Why the Obvious Fix Falls Short

"Raise the limit." This is what nearly everyone tries first, and it partly works, which is what makes it misleading. Doubling the limit doubles the budget per window, so the freeze starts later and lasts less time. But if your runtime thinks it has 32 or 64 cores, it can still burn through a 2-core budget in a few milliseconds. You end up paying for capacity you don't use, and the spikes remain.

"Scale out with an HPA." The HPA reads averaged CPU usage, the same metric that says everything is fine. At 30% utilization it will never add pods. Even if you lower the target to force scaling, every new pod has the same per-window problem.

"Look at the CPU graph more closely." Prometheus scrapes every 15–60 seconds, and rate() averages over minutes. A 5ms burst followed by a 95ms freeze averages out to a flat, healthy-looking line. You cannot see this problem in usage metrics at any zoom level. You have to look at the throttling counters.

How It Actually Works

A CPU limit is enforced by the Linux CFS bandwidth controller, not by Kubernetes. limits.cpu: 1 becomes cpu.max = 100000 100000, which means "100ms of CPU time per 100ms period", summed across all threads.

The quota is not "one core". It is a pool of CPU-time that any number of threads can draw from at once. Eight threads running in parallel drain 100ms of quota in 12.5ms of wall-clock time. The kernel then removes every thread in the cgroup from the run queue until the next period starts. Requests in flight sit frozen for the remaining 87.5ms.

Why does an app run eight threads, or 48? Because many runtimes size their thread pools from the node's core count, not the container's quota:

Go before 1.25: GOMAXPROCS = node cores. Go 1.25+ reads the cgroup limit.
JVM: container-aware since JDK 10, unless someone set -XX:ActiveProcessorCount or -XX:-UseContainerSupport.
Node.js: os.availableParallelism() / os.cpus().length often report host cores, so worker pools sized from them oversubscribe.
Python: multiprocessing.cpu_count() and gunicorn/uvicorn worker formulas built on it.

sequenceDiagram
    participant K as CFS (quota 100ms / period)
    participant T as 8 app threads
    participant R as Incoming request
    Note over K: Period starts, quota = 100ms
    R->>T: request arrives (t=0ms)
    T->>K: 8 threads run in parallel
    Note over K: 8 x 12.5ms = 100ms used at t=12.5ms
    K-->>T: THROTTLE all threads
    Note over T,R: Frozen 87.5ms, node CPU idle
    Note over K: t=100ms new period, quota refilled
    K->>T: threads resume
    T->>R: response at ~105ms instead of ~15ms

CPU requests work differently. They set cpu.weight, which only matters when the node is contended. They guarantee a share and never cap you. This is why many teams set requests and leave out CPU limits.

How do I confirm it's throttling?

# Fraction of periods in which the container was throttled
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))

Anything above roughly 5–10% on a latency-sensitive service is worth fixing. Inside the container, cat /sys/fs/cgroup/cpu.stat shows nr_throttled and throttled_usec directly.

Before and After

Before: a Go 1.22 service on a 48-core node with a 1-CPU limit.

# BEFORE: runtime sees 48 cores, quota allows 1 core-equivalent per 100ms
resources:
  requests:
    cpu: 500m
  limits:
    cpu: "1"        # 100ms quota per 100ms period
# No GOMAXPROCS -> Go spins up 48 Ps; a burst drains the quota in ~2ms
# and every goroutine freezes for ~98ms.

After, option A: keep the limit and make the runtime match it.

# AFTER (A): parallelism matches quota, so bursts can't outrun it
resources:
  requests:
    cpu: 500m
  limits:
    cpu: "1"
env:
  - name: GOMAXPROCS          # CHANGED: derive from the limit
    valueFrom:
      resourceFieldRef:
        resource: limits.cpu
        divisor: "1"          # rounds up to whole CPUs
# Java: drop any -XX:ActiveProcessorCount override.
# Node: size worker pools from the limit, not os.cpus().length.

After, option B: drop the CPU limit and rely on the request.

# AFTER (B): no CFS quota at all; request guarantees fair share under contention
resources:
  requests:
    cpu: "1"        # CHANGED: set request to real steady-state need
    memory: 512Mi
  limits:
    memory: 512Mi   # keep memory limits; memory is not compressible
# Still set GOMAXPROCS to ~request so the scheduler doesn't thrash.

When NOT to Use This

Multi-tenant or untrusted workloads. Removing limits lets one noisy pod take all idle CPU on a node. Keep limits there, and fix the thread count instead (option A).
Namespaces with LimitRange/ResourceQuota requiring limits. Option B may be rejected by admission. Use option A, or negotiate the policy.
Guaranteed QoS / static CPU manager. If you need pinned exclusive cores (low-latency trading, DPDK), you need requests == limits with integer CPUs and cpuManagerPolicy: static. That is a different mechanism with no CFS quota on pinned cores.
Batch jobs where latency doesn't matter. Throttling only slows throughput, which is often acceptable. Don't optimize what nobody waits on.
When the throttle ratio is near zero. If spikes persist without throttling, look elsewhere: GC, lock contention, noisy neighbours on memory bandwidth, or a slow dependency.

Gotchas

Fractional limits round badly. limits.cpu: 1500m with divisor: "1" gives GOMAXPROCS=2, which can still burst past 150ms per period. Prefer whole-number limits for services that set thread counts from them.
Sidecars share nothing, but you pay for them anyway. Quotas are per container. An Envoy/Istio sidecar with its own small limit can throttle and add latency to every request even when your app is fine. Check the sidecar's throttle ratio too.
Startup and JIT bursts. JVMs and Node apps compile heavily at boot. A tight limit stretches startup and can fail readiness or liveness probes, which leads to restart loops. A startupProbe with a generous threshold helps.
Go 1.25 still respects explicit GOMAXPROCS. If an old env var sets it to 64, the new container-aware default is overridden. Grep your manifests and base images.
Throttling is invisible to APM traces. A trace shows a span that took 100ms with no work inside it. If the gaps are suspiciously ~100ms-aligned, suspect CFS before suspecting the code.
Kernel cpu.cfs_burst_us exists but is rarely available. Burst budgets can absorb short spikes on newer kernels, but most managed Kubernetes platforms don't expose them. Don't plan around them.

Key takeaway: A CPU limit is a budget per 100ms window, not an average, so match your runtime's thread count to the limit or drop the limit and keep the request.

Real-world challenge

A Go API pod has requests.cpu: 500m and limits.cpu: 1 and runs on 48-core nodes. It is built with Go 1.22. Grafana shows average CPU of about 0.35 cores, but p99 latency jumps from 15ms to 110ms every few seconds under moderate load. Raising the limit to 2 brought p99 down to around 60ms, but the spikes are still there. The team is about to ask for a bigger node pool. What is going on, and what should they do?

Diagnose

Check the throttle ratio, not CPU usage: rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]). A value above roughly 5–10% on a latency-sensitive service is the smoking gun.
Check GOMAXPROCS. Go before 1.25 ignores cgroup limits, so on a 48-core node it schedules goroutines across 48 OS threads.
Do the math. With a limit of 1, the budget is 100ms of CPU per 100ms period. If 48 threads are busy during a burst, that budget is gone in about 2ms. Every request is then frozen for about 98ms. A ~100ms latency jump is exactly that freeze. Raising the limit to 2 only moved the freeze point to about 4ms.

Fix

Tell Go how many CPUs it actually has:

env:
  - name: GOMAXPROCS
    valueFrom:
      resourceFieldRef:
        resource: limits.cpu
        divisor: "1"   # rounds up to a whole CPU

You can also upgrade to Go 1.25+, which reads the cgroup limit by default, or import go.uber.org/automaxprocs. After the change, put the limit back to 1. The throttle ratio should fall to near zero, and you don't need a bigger node pool.