The CFS quota container throttling problem
Well, this is quite a messy one:
Almost all services at Twitter run on Linux with the CFS scheduler, using CFS bandwidth control quota for isolation, with default parameters. The intention is to allow different services to be colocated on the same boxes without having one service’s runaway CPU usage impact other services and to prevent services on empty boxes from taking all of the CPU on the box, resulting in unpredictable performance, which service owners found difficult to reason about before we enabled quotas. The quota mechanism limits the amortized CPU usage of each container, but it doesn’t limit how many cores the job can use at any given moment. Instead, if a job “wants to” use more than that many cores over a quota timeslice, it will use more cores than its quota for a short period of time and then get throttled, i.e., basically get put to sleep, in order to keep its amortized core usage below the quota, which is disastrous for tail latency1. Since the vast majority of services at Twitter use thread pools that are much larger than their mesos core reservation, when jobs have heavy load, they end up requesting and then using more cores than their reservation and then throttling. This causes services that are provisioned based on load test numbers or observed latency under load to over provision CPU to avoid violating their SLOs. They either have to ask for more CPUs per shard than they actually need or they have to increase the number of shards they use.
Note that Kubernetes uses CFS to implement CPU quotas by default, too. In the twitter thread about this post, a commenter noted: “‘By shrinking the CFS period, the worst case time between quota exhaustion causing throttling and the process group being able to run again is reduced proportionately’. Our P99s at previous gig reduced in line after I petitioned cloud provider to adjust setting.” — this at least seems like a relatively easy setting to tune.(tags: cgroups kubernetes linux k8s cfs scheduling containers quotas)