<p>On my GCE Kubernetes cluster I can no longer create pods.</p>
<pre><code>Warning FailedScheduling    pod (www.caveconditions.com-f1be467e31c7b00bc983fbe5efdbb8eb-438ef) failed to fit in any node
fit failure on node (gke-prod-cluster-default-pool-b39c7f0c-c0ug): Insufficient CPU
</code></pre>
<p>Looking at the allocated stats of that node</p>
<pre><code>Non-terminated Pods:        (8 in total)
  Namespace         Name                                        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                                        ------------    ----------  --------------- -------------
  default           dev.caveconditions.com-n80z8                            100m (10%)  0 (0%)      0 (0%)      0 (0%)
  default           lamp-cnmrc                                  100m (10%)  0 (0%)      0 (0%)      0 (0%)
  default           mongo-2-h59ly                                   200m (20%)  0 (0%)      0 (0%)      0 (0%)
  default           www.caveconditions.com-tl7pa                            100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           fluentd-cloud-logging-gke-prod-cluster-default-pool-b39c7f0c-c0ug       100m (10%)  0 (0%)      200Mi (5%)  200Mi (5%)
  kube-system           kube-dns-v17-qp5la                              110m (11%)  110m (11%)  120Mi (3%)  220Mi (5%)
  kube-system           kube-proxy-gke-prod-cluster-default-pool-b39c7f0c-c0ug              100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kubernetes-dashboard-v1.1.0-orphh                       100m (10%)  100m (10%)  50Mi (1%)   50Mi (1%)
Allocated resources:
  (Total limits may be over 100%, i.e., overcommitted. More info: http://releases.k8s.io/HEAD/docs/user-guide/compute-resources.md)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  910m (91%)    210m (21%)  370Mi (9%)  470Mi (12%)
</code></pre>
<p>Sure I have 91% allocated and can not fit another 10% into it. But is it not possible to over commit  resources?</p>
<p>The usage of the server is at about 10% CPU average</p>
<p><a href="https://i.stack.imgur.com/wBSuc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wBSuc.png" alt="enter image description here" /></a></p>
<p>What changes do I need to make for my Kubernetes cluster to be able to create more pods?</p>

<p>I recently had this same issue. After some research, I found that GKE has a default <code>LimitRange</code> with CPU requests limit set to <code>100m</code>.</p>
<p>You can validate this by running <code>kubectl get limitrange -o=yaml</code>.
It's going to display something like this:</p>
<pre><code>apiVersion: v1
items:
- apiVersion: v1
  kind: LimitRange
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;kind&quot;:&quot;LimitRange&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;limits&quot;,&quot;namespace&quot;:&quot;default&quot;},&quot;spec&quot;:{&quot;limits&quot;:[{&quot;defaultRequest&quot;:{&quot;cpu&quot;:&quot;100m&quot;},&quot;type&quot;:&quot;Container&quot;}]}}
    creationTimestamp: 2017-11-16T12:15:40Z
    name: limits
    namespace: default
    resourceVersion: &quot;18741722&quot;
    selfLink: /api/v1/namespaces/default/limitranges/limits
    uid: dcb25a24-cac7-11e7-a3d5-42010a8001b6
  spec:
    limits:
    - defaultRequest:
        cpu: 100m
      type: Container
kind: List
metadata:
  resourceVersion: &quot;&quot;
  selfLink: &quot;&quot;
</code></pre>
<p>This limit is applied to every container. So, for instance, if you have a 4 cores node and each pod creates 2 containers, it will allow only for around ~20 pods to be created (4 cpus = 4000m -&gt; / 100m = 40 -&gt; / 2 = 20).</p>
<p>The &quot;fix&quot; here is to change the default <code>LimitRange</code> to one that better fits your use-case and then remove old pods allowing them to be recreated with the updated values. Another (and probably better) option is to directly set the CPU limits on each deployment/pod definition you have.</p>
<p>Some reading material:</p>
<p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit</a></p>
<p><a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/#create-a-limitrange-and-a-pod" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/#create-a-limitrange-and-a-pod</a></p>
<p><a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run</a></p>
<p><a href="https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits" rel="nofollow noreferrer">https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits</a></p>
