<p>I am running a cluster on GKE with a single node pool. It has 3 nodes and can scale from 1 to 99 nodes. The cluster uses the <code>nginx-ingress</code> controller</p>

<p>On this cluster, I want to deploy <em>apps</em>. An <em>app</em> is scoped by a namespace and consists of 3 <code>deployments</code> and one <code>ingress</code> (defining paths to access the application from the internet). Each deployment runs a single replica of a container. </p>

<p>Deploying a couple of apps works fine, but deploying <em>many</em> apps (requiring the node pool to scale up) breaks everything:</p>

<p><strong>All pods start having warnings</strong> (including those successfully deployed earlier)</p>

<pre><code>kubectl get pods --namespace bcd
NAME                       READY     STATUS    RESTARTS   AGE
actions-664b7d79f5-7qdkw   1/1       Unknown   1          35m
actions-664b7d79f5-v8s2m   1/1       Running   1          18m
core-85cb74f89b-ns49z      1/1       Unknown   1          35m
core-85cb74f89b-qqzfp      1/1       Running   1          18m
nlu-77899ddbf-8pd7k        1/1       Running   1          27m
</code></pre>

<p><strong>All nodes becomes <em>unready</em></strong>:</p>

<pre><code>kubectl get nodes
NAME                                              STATUS     ROLES     AGE       VERSION
gke-clients-projects-default-pool-f9af73d4-gzwr   NotReady   &lt;none&gt;    42m       v1.9.7-gke.6
gke-clients-projects-default-pool-f9af73d4-p5l2   NotReady   &lt;none&gt;    21m       v1.9.7-gke.6
gke-clients-projects-default-pool-f9af73d4-wnxc   NotReady   &lt;none&gt;    37m       v1.9.7-gke.6
</code></pre>

<p>Deleting the namespace to remove all resources from the cluster also seems to fail as after a long while the pods remain active but still in an unknown state.</p>

<p>How can I safely add more apps and let the cluster autoscale?</p>

znat
<p>The reason seems to be that not knowing the resources needed for each pod, the scheduler schedules them on any available node, potentially exhausting available resources and putting the Docker daemon in an inconsistent state.</p>

<p>The solution is to specify resources requests and limits: <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container</a></p>
