<p>I have a cluster of 3 nodes running Kubernetes 1.6.1, each has 2 CPU and 4G RAM.</p>

<p>I am constantly redeploying my application with the same Docker tag by updating pod template hash by replacing environment variable value that is passed to the container.</p>

<p><code>sed "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" nginx-deployment.yml | kubectl replace -f -</code></p>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        env:
        - name: FOR_GODS_SAKE_PLEASE_REDEPLOY
          value: 'THIS_STRING_IS_REPLACED_DURING_BUILD'
</code></pre>

<p>If I do this for a few hundred times, I can't redeploy any more - new pods are in Pending state.
<code>kubectl get events</code> produces the following:
</p>

<pre><code>Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  1h        50s     379 default-scheduler           Warning     
FailedScheduling    No nodes are available that match all of the following predicates:: Insufficient pods (3). 
</code></pre>

<p>At the same time I can see about 200 Exited nginx containers on every Kube node.</p>

<p>Looking in kube-controller-manager logs I can see that PodGC is trying to delete some pods, but they are not found.
</p>

<pre><code>I0516 12:53:41.137311       1 gc_controller.go:175] Found unscheduled terminating Pod nginx-deployment-2927112463-qczvv not assigned to any Node. Deleting.
I0516 12:53:41.137320       1 gc_controller.go:62] PodGC is force deleting Pod: default:nginx-deployment-2927112463-qczvv
E0516 12:53:41.190592       1 gc_controller.go:177] pods "nginx-deployment-2927112463-qczvv" not found
I0516 12:53:41.195020       1 gc_controller.go:175] Found unscheduled terminating Pod nginx-deployment-3265736979-jrpzb not assigned to any Node. Deleting.
I0516 12:53:41.195048       1 gc_controller.go:62] PodGC is force deleting Pod: default:nginx-deployment-3265736979-jrpzb
E0516 12:53:41.238307       1 gc_controller.go:177] pods "nginx-deployment-3265736979-jrpzb" not found
</code></pre>

<p>Is there anything I can do to prevent that from happening?</p>

<p>Kubernetes allows you to tweak the garbage collection flags of kubelet. This can be done via changing the flags <code>--maximum-dead-containers</code> or <code>--maximum-dead-containers-per-container</code>. Read more about it in docs here:</p>

<ul>
<li><a href="https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/" rel="nofollow noreferrer">Configuring kubelet Garbage Collection</a></li>
<li><a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="nofollow noreferrer">Configuring Out Of Resource Handling</a></li>
<li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/" rel="nofollow noreferrer">Garbage Collection</a></li>
</ul>
