<p>My Kubernetes cluster has memory pressure limits that I need to fix (at a later time).</p>
<p>There are sometimes anywhere from a few evicted pods to dozens.
I created a Cronjob spec for clearing up the evicted pods. I tested the command inside and it works fine from powershell.</p>
<p>However, it doesn't matter if I specify a namespace in the spec or not, deploy it to every namespace that exists, the script doesn't seem to delete my evicted pods.</p>
<p>Original Script:</p>
<pre><code>---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: delete-evicted-pods
spec:
  schedule: &quot;*/30 * * * *&quot;
  failedJobsHistoryLimit: 1
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: kubectl-runner
            image: bitnami/kubectl:latest
            command: [&quot;sh&quot;, &quot;-c&quot;, &quot;kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f -&quot;]
          restartPolicy: OnFailure
</code></pre>
<p>I tried creating the script with associated RBAC, with no luck either.</p>
<pre><code>kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: development
  name: cronjob-runner
rules:
- apiGroups:
  - extensions
  - apps
  resources:
  - deployments
  verbs:
  - 'patch'

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cronjob-runner
  namespace: development
subjects:
- kind: ServiceAccount
  name: sa-cronjob-runner
  namespace: development
roleRef:
  kind: Role
  name: cronjob-runner
  apiGroup: &quot;&quot;

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-cronjob-runner
  namespace: development
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: delete-all-failed-pods
spec:
  schedule: &quot;*/30 * * * *&quot;
  failedJobsHistoryLimit: 1
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-cronjob-runner
          containers:
          - name: kubectl-runner
            image: bitnami/kubectl:latest
            command: 
              - /bin/sh
              - -c
              - kubectl get pods --all-namespaces --field-selector 'status.phase==Failed' -o json | kubectl delete -f -
          restartPolicy: OnFailure 
</code></pre>
<p>I realize I should have better memory limits defined, but this functionality was working before I upgraded k8s to 1.16 from 1.14.</p>
<p>Is there something I'm doing wrong or missing? If it helps, I'm running in Azure (AKS).</p>
<p><a href="https://i.stack.imgur.com/s9352.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/s9352.png" alt="All Namespaces Pods" /></a></p>

<p>It sounds like after the upgrade this:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl get pods --all-namespaces --field-selector 'status.phase==Failed'`
</code></pre>
<p>is not picking up your failed pods anymore. It could be:</p>
<ul>
<li>kubectl/apiserver version mismatch</li>
<li>credentials/service account permissions</li>
<li>(?)</li>
</ul>
<p>You can try running a debug pod to verify:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl run -i --tty --rm debug --image=bitnami/kubectl:latest --restart=Never -- get pods --all-namespaces --field-selector 'status.phase==Failed'
</code></pre>
<p>Every Job in Kubernetes creates a Pod, so you can also look at the logs for your <code>kubectl-runner</code> pods:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl logs kubectl-runner-xxxxx
</code></pre>
<p>Update:</p>
<p>Based on the log files it looks like the <code>default:default</code> service account doesn't have enough permissions this would fix it:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl create clusterrolebinding myadmin-binding --clusterrole=cluster-admin --serviceaccount=default:default

</code></pre>
<p>But then if you'd like to be more restrictive you will have to create a more limited ClusterRole or Role (if you want it limited to a namespace)</p>
