<p>There is a kubernetes cluster with 100 nodes, I have to clean the specific images manually, I know the kubelet garbage collect may help, but it isn't applied in my case.
After browsing the internet , I found a solution - docker in docker, to solve my problem.</p>
<p>I just wanna remove the image in each node one time, is there any way to run a job in each node one time?</p>
<p>I checked the kubernetes labels and podaffinity, but still no ideas, any body could help?</p>
<p>Also, I tried to use daemonset to solve the problem, but turns out that it can only remove the image for a part of nodes instead of all nodes, I don't what might be the problem...</p>
<p>here is the daemonset example:</p>
<pre class="lang-yaml prettyprint-override"><code>kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: test-ds
  labels:
    k8s-app: test
spec:
  selector:
    matchLabels:
      k8s-app: test
  template:
    metadata:
      labels:
        k8s-app: test
    spec:
      containers:
      - name: test
        env:
        - name: DELETE_IMAGE_NAME
          value: &quot;nginx&quot;
        image: busybox
        command: ['sh', '-c', 'curl --unix-socket /var/run/docker.sock -X DELETE http://localhost/v1.39/images/$(DELETE_IMAGE_NAME)']
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-sock-volume
        ports:
        - containerPort: 80
      volumes:
      - name: docker-sock-volume
        hostPath:
          # location on host
          path: /var/run/docker.sock
</code></pre>

<p>If you want to run you job on single specific Node you can us the <strong>Nodeselector</strong> in POD spec</p>
<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: test
spec:
  schedule: &quot;*/1 * * * *&quot;
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: test
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure
          nodeSelector: 
            name: node3
</code></pre>
<p>daemon set ideally should resolve your issues, as it creates the PODs on each available Node in the cluster.</p>
<p>You can read more about the <strong>affinity</strong> at here : <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/</a></p>
<blockquote>
<p><strong>nodeSelector</strong> provides a very simple way to constrain pods to nodes
with particular labels. The affinity/anti-affinity feature, greatly
expands the types of constraints you can express. The key enhancements
are</p>
<p>The affinity/anti-affinity language is more expressive. The language
offers more matching rules besides exact matches created with a
logical AND operation;</p>
</blockquote>
<p>You can use the <strong>Affinity</strong> in Job YAML something like</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/e2e-az-name
            operator: In
            values:
            - e2e-az1
            - e2e-az2
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: another-node-label-key
            operator: In
            values:
            - another-node-label-value
  containers:
  - name: with-node-affinity
    image: k8s.gcr.io/pause:2.0
</code></pre>
<p><strong>Update</strong></p>
<p>Now if you have issue with the Deamon affinity with the Job is also useless, as Job will create the Single POD which will get schedule to Single node as per affinity. Either create 100 job with different affinity rules or you use <strong>Deployment</strong> + Affinity to schedule the Replicas on different nodes.</p>
<p>We will create one Deployment with <strong>POD affinity</strong> and make sure, <strong>multiple</strong> PODs of a <strong>single</strong> deployment won't get scheduled on <strong>one Node</strong>.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  labels:
    app: test
spec:
  replicas: 100
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: test
        image: &lt;Image&gt;
        ports:
        - containerPort: 80
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: &quot;app&quot;
                operator: In
                values:
                - test
            topologyKey: &quot;kubernetes.io/hostname&quot;
</code></pre>
<p>Try using this deployment template and replace your image here. You can reduce <strong>replicas</strong> first to <strong>10</strong> instead of <strong>100</strong> to check it's spreading PODs or not.</p>
<p>Read more at : <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#an-example-of-a-pod-that-uses-pod-affinity" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#an-example-of-a-pod-that-uses-pod-affinity</a></p>
<p><strong>Extra</strong> :</p>
<p>You can also write and use your custom CRD : <a href="https://github.com/darkowlzz/daemonset-job" rel="nofollow noreferrer">https://github.com/darkowlzz/daemonset-job</a> which will behave as <strong>daemon set</strong> and <strong>job</strong></p>
