<p>I created a two nodes clusters and I created a new job using the busybox image that sleeps for 300 secs. I checked on which node this job is running using</p>
<p><code>kubectl get pods -o wide</code></p>
<p>I deleted the node but surprisingly the job was still finishing to run on the same node. Any idea if this is a normal behavior? If not how can I fix it?</p>

Luke Solomon
<p>Jobs aren't scheduled or running on nodes. The role of a job is just to define a policy by making sure that a pod with certain specifications exists and ensure that it runs till the completion of the task whether it completed successfully or not.</p>
<p>When you create a job, you are declaring a policy that the built-in <code>job-controller</code> will see and will create a pod for. Then the built-in <code>kube-scheduler</code> will see this pod without a node and patch the pod to it with a node's identity. The <code>kubelet</code> will see a pod with a node matching it's own identity and hence a container will be started. As the container will be still running, the <code>control-plane</code> will know that the node and the pod still exist.</p>
<p>There are two ways of breaking a node, one with a drain and the second without a drain. The process of breaking a node without draining is identical to a network cut or a server crash. The api-server will keep the node resource for a while, but it 'll cease being Ready. The pods will be then terminated slowly. However, when you drain a node, it looks as if you are preventing new pods from scheduling on to the node and deleting the pods using <code>kubectl delete pod</code>.</p>
<p>In both ways, the pods will be deleted and you will be having a job that hasn't run to completion and doesn't have a pod, therefore <code>job-controller</code> will make a new pod for the job and the job's <code>failed-attempts</code> will be increased by 1, and the loop will start over again.</p>
