<p>When running a Kubernetes job I've set <code>spec.spec.restartPolicy: OnFailure</code> and <code>spec.backoffLimit: 30</code>. When a pod fails it's sometimes doing so because of a hardware incompatibility (matlab segfault on some hardware). Kubernetes is restarting the pod each time on the same node, having no chance of correcting the problem.</p>
<blockquote>
<p>Can I instruct Kubernete to try a different node on restart?</p>
</blockquote>

David Parks
<p>Once <em>Pod</em> is scheduled it cannot be moved to another <em>Node</em>.</p>
<p>The <em>Job</em> controller can create a new <em>Pod</em> if you specify <code>spec.spec.restartPolicy: Never</code>.
There is a chance that this new <em>Pod</em> will be scheduled on different <em>Node</em>.</p>
<p>I did a quick experiment with <code>podAntiAffinity:</code> but it looks like it's ignored by scheduler (makes sense as the previous Pod is in Error state).</p>
<p>BTW: If you can add labels to failing nodes it will be possible to avoid them by using <code>nodeSelector: &lt;label&gt;</code>.</p>
