<p>When scheduling a kubernetes Job and Pod, if the Pod can't be placed the explanation available from <code>kubectl describe pods PODNAME</code> looks like:</p>

<pre><code>Warning  FailedScheduling  &lt;unknown&gt;  default-scheduler  0/172 nodes are available: 
1 Insufficient pods, 1 node(s) were unschedulable, 11 Insufficient memory, 
30 Insufficient cpu, 32 node(s) didn't match node selector, 97 Insufficient nvidia.com/gpu.
</code></pre>

<p>That's useful but a little too vague. I'd like more detail than that. </p>

<blockquote>
  <p>Specifically can I list all nodes with the reason the pod wasn't scheduled to each particular node? </p>
</blockquote>

<p>I was recently changing labels and the node selector and want to determine if I made a mistake somewhere in that process or if the nodes I need really are just busy.</p>

David Parks
<p>You can find more details related to problems with scheduling particular <code>Pod</code> in <code>kube-scheduler</code> logs. If you set up your cluster with <strong>kubeadm</strong> tool, <code>kube-scheduler</code> as well as other key components of the cluster is deployed as a system <code>Pod</code>. You can list such <code>Pods</code> with the following command:</p>

<pre><code>kubectl get pods -n kube-system
</code></pre>

<p>which will show you among others your <code>kube-scheduler</code> <code>Pod</code>:</p>

<pre><code>NAME                                          READY   STATUS              RESTARTS   AGE
kube-scheduler-master-ubuntu-18-04            1/1     Running             0          2m37s
</code></pre>

<p>Then you can check its logs. In my example the command will look as follows:</p>

<pre><code>kubectl logs kube-scheduler-master-ubuntu-18-04 -n kube-system
</code></pre>

<p>You should find there the information you need.</p>

<hr>

<p>One more thing...</p>

<p>If you've already verified it, just ignore this tip</p>

<p>Let's start from the beginning...</p>

<p>I've just created a simple job from the example you can find <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" rel="nofollow noreferrer">here</a>:</p>

<pre><code>kubectl apply -f https://k8s.io/examples/controllers/job.yaml

job.batch/pi created
</code></pre>

<p>If I run:</p>

<pre><code>kubectl get jobs
</code></pre>

<p>it shows me:</p>

<pre><code>NAME   COMPLETIONS   DURATION   AGE
pi     0/1           17m        17m
</code></pre>

<p>Hmm... completions 0/1 ? Something definitely went wrong. Let's check it.</p>

<pre><code>kubectl describe job pi
</code></pre>

<p>tells me basically nothing. In it's events I can see only:</p>

<pre><code>Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  18m   job-controller  Created pod: pi-zxp4p
</code></pre>

<p>as if everything went well... but we already know it didn't. So let's investigate further. As you probably know, <code>job-controller</code> creates <code>Pods</code> that run to completion to perform certain task. From the perspective of the <code>job-controller</code> everything went well (we've just seen it in it's events):</p>

<pre><code>Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  23m   job-controller  Created pod: pi-zxp4p
</code></pre>

<p>It did it's part of the task and reported that everything went fine. But it's just part of the whole task. It passed actual <code>Pod</code> creation task further to the <code>kube-scheduler</code> controller as being just a <code>job-controller</code> it isn't responsible (and doesn't even have enough privileges) to schedule the actual <code>Pod</code> on particular node. If we run:</p>

<pre><code>kubectl get pods
</code></pre>

<p>we can see one <code>Pod</code> in a <code>Pending</code> state:</p>

<pre><code>NAME       READY   STATUS    RESTARTS   AGE
pi-zxp4p   0/1     Pending   0          30m
</code></pre>

<p>Let's describe it:</p>

<pre><code>kubectl describe pod pi-zxp4p
</code></pre>

<p>In events we can see some very important and specific info:</p>

<pre><code>Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  20s (x24 over 33m)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
</code></pre>

<p>so now we know the actual reason why our Pod couldn't be scheduled.</p>

<p>Pay attention to different fields of the event:</p>

<p><code>From</code>: <code>default-scheduler</code> - it means that the message was originated from our <code>kube-scheduler</code>.</p>

<p><code>Type</code>: <code>Warning</code>, which isn't as important as <code>Critical</code> or <code>Error</code> so chances are that it may not appear in <code>kube-scheduler</code> logs if the last one was started with the default level of log verbosity.</p>

<p>You can read <a href="https://github.com/ravigadde/kube-scheduler/blob/master/docs/devel/logging.md" rel="nofollow noreferrer">here</a> that:</p>

<blockquote>
  <p>As per the comments, the practical default level is V(2). Developers
  and QE environments may wish to run at V(3) or V(4). If you wish to
  change the log level, you can pass in -v=X where X is the desired
  maximum level to log.</p>
</blockquote>
