<p>Without any knows changes in our Kubernetes 1.6 cluster all new or restarted pods are not scheduled anymore. The error I get is:</p>

<pre><code>No nodes are available that match all of the following predicates:: MatchInterPodAffinity (10), PodToleratesNodeTaints (2).
</code></pre>

<p>Our cluster was working perfectly before and I really cannot see any configuration changes that have been made before that occured.</p>

<p>Things I already tried:</p>

<ul>
<li>restarting the master node</li>
<li>restarting kube-scheduler</li>
<li>deleting affected pods, deployments, stateful sets</li>
</ul>

<p>Some of the pods do have anti-affinity settings that worked before, but most pods do not have any affinity settings.</p>

<p>Cluster Infos:</p>

<ul>
<li>Kubernetes 1.6.2</li>
<li>Kops on AWS</li>
<li>1 master, 8 main-nodes, 1 tainted data processing node</li>
</ul>

<p>Is there any known cause to this? </p>

<p>What are settings and logs I could check that could give more insight? </p>

<p>Is there any possibility to debug the scheduler?</p>

<p>The problem was that a Pod got stuck in deletion. That caused kube-controller-manager to stop working.</p>

<p>Deletion didn't work because the Pod/RS/Deployment in question had limits that conflicted with the maxLimitRequestRatio that we had set after the creation. A bug report is on the way.</p>

<p>The solution was to increase maxLimitRequestRatio and eventually restart kube-controller-manager.</p>
