<p>Good afternoon</p>
<p>I have two pods that are not lifting due to the following errors:</p>
<pre><code>prometheus-k8s-0 0/3 Pending 0 56s
prometheus-k8s-1 2/3 Running 0 69s
</code></pre>
<p><code>Warning FailedScheduling &lt;unknown&gt; default-scheduler 0/4 nodes are available: 1 node (s) had taint {node.kubernetes.io/memory-pressure:}, that the pod didn't tolerate, 3 node (s) had volume node affinity conflict</code>.</p>
<pre><code>kubectl describe nodes aks-bbagentpool-39730754-vmss000000 | grep -i taint
Taints: node.kubernetes.io/memory-pressure:NoSchedule
</code></pre>
<p>I do not see high consumption in any of the nodes:</p>
<pre><code>kubectl top nodes

NAME CPU (cores) CPU% MEMORY (bytes) MEMORY%

aks-bbagentpool-39730754-vmss000000 153m 8% 920Mi 17%
aks-bbagentpool-39730754-vmss000001 1854m 97% 5357Mi 99%
aks-bbinfra-39730754-vmss000000 211m 5% 2038Mi 16%
aks-bbinfra-39730754-vmss000001 111m 2% 1706Mi 13%
</code></pre>
<p>The prometheus solution is from Coreos and I see that the daemonset is operational</p>
<pre><code>kubectl get daemonsets.apps -n monitoring

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE

node-exporter 4 4 4 4 4 kubernetes.io/os=linux 3d13h
</code></pre>
<p>I want to know the reason for this failure and how I can resolve the status of the nodes.</p>

<p>Taint <strong>node.kubernetes.io/memory-pressure</strong> will be automatically added by the node controller when there is a no enough memory to schedule any new pods.</p>
<p>In your case the particular node would have been reached memory issue at some point and this taint has been added by node controller to prevent the node from attracting any new pod scheduling, and in due course the memory would have been released and came back to normal. Taints can be applied automatically by controller based on meeting certain condition but it will not be removed automatically.</p>
<p>You got two options to fix this.</p>
<ol>
<li>remove the taint from the node <code>kubectl taint nodes aks-bbagentpool-39730754-vmss000000 node.kubernetes.io/memory-pressure:NoSchedule-</code></li>
<li>Or add toleration to your pod (which I personally don't prefer)</li>
</ol>
<p>You can get more details on <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/" rel="nofollow noreferrer">Docs</a> and <a href="https://sysdig.com/blog/kubernetes-pod-evicted/" rel="nofollow noreferrer">Blog</a></p>
