<p>Kubernetes Dashboard works fine before a large number of Pods (more than 1000) are scheduled using a Deployment.  </p>

<p>When many Pods are run, I get the following error when trying to click on any page in the Dashboard:<br>
<code>Error: 'dial tcp &lt;ip&gt;:9090: getsockopt: connection refused'
Trying to reach: 'http://&lt;ip&gt;:9090/api/v1/workload/default?itemsPerPage=10&amp;page=1'</code>  </p>

<p>Does increasing the spec of the master node help?  </p>

<p>I am thinking the master node cannot handle the network load?  </p>

<p><strong>Update:</strong><br>
might be related, but I'm not sure, when I ran the cluster (I'm using <code>kops</code>) on <em>many</em> medium sized instances instead of a few very large ones, this problem is reduced (I get this error page less).<br>
It seems like the Pod controller on each Node (or whatever you call it) cannot handle so many pods, since a very highly specced Node has many Pods on it whereas a medium sized one will have fewer Pods scheduled on it.  </p>

cryanbhu
<p>It definitely looks like the kubernetes-dashboard pod experienced lack of resources. The possible solution would be to run kubernetes-dashboard pod on the dedicated node or on the node with less resource consumers. </p>

<p><a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">Taints</a> or <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/" rel="nofollow noreferrer">Affinity</a> features might be helpful to achieve that. </p>
