<p>I have kubeadm cluster deployed in CentOS VM. while trying to deploy <code>ingress controller</code> following <a href="https://github.com/kubernetes/ingress-nginx/blob/master/deploy/README.md" rel="nofollow noreferrer">github</a> i noticed that i'm unable to see logs: </p>

<pre><code>kubectl logs -n ingress-nginx nginx-ingress-controller-697f7c6ddb-x9xkh --previous

Error from server: Get https://192.168.56.34:10250/containerLogs/ingress-nginx/nginx-ingress-controller-697f7c6ddb-x9xkh/nginx-ingress-controller?previous=true: dial tcp 192.168.56.34:10250: getsockopt: connection timed out
</code></pre>

<p>In 192.168.56.34 (node1) netstat returns: </p>

<pre><code>tcp6       0      0 :::10250                :::*                    LISTEN      1068/kubelet
</code></pre>

<p>In fact i'm unable to see any logs despite the status of the pod.</p>

<p>I disabled both the <code>firewalld</code> and  <code>SELinux</code>. </p>

<p>I used <code>proxy</code> to enable kubernertes to download images, now i removed the proxy.</p>

<p>When navigating to the url in the error above i get <code>Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy)</code></p>

<p>I'm also able to fetch my nodes: </p>

<pre><code>kubectl get node
NAME         STATUS     ROLES     AGE       VERSION
k8s-master   Ready      master    32d       v1.9.3
k8s-node1    Ready      &lt;none&gt;    30d       v1.9.3
k8s-node2    NotReady   &lt;none&gt;    32d       v1.9.3
</code></pre>

<blockquote>
  <p>getsockopt: connection timed out</p>
</blockquote>

<p>Is 99.99999% a firewall issue. If it was "connection refused" then showing the output of netstat would be meaningful, but (as you can see) <code>kubelet</code> is listening on that port just fine -- it's the networking configuration between the machine that is running <code>kubectl</code> and "192.168.56.34" that is incorrectly configured to allow traffic.</p>

<p>The apiserver expects that everyone who would want to view logs (or use <code>kubectl exec</code>) can reach that port on every Node in the cluster; so be sure you don't just fix the firewall rule(s) for that one Node -- fix it for all of them.</p>
