<p>I have a cluster that was recently upgraded. Since then none of the pods running in the cluster are able to get a response from <a href="https://kubernetes.default/healthz" rel="nofollow noreferrer">https://kubernetes.default/healthz</a>.</p>

<p>To be clear, they can <strong>resolve</strong> the URL, but it constantly times out of comes back with connection refused.</p>

<p>I have verified that the api-server is running as I can get a response from <a href="http://localhost:8080/healthz" rel="nofollow noreferrer">http://localhost:8080/healthz</a>, but I can't get anything from within a pod.</p>

<p>I've checked all the scripts and configs and compared them to the other clusters that were upgraded at the same time and there appears to be nothing different. </p>

<p>I'm sure it's something small I've overlooked but I don't know where else to look. </p>

<p>Additional information:</p>

<ul>
<li>setup was with kops </li>
<li>runs in AWS (not the managed services but raw)</li>
<li>the above queries work as expected in other clusters</li>
<li>upgrade was from 1.11 to 1.13 (yes, I know that is frowned upon. At least now I do.)</li>
<li>other clusters had same upgrade path</li>
</ul>

<p>[edit] </p>

<p>providing /etc/resolv.conf</p>

<pre><code>cat /etc/resolv.conf
nameserver 100.64.0.10
search jenkins.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
options ndots:5
</code></pre>

<p>Check the image I uploaded, you can only resolve the <strong>dns</strong> <code>kubernetes</code> from within a pod in the <code>default</code> namespace. For other namespaces use <code>kubernetes.default.svc</code> like in the following curl. You have the certificate and token already mounted in the pod as well. </p>

<p><code>curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -H "Accept: application/json"  https://kubernetes.default.svc/api/</code></p>

<p><a href="https://i.stack.imgur.com/trffc.png" rel="noreferrer"><img src="https://i.stack.imgur.com/trffc.png" alt="Image"></a></p>
