<p>I'm running a Kubernetes cluster, which has worked fine for several months. Now, today, when I was about to deploy some updates, I get timeouts from the server.</p>

<p>Running <code>$ kubectl get nodes</code> yields</p>

<pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)
</code></pre>

<p>Running <code>$ kubectl get pods --all-namespaces</code> yields</p>

<pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)
</code></pre>

<p>Running <code>$ kubectl get deployments</code> yields</p>

<pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get deployments.extensions)
</code></pre>

<p>Running <code>$ kubectl get svc</code> yields</p>

<pre><code>Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get services)
</code></pre>

<p>Running <code>$ kubectl cluster-info</code> yields (note no output after the master)</p>

<pre><code>Kubernetes master is running at https://cluster.mysite.com

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
</code></pre>

<p>As I get these timeouts for every command, troubleshooting is impossible. </p>

<p>How can I continue from here to access my servers? I'm using <code>kube-aws</code>, and an AWS CloudFormation VPC.</p>

<p>Thanks for your time.</p>

<p><strong>EDIT</strong>:</p>

<p>As per request, I ran <code>$ kubectl get pods -v 7</code> and after a bunch of cache returns got this:</p>

<pre><code>I0103 16:51:32.196859 25644 round_trippers.go:414] GET cluster.mysite.com/api/v1/nodes
I0103 16:51:32.196888 25644 round_trippers.go:421] Request Headers: 
I0103 16:51:32.196894 25644 round_trippers.go:424]     Accept: application/json
I0103 16:51:32.196899 25644 round_trippers.go:424]     User-Agent: kubectl/v1.8.3 (darwin/amd64) kubernetes/f0efb3c
I0103 16:52:32.239841 25644 round_trippers.go:439]     Response Status: 504 Gateway Timeout in 60044 milliseconds
</code></pre>

<p>I also ran <code>$ kubectl cluster-info dump -v 7</code> and got:</p>

<pre><code>I0103 16:51:32.196888   25644 round_trippers.go:421] Request Headers:
I0103 16:51:32.196894   25644 round_trippers.go:424]     Accept: application/json
I0103 16:51:32.196899   25644 round_trippers.go:424]     User-Agent: kubectl/v1.8.3 (darwin/amd64) kubernetes/f0efb3c
I0103 16:52:32.239841   25644 round_trippers.go:439] Response Status: 504 Gateway Timeout in 60044 milliseconds
I0103 16:52:32.242362   25644 helpers.go:207] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)",
  "reason": "Timeout",
  "details": {
    "kind": "nodes",
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "{\"metadata\":{},\"status\":\"Failure\",\"message\":\"The list operation against nodes could not be completed at this time, please try again.\",\"reason\":\"ServerTimeout\",\"details\":{\"name\":\"list\",\"kind\":\"nodes\"},\"code\":500}"
      }
    ]
  },
  "code": 504
}]
</code></pre>

<p><strong>EDIT 2:</strong>
Okay, now I'm just getting <code>Unable to connect to the server: EOF</code> on every request and I'm starting to get scared. This is a production cluster and I can't even access it to try to troubleshoot. Anyone have a hint on how to proceed?</p>

<p><strong>EDIT 3:</strong>
I've gotten as far as realizing that the etcd cluster was not working properly, with 2/3 nodes out of sync. Restarting one node had it properly joining the cluster again, but the second one can't start the services. The services that don't start are:</p>

<ul>
<li>etcdadm-check.service</li>
<li>etcdadm-save.service</li>
<li>etcdadm-update-status.service</li>
<li>user@0.service</li>
</ul>

<p>The three former ones all give the error <code>etcdadm-check.service: Control process exited, code=exited status=3</code> and the last one gives <code>user@0.service: Start request repeated too quickly.</code>.</p>

<p>Any hints on how to handle this?</p>

<p>Also, after restoring the second etcd, I get <code>Unable to connect to the server: x509: certificate signed by unknown authority</code> when running any <code>kubectl</code> commands. Does this signify data loss? My certificates are still valid for over half a year, and I haven't changed anything about them.</p>

<p><strong>EDIT 4</strong>:
I still have the etcd-issue, but am following the instructions in camil's answer at this time, will update with the result. However, I solved the issue with the certificates not being valid simply by re-running <code>$ kube-aws render credentials</code> with the proper paths to my intermediate root CA, so that issue is solved.</p>

<p>To avoid the timeouts, you can pass this flag <code>--request-timeout='1s'</code>. This will allow further debugging. </p>

<p>I see you are running <code>kube-aws</code>,so it will be safe to terminate the master instances (at least one, if you run multiple masters). The ASG will replace them automatically. You can do this also with the ETCD nodes.</p>

<p>If the issue still persists, then you have to ssh into masters and check the logs and services by running commands like:</p>

<pre><code>journalctl -xe
systemctl status -l kubelet.service
systemctl status -l flanneld.service
systemctl status -l docker.service
rkt list
</code></pre>

<p>You can also use this function to debug using <code>kubectl</code> from inside the masters:</p>

<pre><code>kubectl() {
/usr/bin/docker run --rm --net=host \
  -v /etc/resolv.conf:/etc/resolv.conf \
  -v /srv/kube-aws/plugins:/srv/kube-aws/plugins \
  quay.io/coreos/hyperkube:v1.9.0_coreos.0 /hyperkube kubectl "$@"
}
</code></pre>

<p>Then try these commands: </p>

<pre><code>kubectl get componentstatus
kubectl cluster-info
kubectl get pods -n kube-system
kubectl get events -n kube-system
</code></pre>

<p>Check the connectivity to ETCD from masters</p>

<pre><code>export $(cat /etc/etcd-environment | tr -d "'")

/usr/bin/etcdctl \
--ca-file=/etc/kubernetes/ssl/etcd-trusted-ca.pem \
--cert-file=/etc/kubernetes/ssl/etcd-client.pem \
--key-file=/etc/kubernetes/ssl/etcd-client-key.pem \
--endpoints="${ETCD_ENDPOINTS}" \
cluster-health
</code></pre>
