<p>OVERVIEW:: I am studying for the Kubernetes Administrator certification. To complete the training course, I created a dual node Kubernetes cluster on Google Cloud, 1 master and 1 slave. As I don't want to leave the instances alive all the time, I took snapshots of them to deploy new instances with the Kubernetes cluster already setup. I am aware that I would need to update the ens4 ip used by kubectl, as this will have changed, which I did.</p>

<p>ISSUE:: When I run "kubectl get pods --all-namespaces" I get the error "The connection to the server localhost:8080 was refused - did you specify the right host or port?"</p>

<p>QUESTION:: Would anyone have had similar issues and know if its possible to recreate a Kubernetes cluster from snapshots?</p>

<p>Adding -v=10 to command, the url matches info in .kube/config file</p>

<blockquote>
  <p>kubectl get pods --all-namespaces -v=10
  I0214 17:11:35.317678    6246 loader.go:375] Config loaded from file:  /home/student/.kube/config
  I0214 17:11:35.321941    6246 round_trippers.go:423] curl -k -v -XGET  -H "User-Agent: kubectl/v1.16.1 (linux/amd64) kubernetes/d647ddb" -H "Accept: application/json, <em>/</em>" '<a href="https://k8smaster:6443/api?timeout=32s" rel="nofollow noreferrer">https://k8smaster:6443/api?timeout=32s</a>'
  I0214 17:11:35.333308    6246 round_trippers.go:443] GET <a href="https://k8smaster:6443/api?timeout=32s" rel="nofollow noreferrer">https://k8smaster:6443/api?timeout=32s</a>  in 11 milliseconds
  I0214 17:11:35.333335    6246 round_trippers.go:449] Response Headers:
  I0214 17:11:35.333422    6246 cached_discovery.go:121] skipped caching discovery info due to Get <a href="https://k8smaster:6443/api?timeout=32s" rel="nofollow noreferrer">https://k8smaster:6443/api?timeout=32s</a>: dial tcp 10.128.0.7:6443: connect: connection refused
  I0214 17:11:35.333858    6246 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, <em>/</em>" -H "User-Agent: kubectl/v1.16.1 (linux/amd64) kubernetes/d647ddb" '<a href="https://k8smaster:6443/api?timeout=32s" rel="nofollow noreferrer">https://k8smaster:6443/api?timeout=32s</a>'
  I0214 17:11:35.334234    6246 round_trippers.go:443] GET <a href="https://k8smaster:6443/api?timeout=32s" rel="nofollow noreferrer">https://k8smaster:6443/api?timeout=32s</a>  in 0 milliseconds
  I0214 17:11:35.334254    6246 round_trippers.go:449] Response Headers:
  I0214 17:11:35.334281    6246 cached_discovery.go:121] skipped caching discovery info due to Get <a href="https://k8smaster:6443/api?timeout=32s" rel="nofollow noreferrer">https://k8smaster:6443/api?timeout=32s</a>: dial tcp 10.128.0.7:6443: connect: connection refused
  I0214 17:11:35.334303    6246 shortcut.go:89] Error loading discovery information: Get <a href="https://k8smaster:6443/api?timeout=32s" rel="nofollow noreferrer">https://k8smaster:6443/api?timeout=32s</a>: dial tcp 10.128.0.7:6443: connect: connection refused</p>
</blockquote>

liam08
<p>I replicated you issue and wrote this step by step debugging process for you so you can see what was my thinking.</p>

<p>I created 2 node cluster (master + worker) with kubeadm and made a snapshot.
Then I deleted all nodes and recreated them from snapshots.</p>

<p>After recreating master node from snapshot I started seeing the same error you are seeing:</p>

<pre><code>@kmaster ~]$ kubectl get po -v=10
I0217 11:04:38.397823    3372 loader.go:375] Config loaded from file:  /home/user/.kube/config
I0217 11:04:38.398909    3372 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://10.156.0.20:6443/api?timeout=32s'
^C
</code></pre>

<p>The connection was hanging so I interrupted it (ctrl+c).
First I noticed was that IP address of where kubectl was connecting was different than node ip, so I modified <code>.kube/config</code> file providing proper IP.</p>

<p>After doing this, here is what running kubectl showed:</p>

<pre><code>$ kubectl get po -v=10
I0217 11:26:57.020744   15929 loader.go:375] Config loaded from file:  /home/user/.kube/config
...
I0217 11:26:57.025155   15929 helpers.go:221] Connection error: Get https://10.156.0.23:6443/api?timeout=32s: dial tcp 10.156.0.23:6443: connect: connection refused
F0217 11:26:57.025201   15929 helpers.go:114] The connection to the server 10.156.0.23:6443 was refused - did you specify the right host or port?
</code></pre>

<p>As you see, connection to apiserver was beeing refused so I checked if apiserver was running:</p>

<pre><code>$ sudo docker ps -a | grep apiserver
5e957ff48d11        90d27391b780             "kube-apiserver --ad…"   24 seconds ago      Exited (2) 3 seconds ago                           k8s_kube-apiserver_kube-apiserver-kmaster_kube-system_997514ff25ec38012de6a5be7c43b0ae_14
d78e179f1565        k8s.gcr.io/pause:3.1     "/pause"                 26 minutes ago      Up 26 minutes                                      k8s_POD_kube-apiserver-kmaster_kube-system_997514ff25ec38012de6a5be7c43b0ae_1
</code></pre>

<p>api-server was exiting for some reason.
I checked its logs (I am only including relevant logs for readability):</p>

<pre><code>$ sudo docker logs 5e957ff48d11
...
W0217 11:30:46.710541       1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379 0  &lt;nil&gt;}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
panic: context deadline exceeded
</code></pre>

<p>Notice apiserver was trying to connect to etcd (notice port: 2379) and receiving connection refused.
My first guess was etcd wasn't running, so I checked etcd container:</p>

<pre><code>$ sudo docker ps -a | grep etcd
4a249cb0743b        303ce5db0e90             "etcd --advertise-cl…"   2 minutes ago        Exited (1) 2 minutes ago                           k8s_etcd_etcd-kmaster_kube-system_9018aafee02ebb028a7befd10063ec1e_19
b89b7e7227de        k8s.gcr.io/pause:3.1     "/pause"                 30 minutes ago       Up 30 minutes                                      k8s_POD_etcd-kmaster_kube-system_9018aafee02ebb028a7befd10063ec1e_1
</code></pre>

<p>I was right: <em>Exited (1) 2 minutes ago</em>. I checked its logs:</p>

<pre><code>$ sudo docker logs 4a249cb0743b
...
2020-02-17 11:34:31.493215 C | etcdmain: listen tcp 10.156.0.20:2380: bind: cannot assign requested address
</code></pre>

<p>etcd was trying to bind with old IP address.</p>

<p>I modified <code>/etc/kubernetes/manifests/etcd.yaml</code> and changed old IP address to new IP everywhere in file.</p>

<p>Quick <code>sudo docker ps | grep etcd</code> showed its running.
After a while apierver also started running.</p>

<p>Then I tried running kubectl:</p>

<pre><code>$ kubectl get po
Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 10.156.0.20, not 10.156.0.23
</code></pre>

<p>Invalid apiserver certificate. SSL certificate was genereated for old IP so that would mean I need to generate new certificate with new IP.</p>

<pre><code>$ sudo kubeadm init phase certs apiserver
...
[certs] Using existing apiserver certificate and key on disk
</code></pre>

<p>That's not what I expected. I wanted to generate new certificates, not use old ones.</p>

<p>I deleted old certificates:</p>

<pre><code>$ sudo rm /etc/kubernetes/pki/apiserver.crt \
          /etc/kubernetes/pki/apiserver.key
</code></pre>

<p>And tried to generate certificates one more time:</p>

<pre><code>$ sudo kubeadm init phase certs apiserver
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kmaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.156.0.23]
</code></pre>

<p>Looks good. Now let's try using kubectl:</p>

<pre><code>$ kubectl get no
NAME          STATUS   ROLES    AGE    VERSION
instance-21   Ready    master   102m   v1.17.3
instance-22   Ready    &lt;none&gt;   95m    v1.17.3
</code></pre>

<p>As you can see now its working.</p>
