<p>I have updated the master of my CentOS 7 based Kubernetes cluster from version 1.15 to 1.16.8. I have followed the official <a href="https://v1-16.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/" rel="nofollow noreferrer">manual</a>.</p>

<p>After the update was successfully completed, I restarted the master. Since then I get a message with every kubectl command that the API is not available. The system information of kubelet gives me the same messages. </p>

<pre class="lang-sh prettyprint-override"><code>$ kubectl get pods
The connection to the server 10.6.231.50:6443 was refused - did you specify the right host or port?
$
$
$ journalctl -u kubelet.service -f
-- Logs begin at Fri 2020-01-31 13:50:09 CET. --
Jan 31 16:52:40 orbisos010 kubelet[14818]: E0131 16:52:40.137787   14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://10.6.231.50:6443/api/v1/nodes?fieldSelector=metadata.name%3Dorbisos010&amp;limit=500&amp;resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:41 orbisos010 kubelet[14818]: E0131 16:52:41.130807   14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://10.6.231.50:6443/api/v1/services?limit=500&amp;resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:41 orbisos010 kubelet[14818]: E0131 16:52:41.137195   14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.6.231.50:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dorbisos010&amp;limit=500&amp;resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:41 orbisos010 kubelet[14818]: E0131 16:52:41.138917   14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://10.6.231.50:6443/api/v1/nodes?fieldSelector=metadata.name%3Dorbisos010&amp;limit=500&amp;resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:42 orbisos010 kubelet[14818]: E0131 16:52:42.132181   14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://10.6.231.50:6443/api/v1/services?limit=500&amp;resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
Jan 31 16:52:42 orbisos010 kubelet[14818]: E0131 16:52:42.138261   14818 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.6.231.50:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dorbisos010&amp;limit=500&amp;resourceVersion=0: dial tcp 10.6.231.50:6443: connect: connection refused
</code></pre>

<p>I have configured the master with the following command:</p>

<pre class="lang-sh prettyprint-override"><code>$ kubeadm init \
  --apiserver-advertise-address=10.6.231.50 \
  --pod-network-cidr=10.244.0.0/16 \
  --service-cidr=10.96.0.0/12 \
  --service-dns-domain=k8s.example.local
</code></pre>

<p>Additionally I set in <code>/etc/sysconfig/kubelet</code> EXTRA_ARGS to define crio as container runtime and systemd as cgroups driver.</p>

<pre class="lang-sh prettyprint-override"><code>KUBELET_EXTRA_ARGS=--container-runtime-endpoint=unix:///var/run/crio/crio.sock --cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice
</code></pre>

<p>Version 1.13.11 of crio is installed.</p>

<pre class="lang-sh prettyprint-override"><code>$ crio --version
crio version 1.13.11
</code></pre>

<p><strong>EDIT: swap partitions are disabled/removed</strong></p>

<pre class="lang-sh prettyprint-override"><code>$ cat /proc/swaps 
Filename                Type        Size    Used    Priority
$ cat /etc/fstab 
#
# /etc/fstab
# Created by anaconda on Fri Apr  5 09:20:29 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=1725c823-3764-4e25-ba35-c638d6d34335 /                       ext4    defaults        1 1
</code></pre>

<p><strong>EDIT: Containers are currently not running after upgrade - k8s-api-server logs</strong></p>

<pre class="lang-sh prettyprint-override"><code>$ crictl logs 7c8d8481003b9
I0313 07:35:07.782793       1 main.go:514] Determining IP address of default interface
I0313 07:35:07.785403       1 main.go:527] Using interface with name eth0 and address 10.6.231.50
I0313 07:35:07.785455       1 main.go:544] Defaulting external address to interface address (10.6.231.50)
I0313 07:35:07.900430       1 kube.go:126] Waiting 10m0s for node controller to sync
I0313 07:35:07.900538       1 kube.go:309] Starting kube subnet manager
I0313 07:35:08.901017       1 kube.go:133] Node controller sync successful
I0313 07:35:08.901094       1 main.go:244] Created subnet manager: Kubernetes Subnet Manager - orbisos010
I0313 07:35:08.901112       1 main.go:247] Installing signal handlers
I0313 07:35:08.903298       1 main.go:386] Found network config - Backend type: vxlan
I0313 07:35:08.903501       1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
I0313 07:35:08.996070       1 main.go:351] Current network or subnet (10.244.0.0/16, 10.244.0.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I0313 07:35:09.005711       1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0313 07:35:09.007865       1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0313 07:35:09.082996       1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0313 07:35:09.084506       1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully
I0313 07:35:09.087178       1 main.go:317] Wrote subnet file to /run/flannel/subnet.env
I0313 07:35:09.087192       1 main.go:321] Running backend.
I0313 07:35:09.087202       1 main.go:339] Waiting for all goroutines to exit
I0313 07:35:09.087227       1 vxlan_network.go:60] watching for new subnet leases
I0313 07:35:09.183701       1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0313 07:35:09.183790       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0313 07:35:09.282337       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0313 07:35:09.282804       1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0313 07:35:09.282825       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.285677       1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
I0313 07:35:09.286809       1 iptables.go:167] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.288137       1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
I0313 07:35:09.289182       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.382765       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0313 07:35:09.488057       1 iptables.go:155] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0313 07:35:09.491426       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0313 07:35:09.589312       1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
I0313 07:35:09.687180       1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
E0316 07:20:25.108176       1 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to watch *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=6128136&amp;timeoutSeconds=450&amp;watch=true: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:20:26.112371       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:50:17.738298       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=11, ErrCode=NO_ERROR, debug=""
E0316 07:50:17.739007       1 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to watch *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=6131353&amp;timeoutSeconds=510&amp;watch=true: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:50:18.739790       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0316 07:50:19.741013       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:20.742425       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:21.748815       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:22.751387       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:23.752835       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:24.754022       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:25.755338       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:26.756589       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:27.757552       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:28.759233       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:29.760553       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:30.761582       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:31.762942       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:32.764278       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:33.765529       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:34.782413       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:35.783639       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:36.785241       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:37.787396       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
I0316 07:50:37.884172       1 main.go:370] shutdownHandler sent cancel signal...
E0316 07:50:38.788705       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:39.790020       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:40.791467       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:41.792689       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:42.793788       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:43.795046       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:44.797485       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:45.799345       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:46.800897       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
E0316 07:50:47.801961       1 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:310: Failed to list *v1.Node: Get https://10.96.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 10.96.0.1:443: connect: network is unreachable
</code></pre>

<p>Any Ideas why I can not connect to 10.6.231.50:6443? Are there any breaking changes in Version 1.16.8?</p>

<p>Volker</p>

<p>This issue usually happens when swapping is enabled so run: <code>swapoff -a</code> on your node. After restarting your node swapping was enabled so you just had to run this command again to disable it. </p>
