<p>I just installed from scratch a small Kubernetes test cluster in a 4 Armbian/Odroid_MC1 (Debian 10) nodes. The install process is this <a href="https://github.com/rodolfoap/odroid-k8s-armbian/blob/master/README.md" rel="nofollow noreferrer">1</a>, nothing fancy or special, adding k8s apt repo and install with apt.</p>
<p>The problem is that the <strong>API server</strong> dies constantly, like every 5 to 10 minutes, after the <strong>controller-manager</strong> and the <strong>scheduler</strong> die together, who seem to stop simultaneously before. Evidently, the API becomes unusable for like a minute. All three services do restart, and things run fine for the next four to nine minutes, when the loop repeats. Logs are here <a href="https://gist.github.com/rodolfoap/f14a8974ad96314ab162bd97478c9198" rel="nofollow noreferrer">2</a>. This is an excerpt:</p>
<pre><code>$ kubectl get pods -o wide --all-namespaces

The connection to the server 192.168.1.91:6443 was refused - did you specify the right host or port?

(a minute later)

$ kubectl get pods -o wide --all-namespaces

NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE   IP             NODE   NOMINATED NODE   READINESS GATES
kube-system   coredns-74ff55c5b-8pm9r       1/1     Running   2          88m   10.244.0.7     mc1    &lt;none&gt;           &lt;none&gt;
kube-system   coredns-74ff55c5b-pxdqz       1/1     Running   2          88m   10.244.0.6     mc1    &lt;none&gt;           &lt;none&gt;
kube-system   etcd-mc1                      1/1     Running   2          88m   192.168.1.91   mc1    &lt;none&gt;           &lt;none&gt;
kube-system   kube-apiserver-mc1            0/1     Running   12         88m   192.168.1.91   mc1    &lt;none&gt;           &lt;none&gt;
kube-system   kube-controller-manager-mc1   1/1     Running   5          31m   192.168.1.91   mc1    &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-fxg2s         1/1     Running   5          45m   192.168.1.94   mc4    &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-jvvmp         1/1     Running   5          48m   192.168.1.92   mc2    &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-qlvbc         1/1     Running   6          45m   192.168.1.93   mc3    &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-ssb9t         1/1     Running   3          77m   192.168.1.91   mc1    &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-7t9ff              1/1     Running   2          45m   192.168.1.93   mc3    &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-8jhc7              1/1     Running   2          88m   192.168.1.91   mc1    &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-cg75m              1/1     Running   2          45m   192.168.1.94   mc4    &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-mq8j7              1/1     Running   2          48m   192.168.1.92   mc2    &lt;none&gt;           &lt;none&gt;
kube-system   kube-scheduler-mc1            1/1     Running   5          31m   192.168.1.91   mc1    &lt;none&gt;           &lt;none&gt;

$ docker ps -a # (check the exited and restarted services)

    CONTAINER ID   NAMES                                                                                                    STATUS                            IMAGE                  NETWORKS   PORTS
    0e179c6495db   k8s_kube-apiserver_kube-apiserver-mc1_kube-system_c55114bd57b1bf357c8f4c0d749ae105_13                    Up About a minute                 66eaad223e2c                      
    2ccb014beb73   k8s_kube-scheduler_kube-scheduler-mc1_kube-system_fe362b2b6b08ca576b7416df7f2e7845_6                     Up 3 minutes                      21e17680ca2d                      
    3322f6ec1546   k8s_kube-controller-manager_kube-controller-manager-mc1_kube-system_17cf17caf36ba27e3d2ec4f113a0cf6f_6   Up 3 minutes                      a1ab72ce4ba2                      
    583129da455f   k8s_kube-apiserver_kube-apiserver-mc1_kube-system_c55114bd57b1bf357c8f4c0d749ae105_12                    Exited (137) About a minute ago   66eaad223e2c                      
    72268d8e1503   k8s_install-cni_kube-flannel-ds-ssb9t_kube-system_dbf3513d-dad2-462d-9107-4813acf9c23a_0                 Exited (0) 5 minutes ago          263b01b3ca1f                      
    fe013d07f186   k8s_kube-controller-manager_kube-controller-manager-mc1_kube-system_17cf17caf36ba27e3d2ec4f113a0cf6f_5   Exited (255) 3 minutes ago        a1ab72ce4ba2                      
    34ef8757b63d   k8s_kube-scheduler_kube-scheduler-mc1_kube-system_fe362b2b6b08ca576b7416df7f2e7845_5                     Exited (255) 3 minutes ago        21e17680ca2d                      
    fd8e0c0ba27f   k8s_coredns_coredns-74ff55c5b-8pm9r_kube-system_3b813dc9-827d-4cf6-88cc-027491b350f1_2                   Up 32 minutes                     15c1a66b013b                      
    f44e2c45ed87   k8s_coredns_coredns-74ff55c5b-pxdqz_kube-system_c3b7fbf2-2064-4f3f-b1b2-dec5dad904b7_2                   Up 32 minutes                     15c1a66b013b                      
    04fa4eca1240   k8s_POD_coredns-74ff55c5b-8pm9r_kube-system_3b813dc9-827d-4cf6-88cc-027491b350f1_42                      Up 32 minutes                     k8s.gcr.io/pause:3.2   none       
    f00c36d6de75   k8s_POD_coredns-74ff55c5b-pxdqz_kube-system_c3b7fbf2-2064-4f3f-b1b2-dec5dad904b7_42                      Up 32 minutes                     k8s.gcr.io/pause:3.2   none       
    a1d6814e1b04   k8s_kube-flannel_kube-flannel-ds-ssb9t_kube-system_dbf3513d-dad2-462d-9107-4813acf9c23a_3                Up 32 minutes                     263b01b3ca1f                      
    94b231456ed7   k8s_kube-proxy_kube-proxy-8jhc7_kube-system_cc637e27-3b14-41bd-9f04-c1779e500a3a_2                       Up 33 minutes                     377de0f45e5c                      
    df91856450bd   k8s_POD_kube-flannel-ds-ssb9t_kube-system_dbf3513d-dad2-462d-9107-4813acf9c23a_2                         Up 34 minutes                     k8s.gcr.io/pause:3.2   host       
    b480b844671a   k8s_POD_kube-proxy-8jhc7_kube-system_cc637e27-3b14-41bd-9f04-c1779e500a3a_2                              Up 34 minutes                     k8s.gcr.io/pause:3.2   host       
    1d4a7bcaad38   k8s_etcd_etcd-mc1_kube-system_14b7b6d6446e21cc57f0b40571ae3958_2                                         Up 35 minutes                     2e91dde7e952                      
    e5d517a9c29d   k8s_POD_kube-controller-manager-mc1_kube-system_17cf17caf36ba27e3d2ec4f113a0cf6f_1                       Up 35 minutes                     k8s.gcr.io/pause:3.2   host       
    3a3da7dbf3ad   k8s_POD_kube-apiserver-mc1_kube-system_c55114bd57b1bf357c8f4c0d749ae105_2                                Up 35 minutes                     k8s.gcr.io/pause:3.2   host       
    eef29cdebf5f   k8s_POD_etcd-mc1_kube-system_14b7b6d6446e21cc57f0b40571ae3958_2                                          Up 35 minutes                     k8s.gcr.io/pause:3.2   host       
    3631d43757bc   k8s_POD_kube-scheduler-mc1_kube-system_fe362b2b6b08ca576b7416df7f2e7845_1                                Up 35 minutes                     k8s.gcr.io/pause:3.2   host       
</code></pre>
<p>I see no weird issues on the logs (I'm a k8s beginner). This was working until a month ago, when I've reinstalled this for practicing, this is probably my tenth install attempt, I've tried different options, versions and googled a lot, but can't find no solution.</p>
<p>What could be the reason? What else can I try? How can I get to the root of the problem?</p>
<h1>UPDATE 2021/02/06</h1>
<p>The problem is not occurring anymore. Apparently, the issue was the version in this specific case. Didn't filed an issue because I didn't found clues regarding what specific issue to report.</p>
<p>The installation procedure in all cases was this:</p>
<pre><code># swapoff -a
# curl -sL get.docker.com|sh
# usermod -aG docker rodolfoap
# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
# echo &quot;deb http://apt.kubernetes.io/ kubernetes-xenial main&quot; &gt; /etc/apt/sources.list.d/kubernetes.list
# apt-get update
# apt-get install -y kubeadm kubectl kubectx # Master
# kubeadm config images pull
# kubeadm init --apiserver-advertise-address=0.0.0.0 --pod-network-cidr=10.244.0.0/16 
</code></pre>
<ul>
<li>Armbian-20.08.1 worked fine. My installation procedure has not changed since.</li>
<li>Armbian-20.11.3 had the issue: the API, scheduler and coredns restarted every 5 minutes, blocking the access to the API 5 of each 8 minutes, average..</li>
<li>Armbian-21.02.1 works fine. Worked at the first install, same procedure.</li>
</ul>
<p>All versions were updated to the last kernel, at the moment of the install, current is 5.10.12-odroidxu4.</p>
<p>As you can see, after around two hours, no API reboots:</p>
<pre><code>NAMESPACE    NAME                         READY  STATUS   RESTARTS  AGE   IP            NODE  LABELS
kube-system  coredns-74ff55c5b-gnvf2      1/1    Running  0         173m  10.244.0.2    mc1   k8s-app=kube-dns,pod-template-hash=74ff55c5b
kube-system  coredns-74ff55c5b-wvnnz      1/1    Running  0         173m  10.244.0.3    mc1   k8s-app=kube-dns,pod-template-hash=74ff55c5b
kube-system  etcd-mc1                     1/1    Running  0         173m  192.168.1.91  mc1   component=etcd,tier=control-plane
kube-system  kube-apiserver-mc1           1/1    Running  0         173m  192.168.1.91  mc1   component=kube-apiserver,tier=control-plane
kube-system  kube-controller-manager-mc1  1/1    Running  0         173m  192.168.1.91  mc1   component=kube-controller-manager,tier=control-plane
kube-system  kube-flannel-ds-c4jgv        1/1    Running  0         123m  192.168.1.93  mc3   app=flannel,controller-revision-hash=64465d999,pod-template-generation=1,tier=node
kube-system  kube-flannel-ds-cl6n5        1/1    Running  0         75m   192.168.1.94  mc4   app=flannel,controller-revision-hash=64465d999,pod-template-generation=1,tier=node
kube-system  kube-flannel-ds-z2nmw        1/1    Running  0         75m   192.168.1.92  mc2   app=flannel,controller-revision-hash=64465d999,pod-template-generation=1,tier=node
kube-system  kube-flannel-ds-zqxh7        1/1    Running  0         150m  192.168.1.91  mc1   app=flannel,controller-revision-hash=64465d999,pod-template-generation=1,tier=node
kube-system  kube-proxy-bd596             1/1    Running  0         75m   192.168.1.94  mc4   controller-revision-hash=b89db7f56,k8s-app=kube-proxy,pod-template-generation=1
kube-system  kube-proxy-n6djp             1/1    Running  0         75m   192.168.1.92  mc2   controller-revision-hash=b89db7f56,k8s-app=kube-proxy,pod-template-generation=1
kube-system  kube-proxy-rf4cr             1/1    Running  0         173m  192.168.1.91  mc1   controller-revision-hash=b89db7f56,k8s-app=kube-proxy,pod-template-generation=1
kube-system  kube-proxy-xhl95             1/1    Running  0         123m  192.168.1.93  mc3   controller-revision-hash=b89db7f56,k8s-app=kube-proxy,pod-template-generation=1
kube-system  kube-scheduler-mc1           1/1    Running  0         173m  192.168.1.91  mc1   component=kube-scheduler,tier=control-plane
</code></pre>
<p>Cluster is fully functional :)</p>

RodolfoAP
<p>I have the same problem, but with Ubuntu:</p>
<pre><code>PRETTY_NAME=&quot;Ubuntu 22.04 LTS&quot;
NAME=&quot;Ubuntu&quot;
VERSION_ID=&quot;22.04&quot;
VERSION=&quot;22.04 (Jammy Jellyfish)&quot;
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL=&quot;https://www.ubuntu.com/&quot;
SUPPORT_URL=&quot;https://help.ubuntu.com/&quot;
BUG_REPORT_URL=&quot;https://bugs.launchpad.net/ubuntu/&quot;
PRIVACY_POLICY_URL=&quot;https://www.ubuntu.com/legal/terms-and-policies/privacy-policy&quot;
UBUNTU_CODENAME=jammy
</code></pre>
<p>The cluster works good for:</p>
<ul>
<li>Ubuntu 20.04 LTS</li>
<li>Ubuntu 18.04 LTS
Thought it will help someone else who is running ubuntu instead of Armbian.</li>
</ul>
<p>Solution for ubuntu (possible for Armbian too) is here: <a href="https://stackoverflow.com/questions/72567945/issues-with-stability-with-kubernetes-cluster-before-adding-networking">Issues with &quot;stability&quot; with Kubernetes cluster before adding networking</a></p>
<p>Apparently it is a problem with the config of containerd on those versions.</p>
<p><strong>UPDATE:</strong></p>
<p>The problem is that if you use sudo apt install containerd, you will install the <strong>version v1.5.9</strong> which has the option <code>SystemdCgroup = false</code> that worked, in my case, in Ubuntu 20.04 but on the Ubuntu 22.04 doesn't work. But if you change it to <code>SystemdCgroup = true</code>it works.(this feature is updated in <strong>containerd v1.6.2</strong> so that it is set on true). This will hopefully fix your problem too.</p>
