<p><strong>Context</strong></p>

<p>I installed <code>Docker</code> following <a href="https://thenewstack.io/how-to-deploy-a-kubernetes-cluster-with-ubuntu-server-18-04/" rel="nofollow noreferrer">this</a> instruction on my <code>Ubuntu 18.04 LTS (Server)</code> and later on <code>Kubernetes</code> followed via <code>kubeadm</code>. After initializing (<code>kubeadm init --pod-network-cidr=10.10.10.10/24</code>) and joining a second node (I got a two node cluster for the start) I cannot get my <em>coredns</em> as well as the later applied <em>Web UI (Dashboard)</em> to actually go into status <em>Running</em>.</p>

<p>As pod network I tried both, <em>Flannel</em> (<code>kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml</code>) and <em>Weave Net</em> - Nothing changed. It still shows status <em>ContainerCreating</em>, even after hours of waiting:</p>

<p><a href="https://i.stack.imgur.com/6n3bV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6n3bV.png" alt="enter image description here"></a></p>

<p><strong>Question</strong></p>

<p>Why doesn't the container creation work as expected and what might be the root cause for this? And most importantly: How do I solve this?</p>

<p><strong>Edit</strong></p>

<p>Summing up my answer below, here are the reasons why:</p>

<ul>
<li>Docker used <code>cgroups</code> instead of <code>systemd</code></li>
<li>I did not configure <code>iptables</code> correctly</li>
<li>I used a wrong <code>kubeadm init</code> since <em>flannels</em> standard-yaml requires <code>--pod-network-cidr</code> to be <code>10.244.0.0/16</code></li>
</ul>

<p>Since answering this questions took me a lot of time, I wanted to share what got me out of this. There might be some more code than necessary, but I also want this to be in one place if I or someone else has to redo all steps.</p>

<p><br>
<br>
<strong>First it all started with Docker...</strong>
<br>
<br></p>

<p>I figured out that it presumably all started with the way I installed <em>Docker</em>. Following the linked online-instructions I used <code>sudo apt-get install docker.io</code> in order to install <em>Docker</em> and used it with <code>cgroups</code> by doing <code>sudo usermod -aG docker $USER</code>.</p>

<p>Well, taking a look at the official instructions from <em>Kubernetes</em> this was a mistake: <code>systemd</code> is the recommended way to go!</p>

<p>So I completly purged all I ever did with docker by following <a href="https://askubuntu.com/a/1021506/968010">these great instructions</a> from 
Mayur Bhandare:</p>

<pre class="lang-sh prettyprint-override"><code>sudo apt-get purge -y docker-engine docker docker.io docker-ce  
sudo apt-get autoremove -y --purge docker-engine docker docker.io docker-ce  
sudo rm -rf /var/lib/docker /etc/docker
sudo rm /etc/apparmor.d/docker
sudo groupdel docker
sudo rm -rf /var/run/docker.sock

# Reboot to be sure
</code></pre>

<p>Afterwards I installed reinstalled <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/" rel="nofollow noreferrer">the official way</a> (keep in mind that this might change in the future):</p>

<pre class="lang-sh prettyprint-override"><code># Install Docker CE
## Set up the repository:
### Install packages to allow apt to use a repository over HTTPS
apt-get update &amp;&amp; apt-get install -y \
  apt-transport-https ca-certificates curl software-properties-common gnupg2

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

### Add Docker apt repository.
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"

## Install Docker CE.
apt-get update &amp;&amp; apt-get install -y \
  containerd.io=1.2.10-3 \
  docker-ce=5:19.03.4~3-0~ubuntu-$(lsb_release -cs) \
  docker-ce-cli=5:19.03.4~3-0~ubuntu-$(lsb_release -cs)

# Setup daemon.
cat &gt; /etc/docker/daemon.json &lt;&lt;EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker
</code></pre>

<p>Note that this explicitly uses <code>systemd</code>!</p>

<p><br>
<br>
<strong>... and then it went on with Flannel...</strong>
<br>
<br></p>

<p>Above I wrote my <code>sudo kubeadm init</code> was done with <code>--pod-network-cidr=10.10.10.10/24</code> since the latter was the IP of my master. 
Well, as pointed out <a href="https://gravitational.com/blog/kubernetes-flannel-dashboard-bomb/" rel="nofollow noreferrer">here</a>  <strong>not using the <a href="https://coreos.com/flannel/docs/latest/kubernetes.html" rel="nofollow noreferrer">official recommended</a> <code>--pod-network-cidr=10.244.0.0/16</code> results in an error</strong> for example using <code>kubectl proxy</code> or the container-creation when using the provided <code>kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml</code>.
This is due to the fact that <code>10.244.0.0/16</code> is hard-linked in the <code>.yaml</code> and, hence, mandatory - Or you just change it in the <code>.yaml</code>.</p>

<p>In order to get rid of the false configuration I did a full reset.
This can be achieved using <code>sudo kubeadm reset</code> and by deleting the config with <code>sudo rm -r ~/.kube/config</code>.
Anyhow, since I screwed it so much, I did a full reset by uninstalling and reinstalling <code>kubeadm</code> and making sure it did use <code>iptables</code> this time (which I also forgot to do before...). </p>

<p><a href="https://stackoverflow.com/a/49253264/3757672">Here</a> is a nice link how to fully uninstall all kubeadm-parts.</p>

<pre class="lang-sh prettyprint-override"><code>kubeadm reset
sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*   
sudo apt-get autoremove  
sudo rm -rf ~/.kube
</code></pre>

<p>For the sake of completeness, here is the reinstall as well:</p>

<pre class="lang-sh prettyprint-override"><code># ensure legacy binaries are installed
sudo apt-get install -y iptables arptables ebtables

# switch to legacy versions
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

# Install Kubernetes with kubeadm
sudo apt-get update &amp;&amp; sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat &lt;&lt;EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

#reboot
</code></pre>

<p><br>
<br>
<strong>... and finally it worked!</strong>
<br>
<br></p>

<p>After the clean reinstallation I did the following:</p>

<pre><code># Initialize with correct cidr
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml
</code></pre>

<p>And then be astouned by the result:</p>

<pre><code>kubectl get pods --all-namespaces
</code></pre>

<p><a href="https://i.stack.imgur.com/pyQy4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pyQy4.png" alt="enter image description here"></a></p>

<p>On a site note: This also resolved the <code>/run/flannel/subnet.env: no such file or directory</code>-error I encountered prior to these steps when describing the uncreated coredns.</p>
