<p>I ran Kubespray in lxc containers with below configuration:<em>(server_ram:8G | all nodes in ubuntu:18.04)</em></p>

<pre><code>|  NAME   |  STATE  |         IPV4  
+---------+---------+-------------------         
| ansible | RUNNING | 10.21.185.23 (eth0)  
| node1   | RUNNING | 10.21.185.158 (eth0)  
| node2   | RUNNING | 10.21.185.186 (eth0)   
| node3   | RUNNING | 10.21.185.65 (eth0)  
| node4   | RUNNING | 10.21.185.106 (eth0)  
| node5   | RUNNING | 10.21.185.14 (eth0) 
</code></pre>

<blockquote>
  <p>In root@ansible: when i ran kubespray command to build cluster i encountered with this Error:</p>
</blockquote>

<pre><code>
TASK [kubernetes/preinstall : Disable swap] ******************
fatal: [node1]: FAILED! =&gt; {"changed": true, "cmd": ["/sbin/swapoff", "-a"], "delta": "0:00:00.020302", "end": "2020-05-13 07:21:24.974910", "msg": "non-zero return code", "rc": 255, "start": "2020-05-13 07:21:24.954608", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
fatal: [node2]: FAILED! =&gt; {"changed": true, "cmd": ["/sbin/swapoff", "-a"], "delta": "0:00:00.010084", "end": "2020-05-13 07:21:25.051443", "msg": "non-zero return code", "rc": 255, "start": "2020-05-13 07:21:25.041359", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
fatal: [node3]: FAILED! =&gt; {"changed": true, "cmd": ["/sbin/swapoff", "-a"], "delta": "0:00:00.008382", "end": "2020-05-13 07:21:25.126695", "msg": "non-zero return code", "rc": 255, "start": "2020-05-13 07:21:25.118313", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
fatal: [node4]: FAILED! =&gt; {"changed": true, "cmd": ["/sbin/swapoff", "-a"], "delta": "0:00:00.006829", "end": "2020-05-13 07:21:25.196145", "msg": "non-zero return code", "rc": 255, "start": "2020-05-13 07:21:25.189316", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
</code></pre>

<blockquote>
  <p>lxc containers configuration:(include:node1,node2,node3,node4,node5)</p>
</blockquote>

<pre><code>architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20200506)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20200506"
  image.version: "18.04"
  limits.cpu: "2"
  limits.memory: 2GB
  limits.memory.swap: "false"
  linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
  raw.lxc: "lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw
    sys:rw"
  security.nesting: "true"
  security.privileged: "true"
  volatile.base_image: 93b9eeb85479af2029203b4a56a2f1fdca6a0e1bf23cdc26b567790bf0f3f3bd
  volatile.eth0.hwaddr: 00:16:3e:5a:91:9a
  volatile.idmap.base: "0"
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
</code></pre>

<blockquote>
  <p>When i try to swapoff manually in nodes i receive nothing.</p>
</blockquote>

<pre><code>root@node1:~# /sbin/swapoff -a
root@node1:~#
</code></pre>

<p>it will be so helpful if anyone has an idea.</p>

Sajjad Hadafi
<p>I divided this answer on 2 parts: </p>

<ul>
<li><strong>TL;DR</strong> Why Kubespray fails on <code>swapoff -a</code></li>
<li>How to install Kubernetes with Kubespray on LXC containers </li>
</ul>

<hr>

<h2>TL;DR</h2>

<p><code>Kubespray</code> fails because he gets non exit zero code (255) when running <code>swapoff -a</code>. </p>

<blockquote>
  <p>A non-zero exit status indicates failure. This seemingly counter-intuitive scheme is used so there is one well-defined way to indicate success and a variety of ways to indicate various failure modes. </p>
  
  <p> <em><a href="https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html" rel="nofollow noreferrer">Gnu.org: Exit Status</a></em>  </p>
</blockquote>

<p>Even if you set <code>limits.memory.swap: "false"</code> in the profile associated with the containers it will still produce this error. </p>

<p>There is a workaround for it by disabling swap in your <strong>host</strong> system. 
You can do it by: </p>

<ul>
<li><code>$ swapoff -a</code></li>
<li>delete line associated with swap in <code>/etc/fstab</code></li>
<li><code>$ reboot</code></li>
</ul>

<p>After that your container should produce <strong>zero exit code</strong> when issuing
<code>$ swapoff -a</code></p>

<hr>

<h2>How to install Kubernetes with Kubespray on LXC containers</h2>

<p><strong>Assuming that you created your <code>lxc</code> containers and have full ssh access to them</strong>, there are still things to take into consideration before running <code>kubespray</code>. </p>

<p>I ran <code>kubespray</code> on <code>lxc</code> containers and stumbled upon issues with: </p>

<ul>
<li>storage space</li>
<li>docker packages</li>
<li><code>kmsg</code></li>
<li>kernel modules </li>
<li><code>conntrack</code> </li>
</ul>

<h3>Storage space</h3>

<p>Please make sure you have enough storage within your storage pool as lack of it will result in failure to provision the cluster. Default storage pool size could be not big enough to hold 5 nodes. </p>

<h3>Docker packages</h3>

<p>When provisioning the cluster please make sure that you have the newest <code>kubespray</code> version available as the older ones had an issue with docker packages not compatible with each other. </p>

<h3>Kmsg</h3>

<blockquote>
  <p>The /dev/kmsg character device node provides userspace access to the kernel's printk buffer.</p>
  
  <p> <em><a href="https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg" rel="nofollow noreferrer">Kernel.org: Documentation: dev-kmsg</a></em>  </p>
</blockquote>

<p>By default <code>kubespray</code> will fail to provision the cluster when the <code>/dev/kmsg</code> is not available on the node (lxc container). </p>

<p><code>/dev/kmsg</code> is not available on <code>lxc</code> container and this will cause a failure of <code>kubespray</code> provisioning. </p>

<p>There is a workaround for it. <strong>In each <code>lxc</code> container run</strong>: </p>

<blockquote>
  <pre class="lang-sh prettyprint-override"><code># Hack required to provision K8s v1.15+ in LXC containers
mknod /dev/kmsg c 1 11
chmod +x /etc/rc.d/rc.local
echo 'mknod /dev/kmsg c 1 11' &gt;&gt; /etc/rc.d/rc.local
</code></pre>
  
  <p> <em><a href="https://github.com/justmeandopensource/kubernetes/blob/master/lxd-provisioning/bootstrap-kube.sh" rel="nofollow noreferrer">Github.com: Justmeandopensource: lxd-provisioning: bootstrap-kube.sh</a></em>  </p>
</blockquote>

<p>I tried other workarounds like: </p>

<ul>
<li>add <code>lxc.kmsg = 1</code> to <code>/etc/lxc/default.conf</code> - <a href="https://github.com/lxc/lxd/issues/4393#issuecomment-378181793" rel="nofollow noreferrer">deprecated</a> </li>
<li>running <code>echo 'L /dev/kmsg - - - - /dev/console' &gt; /etc/tmpfiles.d/kmsg.conf</code> inside the container and then restarting is causing the <code>systemd-journald</code> to sit at 100% usage of a core.</li>
</ul>

<h3>Kernel modules</h3>

<blockquote>
  <p>The LXC/LXD system containers do not load kernel modules for their own use.
  What you do, is get the host it load the kernel module, and this module could be available in the container.</p>
  
  <p><a href="https://discuss.linuxcontainers.org/t/how-to-add-kernel-modules-into-an-lxc-container/5033/3" rel="nofollow noreferrer">Linuxcontainers.org: How to add kernel modules to LXC container</a></p>
</blockquote>

<p><code>Kubespray</code> will check if certain kernel modules are available within your nodes. </p>

<p>You will need to add following modules <strong>on your host</strong>: </p>

<ul>
<li><code>ip_vs</code></li>
<li><code>ip_vs_sh</code></li>
<li><code>ip_vs_rr</code></li>
<li><code>ip_vs_wrr</code></li>
</ul>

<p>You can add above modules with <code>$ modprobe MODULE_NAME</code> or follow this link: <a href="https://www.cyberciti.biz/faq/linux-how-to-load-a-kernel-module-automatically-at-boot-time/" rel="nofollow noreferrer">Cyberciti.biz: Linux how to load a kernel module automatically</a>.</p>

<h3>Conntrack</h3>

<p>You will need to install <code>conntrack</code> and load a module named <code>nf_conntrack</code>: </p>

<ul>
<li><code>$ apt install conntrack -y</code></li>
<li><code>modprobe nf_conntrack</code></li>
</ul>

<p>Without above commands <code>kubespray</code> will fail on step of checking the availability of <code>conntrack</code>. </p>

<p><strong>With this change in place you should be able to run Kubernetes cluster with <code>kubespray</code> within <code>lxc</code> environment and get output of nodes similar to this:</strong></p>

<pre class="lang-sh prettyprint-override"><code>root@k8s1:~# kubectl get nodes -o wide
NAME   STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k8s1   Ready    master   14h   v1.18.2   10.224.47.185   &lt;none&gt;        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7
k8s2   Ready    master   14h   v1.18.2   10.224.47.98    &lt;none&gt;        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7
k8s3   Ready    &lt;none&gt;   14h   v1.18.2   10.224.47.46    &lt;none&gt;        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7
k8s4   Ready    &lt;none&gt;   14h   v1.18.2   10.224.47.246   &lt;none&gt;        Ubuntu 18.04.4 LTS   5.4.0-31-generic   docker://18.9.7 
</code></pre>
