<p>I would like to learn Kubernetes and would like to setup it on my laptop.</p>

<p>The architecture would be as follows:</p>

<p><a href="https://i.stack.imgur.com/MRVWw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/MRVWw.png" alt="enter image description here"></a></p>

<ul>
<li>Create 4 Ubuntu 18.04 server VM's instances on my laptop</li>
<li>3 of 4 VM's will be Kubernetes Clusters and 1 VM wilk be the base</li>
<li>Access via SSH the base VM</li>
</ul>

<p>For virtualization, I am using Virtual Box.</p>

<p>The question is, how to achieve it?</p>

<p>To set up Kubernetes Cluster on Ubuntu Servers with Virtualbox and Kubeadm follow this steps:  </p>

<h2>Prerequisites:</h2>

<ul>
<li>Virtual machines with specification of minimum:  

<ul>
<li>2 cores and 2GB RAM for master node  </li>
<li>1 core and 1GB for each of worker nodes  </li>
</ul></li>
<li>Ubuntu Server 18.04 installed on all virtual machines  </li>
<li>OpenSSH Server installed on all virtual machines  </li>
</ul>

<p>All of the virtual machines need to communicate with the Internet, main host and each other. It can be done through various means like: bridged networking, virtual hosts adapters etc. The networking scheme example below can be adjusted.  </p>

<p><a href="https://i.stack.imgur.com/iW04g.png" rel="nofollow noreferrer">Network scheme</a></p>

<h2>Ansible:</h2>

<p>You can do all things manually but to speed up the configuration process you can use automation tool like Ansible. It can be installed on the virtualization host, another virtual machine etc.  </p>

<h3>Installation steps to reproduce on host</h3>

<ul>
<li>Refresh the information about packages in repository:<br>
<code>$ sudo apt update</code>  </li>
<li>Install package manager for Python3:<br>
<code>$ sudo apt install python3-pip</code>  </li>
<li>Install Ansible package:<br>
<code>$ sudo pip3 install ansible</code>  </li>
</ul>

<h2>Configuring SSH key based access:</h2>

<h3>Generating key pairs</h3>

<p>To be able to connect to virtual machines without password you need to configure ssh keys. Command below will create a pair of ssh keys (private and public) and allow you to login to different systems without providing password.<br>
<code>$ ssh-keygen -t rsa -b 4096</code><br>
These keys will be created in default location: <strong>/home/USER/.ssh</strong></p>

<h3>Authorization of keys on virtual machines</h3>

<p>Next step is to upload newly created ssh keys to all of the virtual machines.<br>
<strong>For each of virtual machine you need to invoke:</strong><br>
<code>$ ssh-copy-id USER@IP_ADDRESS</code><br>
This command will copy your public key to the authorized_keys file and will allow you to login without password.  </p>

<h3>SSH root access</h3>

<p>By default root account can't be accessed with ssh only by password. It can be accessed with ssh keys (which you created earlier). Assuming the default configurations of files you can copy the ssh directory from user to root directory.</p>

<p><strong>This step needs to invoked on all virtual machines:</strong><br>
<code>$ sudo cp -r /home/USER/.ssh /root/</code>  </p>

<p>You can check it by running below command on main host:<br>
<code>$ ssh root@IP_ADDRESS</code>  </p>

<p>If you can connect without password it means that the keys are configured correctly.  </p>

<h2>Checking connection between virtual machines and Ansible:</h2>

<h3>Testing the connection</h3>

<p>You need to check if Ansible can connect to all of the virtual machines. To do that you need 2 things:  </p>

<ul>
<li><strong>Hosts</strong> file with information about hosts (virtual machines in that case)  </li>
<li><strong>Playbook</strong> file with statements what you require from Ansible to do  </li>
</ul>

<p>Example hosts file: </p>

<pre><code>[kubernetes:children]  
master  
nodes  

[kubernetes:vars]  
ansible_user=root  
ansible_port=22  

[master]  
kubernetes-master ansible_host=10.0.0.10  

[nodes]  
kubernetes-node1 ansible_host=10.0.0.11  
kubernetes-node2 ansible_host=10.0.0.12  
kubernetes-node3 ansible_host=10.0.0.13
</code></pre>

<p>Hosts file consists of 2 main groups of hosts:</p>

<ul>
<li>master - group created for master node  </li>
<li>nodes - group created for worker nodes  </li>
</ul>

<p>Variables specific to group are stored in section <strong>[kubernetes:vars]</strong>.  </p>

<p>Example playbook:</p>

<pre><code>- name: Playbook for checking connection between hosts  
  hosts: all  
  gather_facts: no  

  tasks:
  - name: Task to check the connection  
    ping:
</code></pre>

<p>Main purpose of above playbook is to check connection between host and virtual machines.<br>
You can test the connection by invoking command:<br>
<code>$ ansible-playbook -i hosts_file ping.yaml</code>  </p>

<p>Output of this command should be like this:  </p>

<pre class="lang-sh prettyprint-override"><code>PLAY [Playbook for checking connection between hosts] *****************************************************  

TASK [Task to check the connection] ***********************************************************************  

ok: [kubernetes-node1]  
ok: [kubernetes-node2]  
ok: [kubernetes-node3]  
ok: [kubernetes-master]  

PLAY RECAP ************************************************************************************************  

kubernetes-master : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0  
kubernetes-node1 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0  
kubernetes-node2 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0  
kubernetes-node3 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0  
</code></pre>

<p>The output above proves that connection between Ansible and virtual machines have been successful.  </p>

<h2>Configuration before cluster deployment:</h2>

<h3>Configure hostnames</h3>

<p>Hostnames can be configured with Ansible. Each vm should connect with each vm by their hostnames. Ansible can modify hostnames as well as /etc/hosts file. 
Example playbook: <a href="https://pastebin.com/4KZWVA3B" rel="nofollow noreferrer">hostname.yaml</a></p>

<h3>Disable SWAP</h3>

<p>Swap needs to be disabled when working with Kubernetes.
Example playbook: <a href="https://pastebin.com/6W1ZcTTp" rel="nofollow noreferrer">disable_swap.yaml</a></p>

<h3>Additional software installation</h3>

<p>Some packages are required before provisioning. All of them can be installed with Ansible:<br>
Example playbook: <a href="https://pastebin.com/NHTRiFiF" rel="nofollow noreferrer">apt_install.yaml</a></p>

<h3>Container Runtime Interface</h3>

<p>In this example you will install Docker as your CRI.
Playbook  <a href="https://pastebin.com/CP6SXw2Z" rel="nofollow noreferrer">docker_install.yaml</a> will:</p>

<ul>
<li>Add apt signing key for Docker</li>
<li>Add Docker's repository </li>
<li>Install Docker with specific version (latest recommended) </li>
</ul>

<h3>Docker configuration</h3>

<blockquote>
  <p><strong>[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd"</strong></p>
</blockquote>

<p>When deploying Kubernetes cluster kubeadm will give above warning about Docker cgroup driver. Playbook <a href="https://pastebin.com/PwijtqAh" rel="nofollow noreferrer">docker_configure.yaml</a> was created to resolve this issue. </p>

<h3>Kubernetes tools installation</h3>

<p>There are some core components of Kubernetes that need to be installed before cluster deployment. Playbook <a href="https://pastebin.com/HgzJGzN8" rel="nofollow noreferrer">kubetools_install.yaml</a> will: </p>

<ul>
<li>For master and worker nodes: 

<ul>
<li>Add apt signing key for Kubernetes</li>
<li>Add Kubernetes repository </li>
<li>Install kubelet and kubeadm</li>
</ul></li>
<li>Additionally for master node:

<ul>
<li>Install kubectl </li>
</ul></li>
</ul>

<h3>Reboot</h3>

<p>Playbook <a href="https://pastebin.com/ppKKw81k" rel="nofollow noreferrer">reboot.yaml</a> will reboot all the virtual machines. </p>

<h2>Cluster deployment:</h2>

<h3>Cluster initalization</h3>

<p>After successfully completing all the steps above, cluster can be created. Command below will initialize a cluster: </p>

<p><code>$ kubeadm init --apiserver-advertise-address=IP_ADDRESS_OF_MASTER_NODE --pod-network-cidr=192.168.0.0/16</code></p>

<p>Kubeadm can give warning about number of CPU's. It can be ignored by passing additional argument to kubeadm init command: 
<code>--ignore-preflight-errors=NumCPU</code></p>

<p>Sucessful kubeadm provisioning should output something similar to this: </p>

<pre><code>Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.10:6443 --token SECRET-TOKEN \
    --discovery-token-ca-cert-hash sha256:SECRET-CA-CERT-HASH
</code></pre>

<p>Copy kubeadm join command for all the worker nodes: </p>

<pre><code>kubeadm join 10.0.0.10:6443 --token SECRET-TOKEN \
    --discovery-token-ca-cert-hash sha256:SECRET-CA-CERT-HASH
</code></pre>

<p>Run commands below as regular user: </p>

<pre><code> mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config
</code></pre>

<h3>Deploying Container Network Interface (CNI)</h3>

<p>CNI is responsible for networking between pods and nodes. There are  many examples like: </p>

<ul>
<li>Flannel</li>
<li>Calico</li>
<li>Weave</li>
<li>Multus</li>
</ul>

<p>Command below will install Calico:</p>

<p><code>$ kubectl apply -f https://docs.projectcalico.org/v3.10/manifests/calico.yaml</code></p>

<h3>Provisioning worker nodes</h3>

<p>Run previously stored command from kubeadm init output <strong>on all worker nodes</strong>: </p>

<pre><code>kubeadm join 10.0.0.10:6443 --token SECRET-TOKEN \
    --discovery-token-ca-cert-hash sha256:SECRET-CA-CERT-HASH
</code></pre>

<p>All of the worker nodes should output: </p>

<pre><code>This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
</code></pre>

<h2>Testing:</h2>

<p>Run below command on master node as regular user to check if nodes are properly connected: </p>

<p><code>$ kubectl get nodes</code></p>

<p>Output of this command: </p>

<pre><code>NAME                STATUS   ROLES    AGE    VERSION
kubernetes-master   Ready    master   115m   v1.16.2
kubernetes-node1    Ready    &lt;none&gt;   106m   v1.16.2
kubernetes-node2    Ready    &lt;none&gt;   105m   v1.16.2
kubernetes-node3    Ready    &lt;none&gt;   105m   v1.16.2
</code></pre>

<p>Above output concludes that all the nodes are configured correctly. </p>

<p><strong>Pods can now be deployed on the cluster!</strong></p>
