<p>We are trying to install kubernetes in SUSE enterprize linux server v15. We found that there is no way to install k8s using kubeadm. SUSE provides Container as a service Platform(CaasP) to install k8s.</p>
<p>All we are having is few virtual machines and suse subscription. Can we install CaasP in it?
We could not find any documentation to install it in virtual machines.</p>
<p>Is there any way documentation to do step by step CaasP installation in Virtual machines?</p>

Samselvaprabu
<h3>Kubeadm on SLES</h3>
<p>It is possible to install Kubernetes on SUSE Linux Enterprise Server 15 using <code>kubeadm</code>.<br />
You can find a step-by-step example below.</p>
<p>The example was tested on the following cloud VM images:</p>
<p><strong>GCP</strong> :</p>
<ul>
<li>SUSE Linux Enterprise Server 15 SP1 x86_x64</li>
</ul>
<p><strong>AWS</strong> :</p>
<ul>
<li>openSUSE-Leap-15.2-v20200710-HVM-x86_64-548f7b74-f1d6-437e-b650-f6315f6d8aa3-ami-0f5745b812a5b7654.4 - ami-023643495f15f104b</li>
<li>suse-sles-15-sp1-v20200615-hvm-ssd-x86_64 - ami-0044ae6906d786f4b</li>
</ul>
<p><strong>Azure</strong> :</p>
<ul>
<li>SUSE Enterprise Linux 15 SP1 +Patching</li>
</ul>
<p>So, it has a good chance to be used with other images with only a few changes.</p>
<p>It was also tested on <strong>Vagrant</strong> box <code>trueability/sles-15-sp1</code> , and it was required a few additional steps, because of expired subscription keys. I used <a href="https://en.opensuse.org/Package_repositories" rel="nofollow noreferrer">OSS repositories</a> and ignored expiration errors:</p>
<pre><code># add OSS repository for software installation 
$ zypper addrepo http://download.opensuse.org/distribution/leap/15.2/repo/oss/ public

# add repository for installing newer Docker version

$ zypper addrepo https://download.opensuse.org/repositories/Virtualization:containers/openSUSE_Leap_15.0/Virtualization:containers.repo virt

# install symbols required by Docker:

$ zypper install libseccomp-devel

# turn off all swap partitions. Comment appropriate /etc/fstab entry as well.

$ swapoff -a

# Rest of the steps is similar except additional argument during cluster initialization. 

# This box is using btrfs for /var/lib/docker and kubeadm complains about it. 
# I've just asked kubeadm to ignore that fact. 
# Even with btrfs it can start and run pods, but there might be some problems with Persistent Volumes usage, 
# so consider using additional xfs or ext4 partition for /var/lib/docker

$ kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all

</code></pre>
<h3>Cloud VMs:</h3>
<p>Cloud SLES 15 SP1 images uses xfs for their / file system and don't use swap out of the box, and kubeadm passes all pre-flight checks without errors.</p>
<pre><code># become root

$ sudo -s

# install docker

$ zypper refresh
$ zypper install docker

# configure sysctl for Kubernetes

$ cat &lt;&lt;EOF &gt;&gt; /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1
EOF

# add Google repository for installing Kubernetes packages
#$ zypper addrepo --type yum --gpgcheck-strict --refresh https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 google-k8s

#or

$ cat &lt;&lt;EOF &gt; /etc/zypp/repos.d/google-k8s.repo
[google-k8s]
name=google-k8s
enabled=1
autorefresh=1
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
type=rpm-md
gpgcheck=1
repo_gpgcheck=1
pkg_gpgcheck=1
EOF

# import Google repository keys

$ rpm --import https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
$ rpm --import https://packages.cloud.google.com/yum/doc/yum-key.gpg
$ rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --&gt; %{summary}\n'

# the following repository was needed only for GCP image
# other images was able successfully install conntrack-tools using existing repository

$ zypper addrepo https://download.opensuse.org/repositories/security:netfilter/SLE_12/security:netfilter.repo conntrack
$ zypper refresh conntrack

# conntrack presence is checked during kubeadm pre-flight checks 
# but zypper unable to find appropriate dependency for kubelet, 
# so let's install it manually

$ zypper install conntrack-tools

# refresh Google repository cache and check if we see several versions of Kubernetes packages to choose from

$ zypper refresh google-k8s
$ zypper packages --repo google-k8s

# install latest available kubelet package
# ignore conntrack dependency and install kubelet (Solution 2 in my case)

$ zypper install kubelet

# install kubeadm package. kubectl and cri-tools are installed as kubeadm dependency

$ zypper install kubeadm

# force docker to use systemd cgroup driver and overlay2 storage driver. 
# Check the links in the end of the answer for details. 
# BTW, kubelet would work even with default content of the file.

$ cat &gt; /etc/docker/daemon.json &lt;&lt;EOF
{
  &quot;exec-opts&quot;: [&quot;native.cgroupdriver=systemd&quot;],
  &quot;log-driver&quot;: &quot;json-file&quot;,
  &quot;log-opts&quot;: {
    &quot;max-size&quot;: &quot;100m&quot;
  },
  &quot;storage-driver&quot;: &quot;overlay2&quot;
}
EOF

# Not sure if it's necessary it was taken from the Kubernetes documentation

$ mkdir -p /etc/systemd/system/docker.service.d

# lets start and enable docker and kubelet services

$ systemctl start docker.service
$ systemctl enable docker.service
$ systemctl enable kubelet.service

# apply configured earlier sysctl settings. 
# net.bridge.bridge-nf-call-iptables becomes available after successfully starting
# Docker service 

$ sysctl -p

# Now it's time to initialize Kubernetes master node. 
# Ignore pre-flight checks for Vagrant box.

$ kubeadm init --pod-network-cidr=10.244.0.0/16

# prepare kubectl configuration to connect the cluster

$  mkdir -p $HOME/.kube
$  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$  sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Check if api-server responds to our requests. 
# At this moment it's fine to see master node in NotReady state.

$ kubectl get nodes

# Deploy Flannel network addon

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# remove taint from the master node. 
# It allows master node to run application pods. 
# At least one worker node is required if this step is skipped.

$ kubectl taint nodes --all node-role.kubernetes.io/master-

# run test pod to check if everything works fine

$ kubectl run nginx1 --image=nginx

# after some time... ~ 3-5 minutes

# check the pods' state 

$ kubectl get pods -A -o wide
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
default       nginx1                              1/1     Running   0          74s     10.244.0.4   suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-66bff467f8-vc2x4            1/1     Running   0          2m26s   10.244.0.2   suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-66bff467f8-w4jvq            1/1     Running   0          2m26s   10.244.0.3   suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   etcd-suse-test                      1/1     Running   0          2m41s   10.4.0.4     suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   kube-apiserver-suse-test            1/1     Running   0          2m41s   10.4.0.4     suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   kube-controller-manager-suse-test   1/1     Running   0          2m41s   10.4.0.4     suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   kube-flannel-ds-amd64-mbfxp         1/1     Running   0          2m12s   10.4.0.4     suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-cw5xm                    1/1     Running   0          2m26s   10.4.0.4     suse-test   &lt;none&gt;           &lt;none&gt;
kube-system   kube-scheduler-suse-test            1/1     Running   0          2m41s   10.4.0.4     suse-test   &lt;none&gt;           &lt;none&gt;

# check if the test pod is working fine

# curl 10.244.0.4
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
...skipped...

# basic Kubernetes installation is done

</code></pre>
<h3>Additional materials:</h3>
<ul>
<li><p><a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/" rel="nofollow noreferrer">Container runtimes</a> (Kubernetes documentation page)</p>
</li>
<li><p><a href="https://stackoverflow.com/questions/57589205/giving-error-saying-unsupported-graph-driver-btrfs-in-sles-when-try-to-kubead">Giving error saying “unsupported graph driver: btrfs” in SLES when try to Kubeadm init</a><br />
(If you have <code>btrfs</code> <strong>/</strong> partition, you can mount additional <code>xfs</code> or <code>ext4</code> partition for <code>/var/lib/docker</code> to use <code>overlay2</code> Docker storage driver)</p>
</li>
<li><p><a href="https://www.suse.com/support/kb/doc/?id=000019184" rel="nofollow noreferrer">OverlayFS support for SLES 12</a> (I would expect that SLES 15 support it as well.)</p>
</li>
<li><p><a href="https://success.docker.com/article/compatibility-matrix" rel="nofollow noreferrer">Docker Compatibility Matrix</a></p>
<pre><code>  OS Distribution (x86_64): SLES 15
  Enterprise Engine: 19.03.x
  UCP: 3.2.x
  DTR: 2.7.x
  Storage Driver: overlay2,btrfs
  Orchestration: Swarm mode, Kubernetes
  DTR Storage Backend: NFSv4, NFSv3, Amazon S3, S3 Compliant Alternatives, 
              Azure Storage (Blob), Google Cloud Storage, OpenStack Swift, 
              Local Filesystem
</code></pre>
</li>
<li><p><a href="https://documentation.suse.com/sles/15-SP1/pdf/book-sles-docker_color_en.pdf" rel="nofollow noreferrer">SUSE: Docker Open Source Engine Guide</a> (very useful book)</p>
</li>
<li><p><a href="https://docs.docker.com/storage/storagedriver/overlayfs-driver/" rel="nofollow noreferrer">Use the OverlayFS storage driver</a></p>
</li>
<li><p><a href="https://ostechnix.com/install-linux-kernel-4-12-opensuse/" rel="nofollow noreferrer">Install Linux Kernel 4.12 in openSUSE</a> (in case you want to add AUFS support to the Linux kernel)</p>
</li>
</ul>
<h3>Materials about SUSE CaaSP</h3>
<ul>
<li><a href="https://www.youtube.com/watch?v=xAKfexcX8lE" rel="nofollow noreferrer">SUSE CaaS Platform Setup - Nothing to Everything in 1 Hour</a>
(video is quite old but very useful)</li>
<li><a href="https://www.suse.com/products/caas-platform/download/" rel="nofollow noreferrer">CaaSP download page, 60 days free trial</a></li>
</ul>
