<p>I know this question is asked many times, but all about docker, this time is crio.</p>

<pre><code>CentOS Linux release 7.6
CRI-O Version: 1.16.1
Kubernetes: v1.16.3
KubeAdm: v1.16.3
</code></pre>

<p>CoreDNS pods are in Error/CrashLoopBackOff state, and audit.log shows selinux prevents CoreDNS to read from /var/lib/kubelet/container_id/volumes/</p>

<pre><code>type=AVC msg=audit(1576203392.727:1431): avc: denied { read } for pid=15866 comm="coredns" name="Corefile" dev="dm-0" ino=35369330 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1

type=AVC msg=audit(1576203392.727:1431): avc: denied { open } for pid=15866 comm="coredns" path="/etc/coredns/..2019_12_13_02_13_30.965446608/Corefile" dev="dm-0" ino=35369330 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1

type=AVC msg=audit(1576203393.049:1432): avc: denied { open } for pid=15866 comm="coredns" path="/var/run/secrets/kubernetes.io/serviceaccount/..2019_12_13_02_13_30.605147375/token" dev="tmpfs" ino=124481 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=1
</code></pre>

<p>if I use docker newer than 1.7, it works fine, I assume this may related with the patch of mounting volume with z/Z option.</p>

<p>I can add policy like underneath, but it will compromise security.</p>

<pre><code>module coredns 0.1;

require {
  type tmpfs_t;
  type container_t;
  type var_lib_t;

  class file { open read };
}

allow container_t tmpfs_t:file open;
allow container_t var_lib_t:file { open read };
</code></pre>

<p>any better solution exists? just like docker, with a little efforts and don't compromise security.</p>

<p>I've looked into it and it seems that the problem lays in <strong>kubelet version</strong>. Let me elaborate on that: </p>

<p><a href="https://github.com/kubernetes/kubernetes/issues/83679" rel="nofollow noreferrer">SELinux Volumes not relabeled in 1.16</a> - this link is providing more details about the issue. </p>

<p>I tried to reproduce this coredns issue on different versions of Kubernetes.</p>

<p>Issue shows on version 1.16 and newer. It seems to work properly with SELinux enabled on the version 1.15.6 </p>

<p>For this to work you will need working CentOS and CRI-O environment.</p>

<p>CRI-O version: </p>

<pre><code>Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.16.2
RuntimeApiVersion:  v1alpha1
</code></pre>

<p>To deploy this insfrastructure I followed this site for the most part: <a href="https://kubevirt.io/2019/KubeVirt_k8s_crio_from_scratch.html" rel="nofollow noreferrer">KubeVirt</a> </p>

<h2>Kubernetes v1.15.7</h2>

<p><strong>Steps to reproduce</strong>: </p>

<ul>
<li>Disable SELinux and restart machine:

<ul>
<li><code>$ setenforce 0</code></li>
<li><code>$ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config</code></li>
<li><code>$ reboot</code></li>
</ul></li>
<li>Check if SELinux is disabled by invoking command: <code>$ sestatus</code></li>
<li>Install packages with <code>$ yum install INSERT_PACKAGES_BELOW</code>

<ul>
<li>kubelet-1.15.7-0.x86_64</li>
<li>kubeadm-1.15.7-0.x86_64</li>
<li>kubectl-1.15.7-0.x86_64</li>
</ul></li>
<li>Initialize Kubernetes cluster with following command <code>$ kubeadm init --pod-network-cidr=10.244.0.0/16</code></li>
<li>Wait for cluster to initialize correctly and follow kubeadm instructions to connect to cluster</li>
<li>Apply Flannel CNI <code>$ kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml</code></li>
</ul>

<p>Check if coredns pods are running correctly with command: 
<code>$ kubectl get pods -A</code></p>

<p>It should give similar output to that: </p>

<pre><code>NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE
kube-system   coredns-5c98db65d4-2c7lt                     1/1     Running   2          7m59s
kube-system   coredns-5c98db65d4-5dp9s                     1/1     Running   2          7m59s
kube-system   etcd-centos-kube-master                      1/1     Running   2          7m20s
kube-system   kube-apiserver-centos-kube-master            1/1     Running   2          7m4s
kube-system   kube-controller-manager-centos-kube-master   1/1     Running   2          6m55s
kube-system   kube-flannel-ds-amd64-mzh27                  1/1     Running   2          7m14s
kube-system   kube-proxy-bqll8                             1/1     Running   2          7m58s
kube-system   kube-scheduler-centos-kube-master            1/1     Running   2          6m58s
</code></pre>

<p>Coredns pods in kubernetes cluster with <strong>SELinux disabled</strong> are working properly. </p>

<p><strong>Enable SELinux</strong>:</p>

<p>From root account invoke commands to enable SELinux and restart the machine: </p>

<ul>
<li><code>$ setenforce 1</code></li>
<li><code>$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config</code></li>
<li><code>$ reboot</code></li>
</ul>

<p>Check if coredns pods are running correctly. They should <strong>not get crashloopbackoff error</strong> when running: 
<code>kubectl get pods -A</code></p>

<h2>Kubernetes v1.16.4</h2>

<p><strong>Steps to reproduce</strong>: </p>

<ul>
<li>Run <code>$ kubeadm reset</code> if coming from another another version</li>
<li>Remove old Kubernetes packages with <code>$ yum remove OLD_PACKAGES</code></li>
<li>Disable SELinux and restart machine:

<ul>
<li><code>$ setenforce 0</code></li>
<li><code>$ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config</code></li>
<li><code>$ reboot</code></li>
</ul></li>
<li>Check if SELinux is disabled by invoking command: <code>$ sestatus</code></li>
<li>Install packages with <code>$ yum install INSERT_PACKAGES_BELOW</code>

<ul>
<li>kubelet-1.16.4-0.x86_64</li>
<li>kubeadm-1.16.4-0.x86_64</li>
<li>kubectl-1.16.4-0.x86_64</li>
</ul></li>
<li>Initialize Kubernetes cluster with following command <code>$ kubeadm init --pod-network-cidr=10.244.0.0/16</code></li>
<li>Wait for cluster to initialize correctly and follow kubeadm instructions to connect to cluster </li>
<li>Apply Flannel CNI <code>$ kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml</code></li>
</ul>

<p>Check if coredns pods are running correctly with command: 
<code>$ kubectl get pods -A</code></p>

<p>It should give similar output to that: </p>

<pre><code>NAMESPACE     NAME                                         READY   STATUS             RESTARTS   AGE
kube-system   coredns-5644d7b6d9-fgbkl                     1/1     Running            1          13m
kube-system   coredns-5644d7b6d9-x6h4l                     1/1     Running            1          13m
kube-system   etcd-centos-kube-master                      1/1     Running            1          12m
kube-system   kube-apiserver-centos-kube-master            1/1     Running            1          12m
kube-system   kube-controller-manager-centos-kube-master   1/1     Running            1          12m
kube-system   kube-proxy-v52ls                             1/1     Running            1          13m
kube-system   kube-scheduler-centos-kube-master            1/1     Running            1          12m
</code></pre>

<p><strong>Enable SELinux</strong>:</p>

<p>From root account invoke commands to enable SELinux and restart the machine: </p>

<ul>
<li><code>$ setenforce 1</code></li>
<li><code>$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config</code></li>
<li><code>$ reboot</code></li>
</ul>

<p>After reboot coredns pods <strong>should enter crashloopbackoff  state</strong> as shown below: </p>

<pre><code>NAMESPACE     NAME                                         READY   STATUS             RESTARTS   AGE
kube-system   coredns-5644d7b6d9-fgbkl                     0/1     CrashLoopBackOff   25         113m
kube-system   coredns-5644d7b6d9-x6h4l                     0/1     CrashLoopBackOff   25         113m
kube-system   etcd-centos-kube-master                      1/1     Running            1          112m
kube-system   kube-apiserver-centos-kube-master            1/1     Running            1          112m
kube-system   kube-controller-manager-centos-kube-master   1/1     Running            1          112m
kube-system   kube-proxy-v52ls                             1/1     Running            1          113m
kube-system   kube-scheduler-centos-kube-master            1/1     Running            1          112m
</code></pre>

<p>Logs from the pod <code>coredns-5644d7b6d9-fgbkl</code> show: </p>

<pre><code>plugin/kubernetes: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
</code></pre>
