<p>I'm having issues with emptydir:</p>
<p>copying any file from an init container to an emptydir makes it unexecutable in the main container
happens on 3 of my 4 nodes, no issues on single master node
the master node that runs it successfully shows a difference selinux label on some files
Cluster Info:</p>
<ul>
<li>RHEL 7, SELinux permissive, VM's on internal datacenter</li>
<li>kubernetes 1.20, deployed by kubeadm</li>
<li>4 node cluster (1 is a master)</li>
</ul>
<p>Ex Pod Spec (fails on 3 nodes, successful on single master node):</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: emptydir-test
  namespace: default
spec:
  initContainers:
  - command:
    - &quot;bash&quot;
    - -c
    - &quot;cp $(which ls) /empty-dir/empty-dir-ls ; cp $(which ls) /mem-dir/mem-ls&quot;
    image: ubuntu
    imagePullPolicy: IfNotPresent
    name: init
    volumeMounts:
    - mountPath: /empty-dir
      name: empty-dir
    - mountPath: /mem-dir
      name: mem-dir
  containers:
  - securityContext:
      privileged: true
    command:
    - &quot;bash&quot;
    - -c
    - &quot;id ; ls -alhZ /mem-dir; /mem-dir/mem-ls -alhZ /root ; ls -alhZ /empty-dir ; /empty-dir/empty-dir-ls -alhZ /root&quot;    
    image: ubuntu
    imagePullPolicy: IfNotPresent
    name: emptydir-test
    volumeMounts:
    - mountPath: /empty-dir
      name: empty-dir
    - mountPath: /mem-dir
      name: mem-dir
  volumes:
  - emptyDir:
      medium: Memory  
    name: mem-dir
  - emptyDir: {}
    name: empty-dir
</code></pre>
<p>logs showing failure:</p>
<pre><code>uid=0(root) gid=0(root) groups=0(root)
total 140K
drwxrwxrwt. 2 root root system_u:object_r:tmpfs_t:s0            60 Apr  8 00:32 .
drwxr-xr-x. 1 root root system_u:object_r:unlabeled_t:s0        49 Apr  8 00:32 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0 139K Apr  8 00:32 mem-ls
total 8.0K
drwx------. 2 root root system_u:object_r:unlabeled_t:s0   37 Apr  1 01:26 .
drwxr-xr-x. 1 root root system_u:object_r:unlabeled_t:s0   49 Apr  8 00:32 ..
-rw-r--r--. 1 root root system_u:object_r:unlabeled_t:s0 3.1K Dec  5  2019 .bashrc
-rw-r--r--. 1 root root system_u:object_r:unlabeled_t:s0  161 Dec  5  2019 .profile
total 140K
drwxrwxrwx. 2 root root system_u:object_r:container_file_t:s0   25 Apr  8 00:32 .
drwxr-xr-x. 1 root root system_u:object_r:unlabeled_t:s0        49 Apr  8 00:32 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0 139K Apr  8 00:32 empty-dir-ls
bash: /empty-dir/empty-dir-ls: Permission denied
</code></pre>
<p>logs showing success:</p>
<pre><code>uid=0(root) gid=0(root) groups=0(root)
total 140K
drwxrwxrwt. 2 root root system_u:object_r:tmpfs_t:s0             60 Apr  8 00:34 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0  139K Apr  8 00:34 mem-ls
total 8.0K
drwx------. 2 root root system_u:object_r:container_share_t:s0   37 Apr  1 01:26 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0 3.1K Dec  5  2019 .bashrc
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0  161 Dec  5  2019 .profile
total 140K
drwxrwxrwx. 2 root root system_u:object_r:container_file_t:s0    25 Apr  8 00:34 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0  139K Apr  8 00:34 empty-dir-ls
total 8.0K
drwx------. 2 root root system_u:object_r:container_share_t:s0   37 Apr  1 01:26 .
drwxr-xr-x. 1 root root system_u:object_r:container_share_t:s0   49 Apr  8 00:34 ..
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0 3.1K Dec  5  2019 .bashrc
-rw-r--r--. 1 root root system_u:object_r:container_share_t:s0  161 Dec  5  2019 .profile
</code></pre>
<p>I noticed the different selinux labels for the root directory between the failed (system_u:object_r:unlabeled_t:s0) and successful (system_u:object_r:container_share_t:s0) container logs. But have confirmed all nodes are in permissive mode, so not sure if selinux can/is still somehow affecting it.</p>
<p>Any direction or advice is greatly appreciated!</p>

Eric Gagnon
<p>After days of googling and troubleshooting, I've resolved this - the problem was:
/var was mounted with noexec in fstab.</p>
<p>My single master node was provisioned before the others. During that time, company policy must've adopted a default noexec mount option on /var to improve security. You can either remove noexec from the /var entry in fstab (with IT's approval), or create a new mount for /var/lib/kubelet and mount that without noexec. either way, you'll need to reboot.</p>
<p>For anyone that runs into this issue, I also found several other potential causes:</p>
<ul>
<li>ACL's - if you're using ACL's anywhere, make sure you don't have entries for /var/lib/kubelet:</li>
</ul>
<p><code>getfacl -e /var/lib/kubelet/pods</code></p>
<ul>
<li>SELinux - if you're running in enforcing mode, check your selinux labels by -Z option with ls. These labels seem fine, not sure if there's a range of acceptable labels, but may be worth checking.</li>
</ul>
<pre><code>ls -alhZ /var/lib/kubelet/pods/
drwxr-xr-x. root root system_u:object_r:container_file_t:s0 .
drwxr-xr-x. root root system_u:object_r:container_file_t:s0 ..
drwxr-x---. root root system_u:object_r:container_file_t:s0 [poduid]
drwxr-x---. root root system_u:object_r:container_file_t:s0 [poduid]
</code></pre>
<ul>
<li><p>Pod Security Policies: if you still have these, they're deprecated anyway.<br />
<a href="https://kubernetes.io/docs/concepts/policy/pod-security-policy/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/policy/pod-security-policy/</a></p>
</li>
<li><p>Pod Security Context:  make sure these are configured properly<br />
<a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/security-context/</a></p>
</li>
<li><p>Noexec or other mount options on /var/lib/kubelet/pods - this was my problem. Assuming your system uses fstab to mount on boot, remove noexec from the mount point that contains /var/lib/kubelet/pods. Or, create a new mount point at /var/lib/kubelet/pods and omit noexec from its fstab entry.</p>
</li>
</ul>
<pre><code># edit emptydir-test.yaml to specify node as needed, then
kubectl apply -f emptydir-test.yaml
kubectl get po -o wide # confirm it runs/fails on node
# ssh to node
ls -alhZ /var/lib/kubelet/pods/$(kubectl get po emptydir-test -o jsonpath='{.metadata.uid}')/volumes/kubernetes.io~empty-dir/empty-dir
# drwxrwxrwx. root root system_u:object_r:container_file_t:s0 .
# drwxr-xr-x. root root system_u:object_r:container_file_t:s0 ..
# -rwxr-xr-x. root root system_u:object_r:container_file_t:s0 empty-dir-ls
sudo /var/lib/kubelet/pods/$(kubectl get po emptydir-test -o jsonpath='{.metadata.uid}')/volumes/kubernetes.io~empty-dir/empty-dir/empty-dir-ls -alh /
# output should either be the result of ls -alh on / or a permission denied error
# if it's a permission denied error, you may have a noexec mount options issue
# check current mounts for noexec option:
sudo findmnt -l | grep &quot;/var&quot;  | grep noexec
# look for /var/lib or a mount along /var/lib/kubelet/pods with noexec option
# check fstab:
grep &quot;/var&quot; /etc/fstab
# /dev/mapper/vg01-lv_var  /var     xfs     nodev,noexec,nosuid   1 2
# either remove noexec from the mount options or create a new mount point for /var/lib/kubelet and omit noexec
# then reboot to take effect and run the emptydir-test pod on affected nodes
</code></pre>
<p>I have no doubt there's other features driving security through inaccessibility, these are just what I found while solving my issue.</p>
