<p>I'm deploying a device plugin for FPGAs on a local kubernetes cluster. Essentially it is just a daemon set, so each node in the cluster (barring master nodes) will have one pod of this deployment.</p>
<p>The pods need to access the device trees of the hosts (nodes), they also need to access the kubelet socket. So I mount two specific directories from the hosts to the containers, as follows:</p>
<pre><code>      containers:
      - image: uofthprc/fpga-k8s-deviceplugin
        name: fpga-device-plugin-ctr
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
          - name: device-info
            mountPath: /sys/firmware/devicetree/base
            readOnly: true
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins
        - name: device-info
          hostPath:
            path: /sys/firmware/devicetree/base
</code></pre>
<p>For some reason, the <code>/var/lib/kubelet/device-plugins</code> mounts fine, and is fully accessible from within the containers, while the <code>/sys/firmware/devicetree/base</code> is not!. The following is the output of attaching to one of the containers <code>kubectl exec -it fpga-device-plugin-ds-hr6s5 -n device-plugins -- /bin/sh</code>:</p>
<pre><code>/work # ls /var/lib/kubelet/device-plugins
DEPRECATION                  kubelet.sock                 kubelet_internal_checkpoint
/work # ls /sys/firmware/devicetree/base
ls: /sys/firmware/devicetree/base: No such file or directory
/work # ls /sys/firmware/devicetree
ls: /sys/firmware/devicetree: No such file or directory
/work # ls /sys/firmware
/work # ls /sys
block     bus       class     dev       devices   firmware  fs        kernel    module    power
/work # 
</code></pre>
<p>I'm not sure why this happens, but I tested this with Read Only permissions, Read Write permissions, and without the mount at all. In all three cases, there's nothing inside the path <code>/sys/firmware</code> in the containers. On the hosts, I'm 100% sure that the path <code>/sys/firmware/devicetree/base/</code> exists and includes the files I want.</p>
<p>Here's the output of <code>describe pods</code> on one of the containers:</p>
<pre><code>Name:                 fpga-device-plugin-ds-hr6s5
Namespace:            device-plugins
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 mpsoc2/10.84.31.12
Start Time:           Wed, 20 May 2020 22:56:25 -0400
Labels:               controller-revision-hash=cfbc8976f
                      name=fpga-device-plugin-ds
                      pod-template-generation=1
Annotations:          cni.projectcalico.org/podIP: 10.84.32.223/32
                      cni.projectcalico.org/podIPs: 10.84.32.223/32
Status:               Running
IP:                   10.84.32.223
IPs:
  IP:           10.84.32.223
Controlled By:  DaemonSet/fpga-device-plugin-ds
Containers:
  fpga-device-plugin-ctr:
    Container ID:   docker://629ab2fd7d05bc17e6f566912b127eec421f214123309c10674c40ed2839d1cf
    Image:          uofthprc/fpga-k8s-deviceplugin
    Image ID:       docker-pullable://uofthprc/fpga-k8s-deviceplugin@sha256:06f9e46470219d5cfb2e6233b1473e9f1a2d3b76c9fd2d7866f7a18685b60ea3
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Running
      Started:      Wed, 20 May 2020 22:56:29 -0400
    Ready:          True
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /sys/firmware/devicetree/base from device-info (ro)
      /var/lib/kubelet/device-plugins from device-plugin (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dwbsm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  device-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/device-plugins
    HostPathType:  
  device-info:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/firmware/devicetree/base
    HostPathType:  
  default-token-dwbsm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dwbsm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type    Reason     Age        From               Message
  ----    ------     ----       ----               -------
  Normal  Scheduled  &lt;unknown&gt;  default-scheduler  Successfully assigned device-plugins/fpga-device-plugin-ds-hr6s5 to mpsoc2
  Normal  Pulling    23s        kubelet, mpsoc2    Pulling image &quot;uofthprc/fpga-k8s-deviceplugin&quot;
  Normal  Pulled     23s        kubelet, mpsoc2    Successfully pulled image &quot;uofthprc/fpga-k8s-deviceplugin&quot;
  Normal  Created    23s        kubelet, mpsoc2    Created container fpga-device-plugin-ctr
  Normal  Started    22s        kubelet, mpsoc2    Started container fpga-device-plugin-ctr
</code></pre>
<p>As far as I can see, no problems in it.</p>
<p>I'm using kubernetes (kubeadm installed, not microk8s) version 1.18.2 for the client and server. The nodes at question are ARM64 nodes with Ubuntu 16.04, using a 4.14.0 kernel. The containers are all alpine:3.11 with a simple binary copied inside them. I have no idea why the mount is not working, any help would certainly be appreciated.</p>
<h2>Edit1:</h2>
<p>The permissions of <code>/sys/firmware/devicetree/base/</code> on the hosts are as follows:</p>
<pre><code>savi@mpsoc10:~$ ls -alh /sys/firmware/devicetree/base/
total 0
drwxr-xr-x 36 root root  0 May 20 21:32 .
drwxr-xr-x  3 root root  0 May 20 21:32 ..
-r--r--r--  1 root root  4 May 20 21:32 #address-cells
drwxr-xr-x  2 root root  0 May 20 21:32 aliases
drwxr-xr-x 56 root root  0 May 20 21:32 amba
drwxr-xr-x  3 root root  0 May 20 21:32 amba_apu@0
drwxr-xr-x  2 root root  0 May 20 21:32 aux_ref_clk
-r--r--r--  1 root root 15 May 20 21:32 board
drwxr-xr-x  2 root root  0 May 20 21:32 chosen
drwxr-xr-x  2 root root  0 May 20 21:32 clk
-r--r--r--  1 root root 12 May 20 21:32 compatible
drwxr-xr-x  6 root root  0 May 20 21:32 cpu_opp_table
drwxr-xr-x  7 root root  0 May 20 21:32 cpus
drwxr-xr-x  2 root root  0 May 20 21:32 dcc
drwxr-xr-x  2 root root  0 May 20 21:32 dp_aclk
drwxr-xr-x  2 root root  0 May 20 21:32 edac
drwxr-xr-x  2 root root  0 May 20 21:32 fclk0
drwxr-xr-x  2 root root  0 May 20 21:32 fclk1
drwxr-xr-x  2 root root  0 May 20 21:32 fclk2
drwxr-xr-x  2 root root  0 May 20 21:32 fclk3
drwxr-xr-x  3 root root  0 May 20 21:32 firmware
drwxr-xr-x  2 root root  0 May 20 21:32 fpga-full
drwxr-xr-x  2 root root  0 May 20 21:32 gt_crx_ref_clk
drwxr-xr-x  2 root root  0 May 20 21:32 mailbox@ff990400
drwxr-xr-x  2 root root  0 May 20 21:32 memory
-r--r--r--  1 root root  1 May 20 21:32 name
drwxr-xr-x  3 root root  0 May 20 21:32 nvmem_firmware
drwxr-xr-x  2 root root  0 May 20 21:32 pcap
drwxr-xr-x  2 root root  0 May 20 21:32 pmu
drwxr-xr-x 31 root root  0 May 20 21:32 power-domains
drwxr-xr-x  2 root root  0 May 20 21:32 psci
drwxr-xr-x  2 root root  0 May 20 21:32 pss_alt_ref_clk
drwxr-xr-x  2 root root  0 May 20 21:32 pss_ref_clk
drwxr-xr-x  2 root root  0 May 20 21:32 reset-controller
drwxr-xr-x  2 root root  0 May 20 21:32 sha384
-r--r--r--  1 root root  4 May 20 21:32 #size-cells
drwxr-xr-x  2 root root  0 May 20 21:32 __symbols__
drwxr-xr-x  2 root root  0 May 20 21:32 timer
-r--r--r--  1 root root 10 May 20 21:32 vendor
drwxr-xr-x  2 root root  0 May 20 21:32 video_clk
drwxr-xr-x  2 root root  0 May 20 21:32 zynqmp-power
drwxr-xr-x  2 root root  0 May 20 21:32 zynqmp_rsa
</code></pre>
<p>some of the files inside it are read only, which is what prompted me to use read only permissions for the volume mount in the first place.</p>
<p>The following is the permissions of <code>/sys</code> and <code>/sys/firmware</code> on the containers:</p>
<pre><code>/work # ls -alh /sys/
total 4K     
dr-xr-xr-x   12 root     root           0 May 21 02:56 .
drwxr-xr-x    1 root     root        4.0K May 21 02:56 ..
drwxr-xr-x    2 root     root           0 May 21 03:08 block
drwxr-xr-x   32 root     root           0 May 21 03:08 bus
drwxr-xr-x   64 root     root           0 May 21 03:08 class
drwxr-xr-x    4 root     root           0 May 21 03:08 dev
drwxr-xr-x    9 root     root           0 May 21 03:08 devices
drwxrwxrwt    2 root     root          40 May 21 02:56 firmware
drwxr-xr-x   10 root     root           0 May 21 02:56 fs
drwxr-xr-x    7 root     root           0 May 21 02:56 kernel
drwxr-xr-x  156 root     root           0 May 21 03:08 module
drwxr-xr-x    2 root     root           0 May 21 03:08 power
/work # ls -alh /sys/firmware/
total 0      
drwxrwxrwt    2 root     root          40 May 21 02:56 .
dr-xr-xr-x   12 root     root           0 May 21 02:56 ..
</code></pre>
<h2>EDIT2:</h2>
<p>The output of <code>mount | grep sysfs</code> on containers is:</p>
<pre><code>/work # mount | grep sysfs
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/firmware/devicetree/base type sysfs (ro,relatime)
</code></pre>

<p><code>sysfs</code> is <a href="https://github.com/moby/moby/issues/24000" rel="nofollow noreferrer">mounted as read-only</a>:</p>

<pre><code>mount | grep sysfs
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
</code></pre>

<p>And that's why the volume is not shown in the pod. You can change it to writable volume by using <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="nofollow noreferrer">init container</a> and running both containers as <strong>privileged</strong>. If ran without <code>privileged: true</code> the access will not be modified and volume won't be mounted to a pod:</p>

<pre><code>initContainers:
  - name: mount
    image: nginx:alpine
    command: ["/bin/sh", "-c", "mount -o remount,rw '/sys'"]
    securityContext:
      privileged: true
</code></pre>

<p>This way it will change <code>/sys</code> to writable:</p>

<pre><code>mount | grep sysfs
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys/firmware/ type sysfs (rw,relatime)
</code></pre>
