<p>I need help with making the XFS quotas work in Kubernetes on DigitalOcean.</p>
<p>My problem essentially is that the <code>xfs_quota</code> tool seems to work only when one has also an access to the disk device, not only to the mounted volume. However, whatever I try, I can't seem to get access both to the device and the mount.</p>
<p>I tried both volume mounts and raw block volumes.</p>
<h2>Volume Mounts</h2>
<p>Here's my storage class:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: block-storage-retain-xfs-prjquota
provisioner: dobs.csi.digitalocean.com
parameters:
  fsType: xfs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
mountOptions:
  - prjquota
</code></pre>
<p>Then I claim a new volume and add it to a pod like this:</p>
<pre class="lang-yaml prettyprint-override"><code>volumeClaimTemplates:
  - metadata:
      name: my-storage
      namespace: my-namespace
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: block-storage-retain-xfs-prjquota
      resources:
        requests:
          storage: 1Gi
</code></pre>
<p>and mount it:</p>
<pre class="lang-yaml prettyprint-override"><code>volumeMounts:
  - name: my-storage
    mountPath: &quot;/var/www&quot;
</code></pre>
<p>In the pod, everything gets mounted correctly, I have access to the volume (I can create contents in there) and the mount flag is set correctly:</p>
<pre class="lang-sh prettyprint-override"><code>$ mount | grep -i www
/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc on /var/www type xfs (rw,relatime,attr2,inode64,prjquota)
</code></pre>
<p>However, the disk device is not available in the pod:</p>
<pre class="lang-sh prettyprint-override"><code>$ ls -la /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc
ls: cannot access '/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc': No such file or directory
</code></pre>
<p>(in fact, the whole <code>/dev/disk/</code> directory is not available)</p>
<p>According to my investigation, the lack of access to the device is what makes the XFS tools fail:</p>
<pre class="lang-sh prettyprint-override"><code>$ xfs_quota -x -c 'report -h' /var/www
xfs_quota: cannot setup path for mount /var/www: No such device or address
</code></pre>
<h2>Raw Block Volumes</h2>
<p>I also tried to switch to raw block volumes instead:</p>
<pre class="lang-yaml prettyprint-override"><code>volumeClaimTemplates:
  - metadata:
      name: my-storage
      namespace: my-namespace
    spec:
      accessModes:
        - ReadWriteOnce
      volumeMode: Block
      storageClassName: block-storage-retain-xfs-prjquota
      resources:
        requests:
          storage: 1Gi
</code></pre>
<p>and add it as:</p>
<pre class="lang-yaml prettyprint-override"><code>volumeDevices:
  - name: my-storage
    devicePath: /dev/my-storage
</code></pre>
<p>That gives me the device, but for some reason I can't format it / mount it (neither XFS nor ext4 actually):</p>
<pre class="lang-sh prettyprint-override"><code>$ mkfs.xfs /dev/my-storage
mkfs.xfs: error - cannot set blocksize 512 on block device /dev/my-storage: Permission denied
</code></pre>
<pre class="lang-sh prettyprint-override"><code>$ mkfs.ext4 /dev/my-storage
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 18f07181-737c-4b68-a5fe-ccd7f2c50ff8
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

$ mount /dev/my-storage /var/www
mount: /var/www: cannot mount /dev/my-storage read-only.
</code></pre>
<p>With <code>SYS_ADMIN</code> Linux capability, I can actually format it, but I'm still not able to mount it:</p>
<pre class="lang-sh prettyprint-override"><code>$ mkfs.xfs -f /dev/my-storage
meta-data=/dev/my-storage        isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

$ mount /dev/my-storage /var/www
mount: /var/www: cannot mount /dev/my-storage read-only.
</code></pre>
<p>(Why is the disk device read only?)</p>
<h2>Raw Block Volume - With Partitions</h2>
<p>Ok, so I tried to create a partition and format that. Partition is created successfully, but I don't have access to the partition devices:</p>
<pre class="lang-sh prettyprint-override"><code>$ fdisk -l /dev/my-storage
Disk /dev/my-storage: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk model: Volume
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb4a24142

Device            Boot Start     End Sectors  Size Id Type
/dev/my-storage1        2048 2097151 2095104 1023M 83 Linux
</code></pre>
<p>However, the <code>/dev/my-storage1</code> does not exist:</p>
<pre class="lang-sh prettyprint-override"><code>$ ls -la /dev/my-storage*
brw-rw---- 1 root disk 8, 48 Oct 25 14:42 /dev/my-storage
</code></pre>
<p>I tried running the container as <code>privileged</code> which gave me access to more devices in <code>/dev</code>, but then I didn't see my raw block volume device at all.</p>
<h2>What Next?</h2>
<p>As I see that, any of those would work for me:</p>
<ol>
<li>Getting access to the underlying block device for volume mounts.</li>
<li>Access to the partition device so that I can mount it.</li>
<li>Ability to mount the raw block volume (e.g. by making it not read-only, whatever it means?).</li>
<li>Making the <code>xfs_quota</code> tool NOT require the underlying device.</li>
</ol>
<p>I believe I made it work a few months ago using raw block volumes with partitions, but either I forgot how or something changed on DigitalOcean and I can't seem to be able to create and access partitions anymore.</p>
<p>Any help is hugely appreciated, thank you!</p>

<p>Timo here from the Managed Kubernetes (DOKS) team at DigitalOcean.</p>
<p>What you are missing is the host system mount of the <code>/dev</code> directory. If you add both</p>
<pre><code> volumes:
 - name: device-dir
   hostPath:
     path: /dev
</code></pre>
<p>and</p>
<pre><code>volumeMounts:
  - name: device-dir
    mountPath: /dev
</code></pre>
<p>to the manifest at the right places, things should work as expected.</p>
