<p>I'm running a pod in kubernetes, with hugepages allocated in host and hugepages defined in the pod. The kubernetes worker is in a VM. The VM (host) has huge pages allocated. The pod fails to allocate hugepages though. Application gets SIGBUS when trying to write to the first hugepage allocation.</p>
<p>The pod definition includes hugepages:</p>
<pre class="lang-json prettyprint-override"><code>    securityContext:
      allowPrivilegeEscalation: true
      privileged: true
      runAsUser: 0
      capabilities:
        add: [&quot;SYS_ADMIN&quot;, &quot;IPC_LOCK&quot;]
    resources:
      requests:
        intel.com/intel_sriov_netdevice : 2
        memory: 2Gi
        hugepages-2Mi: 4Gi
      limits:
        intel.com/intel_sriov_netdevice : 2
        memory: 2Gi
        hugepages-2Mi: 4Gi
    volumeMounts:
    - mountPath: /sys
      name: sysfs
    - mountPath: /dev/hugepages
      name: hugepage
      readOnly: false
  volumes:
  - name: hugepage
    emptyDir:
      medium: HugePages
  - name: sysfs
    hostPath:
      path: /sys
    
</code></pre>
<p>The VM hosting the pod has hugepages allocated:</p>
<pre><code>cat /proc/meminfo | grep -i hug
AnonHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
</code></pre>
<p>The following piece of code runs fine in the VM hosting the pod, I can see the hugepages files getting created in <code>/dev/hugepages</code>, also the HugePages_Free counter decreases while the process is running.</p>
<pre class="lang-c prettyprint-override"><code>#include &lt;stdio.h&gt;
#include &lt;sys/mman.h&gt;
#include &lt;errno.h&gt;
#include &lt;stdlib.h&gt;
#include &lt;unistd.h&gt;
#include &lt;fcntl.h&gt;
#define LENGTH (2UL*1024*1024)
#define FILE_NAME &quot;/dev/hugepages/hugepagefile&quot;
static void write_bytes(char *addr)
{
        unsigned long i;

        for (i = 0; i &lt; LENGTH; i++)
                *(addr + i) = (char)i;
}
int main ()
{
   void *addr;
   int i;
   char buf[32];
   int fd;

   for (i = 0 ; i &lt; 16 ; i++ ) {
           sprintf(buf, &quot;%s_%d&quot;, FILE_NAME, i);
           fd = open(buf, O_CREAT | O_RDWR, 0755);
           addr = mmap((void *)(0x0UL), LENGTH, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_HUGETLB , fd, 0);

           printf(&quot;address returned %p \n&quot;, addr);

           if (addr == MAP_FAILED) {
                   perror(&quot;mmap &quot;);
           } else {
                write_bytes(addr);
                //munmap(addr, LENGTH);
                //unlink(FILE_NAME);
           }
           close(fd);
   }
   while (1){}
   return 0;
}

</code></pre>
<p>But if I run the same code in the pod, I get a SIGBUS while trying to write to the first hugepage allocated.</p>
<p>Results on the VM (hosting the pod)</p>
<pre><code>root@k8s-1:~# cat /proc/meminfo | grep -i hug
AnonHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4096
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
root@k8s-1:~# ./mmap  &amp;
[1] 19428
root@k8s-1:~# address returned 0x7ffff7800000
address returned 0x7ffff7600000
address returned 0x7ffff7400000
address returned 0x7ffff7200000
address returned 0x7ffff7000000
address returned 0x7ffff6e00000
address returned 0x7ffff6c00000
address returned 0x7ffff6a00000
address returned 0x7ffff6800000
address returned 0x7ffff6600000
address returned 0x7ffff6400000
address returned 0x7ffff6200000
address returned 0x7ffff6000000
address returned 0x7ffff5e00000
address returned 0x7ffff5c00000
address returned 0x7ffff5a00000

root@k8s-1:~# cat /proc/meminfo | grep -i hug
AnonHugePages:         0 kB
HugePages_Total:    4096
HugePages_Free:     4080
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
</code></pre>
<p>Results in the pod:</p>
<pre><code>Program received signal SIGBUS, Bus error.
0x00005555555547cb in write_bytes ()
(gdb) where
#0  0x00005555555547cb in write_bytes ()
#1  0x00005555555548a6 in main ()

</code></pre>

emartin
<p>This is <a href="https://github.com/kubernetes/kubernetes/issues/71233" rel="nofollow noreferrer">a known problem</a> in K8s.</p>
<p>The culprit is that kubelet doesn't update /sys/fs/cgroup/hugetlb/kubepods/hugetlb.2MB.limit_in_bytes upon Node Status Update which happens every 5 minutes by default. Yet it updates the node's resources correctly after enabling hugepages on the host. This creates the possibility to schedule a workload using hugepages on a node with misconfigured limits in the root cgroup.</p>
<p>Some time ago I made <a href="https://github.com/kubernetes/kubernetes/pull/81774" rel="nofollow noreferrer">this patch</a> to K8s, but it never got accepted. You can try to apply it to your K8s build if it's still applicable. If not I'd appreciate if somebody else rebased it and submitted again. I spent too much time trying to make it in and switched to another project.</p>
