title: Kubernetes cannot mount Glusterfs volumes
body: <p>I have a GlusterFS cluster and a k8s cluster.   On any of the nodes in the k8s cluster, I can execute the following command:</p>
  <pre class="lang-bash prettyprint-override"><code> mount -t glusterfs data1:/sbnstore /mnt/data/ -o xlator-option=&quot;transport.address-family=inet6&quot;
  </code></pre>
  <p>and this works perfectly.  I can create and access files under /mnt/data and they appear in the GlusterFS cluster and replicate among the peers.   Note that the GlusterFS peers do not have IPv4 addresses.  Only IPv6.</p>
  <p>However creating a Kubernetes Pod hangs in the container creation stage.   The following is the manifest:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Pod
  metadata:
    name: shell-testg
    labels:
            alan: testg
  spec:
          containers:
                  - name: ubuntu
                    image: ubuntu:latest
                    command: [&quot;/bin/sleep&quot;, &quot;3650d&quot;]
                    volumeMounts:
                          - name: glusterfs-volume
                            mountPath: /data
          volumes:
                    - name: glusterfs-volume
                      glusterfs:
                          endpoints: glusterfs-cluster
                          path: sbnstore
                          readOnly: no
  </code></pre>
  <p>When this manifest is applied, the pod that is created is permanently stuck in ContainerCreating status.</p>
  <pre class="lang-bash prettyprint-override"><code>$ kc get pod
  NAME                      READY   STATUS              RESTARTS        AGE
  shell-testg               0/1     ContainerCreating   0               12m
  </code></pre>
  <p>Digging for more info (extraneous lines omitted):</p>
  <pre class="lang-bash prettyprint-override"><code>$ kc describe pod shell-testg
  Name:         shell-testg
  Namespace:    default
  ...
      Mounts:
        /data from glusterfs-volume (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxktp (ro)
  ...
  Volumes:
    glusterfs-volume:
      Type:           Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
      EndpointsName:  glusterfs-cluster
      Path:           sbnstore
      ReadOnly:       false
    kube-api-access-kxktp:
      Type:                    Projected (a volume that contains injected data from multiple sources)
      TokenExpirationSeconds:  3607
      ConfigMapName:           kube-root-ca.crt
      ConfigMapOptional:       &lt;nil&gt;
      DownwardAPI:             true
  QoS Class:                   BestEffort
  Node-Selectors:              &lt;none&gt;
  Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                               node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
    Type     Reason       Age                 From     Message
    ----     ------       ----                ----     -------
    Warning  FailedMount  109s (x6 over 17m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[glusterfs-volume], unattached volumes=[glusterfs-volume kube-api-access-kxktp]: timed out waiting for the condition
    Warning  FailedMount  84s                 kubelet  MountVolume.SetUp failed for volume &quot;glusterfs-volume&quot; : mount failed: mount failed: exit status 1
  Mounting command: systemd-run
  Mounting arguments: --description=Kubernetes transient mount for /var/snap/microk8s/common/var/lib/kubelet/pods/2ef51502-7327-443c-abc1-2a0f1ee1b177/volumes/kubernetes.io~glusterfs/glusterfs-volume --scope -- mount -t glusterfs -o auto_unmount,backup-volfile-servers=2001:470:1:999:1:0:30:100:2001:470:1:999:1:0:30:200:2001:470:1:999:1:0:30:300,log-file=/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/glusterfs/glusterfs-volume/shell-testg-glusterfs.log,log-level=ERROR 2001:470:1:841:1:0:30:200:sbnstore /var/snap/microk8s/common/var/lib/kubelet/pods/2ef51502-7327-443c-abc1-2a0f1ee1b177/volumes/kubernetes.io~glusterfs/glusterfs-volume
  Output: Running scope as unit: run-rd7adcd2329cb4ab2a694d02925df1a2f.scope
  [2022-06-23 18:32:35.342172] E [glusterfsd.c:833:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
  ...
  22-06-23 18:32:35.342443] E [glusterfsd.c:833:gf_remember_backup_volfile_server] 0-glusterfs: failed to set volfile server: File exists
  Mounting glusterfs on /var/snap/microk8s/common/var/lib/kubelet/pods/2ef51502-7327-443c-abc1-2a0f1ee1b177/volumes/kubernetes.io~glusterfs/glusterfs-volume failed.
  
  </code></pre>
  <p>So now I am kind of stuck.  Poking through the github issues board this kind of error seems to appear when there is a version incompatibility.  My k8s is 1.23 and gluster is 7.2.</p>
  <p>Any suggestions of what to try next?</p>
  <p><strong>UPDATE</strong></p>
  <p>As it turns out, the issue is that the glusterfs module within Kubernetes is not able to use IPv6.  So I have a workaround which is to convert all the Endpoints to IPv4 addresses.</p>
  <p>Not working:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Endpoints
  metadata:
    name: glusterfs-cluster
  subsets:
  - addresses:
    - ip: 2001:470:1:999:1:0:30:100
    ports:
    - port: 49152
  - addresses:
    - ip: 2001:470:1:999:1:0:30:200
    ports:
    - port: 49152
  - addresses:
    - ip: 2001:470:1:999:1:0:30:300
    ports:
    - port: 49152
  
  </code></pre>
  <p>Working:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Endpoints
  metadata:
    name: glusterfs-cluster
  subsets:
  - addresses:
    - ip: 10.2.12.1
    ports:
    - port: 49152
  - addresses:
    - ip: 10.2.12.2
    ports:
    - port: 49152
  - addresses:
    - ip: 10.2.12.3
    ports:
    - port: 49152
  </code></pre>
  <p>It is interesting to note that automatically provisioned PV/PVCs using the Heketi package <strong>do</strong> work over IPv6.</p>
  <p>I haven't been able to find any way to specify configuration parameters to the glusterfs module that might provide a real solution.   So if anyone has any pointers to that please provide it in an answer to this posting.</p>
  <p>Hopefully this update will help someone else avoid this learning curve.</p>