<p>Is there a way to mount a Kerberos authenticated NFS server inside a Kubernetes pod as the user who created the pod?</p>
<p>We use FreeIPA for user management, and we have a Kubernetes cluster setup for training our deep learning models. We have our data on an NFS, which is authenticated using Kerberos. Here is what we are trying to achieve:</p>
<ol>
<li>Mount kerberized NFS in a pod.</li>
<li>NFS should be mounted with the same permissions as that of the user who deployed the pod.</li>
<li>Uses should not be able to exec into the pods deployed by other users and access their data.</li>
</ol>
<p>We are using GKE for kubernetes and our NFS is in the same VPC.</p>

<p>This is how I do it.</p>
<p>For my approach you need:</p>
<ul>
<li>Workers joined to the kerberso realm. (NFS is mounted bu the worker with the host TGT)</li>
<li>Keytabs stored in a safe place, I use vault and vault-agent-injector</li>
<li>Sidecar container that will manage the credentials</li>
<li>Shared KCM between server and pods sharing the KCM socket, avoiding other pods to have access to the stored TGTs</li>
<li>Krb5 stored as configmap in the namespace</li>
<li>Proper rights in the NFS exported data, owned by the users that exist in the IPA and which uids/gids will be used for running the containers in the pod.</li>
</ul>
<p>The reasons for this approach are:</p>
<ul>
<li>Information secured in vault</li>
<li>Secrets removed from storage and stored only in memory</li>
<li>Each container manages a single process or task.</li>
</ul>
<p>With all this in mind you first write down your Dockerfile for the krb5-sidecar.
command: [&quot;/bin/sh&quot;]
args: [&quot;-c&quot;, &quot;/usr/bin/sleep 3600000&quot;]</p>
<pre><code>    FROM centos:centos7
    
    # install the kerberos client tools
    RUN yum install -y krb5-workstation &amp;&amp; \
     mkdir /krb5 &amp;&amp; chmod 755 /krb5
    
    # add resources, the kinit script and the default krb5 configuration
    ADD entrypoint.sh /entrypoint.sh
    RUN chmod +x /krb-sidecar-entrypoint.sh
    # Little trick here that will allow my container to remove 
    # the vault secrets without root
    RUN chmod u+s /usr/bin/rm
    ENTRYPOINT [&quot;/entrypoint.sh&quot;]
</code></pre>
<p>And this is the entrypoint script that manages</p>
<ol>
<li>Keytab load into KCM memory</li>
<li>Removal from shared <code>/vault/secrets</code> file of the keytab</li>
<li>Renewal of the kerberos ticket depending on your Krb5 policy</li>
</ol>
<pre><code>
# Default value for renewing the TGT ticket
KERBEROS_RENEWAL_TIME=86400 # One day

# Move the keytab into keytabfile
echo  &quot;Generating keytab file&quot;
cat /vault/secrets/${USERNAME}.keytab  | cut -d' ' -f2 | base64 -d &gt; /etc/${USERNAME}.keytab

# Get the TGT
echo &quot;Loading keytab&quot;
kinit -kt /etc/${USERNAME}.keytab ${USERNAME}@${REALM}

# Remove secrets for security reasons
rm -rf /vault/secrets/*
rm -rf /etc/${USERNAME}.keytab
echo &quot;Secrets removed from tmpfs&quot;
while :;
do
  kinit -R
  sleep ${KERBEROS_RENEWAL_TIME}
done
</code></pre>
<p>Of course you need to create PersistentVolumes and PersistentVolumeClaims for the deployment.</p>
<p>PersistentVolume</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: NFS-vol
spec:
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: slow
  mountOptions:
    - sec=krb5
  nfs:
    path: /exports
    server: nfs.server.test
</code></pre>
<p>PersistentVolumeClaim:</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfsvol
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 3Gi
</code></pre>
<p>And finally the Deployment:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-user
spec:
  selector:
    matchLabels:
      test: test
  template:
    metadata:
      labels:
        test: test
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-userKeytab: 'user/keytabs/user'
        vault.hashicorp.com/role: 'nfs'
        vault.hashicorp.com/ca-cert: 'certs/ca.crt'
        vault.hashicorp.com/tls-secret: 'tls-ca'
        vault.hashicorp.com/agent-pre-populate-only: &quot;true&quot;
    spec:
      securityContext:
        # Here we defined the user uid, this user must be present in the NFS server
        runAsUser: 2500
        runAsGroup: 2500
      # This may be needed or not depending on your DNS setup
      hostAliases:
        - ip: &quot;192.168.111.130&quot;
          hostnames:
            - &quot;IPA&quot;
            - &quot;IPA.server&quot;
        - ip: &quot;192.168.111.131&quot;
          hostnames:
            - &quot;nfs&quot;
            - &quot;nfs.serer&quot;
      restartPolicy: Always
      volumes:
      - name: nfs-user
        persistentVolumeClaim:
          claimName: nfs-vol
      - name: krb5
        configMap:
          name: keos-kerberos-config
      - name: kcmsocket
        hostPath:
          path: /var/run/.heim_org.h5l.kcm-socket
          type: File
      containers:
      - name: krb5-sidecar
        image: krb5-sidecar:0.1.0
        env:
        - name: KRB5CCNAME
          value: &quot;KCM:&quot;
        - name: USERNAME
          value: user
        - name: REALM
          value: server
        volumeMounts:
        - name: krb5
          mountPath: &quot;/etc/krb5.conf&quot;
          subPath: &quot;krb5.conf&quot;
        - name: kcmsocket
          mountPath: &quot;/var/run/.heim_org.h5l.kcm-socket&quot;
        lifecycle:
          preStop:
            exec:
              command: [&quot;/usr/bin/kdestroy&quot;]
      - name: mount-nfs-container
        image: nfs-centos:0.2.0
        env:
        - name: KRB5CCNAME
          value: &quot;KCM:&quot;
        volumeMounts:
        - name: nfs-user
          mountPath: &quot;/nfs&quot;
        - name: krb5
          mountPath: &quot;/etc/krb5.conf&quot;
          subPath: &quot;krb5.conf&quot;
        - name: kcmsocket
          mountPath: &quot;/var/run/.heim_org.h5l.kcm-socket&quot;

</code></pre>
