<p>I have an EKS cluster where having one daemon which mounts a s3 bucket to all pods.</p>
<p>Whenever there is some issue or pod restarts, then the mount volume is not accessible and throws the below error.</p>
<pre><code>Transport endpoint is not connected
</code></pre>
<p>For solving this error, I have to manaully unmount the volume and restart the daemon.</p>
<pre><code>umount /mnt/data-s3-fuse
</code></pre>
<p>What could be the permanent solution for this issue?</p>
<p>My Daemon file</p>
<pre><code>apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: s3-provider
  name: s3-provider
  namespace: airflow
spec:
  template:
    metadata:
      labels:
        app: s3-provider
    spec:
      containers:
      - name: s3fuse
        image: image
        lifecycle:
          preStop:
            exec:
              command: [&quot;/bin/sh&quot;,&quot;-c&quot;,&quot;umount -f /opt/airflow/dags&quot;]
        securityContext:
          privileged: true
          capabilities:
            add:
            - SYS_ADMIN
        # use ALL  entries in the config map as environment variables
        envFrom:
        - configMapRef:
            name: s3-config
        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        - name: mntdatas3fs
          mountPath: /opt/airflow/dags:shared
      volumes:
      - name: devfuse
        hostPath:
          path: /dev/fuse
      - name: mntdatas3fs
        hostPath:
          path: /mnt/data-s3-fuse
</code></pre>
<p>and my pod yaml is</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: test-pd
  namespace: airflow
spec:
  containers:
  - image: nginx
    name: s3-test-container
    securityContext:
      privileged: true
    volumeMounts:
    - name: mntdatas3fs
      mountPath: /opt/airflow/dags:shared
    livenessProbe:
      exec:
        command: [&quot;ls&quot;, &quot;/opt/airflow/dags&quot;]
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
  volumes:
  - name: mntdatas3fs
    hostPath:
      path: /mnt/data-s3-fuse
</code></pre>
<p>I am using the below code for the s3 kubernetes fuse.</p>
<p><a href="https://github.com/freegroup/kube-s3" rel="nofollow noreferrer">https://github.com/freegroup/kube-s3</a></p>

<p>Ok, I think I solved it.
It seems like sometimes the pod looses the connection, resulting in &quot;Transport not connected&quot;.
The workaround I found to fix this is to add an init container, that tries to unmount the folder before. That seems to fix the issue.
Note, you want to mount a higher-level folder, so you have access to the node.
Will let it run and see if it comes back, it seems to have fixed the issue once here:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: s3-provider
  name: s3-provider
spec:
  selector:
    matchLabels:
      app: s3-provider
  template:
    metadata:
      labels:
        app: s3-provider
    spec:
      initContainers:
      - name: init-myservice
        image: bash
        command: ['bash', '-c', 'umount -l /mnt/data-s3-fs/root ; true']
        securityContext:
          privileged: true
          capabilities:
            add:
            - SYS_ADMIN
        # use ALL  entries in the config map as environment variables
        envFrom:
        - configMapRef:
            name: s3-config
        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        - name: mntdatas3fs-init
          mountPath: /mnt:shared
      containers:
      - name: s3fuse
        image: 963341077747.dkr.ecr.us-east-1.amazonaws.com/kube-s3:1.0
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command: [&quot;bash&quot;, &quot;-c&quot;, &quot;umount -f /srv/s3-mount/root&quot;]
        securityContext:
          privileged: true
          capabilities:
            add:
            - SYS_ADMIN
        # use ALL  entries in the config map as environment variables
        envFrom:
        - configMapRef:
            name: s3-config
        env:
        - name: S3_BUCKET
          value: s3-mount
        - name: MNT_POINT
          value: /srv/s3-mount/root
        - name: IAM_ROLE
          value: none
        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        - name: mntdatas3fs
          mountPath: /srv/s3-mount/root:shared
      volumes:
      - name: devfuse
        hostPath:
          path: /dev/fuse
      - name: mntdatas3fs
        hostPath:
          type: DirectoryOrCreate
          path: /mnt/data-s3-fs/root
      - name: mntdatas3fs-init
        hostPath:
          type: DirectoryOrCreate
          path: /mnt
</code></pre>
