<p>I have an EKS cluster that is Fargate only. I really don't want to have to manage instances myself. I'd like to deploy Prometheus to it - which requires a persistent volume. <a href="https://aws.amazon.com/blogs/aws/new-aws-fargate-for-amazon-eks-now-supports-amazon-efs/" rel="nofollow noreferrer">As of two months ago this should be possible with EFS</a> (managed NFS share)  I feel that I am almost there but I cannot figure out what the current issue is</p>
<p>What I have done:</p>
<ul>
<li>Set up an EKS fargate cluster and a suitable fargate profile</li>
<li>Set up an EFS with an appropriate security group</li>
<li>Installed the CSI driver and validated the EFS as per <a href="https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html" rel="nofollow noreferrer">AWS walkthough</a></li>
</ul>
<p>All good so far</p>
<p>I set up the persistent volume claims (which I understand must be done statically) with:</p>
<pre><code>kubectl apply -f pvc/
</code></pre>
<p>where</p>
<pre><code>tree pvc/
pvc/
├── two_pvc.yml
└── ten_pvc.yml
</code></pre>
<p>and</p>
<pre><code>cat pvc/*

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv-two
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-ec0e1234
apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv-ten
spec:
  capacity:
    storage: 8Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-ec0e1234
</code></pre>
<p>then</p>
<pre><code>helm upgrade --install myrelease-helm-02 prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass=&quot;efs-sc&quot;,server.persistentVolume.storageClass=&quot;efs-sc&quot;
</code></pre>
<p><strong>What happens?</strong></p>
<p>The prometheus alertmanager comes up fine with its pvc. So do the other pods for this deployment, but the prometheus server goes to crashloopbackoff with</p>
<pre><code>invalid capacity 0 on filesystem
</code></pre>
<p><strong>Diagnostics</strong></p>
<pre><code>kubectl get pv -A
NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                               STORAGECLASS   REASON   AGE
efs-pv-ten                    8Gi        RWO            Retain           Bound      prometheus/myrelease-helm-02-prometheus-server         efs-sc                  11m
efs-pv-two                    2Gi        RWO            Retain           Bound      prometheus/myrelease-helm-02-prometheus-alertmanager   efs-sc                  11m
</code></pre>
<p>and</p>
<pre><code>kubectl get pvc -A
NAMESPACE    NAME                                     STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus   myrelease-helm-02-prometheus-alertmanager   Bound    efs-pv-two   2Gi        RWO            efs-sc         12m
prometheus   myrelease-helm-02-prometheus-server         Bound    efs-pv-ten   8Gi        RWO            efs-sc         12m
</code></pre>
<p><code>describe pod</code> just shows 'error'</p>
<p>lastly, this (from a colleague):</p>
<pre><code>level=info ts=2020-10-09T15:17:08.898Z caller=main.go:346 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.21.0, branch=HEAD, revision=e83ef207b6c2398919b69cd87d2693cfc2fb4127)&quot;
level=info ts=2020-10-09T15:17:08.898Z caller=main.go:347 build_context=&quot;(go=go1.15.2, user=root@a4d9bea8479e, date=20200911-11:35:02)&quot;
level=info ts=2020-10-09T15:17:08.898Z caller=main.go:348 host_details=&quot;(Linux 4.14.193-149.317.amzn2.x86_64 #1 SMP Thu Sep 3 19:04:44 UTC 2020 x86_64 myrelease-helm-02-prometheus-server-85765f9895-vxrkn (none))&quot;
level=info ts=2020-10-09T15:17:08.898Z caller=main.go:349 fd_limits=&quot;(soft=1024, hard=4096)&quot;
level=info ts=2020-10-09T15:17:08.898Z caller=main.go:350 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
level=error ts=2020-10-09T15:17:08.901Z caller=query_logger.go:87 component=activeQueryTracker msg=&quot;Error opening query log file&quot; file=/data/queries.active err=&quot;open /data/queries.active: permission denied&quot;
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7fffeb6e85ee, 0x5, 0x14, 0x30ca080, 0xc000d43620, 0x30ca080)
    /app/promql/query_logger.go:117 +0x4cf
main.main()
    /app/cmd/prometheus/main.go:377 +0x510c
</code></pre>
<p>Beyond the appearance of a permissions issue I am baffled - I know that the storage 'works' and is accessible - the other pod in the deployment seems happy with it - but not this one.</p>

<p>Working now - and writing up here for the common good. Thanks to <a href="https://www.reddit.com/r/aws/comments/j93djz/permissions_error_trying_to_run_prometheus_on_eks/" rel="noreferrer">/u/EmiiKhaos on reddit</a> for the suggestions where to look</p>
<p><strong>Problem:</strong></p>
<p>EFS shares are <code>root:root</code> only and prometheus forbids running pods as root.</p>
<p><strong>Solution:</strong></p>
<ul>
<li>create an EFS access point for each pod requiring a persistent
volume to permit access for a specified user.</li>
<li>Specify these access points for the persistent volumes</li>
<li>apply a suitable security context to run the pods as the matching user</li>
</ul>
<p><strong>Method:</strong></p>
<p>Create 2x EFS access points something like:</p>
<pre><code>{
    &quot;Name&quot;: &quot;prometheuserver&quot;,
    &quot;AccessPointId&quot;: &quot;fsap-&lt;hex01&gt;&quot;,
    &quot;FileSystemId&quot;: &quot;fs-ec0e1234&quot;,
    &quot;PosixUser&quot;: {
        &quot;Uid&quot;: 500,
        &quot;Gid&quot;: 500,
        &quot;SecondaryGids&quot;: [
            2000
        ]
    },
    &quot;RootDirectory&quot;: {
        &quot;Path&quot;: &quot;/prometheuserver&quot;,
        &quot;CreationInfo&quot;: {
            &quot;OwnerUid&quot;: 500,
            &quot;OwnerGid&quot;: 500,
            &quot;Permissions&quot;: &quot;0755&quot;
        }
    }
},
{
    &quot;Name&quot;: &quot;prometheusalertmanager&quot;,
    &quot;AccessPointId&quot;: &quot;fsap-&lt;hex02&gt;&quot;,
    &quot;FileSystemId&quot;: &quot;fs-ec0e1234&quot;,
    &quot;PosixUser&quot;: {
        &quot;Uid&quot;: 501,
        &quot;Gid&quot;: 501,
        &quot;SecondaryGids&quot;: [
            2000
        ]
    },
    &quot;RootDirectory&quot;: {
        &quot;Path&quot;: &quot;/prometheusalertmanager&quot;,
        &quot;CreationInfo&quot;: {
            &quot;OwnerUid&quot;: 501,
            &quot;OwnerGid&quot;: 501,
            &quot;Permissions&quot;: &quot;0755&quot;
        }
    }
}
</code></pre>
<p>Update my persistent volumes:</p>
<pre><code>kubectl apply -f pvc/
</code></pre>
<p>to something like:</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheusalertmanager
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-ec0e1234::fsap-&lt;hex02&gt;
---    
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheusserver
spec:
  capacity:
    storage: 8Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-ec0e1234::fsap-&lt;hex01&gt;
</code></pre>
<p>Re-install prometheus as before:</p>
<pre><code>helm upgrade --install myrelease-helm-02 prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass=&quot;efs-sc&quot;,server.persistentVolume.storageClass=&quot;efs-sc&quot;
</code></pre>
<p>Take an educated guess from</p>
<pre><code>kubectl describe pod myrelease-helm-02-prometheus-server -n prometheus
</code></pre>
<p>and</p>
<pre><code>kubectl describe pod myrelease-helm-02-prometheus-alert-manager -n prometheus
</code></pre>
<p>as to which container needed to be specified when setting the security context. Then apply security context to run pods with appropriate <code>uid:gid</code>, e.g. with</p>
<pre><code>kubectl apply -f setpermissions/
</code></pre>
<p>where</p>
<pre><code>cat setpermissions/*
</code></pre>
<p>gives</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: myrelease-helm-02-prometheus-alertmanager
spec:
  securityContext:
    runAsUser: 501
    runAsGroup: 501
    fsGroup: 501
  volumes:
    - name: prometheusalertmanager
  containers:
    - name: prometheusalertmanager
      image: jimmidyson/configmap-reload:v0.4.0
      securityContext:
        runAsUser: 501
        allowPrivilegeEscalation: false        
apiVersion: v1
kind: Pod
metadata:
  name: myrelease-helm-02-prometheus-server
spec:
  securityContext:
    runAsUser: 500
    runAsGroup: 500
    fsGroup: 500
  volumes:
    - name: prometheusserver
  containers:
    - name: prometheusserver
      image: jimmidyson/configmap-reload:v0.4.0
      securityContext:
        runAsUser: 500
        allowPrivilegeEscalation: false
</code></pre>
