<p>I am running Kubernetes using OKD 4.11 (running on vSphere) and have validated the basic functionality (including dyn. volume provisioning) using applications (like nginx).</p>
<p>I also applied</p>
<p><code>oc adm policy add-scc-to-group anyuid system:authenticated</code></p>
<p>to allow authenticated users to use anyuid (which seems to have been required to deploy the nginx example I was testing with).</p>
<p>Then I installed ECK using <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html" rel="nofollow noreferrer">this quickstart</a> with kubectl to install the CRD and RBAC manifests. This seems to have worked.</p>
<p>Then I deployed the most basic ElasticSearch quickstart example with <code>kubectl apply -f quickstart.yaml</code> using this manifest:</p>
<pre><code>  apiVersion: elasticsearch.k8s.elastic.co/v1
  kind: Elasticsearch
  metadata:
    name: quickstart
  spec:
    version: 8.4.2
    nodeSets:
    - name: default
      count: 1
      config:
        node.store.allow_mmap: false
</code></pre>
<p>The deployment proceeds as expected, pulling image and starting container, but ends in a CrashLoopBackoff with the following error from ElasticSearch at the end of the log:</p>
<pre><code> &quot;elasticsearch.cluster.name&quot;:&quot;quickstart&quot;,
 &quot;error.type&quot;:&quot;java.lang.IllegalStateException&quot;,
 &quot;error.message&quot;:&quot;failed to obtain node locks, tried 
 [/usr/share/elasticsearch/data]; maybe these locations 
 are not writable or multiple nodes were started on the same data path?&quot;
</code></pre>
<p>Looking into the storage, the PV and PVC are created successfully, the output of <code>kubectl get pv,pvc,sc -A -n my-namespace</code> is:</p>
<pre><code>  NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                  STORAGECLASS   REASON   AGE
  persistentvolume/pvc-9d7b57db-8afd-40f7-8b3d-6334bdc07241   1Gi        RWO            Delete           Bound    my-namespace/elasticsearch-data-quickstart-es-default-0   thin                    41m

  NAMESPACE                       NAME                                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  my-namespace                       persistentvolumeclaim/elasticsearch-data-quickstart-es-default-0   Bound    pvc-9d7b57db-8afd-40f7-8b3d-6334bdc07241   1Gi        RWO            thin           41m

  NAMESPACE   NAME                                         PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
              storageclass.storage.k8s.io/thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate              false                  19d
              storageclass.storage.k8s.io/thin-csi         csi.vsphere.vmware.com         Delete          WaitForFirstConsumer   true                   19d
</code></pre>
<p>Looking at the pod yaml, it appears that the volume is correctly attached :</p>
<pre><code>volumes:
  - name: elasticsearch-data
    persistentVolumeClaim:
      claimName: elasticsearch-data-quickstart-es-default-0
  - name: downward-api
    downwardAPI:
      items:
        - path: labels
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.labels
      defaultMode: 420
  ....
  volumeMounts:
    ...
    - name: elasticsearch-data
      mountPath: /usr/share/elasticsearch/data
</code></pre>
<p>I cannot understand why the volume would be read-only or rather why ES cannot create the lock.</p>
<p>I did find this <a href="https://stackoverflow.com/questions/72654220/elasticsearch-kubernetes-crashloopbackoff">similar issue</a>, but I am not sure how to apply the UID permissions (in general I am fairly naive about the way permissions work in OKD) when when working with ECK.</p>
<p>Does anyone with deeper K8s / OKD or ECK/ElasticSearch knowledge have an idea how to better isolate and/or resolve this issue?</p>
<p>Update: I believe this has something to do with <a href="https://github.com/elastic/cloud-on-k8s/issues/2791#issuecomment-608341041" rel="nofollow noreferrer">this issue</a> and am researching the optionas related to OKD.</p>

sprockets
<p>For posterity, the ECK starts an init container that <em>should</em> take care of the <code>chown</code> on the data volume, but can only do so if it is running as root.</p>
<p>The resolution for me was documented here:
<a href="https://repo1.dso.mil/dsop/elastic/elasticsearch/elasticsearch/-/issues/7" rel="nofollow noreferrer">https://repo1.dso.mil/dsop/elastic/elasticsearch/elasticsearch/-/issues/7</a></p>
<p>The manifest now looks like this:</p>
<pre><code>  apiVersion: elasticsearch.k8s.elastic.co/v1
  kind: Elasticsearch
  metadata:
    name: quickstart
  spec:
    version: 8.4.2
    nodeSets:
    - name: default
      count: 1
      config:
        node.store.allow_mmap: false
    # run init container as root to chown the volume to uid 1000
      podTemplate:
        spec:
          securityContext:
            runAsUser: 1000
            runAsGroup: 0
          initContainers:
          - name: elastic-internal-init-filesystem
            securityContext:
              runAsUser: 0
              runAsGroup: 0
</code></pre>
<p>And the pod starts up and can write to the volume as uid 1000.</p>
