<p>Am working on Kubernetes - Elastic search deployment,</p>
<p>I have followed documentation provided by elastic.co (<a href="https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-deploy-elasticsearch.html" rel="nofollow noreferrer">https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-deploy-elasticsearch.html</a>)</p>
<p>My YAML file for elastic is below:</p>
<pre><code>cat &lt;&lt;EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.8.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms2g -Xmx2g
          resources:
            requests:
              memory: 4Gi
              cpu: 0.5
            limits:
              memory: 4Gi
              cpu: 2
EOF
</code></pre>
<p>But am getting below error, when I describe the pod created</p>
<pre><code>Name:           quickstart-es-default-0
Namespace:      default
Priority:       0
Node:           &lt;none&gt;
Labels:         common.k8s.elastic.co/type=elasticsearch
                controller-revision-hash=quickstart-es-default-55759bb696
                elasticsearch.k8s.elastic.co/cluster-name=quickstart
                elasticsearch.k8s.elastic.co/config-hash=178912897
                elasticsearch.k8s.elastic.co/http-scheme=https
                elasticsearch.k8s.elastic.co/node-data=true
                elasticsearch.k8s.elastic.co/node-ingest=true
                elasticsearch.k8s.elastic.co/node-master=true
                elasticsearch.k8s.elastic.co/node-ml=true
                elasticsearch.k8s.elastic.co/statefulset-name=quickstart-es-default
                elasticsearch.k8s.elastic.co/version=7.8.0
                statefulset.kubernetes.io/pod-name=quickstart-es-default-0
Annotations:    co.elastic.logs/module: elasticsearch
Status:         Pending
IP:             
IPs:            &lt;none&gt;
Controlled By:  StatefulSet/quickstart-es-default
Init Containers:
  elastic-internal-init-filesystem:
    Image:      docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      bash
      -c
      /mnt/elastic-internal/scripts/prepare-fs.sh
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_IP:     (v1:status.podIP)
      POD_NAME:  quickstart-es-default-0 (v1:metadata.name)
      POD_IP:     (v1:status.podIP)
      POD_NAME:  quickstart-es-default-0 (v1:metadata.name)
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-bin-local from elastic-internal-elasticsearch-bin-local (rw)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/elasticsearch-config-local from elastic-internal-elasticsearch-config-local (rw)
      /mnt/elastic-internal/elasticsearch-plugins-local from elastic-internal-elasticsearch-plugins-local (rw)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/transport-certificates from elastic-internal-transport-certificates (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
  sysctl:
    Image:      docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      sh
      -c
      sysctl -w vm.max_map_count=262144
    Environment:
      POD_IP:     (v1:status.podIP)
      POD_NAME:  quickstart-es-default-0 (v1:metadata.name)
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Containers:
  elasticsearch:
    Image:       docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    Ports:       9200/TCP, 9300/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      500m
      memory:   4Gi
    Readiness:  exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3
    Environment:
      ES_JAVA_OPTS:              -Xms2g -Xmx2g
      POD_IP:                     (v1:status.podIP)
      POD_NAME:                  quickstart-es-default-0 (v1:metadata.name)
      PROBE_PASSWORD_PATH:       /mnt/elastic-internal/probe-user/elastic-internal-probe
      PROBE_USERNAME:            elastic-internal-probe
      READINESS_PROBE_PROTOCOL:  https
      HEADLESS_SERVICE_NAME:     quickstart-es-default
      NSS_SDB_USE_CACHE:         no
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  elasticsearch-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-data-quickstart-es-default-0
    ReadOnly:   false
  downward-api:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -&gt; labels
  elastic-internal-elasticsearch-bin-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  &lt;unset&gt;
  elastic-internal-elasticsearch-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quickstart-es-default-es-config
    Optional:    false
  elastic-internal-elasticsearch-config-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  &lt;unset&gt;
  elastic-internal-elasticsearch-plugins-local:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  &lt;unset&gt;
  elastic-internal-http-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quickstart-es-http-certs-internal
    Optional:    false
  elastic-internal-probe-user:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quickstart-es-internal-users
    Optional:    false
  elastic-internal-remote-certificate-authorities:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quickstart-es-remote-ca
    Optional:    false
  elastic-internal-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      quickstart-es-scripts
    Optional:  false
  elastic-internal-transport-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quickstart-es-transport-certificates
    Optional:    false
  elastic-internal-unicast-hosts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      quickstart-es-unicast-hosts
    Optional:  false
  elastic-internal-xpack-file-realm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  quickstart-es-xpack-file-realm
    Optional:    false
  elasticsearch-logs:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
    SizeLimit:   &lt;unset&gt;
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  &lt;unknown&gt;          default-scheduler  running &quot;VolumeBinding&quot; filter plugin for pod &quot;quickstart-es-default-0&quot;: pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  &lt;unknown&gt;          default-scheduler  running &quot;VolumeBinding&quot; filter plugin for pod &quot;quickstart-es-default-0&quot;: pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  20m (x3 over 21m)  default-scheduler  0/2 nodes are available: 2 Insufficient memory.

</code></pre>
<p><strong>Question 2:</strong>
I have created two ec2 servers(t2 large). Master and worker.
I am using 300 GB HDD for both the servers.</p>
<p><strong>I have following pv</strong></p>
<pre><code>NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv0001 200Gi RWO Retain Available
</code></pre>
<p>I am using the below code to create a claim for my elastic.</p>
<pre><code>apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.8.0
nodeSets:
name: default count: 1 config: node.master: true node.data: true node.ingest: true node.store.allow_mmap: false volumeClaimTemplates:
metadata: name: elasticsearch-data spec: accessModes:
ReadWriteOnce resources: requests: storage: 200Gi storageClassName: gp2 EOF
</code></pre>
<p><strong>Storage class:</strong>(I created and make it as default)</p>
<pre><code>NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION 
gp2 (default) kubernetes.io/aws-ebs Delete Immediate false
</code></pre>
<p><strong>Kubectl get pv</strong></p>
<pre><code>Labels:          &lt;none&gt;
Annotations:     Finalizers:  [kubernetes.io/pv-protection]
StorageClass:    
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        200Gi
Node Affinity:   &lt;none&gt;
</code></pre>
<p><strong>kubectl get pvc</strong></p>
<pre><code>Namespace:     default
StorageClass:  gp2
Status:        Pending
Volume:        
Labels:        common.k8s.elastic.co/type=elasticsearch
               elasticsearch.k8s.elastic.co/cluster-name=quickstart
               elasticsearch.k8s.elastic.co/statefulset-name=quickstart-es-default
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Mounted By:    quickstart-es-default-0
Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  ProvisioningFailed  61s (x18 over 24m)  persistentvolume-controller  Failed to provision volume with StorageClass &quot;gp2&quot;: Failed to get AWS Cloud Provider. GetCloudProvider returned &lt;nil&gt; instead
</code></pre>
<p><strong>But am getting below error:</strong>
running &quot;VolumeBinding&quot; filter plugin for pod &quot;quickstart-es-default-0&quot;: pod has unbound immediate PersistentVolumeClaims</p>
<p><strong>My volume is in Ec2 EBS</strong></p>

Rajeev Uppala
<p>Based on logs from the pod there are 2 issues you need to fix</p>
<hr />
<h2>Resources</h2>
<pre><code>Warning  FailedScheduling  20m (x3 over 21m)  default-scheduler  0/2 nodes are available: 2 Insufficient memory.
</code></pre>
<p>In the docs you provided it is specified that you need atleast 2GiB of memory, you should try to change your request resources from 4Gi to 2Gi in both, limits and requests. As mentioned in the aws <a href="https://aws.amazon.com/ec2/instance-types/" rel="nofollow noreferrer">documentation</a> t2.large have 2vCPU and 8GiB of memory, so with your current requests it´s almost all resources of the vms.</p>
<hr />
<h2>VolumeBinding</h2>
<pre><code>Warning  FailedScheduling  &lt;unknown&gt;          default-scheduler  running &quot;VolumeBinding&quot; filter plugin for pod &quot;quickstart-es-default-0&quot;: pod has unbound immediate PersistentVolumeClaims
</code></pre>
<p>If you do a <code>kubectl describe</code> on the pv and pvc you should be able to see more detail on why it cannot be bound.</p>
<p>I assume it´s because there is no default storage class.</p>
<p>As mentioned in <a href="https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/" rel="nofollow noreferrer">documentation</a></p>
<blockquote>
<p>Depending on the installation method, your Kubernetes cluster may be deployed with an existing StorageClass that is marked as default. This default StorageClass is then used to dynamically provision storage for PersistentVolumeClaims that do not require any specific storage class. See <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims" rel="nofollow noreferrer">PersistentVolumeClaim documentation</a> for details.</p>
</blockquote>
<p>You can check if you have default storage class with</p>
<pre><code>kubectl get storageclass
</code></pre>
<p>Command which can be used to make your storageclass a default one.</p>
<pre><code>kubectl patch storageclass &lt;name_of_storageclass&gt; -p '{&quot;metadata&quot;: {&quot;annotations&quot;:{&quot;storageclass.kubernetes.io/is-default-class&quot;:&quot;true&quot;}}}'
</code></pre>
<hr />
<p>There is related <a href="https://discuss.elastic.co/t/pod-has-unbound-immediate-persistentvolumeclaims/223788" rel="nofollow noreferrer">issue</a> on elastic discuss about that.</p>
<hr />
<p><strong>EDIT</strong></p>
<p>Quoted from the <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming" rel="nofollow noreferrer">documentation</a></p>
<blockquote>
<p>The Retain reclaim policy allows for manual reclamation of the resource. When the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered &quot;released&quot;. But it is not yet available for another claim because the previous claimant's data remains on the volume. An administrator can manually reclaim the volume with the following steps.</p>
</blockquote>
<ul>
<li>Delete the PersistentVolume. The associated storage asset in external infrastructure (such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume) still exists after the PV is deleted.</li>
<li>Manually clean up the data on the associated storage asset accordingly.</li>
<li>Manually delete the associated storage asset, or if you want to reuse the same storage asset, create a new PersistentVolume with the storage asset definition.</li>
</ul>
<p>So please try to delete your pv and pvc and create them again.</p>
