<p>I'm going through a not very understandable situation.</p>
<blockquote>
<ul>
<li>Environment
<ul>
<li>Two dedicated nodes with azure <em>centos 8.2</em> (2vcpu, 16G ram), not AKS</li>
<li>1 master node, 1 worker node.</li>
<li><em>kubernetes v1.19.3</em></li>
<li><em>helm v2.16.12</em></li>
<li>Helm charts Elastic (<a href="https://github.com/elastic/helm-charts/tree/7.9.3" rel="nofollow noreferrer">https://github.com/elastic/helm-charts/tree/7.9.3</a>)</li>
</ul>
</li>
</ul>
</blockquote>
<p>At the first time, It works fine with below installation.</p>
<pre><code>## elasticsearch, filebeat
# kubectl apply -f pv.yaml
# helm install -f values.yaml --name elasticsearch elastic/elasticsearch
# helm install --name filebeat --version 7.9.3 elastic/filebeat
</code></pre>
<p><strong>curl elasitcsearchip:9200</strong> and <strong>curl elasitcsearchip:9200/_cat/indices</strong>
show right values.</p>
<p>but after rebooting a worker node, it just keeping ready 0/1 and not working.</p>
<p><code>NAME                      READY   STATUS    RESTARTS   AGE</code><br>
<code>elasticsearch-master-0    0/1     Running   10         71m</code><br>
<code>filebeat-filebeat-67qm2   0/1     Running   4          40m</code><br></p>
<p>In this situation, after removing /mnt/data/nodes and rebooting again
then works fine.</p>
<p>elasticsearch pod has nothing special I think.</p>
<pre><code>#describe
{&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2020-10-26T07:49:49,708Z&quot;, &quot;level&quot;: &quot;INFO&quot;, &quot;component&quot;: &quot;o.e.c.r.a.AllocationService&quot;, &quot;cluster.name&quot;: &quot;elasticsearch&quot;, &quot;node.name&quot;: &quot;elasticsearch-master-0&quot;, &quot;message&quot;: &quot;Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[filebeat-7.9.3-2020.10.26-000001][0]]]).&quot;, &quot;cluster.uuid&quot;: &quot;sWUAXJG9QaKyZDe0BLqwSw&quot;, &quot;node.id&quot;: &quot;ztb35hToRf-2Ahr7olympw&quot;  }

#logs
  Normal   SandboxChanged          4m4s (x3 over 4m9s)   kubelet          Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  4m3s                  kubelet          Container image &quot;docker.elastic.co/elasticsearch/elasticsearch:7.9.3&quot; already present on machine
  Normal   Created                 4m1s                  kubelet          Created container configure-sysctl
  Normal   Started                 4m1s                  kubelet          Started container configure-sysctl
  Normal   Pulled                  3m58s                 kubelet          Container image &quot;docker.elastic.co/elasticsearch/elasticsearch:7.9.3&quot; already present on machine
  Normal   Created                 3m58s                 kubelet          Created container elasticsearch
  Normal   Started                 3m57s                 kubelet          Started container elasticsearch
  Warning  Unhealthy               91s (x14 over 3m42s)  kubelet          Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: &quot;wait_for_status=green&amp;timeout=1s&quot; )
Cluster is not yet ready (request params: &quot;wait_for_status=green&amp;timeout=1s&quot; )

#events
6m1s        Normal    Pulled                    pod/elasticsearch-master-0                     Container image &quot;docker.elastic.co/elasticsearch/elasticsearch:7.9.3&quot; already present on machine
6m1s        Normal    Pulled                    pod/filebeat-filebeat-67qm2                    Container image &quot;docker.elastic.co/beats/filebeat:7.9.3&quot; already present on machine
5m59s       Normal    Started                   pod/elasticsearch-master-0                     Started container configure-sysctl
5m59s       Normal    Created                   pod/elasticsearch-master-0                     Created container configure-sysctl
5m59s       Normal    Created                   pod/filebeat-filebeat-67qm2                    Created container filebeat
5m58s       Normal    Started                   pod/filebeat-filebeat-67qm2                    Started container filebeat
5m56s       Normal    Created                   pod/elasticsearch-master-0                     Created container elasticsearch
5m56s       Normal    Pulled                    pod/elasticsearch-master-0                     Container image &quot;docker.elastic.co/elasticsearch/elasticsearch:7.9.3&quot; already present on machine
5m55s       Normal    Started                   pod/elasticsearch-master-0                     Started container elasticsearch
61s         Warning   Unhealthy                 pod/filebeat-filebeat-67qm2                    Readiness probe failed: elasticsearch: http://elasticsearch-master:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.97.133.135
    dial up... ERROR dial tcp 10.97.133.135:9200: connect: connection refused
59s         Warning   Unhealthy                 pod/elasticsearch-master-0                     Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: &quot;wait_for_status=green&amp;timeout=1s&quot; )
Cluster is not yet ready (request params: &quot;wait_for_status=green&amp;timeout=1s&quot; )
</code></pre>
<p>/mnt/data path has chown 1000:1000</p>
<p>and In case of only elastisearch without filebeat, rebooting has no problem.</p>
<p>I can't figure this out at all. :(</p>
<p>What am I missing?</p>
<hr />
<ol>
<li>pv.yaml</li>
</ol>
<pre><code>kind: PersistentVolume
apiVersion: v1
metadata:
  name: elastic-pv
  labels:
    type: local
    app: elastic
spec:
  storageClassName: local-storage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  claimRef: 
    namespace: default
    name: elasticsearch-master-elasticsearch-master-0
  hostPath:
    path: &quot;/mnt/data&quot;
</code></pre>
<ol start="2">
<li>values.yaml</li>
</ol>
<pre><code>---
clusterName: &quot;elasticsearch&quot;
nodeGroup: &quot;master&quot;

# The service that non master groups will try to connect to when joining the cluster
# This should be set to clusterName + &quot;-&quot; + nodeGroup for your master group
masterService: &quot;&quot;

# Elasticsearch roles that will be applied to this nodeGroup
# These will be set as environment variables. E.g. node.master=true
roles:
  master: &quot;true&quot;
  ingest: &quot;true&quot;
  data: &quot;true&quot;

replicas: 1
minimumMasterNodes: 1

esMajorVersion: &quot;&quot;

# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties
esConfig: {}
#  elasticsearch.yml: |
#    key:
#      nestedkey: value
#  log4j2.properties: |
#    key = value

# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: []
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

# Allows you to load environment variables from kubernetes secret or config map
envFrom: []
# - secretRef:
#     name: env-secret
# - configMapRef:
#     name: config-map

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
#  - name: elastic-certificates
#    secretName: elastic-certificates
#    path: /usr/share/elasticsearch/config/certs
#    defaultMode: 0755

image: &quot;docker.elastic.co/elasticsearch/elasticsearch&quot;
imageTag: &quot;7.9.3&quot;
imagePullPolicy: &quot;IfNotPresent&quot;

podAnnotations: {}
  # iam.amazonaws.com/role: es-cluster

# additionals labels
labels: {}
esJavaOpts: &quot;-Xmx1g -Xms1g&quot;

resources:
  requests:
    cpu: &quot;500m&quot;
    memory: &quot;1Gi&quot;
  limits:
    cpu: &quot;1000m&quot;
    memory: &quot;2Gi&quot;

initResources: {}
  # limits:
  #   cpu: &quot;25m&quot;
  #   # memory: &quot;128Mi&quot;
  # requests:
  #   cpu: &quot;25m&quot;
  #   memory: &quot;128Mi&quot;

sidecarResources: {}
  # limits:
  #   cpu: &quot;25m&quot;
  #   # memory: &quot;128Mi&quot;
  # requests:
  #   cpu: &quot;25m&quot;
  #   memory: &quot;128Mi&quot;

networkHost: &quot;0.0.0.0&quot;

volumeClaimTemplate:
  accessModes: [ &quot;ReadWriteOnce&quot; ]
  storageClassName: local-storage
  resources:
    requests:
      storage: 5Gi

rbac:
  create: false
  serviceAccountAnnotations: {}
  serviceAccountName: &quot;&quot;

podSecurityPolicy:
  create: false
  name: &quot;&quot;
  spec:
    privileged: true
    fsGroup:
      rule: RunAsAny
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
      - secret
      - configMap
      - persistentVolumeClaim

persistence:
  enabled: true
  name: elastic-vc
  labels:
    # Add default labels for the volumeClaimTemplate fo the StatefulSet
    app: elastic
  annotations: {}

extraVolumes: []
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: []
  # - name: extras
  #   mountPath: /usr/share/extras
  #   readOnly: true

extraContainers: []
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']

extraInitContainers: []
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: &quot;&quot;

# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: &quot;kubernetes.io/hostname&quot;

# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this &quot;best effort&quot;
antiAffinity: &quot;hard&quot;

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: &quot;Parallel&quot;

# The environment variables injected by service links are not used, but can lead to slow Elasticsearch boot times when
# there are many services in the current namespace.
# If you experience slow pod startups you probably want to set this to `false`.
enableServiceLinks: true

protocol: http
httpPort: 9200
transportPort: 9300

service:
  labels: {}
  labelsHeadless: {}
  type: ClusterIP
  nodePort: &quot;&quot;
  annotations: {}
  httpPortName: http
  transportPortName: transport
  loadBalancerIP: &quot;&quot;
  loadBalancerSourceRanges: []
  externalTrafficPolicy: &quot;&quot;

updateStrategy: RollingUpdate

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000

securityContext:
  capabilities:
    drop:
    - ALL
  #readOnlyRootFilesystem: false
  runAsNonRoot: true
  runAsUser: 1000

# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

# https://www.elastic.co/guide/en/elasticsearch/reference/7.9/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: &quot;wait_for_status=green&amp;timeout=1s&quot;

## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: &quot;&quot;

imagePullSecrets: []
nodeSelector: {}
tolerations: []
  # - effect: NoSchedule
  #   key: node-role.kubernetes.io/master

# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: &quot;true&quot;
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

nameOverride: &quot;&quot;
fullnameOverride: &quot;&quot;

# https://github.com/elastic/helm-charts/issues/63
masterTerminationFix: false

lifecycle: {}
  # preStop:
  #   exec:
  #     command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;echo Hello from the postStart handler &gt; /usr/share/message&quot;]
  # postStart:
  #   exec:
  #     command:
  #       - bash
  #       - -c
  #       - |
  #         #!/bin/bash
  #         # Add a template to adjust number of shards/replicas
  #         TEMPLATE_NAME=my_template
  #         INDEX_PATTERN=&quot;logstash-*&quot;
  #         SHARD_COUNT=8
  #         REPLICA_COUNT=1
  #         ES_URL=http://localhost:9200
  #         while [[ &quot;$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)&quot; != &quot;200&quot; ]]; do sleep 1; done
  #         curl -XPUT &quot;$ES_URL/_template/$TEMPLATE_NAME&quot; -H 'Content-Type: application/json' -d'{&quot;index_patterns&quot;:['\&quot;&quot;$INDEX_PATTERN&quot;\&quot;'],&quot;settings&quot;:{&quot;number_of_shards&quot;:'$SHARD_COUNT',&quot;number_of_replicas&quot;:'$REPLICA_COUNT'}}'

sysctlInitContainer:
  enabled: true

keystore: []

# Deprecated
# please use the above podSecurityContext.fsGroup instead
fsGroup: &quot;&quot;
</code></pre>

<h2>Issue</h2>
<p>There is an issue with elasticsearch <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes" rel="noreferrer">readiness probe</a> when running on single replica cluster.</p>
<pre><code>Warning  Unhealthy               91s (x14 over 3m42s)  kubelet          Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: &quot;wait_for_status=green&amp;timeout=1s&quot; )
Cluster is not yet ready (request params: &quot;wait_for_status=green&amp;timeout=1s&quot; )
</code></pre>
<h2>Solution</h2>
<p>As mentioned <a href="https://github.com/elastic/helm-charts/issues/783#issuecomment-701037663" rel="noreferrer">here</a> by @adinhodovic</p>
<blockquote>
<p>If your running a single replica cluster add the following helm value:</p>
</blockquote>
<pre><code>clusterHealthCheckParams: &quot;wait_for_status=yellow&amp;timeout=1s&quot;
</code></pre>
<blockquote>
<p>Your status will never go green with a single replica cluster.</p>
<p>The following values should work:</p>
</blockquote>
<pre><code>replicas: 1
minimumMasterNodes: 1
clusterHealthCheckParams: 'wait_for_status=yellow&amp;timeout=1s'
</code></pre>
