<p>I am trying to install an elastic statefulset in my GKE cluster but it's throwing an error and am unable to identify the error here this is the log that I had got inside the pod. Can someone help me? I have given the error logs as well as the elasticsearch_statefulset.yml file.</p>
<pre><code>{&quot;type&quot;: &quot;server&quot;, &quot;timestamp&quot;: &quot;2021-12-16T09:30:49,473Z&quot;, &quot;level&quot;: &quot;WARN&quot;, &quot;component&quot;: &quot;o.e.d.SeedHostsResolver&quot;, &quot;cluster.name&quot;: &quot;k8s-logs&quot;, &quot;node.name&quot;: &quot;es-cluster-0&quot;, &quot;message&quot;: &quot;failed to resolve host [es-cluster-2.elasticsearch]&quot;,
&quot;stacktrace&quot;: [&quot;java.net.UnknownHostException: es-cluster-2.elasticsearch&quot;,
&quot;at java.net.InetAddress$CachedAddresses.get(InetAddress.java:800) ~[?:?]&quot;,
&quot;at java.net.InetAddress.getAllByName0(InetAddress.java:1495) ~[?:?]&quot;,
&quot;at java.net.InetAddress.getAllByName(InetAddress.java:1354) ~[?:?]&quot;,
&quot;at java.net.InetAddress.getAllByName(InetAddress.java:1288) ~[?:?]&quot;,
&quot;at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:548) ~[elasticsearch-7.9.1.jar:7.9.1]&quot;,
&quot;at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:490) ~[elasticsearch-7.9.1.jar:7.9.1]&quot;,
&quot;at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:855) ~[elasticsearch-7.9.1.jar:7.9.1]&quot;,
&quot;at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.9.1.jar:7.9.1]&quot;,
&quot;at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]&quot;,
&quot;at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) ~[elasticsearch-7.9.1.jar:7.9.1]&quot;,
&quot;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]&quot;,
&quot;at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]&quot;,
&quot;at java.lang.Thread.run(Thread.java:832) [?:?]&quot;] }
</code></pre>
<p>This is the yml I used to configure the stateful set :</p>
<pre><code>apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: es-cluster
    namespace: kube-logging
  spec:
    serviceName: elasticsearch
    replicas: 3
    selector:
      matchLabels:
        app: elasticsearch
    template:
      metadata:
        labels:
          app: elasticsearch
      spec:
        containers:
        - name: elasticsearch
          image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
          resources: {}
          ports:
          - containerPort: 9200
            name: rest
            protocol: TCP
          - containerPort: 9300
            name: inter-node
            protocol: TCP
          volumeMounts:
          - name: data
            mountPath: /usr/share/elasticsearch/data
          env:
            - name: cluster.name
              value: k8s-logs
            - name: node.name
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: discovery.seed_hosts
              value: &quot;es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch&quot;
            - name: cluster.initial_master_nodes
              value: &quot;es-cluster-0,es-cluster-1,es-cluster-2&quot;
            - name: ES_JAVA_OPTS
              value: &quot;-Xms512m -Xmx512m&quot;
        initContainers:
        - name: fix-permissions
          image: busybox
          command: [&quot;sh&quot;, &quot;-c&quot;, &quot;chown -R 1000:1000 /usr/share/elasticsearch/data&quot;]
          securityContext:
            privileged: true
          volumeMounts:
          - name: data
            mountPath: /usr/share/elasticsearch/data
        - name: increase-vm-max-map
          image: busybox
          command: [&quot;sysctl&quot;, &quot;-w&quot;, &quot;vm.max_map_count=262144&quot;]
          securityContext:
            privileged: true
        - name: increase-fd-ulimit
          image: busybox
          command: [&quot;sh&quot;, &quot;-c&quot;, &quot;ulimit -n 65536&quot;]
          securityContext:
            privileged: true
    volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app: elasticsearch
      spec:
        accessModes: [ &quot;ReadWriteOnce&quot; ]
        storageClassName: standard
        resources:
          requests:
            storage: 10Gi
</code></pre>
<p>Service file which I used :</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: kube-logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  type: LoadBalancer
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node
</code></pre>
<p>Output for kubectl get statefulset -n kube-logging es-cluster -oyaml</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: &quot;2021-12-20T06:31:55Z&quot;
  generation: 1
  name: es-cluster
  namespace: kube-logging
  resourceVersion: &quot;43285&quot;
  uid: a1730c94-1aa5-461c-ba73-9af617ea4c42
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: elasticsearch
  serviceName: elasticsearch-headless
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch
    spec:
      containers:
      - env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: discovery.seed_hosts
          value: es-cluster-0.elasticsearch
        - name: cluster.initial_master_nodes
          value: es-cluster-0
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        image: docker.elastic.co/elasticsearch/elasticsearch:7.9.1
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        image: busybox
        imagePullPolicy: Always
        name: fix-permissions
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: busybox
        imagePullPolicy: Always
        name: increase-vm-max-map
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - command:
        - sh
        - -c
        - ulimit -n 65536
        image: busybox
        imagePullPolicy: Always
        name: increase-fd-ulimit
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: gold
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 1
  currentRevision: es-cluster-6f5bd744c7
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updateRevision: es-cluster-6f5bd744c7
  updatedReplicas: 1
</code></pre>
<p>Output for kubectl get svc -n kube-logging elasticsearch-headless -oyaml :</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{&quot;ingress&quot;:true}'
  creationTimestamp: &quot;2021-12-20T06:31:40Z&quot;
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app: elasticsearch
  name: elasticsearch-headless
  namespace: kube-logging
  resourceVersion: &quot;43384&quot;
  uid: 80d10b9b-1f22-48ce-ba39-7d5801dc91ee
spec:
  clusterIP: 10.8.4.128
  clusterIPs:
  - 10.8.4.128
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: rest
    nodePort: 30625
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: inter-node
    nodePort: 30056
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    app: elasticsearch
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 35.222.11.96
</code></pre>

sidharth vijayakumar
<p>Additionally to the service you created to expose elastic search outside the cluster, you also need an <code>headless</code> service so that each node / Pod of the elastic cluster can communicate with each other.</p>
<p>I would do the following:</p>
<p>First, inside the spec of the <code>StatefulSet</code>, change <code>spec.serviceName</code> to another value, such as for example <code>elasticsearch-headless</code></p>
<p>Second. create the new service with the following:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  # must be the same as the StatefulSet spec.serviceName
  name: elasticsearch-headless
  namespace: kube-logging
  labels:
    app: elasticsearch
spec:
  type: ClusterIP
  # headless service, can be used by elastic Pods to contact each other
  clusterIP: None
  ports:
  - name: rest
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: inter-node
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    app: elasticsearch
</code></pre>
<hr />
<p>Some docs on <a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services" rel="nofollow noreferrer">Headless Services</a></p>
<p>Also, you may be interested into checking <a href="https://github.com/elastic/helm-charts" rel="nofollow noreferrer">HELM Charts</a> and <a href="https://www.elastic.co/elastic-cloud-kubernetes" rel="nofollow noreferrer">ECK</a> because there is several stuff ready to be used in order to deploy production-ready elastic search clusters.</p>
