<p>I create two containers in two pods in k8s, <code>elasticsearch</code> and <code>kibana</code>.
<code>kibana</code> container needs to access elasticsearch endpoint on the port 9200. So I set the env to be <code>http://es-cluster-1.elasticsearch.default.svc.local:9200</code> based on this doc: <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/</a>.</p>
<p>But in kibana logs I can see it can't reach to this endpoint. What did I do wrong?</p>
<p><code>{&quot;type&quot;:&quot;log&quot;,&quot;@timestamp&quot;:&quot;2020-12-31T07:44:05Z&quot;,&quot;tags&quot;:[&quot;error&quot;,&quot;elasticsearch&quot;,&quot;data&quot;],&quot;pid&quot;:6,&quot;message&quot;:&quot;[ConnectionError]: getaddrinfo ENOTFOUND es-cluster-1.elasticsearch-entrypoint.default.svc.local es-cluster-1.elasticsearch-entrypoint.default.svc.local:9200&quot;}</code></p>
<pre><code>
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
spec:
  serviceName: elasticsearch-entrypoint
  replicas: 1
  selector:
    matchLabels:
      name: elasticsearch
  template:
    metadata:
      labels:
        name: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: elasticsearch:7.10.1
          ports:
            - containerPort: 9200
              name: rest
            - containerPort: 9300
              name: inter-node
          env:
            - name: discovery.type
              value: single-node
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kibana
  template:
    metadata:
      labels:
        name: kibana
    spec:
      containers:
        - name: kibana
          image: kibana:7.10.1
          ports:
            - containerPort: 5601
          env:
            - name: ELASTICSEARCH_HOSTS
              value: http://es-cluster-1.elasticsearch-entrypoint.default.svc.local:9200
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-entrypoint
  namespace: default
spec:
  clusterIP: None
  selector:
    name: elasticsearch
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-entrypoint
  namespace: default
spec:
  selector:
    name: kibana
  ports:
  - port: 5601

</code></pre>

Joey Yi Zhao
<p>I tried to run your YAML config, but it doesn't work, the pod keep restarting</p>
<p><a href="https://stackoverflow.com/a/65517865/7166519">like @Dom said</a>, your service should be available at &quot;elasticsearch-entrypoint.svc.cluster.local&quot;, only iff your pod is running</p>
<p>For more regarding DNS, read the docs <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id</a></p>
<p>That being said, lets fix your YAML config,</p>
<p>now that your pod is not running, what should we do? Check the logs and so did I and I found interesting log entry:</p>
<pre><code>ERROR: [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
</code></pre>
<p>So, there must be something wrong with elasticsearch discovery service, ok.
Luckily I found this blog by rancher labs <a href="https://rancher.com/blog/2018/2018-11-22-deploying-elasticsearch/" rel="nofollow noreferrer">https://rancher.com/blog/2018/2018-11-22-deploying-elasticsearch/</a> about running elasticserach on k8s, I read their config and tweak your original config to become</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: es-config
data:
  elasticsearch.yml: |
    network.host: &quot;0.0.0.0&quot;
    cluster.initial_master_nodes: es-cluster-0
    discovery.zen.minimum_master_nodes: 1
    xpack.security.enabled: false
    xpack.monitoring.enabled: false
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
spec:
  serviceName: elasticsearch-entrypoint
  replicas: 1
  selector:
    matchLabels:
      name: elasticsearch
  template:
    metadata:
      labels:
        name: elasticsearch
    spec:
      volumes:
      - name: elasticsearch-config
        configMap:
          name: es-config
          items:
            - key: elasticsearch.yml
              path: elasticsearch.yml
      containers:
        - name: elasticsearch
          image: elasticsearch:7.10.1
          ports:
            - containerPort: 9200
              name: rest
            - containerPort: 9300
              name: inter-node
          volumeMounts:
          - name: elasticsearch-config
            mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
            subPath: elasticsearch.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      name: kibana
  template:
    metadata:
      labels:
        name: kibana
    spec:
      containers:
        - name: kibana
          image: kibana:7.10.1
          ports:
            - containerPort: 5601
          env:
            - name: ELASTICSEARCH_HOSTS
              value: http://elasticsearch-entrypoint.default.svc.cluster.local:9200
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-entrypoint
  namespace: default
spec:
  clusterIP: None
  selector:
    name: elasticsearch
  ports:
  - port: 9200
    name: rest
  - port: 9300
    name: inter-node
---
apiVersion: v1
kind: Service
metadata:
  name: kibana-entrypoint
  namespace: default
spec:
  selector:
    name: kibana
  ports:
  - port: 5601
</code></pre>
<p>The difference is that now the ES pod read config from our specified config map, to make sure the service is running properly we specify livenessProbe in our YAML spec or manually using</p>
<pre><code>/ &gt; dig +search elasticsearch-entrypoint

; &lt;&lt;&gt;&gt; DiG 9.16.6 &lt;&lt;&gt;&gt; +search elasticsearch-entrypoint
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 27045
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: ca540a28cfb103a8 (echoed)
;; QUESTION SECTION:
;elasticsearch-entrypoint.default.svc.cluster.local. IN A

;; ANSWER SECTION:
elasticsearch-entrypoint.default.svc.cluster.local. 30 IN A 10.244.82.52

;; Query time: 1 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Thu Dec 31 09:33:32 UTC 2020
;; MSG SIZE  rcvd: 157

/ #
/ &gt; nc -vz elasticsearch-entrypoint.default.svc.cluster.local. 9200
elasticsearch-entrypoint.default.svc.cluster.local. (10.244.82.52:9200) open
</code></pre>
