<p>I am trying to deploy elasticsearch and kibana to kubernetes using <a href="https://github.com/elastic/helm-charts" rel="nofollow noreferrer">this chart</a> and getting this error inside the kibana container, therefore ingress returns 503 error and container is never ready.</p>
<p>Error:</p>
<pre><code>[2022-11-08T12:30:53.321+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 10.112.130.148:42748, Remote: 10.96.237.95:9200
</code></pre>
<p>Ip adress 10.96.237.95 is a valid elasticsearch service address, and port is right.</p>
<p>When i am doing curl to elasticsearch from inside the kibana container, it successfully returns a response.</p>
<p>Am i missing something in my configurations?</p>
<p><strong>Chart version: 7.17.3</strong></p>
<p>Values for elasticsearch chart:</p>
<pre><code>clusterName: &quot;elasticsearch&quot;
nodeGroup: &quot;master&quot;

createCert: false

roles:
  master: &quot;true&quot;
  data: &quot;true&quot;
  ingest: &quot;true&quot;
  ml: &quot;true&quot;
  transform: &quot;true&quot;
  remote_cluster_client: &quot;true&quot;

protocol: https

replicas: 2

sysctlVmMaxMapCount: 262144

readinessProbe:
   failureThreshold: 3
   initialDelaySeconds: 90
   periodSeconds: 10
   successThreshold: 1
   timeoutSeconds: 10

imageTag: &quot;7.17.3&quot;

extraEnvs:
- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      name: elasticsearch-creds
      key: password
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      name: elasticsearch-creds
      key: username

clusterHealthCheckParams: &quot;wait_for_status=green&amp;timeout=20s&quot;

antiAffinity: &quot;soft&quot;

resources:
  requests:
    cpu: &quot;100m&quot;
    memory: &quot;1Gi&quot;
  limits:
    cpu: &quot;1000m&quot;
    memory: &quot;1Gi&quot;

esJavaOpts: &quot;-Xms512m -Xmx512m&quot;

volumeClaimTemplate:
  accessModes: [&quot;ReadWriteOnce&quot;]
  resources:
    requests:
      storage: 30Gi

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12


secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs
</code></pre>
<p>Values for kibana chart:</p>
<pre><code>elasticSearchHosts: &quot;https://elasticsearch-master:9200&quot;

extraEnvs:
  - name: ELASTICSEARCH_USERNAME
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: username
  - name: ELASTICSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: password
  - name: KIBANA_ENCRYPTION_KEY
    valueFrom:
      secretKeyRef:
        name: encryption-key  
        key: encryption_key

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
    
protocol: https

secretMounts:
  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/kibana/config/certs

imageTag: &quot;7.17.3&quot;

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-issuer
    kubernetes.io/ingress.allow-http: 'false'
  paths:
    - path: /
      pathType: Prefix
      backend:
        serviceName: kibana
        servicePort: 5601
  hosts:
    - host: mydomain.com
      paths:
        - path: /
          pathType: Prefix
          backend:
            serviceName: kibana
            servicePort: 5601
  tls:
    - hosts:
        - mydomain.com
      secretName: mydomain.com
</code></pre>
<p>UPD: tried it with other image version (8.4.1), nothing has changed, i am getting the same error. By the way, logstash is successfully  shipping logs to this elasticsearch instance, so i think problem is in kibana.</p>

<p>Figured it out. It was a complete pain in the ass. I hope these tips will help others:</p>
<ol>
<li><code>xpack.security.http.ssl.enabled</code> should be set to false. I can't find another way around it, but if you do i'd be glad to hear any advices. As i see it, you don't need security for http layer since kibana connects to elastic via transport layer (correct me if i am wrong). Therefore <code>xpack.security.transport.ssl.enabled</code> should be still set to true, but <code>xpack.security.http.ssl.enabled</code> should be set to false. (don't forget to change your <code>protocol</code> field for readinessProbe to http, and also change protocol for elasticsearch in kibana chart to http.</li>
<li><code>ELASTIC_USERNAME</code> env variable is pointless in elasticsearch chart, only password is used, user is always <code>elastic</code></li>
<li><code>ELASTICSEARCH_USERNAME</code> in kibana chart should be actually set to <code>kibana_systems</code> user with according password for that user</li>
</ol>
