<p>I have deployed <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html" rel="nofollow noreferrer">ECK</a> on my kubernetes cluster(all vagrant VMs). The cluster has following config.</p>
<pre><code>NAME       STATUS   ROLES                       AGE   VERSION 
kmaster1   Ready    control-plane,master        27d     v1.21.1 
kworker1   Ready    &lt;none&gt;                      27d     v1.21.1
kworker2   Ready    &lt;none&gt;                      27d     v1.21.1
</code></pre>
<p>I have also setup a loadbalancer with HAProxy. The loadbalancer config is as following(created my own private cert)</p>
<pre><code>frontend http_front
  bind *:80
  stats uri /haproxy?stats
  default_backend http_back

frontend https_front
  bind *:443 ssl crt /etc/ssl/private/mydomain.pem
  stats uri /haproxy?stats
  default_backend https_back


backend http_back
  balance roundrobin
  server kworker1 172.16.16.201:31953
  server kworker2 172.16.16.202:31953
 

backend https_back
  balance roundrobin
  server kworker1 172.16.16.201:31503 check-ssl ssl verify none
  server kworker2 172.16.16.202:31503 check-ssl ssl verify none
</code></pre>
<p>I have also deployed an nginx ingress controller and
31953 is the http port of the nginx controller
31503 is the https port of nginx controller</p>
<pre><code>nginx-ingress    nginx-ingress-controller-service   NodePort    10.103.189.197   &lt;none&gt;        80:31953/TCP,443:31503/TCP   8d    app=nginx-ingress
</code></pre>
<p>I am trying to make the kibana dashboard available outside of the cluster on https. It works fine and I can access it within the cluster. However I am unable to access it via the loadbalancer.</p>
<p>Kibana Pod:</p>
<pre><code>default          quickstart-kb-f74c666b9-nnn27              1/1     Running   4          27d   192.168.41.145   kworker1   &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>I have mapped the loadbalancer to the host</p>
<pre><code>172.16.16.100   elastic.kubekluster.com
</code></pre>
<p>Any request to <a href="https://elastic.kubekluster.com" rel="nofollow noreferrer">https://elastic.kubekluster.com</a> results in the following error(logs from nginx ingress controller  pod)</p>
<pre><code> 10.0.2.15 - - [20/Jun/2021:17:38:14 +0000] &quot;GET / HTTP/1.1&quot; 502 157 &quot;-&quot; &quot;Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0&quot; &quot;-&quot;
2021/06/20 17:38:14 [error] 178#178: *566 upstream prematurely closed connection while reading response header from upstream, client: 10.0.2.15, server: elastic.kubekluster.com, request: &quot;GET / H
TTP/1.1&quot;, upstream: &quot;http://192.168.41.145:5601/&quot;, host: &quot;elastic.kubekluster.com&quot;
</code></pre>
<p>HAproxy logs are following</p>
<pre><code>Jun 20 18:11:45 loadbalancer haproxy[18285]: 172.16.16.1:48662 [20/Jun/2021:18:11:45.782] https_front~ https_back/kworker2 0/0/0/4/4 502 294 - - ---- 1/1/0/0/0 0/0 &quot;GET / HTTP/1.1&quot;
</code></pre>
<p>The ingress is as following</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubekluster-elastic-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: &quot;HTTPS&quot;
    nginx.ingress.kubernetes.io/default-backend: quickstart-kb-http
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-connect-timeout: &quot;600s&quot;
    nginx.ingress.kubernetes.io/proxy-read-timeout: &quot;600s&quot;
    nginx.ingress.kubernetes.io/proxy-send-timeout: &quot;600s&quot;
    nginx.ingress.kubernetes.io/proxy-body-size: 20m
spec:
  tls:
    - hosts:
      - elastic.kubekluster.com
  rules:
  - host: elastic.kubekluster.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: quickstart-kb-http
            port: 
              number: 5601
</code></pre>
<p>I think the request is not reaching the kibana pod because I don't see any logs in the pod. Also I don't understand why Haproxy is sending the request as HTTP instead of HTTPS.
Could you please point to any issues with my configuration?</p>

<p>I hope this helps ... Here is how I set a &quot;LoadBalancer&quot; using nginx and forward traffic to HTTPS services:</p>
<pre><code> kubectl get nodes -o wide 
NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
asd-master-1   Ready    master   72d   v1.19.8   192.168.1.163   213.95.154.199   Ubuntu 20.04.2 LTS   5.8.0-45-generic   docker://20.10.6
asd-node-1     Ready    &lt;none&gt;   72d   v1.19.8   192.168.1.101   &lt;none&gt;           Ubuntu 20.04.1 LTS   5.8.0-45-generic   docker://19.3.15
asd-node-2     Ready    &lt;none&gt;   72d   v1.19.8   192.168.0.5     &lt;none&gt;           Ubuntu 20.04.1 LTS   5.8.0-45-generic   docker://19.3.15
asd-node-3     Ready    &lt;none&gt;   15d   v1.19.8   192.168.2.190   &lt;none&gt;           Ubuntu 20.04.1 LTS   5.8.0-45-generic   docker://19.3.15
</code></pre>
<p>This is the service for nginx:</p>
<pre><code># kubectl get service -n ingress-nginx
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.101.161.113   &lt;none&gt;        80:30337/TCP,443:31996/TCP   72d
</code></pre>
<p>And this is the LoadBalancer configuration:</p>
<pre><code># cat /etc/nginx/nginx.conf
... trimmed ...
stream {
    upstream nginx_http {
        least_conn;
        server asd-master-1:30337 max_fails=3 fail_timeout=5s;
        server asd-node-1:30337 max_fails=3 fail_timeout=5s;
        server asd-node-2:30337 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 80;
        proxy_pass nginx_http;
        proxy_protocol on;
    }

    upstream nginx_https {
        least_conn;
        server 192.168.1.163:31996 max_fails=3 fail_timeout=5s;
        server 192.168.1.101:31996 max_fails=3 fail_timeout=5s;
        server 192.168.0.5:31996 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     443;
        proxy_pass nginx_https;
        proxy_protocol on;
    }

}
</code></pre>
<p>The relevant part is that I am sending the proxy protocol. You will need to configure nginx ingress (in the configuration map) to accept this, and maybe add the correct syntax to haproxy configuration.</p>
<p>This might be something like:</p>
<pre><code>backend https_back
  balance roundrobin
  server kworker1 172.16.16.201:31503 check-ssl ssl verify none send-proxy-v2
  server kworker2 172.16.16.202:31503 check-ssl ssl verify none send-proxy-v2
</code></pre>
<p>Nginx Ingress configuration should be:</p>
<pre><code># kubectl get configmap -n ingress-nginx  nginx-configuration -o yaml
apiVersion: v1
data:
  use-proxy-protocol: &quot;true&quot;
kind: ConfigMap
metadata:
...
</code></pre>
<p>I hope this puts you on the right track.</p>
