<p>Modified the <a href="https://github.com/alerta/docker-alerta/tree/master/contrib/kubernetes/helm/alerta" rel="nofollow noreferrer">helm chart</a> of <code>alerta</code> to spin it up on an <code>istio</code>- enabled GKE cluster.</p>

<p>The alerta pod and its sidecar are created OK</p>

<pre><code>▶ k get pods | grep alerta
alerta-758bc87dcf-tp5nv                        2/2     Running   0          22m
</code></pre>

<p>When I try to access the url that my virtual service is pointing to</p>

<p>I get the following error</p>

<blockquote>
  <p>upstream connect error or disconnect/reset before headers. reset reason: connection termination</p>
</blockquote>

<pre><code>▶ k get vs alerta-virtual-service -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
  creationTimestamp: "2020-04-23T14:45:04Z"
  generation: 1
  name: alerta-virtual-service
  namespace: mynamespace
  resourceVersion: "46844125"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/mynamespace/virtualservices/alerta-virtual-service
  uid: 2a3caa13-3900-4da1-a3a1-9f07322b52b0
spec:
  gateways:
  - mynamespace/istio-ingress-gateway
  hosts:
  - alerta.myurl.com
  http:
  - appendHeaders:
      x-request-start: t=%START_TIME(%s.%3f)%
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: alerta
        port:
          number: 80
    timeout: 60s
</code></pre>

<p>and here is the service</p>

<pre><code>▶ k get svc alerta -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
  creationTimestamp: "2020-04-23T14:45:04Z"
  labels:
    app: alerta
    chart: alerta-0.1.0
    heritage: Tiller
    release: alerta
  name: alerta
  namespace: mynamespace
  resourceVersion: "46844120"
  selfLink: /api/v1/namespaces/mynamespace/services/alerta
  uid: 4d4a3c73-ee42-49e3-a4cb-8c51536a0508
spec:
  clusterIP: 10.8.58.228
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: alerta
    release: alerta
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
</code></pre>

<p>However, when I exec via another pod into the cluster and try to reach the alerta svc endpoint:</p>

<pre><code>/ # curl -IL http://alerta
curl: (56) Recv failure: Connection reset by peer
/ # nc -zv -w 3 alerta 80
alerta (10.8.58.228:80) open
</code></pre>

<p>although as it is evident the port is open</p>

<p>any suggestion?</p>

<p>could it be that the chaining of the 2 proxies is creating issues? nginx behind envoy?</p>

<p>The container logs seem normal</p>

<pre><code>2020-04-23 15:34:40,272 DEBG 'nginx' stdout output: 
ip=\- [\23/Apr/2020:15:34:40 +0000] "\GET / HTTP/1.1" \200 \994 "\-" "\kube-probe/1.15+"
/web | /index.html | &gt; GET / HTTP/1.1
</code></pre>

<p><strong>edit</strong>: Here is a verbose <code>curl</code> with host header explicitly set</p>

<pre><code>/ # curl -v -H "host: alerta.myurl.com" http://alerta:80
* Rebuilt URL to: http://alerta:80/
*   Trying 10.8.58.228...
* TCP_NODELAY set
* Connected to alerta (10.8.58.228) port 80 (#0)
&gt; GET / HTTP/1.1
&gt; host: alerta.myurl.com
&gt; User-Agent: curl/7.57.0
&gt; Accept: */*
&gt;
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
</code></pre>

<p>The <code>nginx</code> config file used by the app/pod is the following FWIW</p>

<pre><code>worker_processes 4;
pid /tmp/nginx.pid;

daemon off;
error_log /dev/stderr info;

events {
        worker_connections 1024;
}

http {
        client_body_temp_path /tmp/client_body;
        fastcgi_temp_path /tmp/fastcgi_temp;
        proxy_temp_path /tmp/proxy_temp;
        scgi_temp_path /tmp/scgi_temp;
        uwsgi_temp_path /tmp/uwsgi_temp;

        include /etc/nginx/mime.types;

        gzip on;
        gzip_disable "msie6";

        log_format main 'ip=\$http_x_real_ip [\$time_local] '
        '"\$request" \$status \$body_bytes_sent "\$http_referer" '
        '"\$http_user_agent"' ;

        log_format scripts '$document_root | $uri | &gt; $request';

        default_type application/octet-stream;

        server {
                listen 8080 default_server;


                access_log /dev/stdout main;
                access_log /dev/stdout scripts;

                location ~ /api {
                        include /etc/nginx/uwsgi_params;
                        uwsgi_pass unix:/tmp/uwsgi.sock;

                        proxy_set_header Host $host:$server_port;
                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                }

                root /web;
                index index.html;
                location / {
                        try_files $uri $uri/ /index.html;
                }
        }
}
</code></pre>

<p><strong>edit 2</strong>: Trying to get the <code>istio</code> authentication policy</p>

<pre><code>✔                                                                                                             18h55m  ⍉
▶ kubectl get peerauthentication.security.istio.io
No resources found.
✔                                                                                                              18h55m
▶ kubectl get peerauthentication.security.istio.io/default -o yaml
Error from server (NotFound): peerauthentications.security.istio.io "default" not found
</code></pre>

<p><strong>edit 3</strong>: when performing <code>curl</code> to the service from within the istio proxy container</p>

<pre><code>▶ k exec -it alerta-758bc87dcf-jzjgj -c istio-proxy bash
istio-proxy@alerta-758bc87dcf-jzjgj:/$ curl -v http://alerta:80
* Rebuilt URL to: http://alerta:80/
*   Trying 10.8.58.228...
* Connected to alerta (10.8.58.228) port 80 (#0)
&gt; GET / HTTP/1.1
&gt; Host: alerta
&gt; User-Agent: curl/7.47.0
&gt; Accept: */*
&gt;
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
</code></pre>

<p>I created new gke cluster with istio 1.5.2, in fact if you check for mtls, there are no resources found</p>

<pre><code>kubectl get peerauthentication --all-namespaces
</code></pre>

<blockquote>
  <p>No resources found.</p>
</blockquote>

<pre><code>kubectl get peerauthentication.security.istio.io/default
</code></pre>

<blockquote>
  <p>Error from server (NotFound): peerauthentications.security.istio.io "default" not found</p>
</blockquote>

<p>So I tried to make this <a href="https://istio.io/docs/tasks/security/authentication/authn-policy/" rel="nofollow noreferrer">example</a> and that clearly shows istio is in strict tls mode when you installed it with <code>global.mtls.enabled=true</code>.</p>

<p>If you add pods,namespaces as mentioned <a href="https://istio.io/docs/tasks/security/authentication/authn-policy/#setup" rel="nofollow noreferrer">here</a> it should be 200 for every request, but it's not</p>

<pre><code>sleep.foo to httpbin.foo: 200
sleep.foo to httpbin.bar: 200
sleep.foo to httpbin.legacy: 200
sleep.bar to httpbin.foo: 200
sleep.bar to httpbin.bar: 200
sleep.bar to httpbin.legacy: 200
sleep.legacy to httpbin.foo: 000
command terminated with exit code 56
sleep.legacy to httpbin.bar: 000
command terminated with exit code 56
sleep.legacy to httpbin.legacy: 200
</code></pre>

<p>So if you change the mtls from strict to permissive with above below yaml</p>

<pre><code>apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: PERMISSIVE
</code></pre>

<p>it works now</p>

<pre><code>sleep.foo to httpbin.foo: 200
sleep.foo to httpbin.bar: 200
sleep.foo to httpbin.legacy: 200
sleep.bar to httpbin.foo: 200
sleep.bar to httpbin.bar: 200
sleep.bar to httpbin.legacy: 200
sleep.legacy to httpbin.foo: 200
sleep.legacy to httpbin.bar: 200
sleep.legacy to httpbin.legacy: 200
</code></pre>

<hr>

<p>Additionaly <a href="https://github.com/istio/istio/issues/14963" rel="nofollow noreferrer">github issue</a> with error you provided.</p>

<hr>

<p>About the question </p>

<blockquote>
  <p>why the pod fails to mtls authenticate with itself, when curling from inside it </p>
</blockquote>

<p>There is a <a href="https://github.com/istio/istio/issues/12551" rel="nofollow noreferrer">github</a> issue about this.</p>

<hr>

<p>Additionally take a look at this <a href="https://istio.io/pt-br/docs/tasks/security/authentication/mutual-tls/#verify-requests" rel="nofollow noreferrer">istio docs</a>.</p>
