<p>I have a mutual TLS enabled Istio mesh. My setup is as follows</p>

<p><a href="https://i.stack.imgur.com/Vcxa4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Vcxa4.png" alt="enter image description here"></a></p>

<ol>
<li>A service running inside a pod (Service container + envoy)</li>
<li>An envoy gateway which stays in front of the above service. An Istio Gateway and Virtual Service attached to this. It routes <code>/info/</code> route to the above service.</li>
<li>Another Istio Gateway configured for ingress using the default istio ingress pod. This also has Gateway+Virtual Service combination. The virtual service directs <code>/info/</code> path to the service described in 2</li>
</ol>

<p>I'm attempting to access the service from the ingress gateway using a curl command such as:</p>

<pre><code>$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v
</code></pre>

<p>But I'm getting a 503 not found error as below:</p>

<pre><code>$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.105.138.94...
* Connected to istio-ingressgateway.istio-system (10.105.138.94) port 80 (#0)
&gt; GET /info/ HTTP/1.1
&gt; Host: istio-ingressgateway.istio-system
&gt; User-Agent: curl/7.47.0
&gt; Accept: */*
&gt; Authorization: Bearer ...
&gt;
&lt; HTTP/1.1 503 Service Unavailable
&lt; content-length: 57
&lt; content-type: text/plain
&lt; date: Sat, 12 Jan 2019 13:30:13 GMT
&lt; server: envoy
&lt;
* Connection #0 to host istio-ingressgateway.istio-system left intact
</code></pre>

<p>I checked the logs of <code>istio-ingressgateway</code> pod and the following line was logged there</p>

<pre><code>[2019-01-13T05:40:16.517Z] "GET /info/ HTTP/1.1" 503 UH 0 19 6 - "10.244.0.5" "curl/7.47.0" "da02fdce-8bb5-90fe-b422-5c74fe28759b" "istio-ingressgateway.istio-system" "-"
</code></pre>

<p>If I logged into istio ingress pod and attempt to send the request with curl, I get a successful 200 OK.</p>

<pre><code># curl hr--gateway-service.default/info/ -H "Authorization: Bearer $token" -v
</code></pre>

<p>Also, I managed to get a successful response for the same curl command when the mesh was created in mTLS disabled mode. There are no conflicts shown in mTLS setup.</p>

<p>Here are the config details for my service mesh in case you need additional info.</p>

<p><strong>Pods</strong></p>

<pre><code>$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                                       READY     STATUS      RESTARTS   AGE
default         hr--gateway-deployment-688986c87c-z9nkh                    1/1       Running     0          37m
default         hr--hr-deployment-596946948d-c89bn                         2/2       Running     0          37m
default         hr--sts-deployment-694d7cff97-gjwdk                        1/1       Running     0          37m
ingress-nginx   default-http-backend-6586bc58b6-8qss6                      1/1       Running     0          42m
ingress-nginx   nginx-ingress-controller-6bd7c597cb-t4rwq                  1/1       Running     0          42m
istio-system    grafana-85dbf49c94-lfpbr                                   1/1       Running     0          42m
istio-system    istio-citadel-545f49c58b-dq5lq                             1/1       Running     0          42m
istio-system    istio-cleanup-secrets-bh5ws                                0/1       Completed   0          42m
istio-system    istio-egressgateway-7d59954f4-qcnxm                        1/1       Running     0          42m
istio-system    istio-galley-5b6449c48f-72vkb                              1/1       Running     0          42m
istio-system    istio-grafana-post-install-lwmsf                           0/1       Completed   0          42m
istio-system    istio-ingressgateway-8455c8c6f7-5khtk                      1/1       Running     0          42m
istio-system    istio-pilot-58ff4d6647-bct4b                               2/2       Running     0          42m
istio-system    istio-policy-59685fd869-h7v94                              2/2       Running     0          42m
istio-system    istio-security-post-install-cqj6k                          0/1       Completed   0          42m
istio-system    istio-sidecar-injector-75b9866679-qg88s                    1/1       Running     0          42m
istio-system    istio-statsd-prom-bridge-549d687fd9-bspj2                  1/1       Running     0          42m
istio-system    istio-telemetry-6ccf9ddb96-hxnwv                           2/2       Running     0          42m
istio-system    istio-tracing-7596597bd7-m5pk8                             1/1       Running     0          42m
istio-system    prometheus-6ffc56584f-4cm5v                                1/1       Running     0          42m
istio-system    servicegraph-5d64b457b4-jttl9                              1/1       Running     0          42m
kube-system     coredns-78fcdf6894-rxw57                                   1/1       Running     0          50m
kube-system     coredns-78fcdf6894-s4bg2                                   1/1       Running     0          50m
kube-system     etcd-ubuntu                                                1/1       Running     0          49m
kube-system     kube-apiserver-ubuntu                                      1/1       Running     0          49m
kube-system     kube-controller-manager-ubuntu                             1/1       Running     0          49m
kube-system     kube-flannel-ds-9nvf9                                      1/1       Running     0          49m
kube-system     kube-proxy-r868m                                           1/1       Running     0          50m
kube-system     kube-scheduler-ubuntu                                      1/1       Running     0          49m
</code></pre>

<p><strong>Services</strong></p>

<pre><code>$ kubectl get svc --all-namespaces
NAMESPACE       NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                                                                   AGE

default         hr--gateway-service                     ClusterIP      10.100.238.144   &lt;none&gt;         80/TCP,443/TCP                                                                                                            39m
default         hr--hr-service                          ClusterIP      10.96.193.43     &lt;none&gt;         80/TCP                                                                                                                    39m
default         hr--sts-service                         ClusterIP      10.99.54.137     &lt;none&gt;         8080/TCP,8081/TCP,8090/TCP                                                                                                39m
default         kubernetes                              ClusterIP      10.96.0.1        &lt;none&gt;         443/TCP                                                                                                                   52m
ingress-nginx   default-http-backend                    ClusterIP      10.109.166.229   &lt;none&gt;         80/TCP                                                                                                                    44m
ingress-nginx   ingress-nginx                           NodePort       10.108.9.180     192.168.60.3   80:31001/TCP,443:32315/TCP                                                                                                44m
istio-system    grafana                                 ClusterIP      10.102.141.231   &lt;none&gt;         3000/TCP                                                                                                                  44m
istio-system    istio-citadel                           ClusterIP      10.101.128.187   &lt;none&gt;         8060/TCP,9093/TCP                                                                                                         44m
istio-system    istio-egressgateway                     ClusterIP      10.102.157.204   &lt;none&gt;         80/TCP,443/TCP                                                                                                            44m
istio-system    istio-galley                            ClusterIP      10.96.31.251     &lt;none&gt;         443/TCP,9093/TCP                                                                                                          44m
istio-system    istio-ingressgateway                    LoadBalancer   10.105.138.94    &lt;pending&gt;      80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31219/TCP,8060:31482/TCP,853:30034/TCP,15030:31544/TCP,15031:32652/TCP   44m
istio-system    istio-pilot                             ClusterIP      10.100.170.73    &lt;none&gt;         15010/TCP,15011/TCP,8080/TCP,9093/TCP                                                                                     44m
istio-system    istio-policy                            ClusterIP      10.104.77.184    &lt;none&gt;         9091/TCP,15004/TCP,9093/TCP                                                                                               44m
istio-system    istio-sidecar-injector                  ClusterIP      10.100.180.152   &lt;none&gt;         443/TCP                                                                                                                   44m
istio-system    istio-statsd-prom-bridge                ClusterIP      10.107.39.50     &lt;none&gt;         9102/TCP,9125/UDP                                                                                                         44m
istio-system    istio-telemetry                         ClusterIP      10.110.55.232    &lt;none&gt;         9091/TCP,15004/TCP,9093/TCP,42422/TCP                                                                                     44m
istio-system    jaeger-agent                            ClusterIP      None             &lt;none&gt;         5775/UDP,6831/UDP,6832/UDP                                                                                                44m
istio-system    jaeger-collector                        ClusterIP      10.102.43.21     &lt;none&gt;         14267/TCP,14268/TCP                                                                                                       44m
istio-system    jaeger-query                            ClusterIP      10.104.182.189   &lt;none&gt;         16686/TCP                                                                                                                 44m
istio-system    prometheus                              ClusterIP      10.100.0.70      &lt;none&gt;         9090/TCP                                                                                                                  44m
istio-system    servicegraph                            ClusterIP      10.97.65.37      &lt;none&gt;         8088/TCP                                                                                                                  44m
istio-system    tracing                                 ClusterIP      10.109.87.118    &lt;none&gt;         80/TCP                                                                                                                    44m
kube-system     kube-dns                                ClusterIP      10.96.0.10       &lt;none&gt;         53/UDP,53/TCP                                                                                                             52m
</code></pre>

<p><strong>Gateway and virtual service described in point 2</strong></p>

<pre><code>$ kubectl describe gateways.networking.istio.io hr--gateway
Name:         hr--gateway
Namespace:    default
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  ...
Spec:
  Selector:
    App:                           hr--gateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http2
      Number:    80
      Protocol:  HTTP2
    Hosts:
      *
    Port:
      Name:      https
      Number:    443
      Protocol:  HTTPS
    Tls:
      Mode:  PASSTHROUGH

$ kubectl describe virtualservices.networking.istio.io hr--gateway
Name:         hr--gateway
Namespace:    default
Labels:       app=hr--gateway
              Annotations:  &lt;none&gt;
API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Metadata:
  ...
Spec:
  Gateways:
    hr--gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Prefix:  /info/
    Rewrite:
      Uri:  /
    Route:
      Destination:
        Host:  hr--hr-service
</code></pre>

<p><strong>Gateway and virtual service described in point 3</strong></p>

<pre><code>$ kubectl describe gateways.networking.istio.io ingress-gateway
Name:         ingress-gateway
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"ingress-gateway","namespace":"default"},"spec":{"sel...
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  ...
Spec:
  Selector:
    Istio:  ingressgateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http2
      Number:    80
      Protocol:  HTTP2

$ kubectl describe virtualservices.networking.istio.io hr--gateway-ingress-vs
Name:         hr--gateway-ingress-vs
Namespace:    default
Labels:       app=hr--gateway
              API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Metadata:
Spec:
  Gateways:
    ingress-gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Prefix:  /info/
    Route:
      Destination:
        Host:  hr--gateway-service
Events:        &lt;none&gt;
</code></pre>

<p>The problem is probably as follows: <em>istio-ingressgateway</em> initiates mTLS to <em>hr--gateway-service</em> on port 80, but <em>hr--gateway-service</em> expects plain HTTP connections.</p>

<p>There are multiple solutions:</p>

<ol>
<li>Define a DestinationRule to instruct clients to disable mTLS on calls to <em>hr--gateway-service</em></li>
</ol>

<pre><code>   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     name: hr--gateway-service-disable-mtls
   spec:
     host: hr--gateway-service.default.svc.cluster.local
     trafficPolicy:
       tls:
         mode: DISABLE
</code></pre>

<ol start="2">
<li>Instruct <em>hr-gateway-service</em> to accept mTLS connections. For that, configure the <a href="https://istio.io/docs/reference/config/networking/v1alpha3/gateway/#Server-TLSOptions" rel="noreferrer">server TLS options</a> on port 80 to be <code>MUTUAL</code> and to use Istio certificates and the private key. Specify <code>serverCertificate</code>, <code>caCertificates</code> and <code>privateKey</code> to be <code>/etc/certs/cert-chain.pem</code>, <code>/etc/certs/root-cert.pem</code>, <code>/etc/certs/key.pem</code>, respectively.</li>
</ol>
