<ul>
<li>Istio: 1.3 (also tried 1.1 before update to 1.3)</li>
<li>K8s: 1.16.2</li>
<li>Cloud provider: DigitalOcean</li>
</ul>

<p>I have a cluster setup with Istio. I have enabled grafana/kiali and also installed kibana and RabbitMQ management UI and for all of those I have gateways and virtual services configured (all in istio-system namespace) along with HTTPS using SDS and cert-manager and all works fine. It means I can access these resources in the browser over HTTPS with a sub domain.</p>

<p>Then I deployed a microservice (part of a real application) and created <code>Service</code>, <code>VirtualService</code> and <code>Gateway</code> resources for it (for now it is the only one service and gateway except rabbitmq which uses different sub domain and differend port). And it is located in default namespace.</p>

<pre><code>$ kubectl get gateway

NAME             AGE
gateway-rabbit   131m
tg-gateway       45m

$ kubectl get po

NAME                           READY     STATUS    RESTARTS   AGE
rabbit-rabbitmq-0              2/2       Running   2          134m
tg-app-auth-79c578b94f-mqsz9   2/2       Running   0          46m
</code></pre>

<p>If I try to connect to my service with port forwarding I can get a success response from <code>localhost:8000/api/me</code> (also healthz, readyz both return 200 and pod has 0 restarts) so it is working fine.</p>

<pre><code>kubectl port-forward $(kubectl get pod --selector="app=tg-app-auth" --output jsonpath='{.items[0].metadata.name}') 8000:8000
</code></pre>

<p>But I can't access it neither via HTTP nor HTTPS. I get <code>404</code> using HTTP and the following response using HTTPS:</p>

<pre><code>*   Trying MYIP...
* TCP_NODELAY set
* Connected to example.com (MYIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443 
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443
</code></pre>

<p>Here are my yaml files:</p>

<p><strong>Gateway:</strong></p>

<pre><code>apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:
  name: tg-gateway
  namespace: default

spec:
  selector:
    istio: ingressgateway

  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - www.example.com
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - www.example.com
    tls:
      mode: SIMPLE
      serverCertificate: sds
      privateKey: sds
      credentialName: tg-certificate

</code></pre>

<p><strong>Service:</strong></p>

<pre><code>apiVersion: v1

kind: Service

metadata:
  name: tg-app-auth
  namespace: default
  labels:
    app: tg-app-auth

spec:
  selector:
    app: tg-app-auth

  ports:
  - name: http
    port: 8000

</code></pre>

<p><strong>VirtualService</strong></p>

<pre><code>apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:
  name: tg-app-auth-vs
  namespace: default

spec:
  hosts:
  - www.example.com

  gateways:
  - tg-gateway

  http:
  - match:
    - port: 443
    - uri:
        prefix: /api/auth
    rewrite:
      uri: /api
    route:
    - destination:
        host: tg-app-auth
        port:
          number: 8000

---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: tg-app-auth-dr
  namespace: default
spec:
  host: tg-app-auth
  trafficPolicy:
    tls:
      mode: DISABLE
</code></pre>

<p>I tried to remove all the HTTPS and TLS details and configure it with HTTP only but still can not get any response. I read all the issues on github but nothing helps and it seems like I have a very silly mistake. All these configurations are pretty much the same as I have for grafana/kibana/kiali/rabbit and all of them works fine.</p>

<p>UPD: Tried to get response with and it also works fine but I can't
get response from LB IP or domain</p>

<pre><code>kubectl exec $(kubectl get pod --selector app=tg-app-auth --output jsonpath='{.items[0].metadata.name}') -c istio-proxy -- curl -v http://$(kubectl get endpoints tg-app-auth -o jsonpath='{.subsets[0].addresses[0].ip}'):8000/api/me
</code></pre>

<pre><code>$ kubectl get endpoints tg-app-auth

NAME          ENDPOINTS          AGE
tg-app-auth   10.244.0.37:8000   22h
</code></pre>

<p><strong>UPD</strong></p>

<p>All statuses are OK. There are a lot more with different ports but I copied 80/443 only.</p>

<pre><code>$ istioctl authn tls-check &lt;pod_name&gt;

cert-manager-webhook.istio-system.svc.cluster.local:443 
istio-galley.istio-system.svc.cluster.local:443
istio-ingressgateway.istio-system.svc.cluster.local:80
istio-ingressgateway.istio-system.svc.cluster.local:443
istio-sidecar-injector.istio-system.svc.cluster.local:443
kubernetes.default.svc.cluster.local:443
</code></pre>

<pre><code>$ kubectl get ingress --all-namespaces
No resources found.

$ kubectl get gateways --all-namespaces

default        gateway-rabbit                    3d2h
default        tg-gateway                        17h
istio-system   gateway-grafana                   3d2h
istio-system   gateway-kiali                     3d2h
istio-system   istio-autogenerated-k8s-ingress   3d2h
logging        gateway-kibana                    3d2h
</code></pre>

<p>Issue was really simple and silly. I had enabled global.k8sIngress.enabled = true in Istio values.yml. After changing it to false all starts working.</p>
