<p>I am using Istio 1.8.0 with on-prem k8s v1.19..We have several microservices running where I am using <code>STRICT</code> mode for peerauthentication. And I can verify that if I use <code>PERMISSIVE</code> mode I did not receive any 503 errors.</p>
<p><a href="https://i.stack.imgur.com/KbaqG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KbaqG.png" alt="Network console" /></a></p>
<p>I really get stuck to find any solution cause I do not want to use <code>PERMISSIVE</code> mode as recommended.</p>
<p>Here is the log for istio ingressgateway.</p>
<pre><code>$kubectl logs -f istio-ingressgateway-75496c97df-44g6l -n istio-system
[2021-01-16T07:28:51.852Z] &quot;GET /config HTTP/1.1&quot; 503 URX &quot;-&quot; 0 95 57 57 &quot;95.0.145.40,10.6.0.21&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36&quot; &quot;foo-bar.example.com&quot; &quot;10.6.25.34:3000&quot; outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local 10.6.5.216:46364 10.6.5.216:8080 10.6.0.21:14387 - -
</code></pre>
<p>Here is the log for my ui microservice.. I did not catch any 503 errors for rest of services</p>
<pre><code>$ kubectl get ns --show-labels
NAME              STATUS   AGE     LABELS 
hub-dev           Active   2d19h   istio-injection=enabled
istio-system      Active   3d5h    istio-injection=disabled

$ kubectl get pods -n hub-dev
NAME                                     READY   STATUS    RESTARTS   AGE
oneapihub-api-dev-79dff67cdb-hx754       3/3     Running   0          15h
oneapihub-auth-dev-76cfcb6cb4-74ljq      3/3     Running   0          15h
oneapihub-backend-dev-d76799bcd-bmwjn    2/2     Running   0          15h
oneapihub-cronjob-dev-6879dbf9b8-wvpnp   3/3     Running   0          15h
oneapihub-mp-dev-864794d446-cfqj7        3/3     Running   0          15h
oneapihub-ui-dev-67d7bb6779-8z4xt        2/2     Running   0          14h
redis-hub-master-0                       2/2     Running   0          15h

$ kubectl logs -f oneapihub-ui-dev-67d7bb6779-8z4xt -n hub-dev -c istio-proxy
[2021-01-15T14:17:24.698Z] &quot;GET /config HTTP/1.1&quot; 503 URX &quot;-&quot; 0 95 65 64 &quot;95.0.145.40,10.6.0.19&quot; &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36&quot; &quot;foo-bar.example.com&quot; &quot;10.6.25.241:3000&quot; outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local 10.6.5.216:37584 10.6.5.216:8080 10.6.0.19:31138 - -
[2021-01-15T14:17:24.817Z] 
</code></pre>
<p>Here is peerauthentication</p>
<pre><code>$ kubectl get peerauthentication -n istio-system -o yaml
apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
  kind: PeerAuthentication
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {&quot;apiVersion&quot;:&quot;security.istio.io/v1beta1&quot;,&quot;kind&quot;:&quot;PeerAuthentication&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;default&quot;,&quot;namespace&quot;:&quot;istio-system&quot;},&quot;spec&quot;:{&quot;mtls&quot;:{&quot;mode&quot;:&quot;STRICT&quot;}}}
    name: default
    namespace: istio-system
  spec:
    mtls:
  mode: STRICT
</code></pre>
<p>Here is the service.yaml of my frontend app.</p>
<pre><code>capel0068340585:~ semural$ kubectl get svc -n hub-dev
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
oneapihub-ui-dev        ClusterIP   10.254.47.95     &lt;none&gt;        80/TCP     48m

# Source: oneapihub-ui/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: RELEASE-NAME-oneapihub-ui
  labels:
    app.kubernetes.io/name: oneapihub-ui
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
    helm.sh/chart: oneapihub-ui-0.1.0
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
      name: http
</code></pre>
<p>Here is the Gateway and VS that I created in same namepace where microservices are running.</p>
<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: hub-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - foo-bar.example.com
apiVersion: networking.istio.io/v1alpha3

kind: VirtualService
metadata:
  name: hub
spec:
  hosts:
    - foo-bar.example.com
  gateways:
    - hub-gateway
  http:
    - route:
      - destination:
          host: oneapihub-ui-dev.hub-dev.svc.cluster.local
          port:
            number: 80
</code></pre>
<p>Here is the proxy config for ingressgateway</p>
<pre><code>$ istioctl proxy-config route istio-ingressgateway-75496c97df-44g6l -n istio-system -o json
[
    {
        &quot;name&quot;: &quot;http.80&quot;,
        &quot;virtualHosts&quot;: [
            {
                &quot;name&quot;: &quot;foo-baexample.com:80&quot;,
                &quot;domains&quot;: [
                    &quot;foo-bar.example.com&quot;,
                    &quot;foo-bar.example.com:*&quot;
                ],
                &quot;routes&quot;: [
                    {
                        &quot;match&quot;: {
                            &quot;prefix&quot;: &quot;/&quot;
                        },
                        &quot;route&quot;: {
                            &quot;cluster&quot;: &quot;outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local&quot;,
                            &quot;timeout&quot;: &quot;0s&quot;,
                            &quot;retryPolicy&quot;: {
                                &quot;retryOn&quot;: &quot;connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes&quot;,
                                &quot;numRetries&quot;: 2,
                                &quot;retryHostPredicate&quot;: [
                                    {
                                        &quot;name&quot;: &quot;envoy.retry_host_predicates.previous_hosts&quot;
                                    }
                                ],
</code></pre>
<p>Here are the outputs of istioctl describe pods.</p>
<pre><code>    $ istioctl x describe pod oneapihub-api-dev-78fbccf48c-5hb4c -n hub-dev
    Pod: oneapihub-api-dev-78fbccf48c-5hb4c
       Pod Ports: 50004 (oneapihub-api), 15090 (istio-proxy)
    --------------------
    Service: oneapihub-api-dev
       Port: http 80/HTTP targets pod port 50004
    
    $ istioctl x describe pod oneapihub-auth-dev-7f8998cd69-gzmnm -n hub-dev
    Pod: oneapihub-auth-dev-7f8998cd69-gzmnm
       Pod Ports: 50002 (oneapihub-auth), 15090 (istio-proxy)
    --------------------
    Service: oneapihub-auth-dev
       Port: http 80/HTTP targets pod port 5000
    
    $ istioctl x describe pod oneapihub-backend-dev-849b4bcd5d-fcm4l -n hub-dev
    Pod: oneapihub-backend-dev-849b4bcd5d-fcm4l
       Pod Ports: 50001 (oneapihub-backend), 15090 (istio-proxy)
    --------------------
    Service: oneapihub-backend-dev
       Port: http 80/HTTP targets pod port 50001
    $ istioctl x describe pod oneapihub-cronjob-dev-58b64d9c68-lv5bk -n hub-dev
    Pod: oneapihub-cronjob-dev-58b64d9c68-lv5bk
       Pod Ports: 50005 (oneapihub-cronjob), 15090 (istio-proxy)
    --------------------
    Service: oneapihub-cronjob-dev
       Port: http 80/HTTP targets pod port 50005
    $ istioctl x describe pod oneapihub-mp-dev-74fd6ffc9f-65gh5 -n hub-dev
    Pod: oneapihub-mp-dev-74fd6ffc9f-65gh5
       Pod Ports: 50003 (oneapihub-mp), 15090 (istio-proxy)
    --------------------
    Service: oneapihub-mp-dev
       Port: http 80/HTTP targets pod port 50003

    $ istioctl x describe pod oneapihub-ui-dev-7fd56f747c-nr5fk -n hub-dev
    Pod: oneapihub-ui-dev-7fd56f747c-nr5fk
       Pod Ports: 3000 (oneapihub-ui), 15090 (istio-proxy)
    --------------------
    Service: oneapihub-ui-dev
       Port: http 80/HTTP targets pod port 3000
    
    
    Exposed on Ingress Gateway http://53.6.48.168
    VirtualService: hub
       1 HTTP route(s)

$ istioctl x describe pod redis-hub-master-0 -n hub-dev
    Pod: redis-hub-master-0
       Pod Ports: 6379 (redis-hub), 15090 (istio-proxy)
    Suggestion: add 'version' label to pod for Istio telemetry.
    --------------------
    Service: redis-hub-headless
       Port: redis 6379/Redis targets pod port 6379
    --------------------
    Service: redis-hub-master
       Port: redis 6379/Redis targets pod port 6379
</code></pre>
<p>Here is my destination rule</p>
<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: backend-destination-rule
  namespace: hub-dev
spec:
  host: oneapihub-backend-dev.hub-dev.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
      sni: oneapihub-backend-dev.hub-dev.svc.cluster.local
</code></pre>

<p>The solution worked for me to add rewrite:authority to the virtual service</p>
<pre><code>  apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: hub
    spec:
      hosts:
        - foo-example.com
      gateways:
        - hub-gateway
      http:
        - route:
          - destination:
              host: oneapihub-ui-dev.hub-dev.svc.cluster.local
              port:
                number: 80
          rewrite:
            authority: oneapihub-backend-dev.hub-dev.svc.cluster.local
</code></pre>
