<p>I am trying to secure a 3rd party application within our EKS cluster using Istio and Azure AD.</p>
<p>My configuration works on a local docker-desktop K8S cluster but when deployed to our EKS it seems that the token is never passed to the istio-proxy on the application's pod and thus never authorizes.</p>
<p>Given my configurations:</p>
<pre><code>apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: marquez-sso
  namespace: marquez
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: marquez
  jwtRules:
    - issuer: &quot;https://sts.windows.net/{{ .Values.sso.tenant }}/&quot;
      audiences: [&quot;{{ .Values.sso.scope }}&quot;]
      jwksUri: &quot;https://login.microsoftonline.com/{{ .Values.sso.tenant }}/discovery/keys?appid={{ .Values.sso.appId.read }}&quot;
#      forwardOriginalToken: true #forward jwt to proxy container - commented out because it didn't forward either.
      outputPayloadToHeader: &quot;x-jwt-payload&quot; #pass header


---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: authorize-marquez-poc
  namespace: marquez
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: marquez
  action: ALLOW
  rules:
    - to:
        - operation:
            methods: [&quot;GET&quot;]
            paths: [&quot;*&quot;]
      when:
        - key: request.auth.claims[roles]
          values: [&quot;poc.read&quot;]
</code></pre>
<p>When I make a request to my app with a valid JWT token containing a &quot;poc.read&quot; role, I would assume that my request would be authenticated and authorized and reach the application.</p>
<p>This happens on my local cluster but when attempted on EKS I get a 403 &quot;RBAC: access denied&quot; response.</p>
<p>Looking at the logs for the gateway I see that the JWT is successfully authenticated (JWT values are redacted):</p>
<pre><code>2021-12-09T16:10:28.399763Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.399806Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.399836Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.400332Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.557660Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.557857Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.558903Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.558975Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.592729Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.592773Z debug   envoy filter    tls:onServerName(), requestedServerName: redacted.com
2021-12-09T16:10:28.647901Z debug   envoy http  [C4469] new stream
2021-12-09T16:10:28.647975Z debug   envoy http  [C4469][S10542422563474009578] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'authorization', 'Bearer redacted-token'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'connection', 'keep-alive'
'content-length', '93'
2021-12-09T16:10:28.648018Z debug   envoy jwt   Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.648063Z debug   envoy jwt   Called Filter : decodeHeaders
2021-12-09T16:10:28.648075Z debug   envoy jwt   Prefix requirement '/' matched.
2021-12-09T16:10:28.648081Z debug   envoy jwt   extract authorizationBearer
2021-12-09T16:10:28.648101Z debug   envoy jwt   origins-0: JWT authentication starts (allow_failed=false), tokens size=1
2021-12-09T16:10:28.648107Z debug   envoy jwt   origins-0: startVerify: tokens size 1
2021-12-09T16:10:28.648111Z debug   envoy jwt   origins-0: Parse Jwt redacted-token
2021-12-09T16:10:28.648222Z debug   envoy jwt   origins-0: Verifying JWT token of issuer https://sts.windows.net/redacted-tenant/
2021-12-09T16:10:28.648271Z debug   envoy jwt   origins-0: JWT token verification completed with: OK
2021-12-09T16:10:28.648282Z debug   envoy jwt   Jwt authentication completed with: OK
2021-12-09T16:10:28.648302Z debug   envoy filter    AuthenticationFilter::decodeHeaders with config
policy {
  origins {
    jwt {
      issuer: &quot;https://sts.windows.net/redacted-tenant/&quot;
    }
  }
  origin_is_optional: true
  principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.648309Z debug   envoy filter    No method defined. Skip source authentication.
2021-12-09T16:10:28.648313Z debug   envoy filter    Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: &quot;https://sts.windows.net/redacted-tenant/&quot;
2021-12-09T16:10:28.648385Z debug   envoy filter    ProcessJwtPayload: json object is {&quot;aio&quot;:&quot;redacted-aio&quot;,&quot;appid&quot;:&quot;redacted-appid1&quot;,&quot;appidacr&quot;:&quot;1&quot;,&quot;aud&quot;:&quot;redacted-aud&quot;,&quot;exp&quot;:1639068956,&quot;iat&quot;:1639065056,&quot;idp&quot;:&quot;https://sts.windows.net/redacted-tenant/&quot;,&quot;iss&quot;:&quot;https://sts.windows.net/redacted-tenant/&quot;,&quot;nbf&quot;:1639065056,&quot;oid&quot;:&quot;redacted-oid&quot;,&quot;rh&quot;:&quot;redacted-rh&quot;,&quot;roles&quot;:[&quot;poc.read&quot;],&quot;sub&quot;:&quot;redacted-oid&quot;,&quot;tid&quot;:&quot;redacted-tenant&quot;,&quot;uti&quot;:&quot;redacted-uti&quot;,&quot;ver&quot;:&quot;1.0&quot;}
2021-12-09T16:10:28.648406Z debug   envoy filter    JWT validation succeeded
2021-12-09T16:10:28.648415Z debug   envoy filter    Set principal from origin: https://sts.windows.net/redacted-tenant//redacted-oid
2021-12-09T16:10:28.648419Z debug   envoy filter    Origin authenticator succeeded
2021-12-09T16:10:28.648524Z debug   envoy filter    Saved Dynamic Metadata:
fields {
  key: &quot;request.auth.audiences&quot;
  value {
    string_value: &quot;redacted-aud&quot;
  }
}
fields {
  key: &quot;request.auth.claims&quot;
  value {
    struct_value {
      fields {
        key: &quot;aio&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-aio&quot;
            }
          }
        }
      }
      fields {
        key: &quot;appid&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-appid1&quot;
            }
          }
        }
      }
      fields {
        key: &quot;appidacr&quot;
        value {
          list_value {
            values {
              string_value: &quot;1&quot;
            }
          }
        }
      }
      fields {
        key: &quot;aud&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-aud&quot;
            }
          }
        }
      }
      fields {
        key: &quot;idp&quot;
        value {
          list_value {
            values {
              string_value: &quot;https://sts.windows.net/redacted-tenant/&quot;
            }
          }
        }
      }
      fields {
        key: &quot;iss&quot;
        value {
          list_value {
            values {
              string_value: &quot;https://sts.windows.net/redacted-tenant/&quot;
            }
          }
        }
      }
      fields {
        key: &quot;oid&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-oid&quot;
            }
          }
        }
      }
      fields {
        key: &quot;rh&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-rh&quot;
            }
          }
        }
      }
      fields {
        key: &quot;roles&quot;
        value {
          list_value {
            values {
              string_value: &quot;poc.read&quot;
            }
          }
        }
      }
      fields {
        key: &quot;sub&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-oid&quot;
            }
          }
        }
      }
      fields {
        key: &quot;tid&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-tenant&quot;
            }
          }
        }
      }
      fields {
        key: &quot;uti&quot;
        value {
          list_value {
            values {
              string_value: &quot;redacted-uti&quot;
            }
          }
        }
      }
      fields {
        key: &quot;ver&quot;
        value {
          list_value {
            values {
              string_value: &quot;1.0&quot;
            }
          }
        }
      }
    }
  }
}
fields {
  key: &quot;request.auth.principal&quot;
  value {
    string_value: &quot;https://sts.windows.net/redacted-tenant//redacted-oid&quot;
  }
}
fields {
  key: &quot;request.auth.raw_claims&quot;
  value {
    string_value: &quot;{\&quot;appid\&quot;:\&quot;redacted-appid1\&quot;,\&quot;aud\&quot;:\&quot;redacted-aud\&quot;,\&quot;ver\&quot;:\&quot;1.0\&quot;,\&quot;sub\&quot;:\&quot;redacted-oid\&quot;,\&quot;nbf\&quot;:1639065056,\&quot;rh\&quot;:\&quot;redacted-rh\&quot;,\&quot;uti\&quot;:\&quot;redacted-uti\&quot;,\&quot;exp\&quot;:1639068956,\&quot;tid\&quot;:\&quot;redacted-tenant\&quot;,\&quot;iat\&quot;:1639065056,\&quot;oid\&quot;:\&quot;redacted-oid\&quot;,\&quot;aio\&quot;:\&quot;redacted-aio\&quot;,\&quot;appidacr\&quot;:\&quot;1\&quot;,\&quot;iss\&quot;:\&quot;https://sts.windows.net/redacted-tenant/\&quot;,\&quot;idp\&quot;:\&quot;https://sts.windows.net/redacted-tenant/\&quot;,\&quot;roles\&quot;:[\&quot;poc.read\&quot;]}&quot;
  }
}
2021-12-09T16:10:28.648551Z debug   envoy router    [C4469][S10542422563474009578] cluster 'outbound|443||marquez.marquez.svc.cluster.local' match for URL '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648603Z debug   envoy router    [C4469][S10542422563474009578] router decoding headers:
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.648642Z debug   envoy pool  queueing stream due to no available connections
2021-12-09T16:10:28.648645Z debug   envoy pool  trying to create new connection
2021-12-09T16:10:28.648649Z debug   envoy pool  creating a new connection
2021-12-09T16:10:28.648708Z debug   envoy client    [C4470] connecting
2021-12-09T16:10:28.648715Z debug   envoy connection    [C4470] connecting to 100.112.69.104:5000
2021-12-09T16:10:28.648876Z debug   envoy connection    [C4470] connection in progress
2021-12-09T16:10:28.648904Z debug   envoy jwt   Called Filter : decodeData
2021-12-09T16:10:28.648921Z debug   envoy http  [C4469][S10542422563474009578] request end stream
2021-12-09T16:10:28.648924Z debug   envoy jwt   Called Filter : decodeData
2021-12-09T16:10:28.648938Z debug   envoy connection    [C4470] connected
2021-12-09T16:10:28.649435Z debug   envoy client    [C4470] connected
2021-12-09T16:10:28.649452Z debug   envoy pool  [C4470] attaching to next stream
2021-12-09T16:10:28.649456Z debug   envoy pool  [C4470] creating stream
2021-12-09T16:10:28.649465Z debug   envoy router    [C4469][S10542422563474009578] pool ready
2021-12-09T16:10:28.650350Z debug   envoy router    [C4469][S10542422563474009578] upstream headers complete: end_stream=false
2021-12-09T16:10:28.650404Z debug   envoy http  [C4469][S10542422563474009578] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'x-envoy-upstream-service-time', '1'
2021-12-09T16:10:28.650422Z debug   envoy client    [C4470] response complete
2021-12-09T16:10:28.650545Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650555Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650558Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650561Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650565Z debug   envoy jwt   Called Filter : onDestroy
2021-12-09T16:10:28.650568Z debug   envoy filter    Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650574Z debug   envoy pool  [C4470] response complete
2021-12-09T16:10:28.650577Z debug   envoy pool  [C4470] saw upstream close connection
2021-12-09T16:10:28.650580Z debug   envoy connection    [C4470] closing data_to_write=0 type=1
2021-12-09T16:10:28.650583Z debug   envoy connection    [C4470] closing socket: 1
2021-12-09T16:10:28.650642Z debug   envoy connection    [C4470] SSL shutdown: rc=0
2021-12-09T16:10:28.650690Z debug   envoy client    [C4470] disconnect. resetting 0 pending requests
2021-12-09T16:10:28.650699Z debug   envoy pool  [C4470] client disconnected, failure reason:
2021-12-09T16:10:28.650747Z debug   envoy pool  [C4470] destroying stream: 0 remaining
</code></pre>
<p>But the logs for the application pod show that the JWT values are never sent from the gateway and thus fails authorization:</p>
<pre><code>2021-12-09T16:10:28.648927Z debug   envoy filter    original_dst: New connection accepted
2021-12-09T16:10:28.648959Z debug   envoy filter    tls inspector: new connection accepted
2021-12-09T16:10:28.649014Z debug   envoy filter    tls:onServerName(), requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local
2021-12-09T16:10:28.649556Z debug   envoy http  [C4227] new stream
2021-12-09T16:10:28.649677Z debug   envoy http  [C4227][S15673186747439282324] request headers complete (end_stream=false):
':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-decorator-operation', 'marquez.marquez.svc.cluster.local:443/api/*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0gUKBkxBQkVMUxLHBSrEBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiIKBWlzdGlvEhkaF21ldGFkYXRhLWluZ3Jlc3NnYXRld2F5ChkKDGlzdGlvLmlvL3JldhIJGgdkZWZhdWx0CjAKG29wZXJhdG9yLmlzdGlvLmlvL2NvbXBvbmVudBIRGg9JbmdyZXNzR2F0ZXdheXMKIQoRcG9kLXRlbXBsYXRlLWhhc2gSDBoKNjU2ZmY3NmQ2YgoSCgdyZWxlYXNlEgcaBWlzdGlvCjwKH3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLW5hbWUSGRoXbWV0YWRhdGEtaW5ncmVzc2dhdGV3YXkKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0ChEKA3NoYRIKGgg2MTRlYTkyYwoiChdzaWRlY2FyLmlzdGlvLmlvL2luamVjdBIHGgVmYWxzZQoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKMgoETkFNRRIqGihtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheS02NTZmZjc2ZDZiLXFkbDJqChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KYAoFT1dORVISVxpVa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9tZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQoXChFQTEFURk9STV9NRVRBREFUQRICKgAKKgoNV09SS0xPQURfTkFNRRIZGhdtZXRhZGF0YS1pbmdyZXNzZ2F0ZXdheQ=='
'x-envoy-peer-metadata-id', 'router~100.112.90.145~metadata-ingressgateway-656ff76d6b-qdl2j.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
2021-12-09T16:10:28.649788Z debug   envoy jwt   Called Filter : setDecoderFilterCallbacks
2021-12-09T16:10:28.649840Z debug   envoy jwt   Called Filter : decodeHeaders
2021-12-09T16:10:28.649853Z debug   envoy jwt   Prefix requirement '/' matched.
2021-12-09T16:10:28.649860Z debug   envoy jwt   extract authorizationBearer
2021-12-09T16:10:28.649865Z debug   envoy jwt   origins-0: JWT authentication starts (allow_failed=false), tokens size=0
2021-12-09T16:10:28.649868Z debug   envoy jwt   origins-0: JWT token verification completed with: Jwt is missing
2021-12-09T16:10:28.649871Z debug   envoy jwt   Jwt authentication completed with: OK
2021-12-09T16:10:28.649895Z debug   envoy filter    AuthenticationFilter::decodeHeaders with config
policy {
  peers {
    mtls {
      mode: PERMISSIVE
    }
  }
  origins {
    jwt {
      issuer: &quot;https://sts.windows.net/redacted-tenant/&quot;
    }
  }
  origin_is_optional: true
  principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
2021-12-09T16:10:28.649905Z debug   envoy filter    [C4227] validateX509 mode PERMISSIVE: ssl=true, has_user=true
2021-12-09T16:10:28.649908Z debug   envoy filter    [C4227] trust domain validation skipped
2021-12-09T16:10:28.649910Z debug   envoy filter    Set peer from X509: cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account
2021-12-09T16:10:28.649915Z debug   envoy filter    Validating request path /api/v1/namespaces/troubleshootistio for jwt issuer: &quot;https://sts.windows.net/redacted-tenant/&quot;
2021-12-09T16:10:28.649917Z debug   envoy filter    No dynamic_metadata found for filter envoy.filters.http.jwt_authn
2021-12-09T16:10:28.649920Z debug   envoy filter    No dynamic_metadata found for filter jwt-auth
2021-12-09T16:10:28.649922Z debug   envoy filter    Origin authenticator failed
2021-12-09T16:10:28.649952Z debug   envoy filter    Saved Dynamic Metadata:
fields {
  key: &quot;source.namespace&quot;
  value {
    string_value: &quot;istio-system&quot;
  }
}
fields {
  key: &quot;source.principal&quot;
  value {
    string_value: &quot;cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account&quot;
  }
}
fields {
  key: &quot;source.user&quot;
  value {
    string_value: &quot;cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account&quot;
  }
}
2021-12-09T16:10:28.650000Z debug   envoy rbac  checking request: requestedServerName: outbound_.443_._.marquez.marquez.svc.cluster.local, sourceIP: 100.112.90.145:40310, directRemoteIP: 100.112.90.145:40310, remoteIP: 10.11.226.29:0,localAddress: 100.112.69.104:5000, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'redacted.com'
':path', '/api/v1/namespaces/troubleshootistio'
':method', 'GET'
':scheme', 'https'
'content-type', 'application/json'
'user-agent', 'PostmanRuntime/7.28.4'
'accept', '*/*'
'cache-control', 'no-cache'
'postman-token', '3318e2c3-7a16-4f35-a4a6-03ca1c30680c'
'accept-encoding', 'gzip, deflate, br'
'content-length', '93'
'x-forwarded-for', '10.11.226.29'
'x-forwarded-proto', 'https'
'x-request-id', '263e9f61-f6a0-4d22-bf67-c5abafcd4d6d'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'dae9d28da5c49193785bcb1128971c0b'
'x-b3-spanid', '785bcb1128971c0b'
'x-b3-sampled', '0'
'x-envoy-original-path', '/api/v1/namespaces/troubleshootistio'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/marquez/sa/default;Hash=0adef9d0a150cbba7db8c026be24a496bc09ff4dd3f30ddc020b5e90d3afb619;Subject=&quot;&quot;;URI=spiffe://cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account'
, dynamicMetadata: filter_metadata {
  key: &quot;istio_authn&quot;
  value {
    fields {
      key: &quot;source.namespace&quot;
      value {
        string_value: &quot;istio-system&quot;
      }
    }
    fields {
      key: &quot;source.principal&quot;
      value {
        string_value: &quot;cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account&quot;
      }
    }
    fields {
      key: &quot;source.user&quot;
      value {
        string_value: &quot;cluster.local/ns/istio-system/sa/metadata-ingressgateway-service-account&quot;
      }
    }
  }
}
2021-12-09T16:10:28.650019Z debug   envoy rbac  enforced denied, matched policy none
2021-12-09T16:10:28.650030Z debug   envoy http  [C4227][S15673186747439282324] Sending local reply with details rbac_access_denied_matched_policy[none]
2021-12-09T16:10:28.650068Z debug   envoy http  [C4227][S15673186747439282324] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'x-envoy-peer-metadata', 'ChsKDkFQUF9DT05UQUlORVJTEgkaB21hcnF1ZXoKGgoKQ0xVU1RFUl9JRBIMGgpLdWJlcm5ldGVzChkKDUlTVElPX1ZFUlNJT04SCBoGMS4xMC4wCpMDCgZMQUJFTFMSiAMqhQMKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB21hcnF1ZXoKJwoaYXBwLmt1YmVybmV0ZXMuaW8vaW5zdGFuY2USCRoHbWFycXVlegomChxhcHAua3ViZXJuZXRlcy5pby9tYW5hZ2VkLWJ5EgYaBEhlbG0KIwoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIJGgdtYXJxdWV6CiEKDWhlbG0uc2gvY2hhcnQSEBoObWFycXVlei0wLjE5LjEKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKIAoRcG9kLXRlbXBsYXRlLWhhc2gSCxoJNzZmOTg3Yzk0CiQKGXNlY3VyaXR5LmlzdGlvLmlvL3Rsc01vZGUSBxoFaXN0aW8KLAofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIJGgdtYXJxdWV6Ci8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoaCgdNRVNIX0lEEg8aDWNsdXN0ZXIubG9jYWwKIQoETkFNRRIZGhdtYXJxdWV6LTc2Zjk4N2M5NC1wNXdjegoWCglOQU1FU1BBQ0USCRoHbWFycXVlegpLCgVPV05FUhJCGkBrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvbWFycXVlei9kZXBsb3ltZW50cy9tYXJxdWV6ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAoaCg1XT1JLTE9BRF9OQU1FEgkaB21hcnF1ZXo='
'x-envoy-peer-metadata-id', 'sidecar~100.112.69.104~marquez-76f987c94-p5wcz.marquez~marquez.svc.cluster.local'
'date', 'Thu, 09 Dec 2021 16:10:28 GMT'
'server', 'istio-envoy'
'connection', 'close'
2021-12-09T16:10:28.650089Z debug   envoy http  [C4227][S15673186747439282324] doEndStream() resetting stream
2021-12-09T16:10:28.650095Z debug   envoy http  [C4227][S15673186747439282324] stream reset
2021-12-09T16:10:28.650177Z debug   envoy wasm  wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2021-12-09T16:10:28.650188Z debug   envoy wasm  wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2021-12-09T16:10:28.650191Z debug   envoy wasm  wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2021-12-09T16:10:28.650194Z debug   envoy wasm  wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2021-12-09T16:10:28.650198Z debug   envoy jwt   Called Filter : onDestroy
2021-12-09T16:10:28.650200Z debug   envoy filter    Called AuthenticationFilter : onDestroy
2021-12-09T16:10:28.650208Z debug   envoy connection    [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650216Z debug   envoy connection    [C4227] setting delayed close timer with timeout 1000 ms
2021-12-09T16:10:28.650230Z debug   envoy connection    [C4227] closing data_to_write=1245 type=2
2021-12-09T16:10:28.650306Z debug   envoy connection    [C4227] write flush complete
2021-12-09T16:10:28.650690Z debug   envoy connection    [C4227] remote early close
2021-12-09T16:10:28.650700Z debug   envoy connection    [C4227] closing socket: 0
2021-12-09T16:10:28.650750Z debug   envoy connection    [C4227] SSL shutdown: rc=0
</code></pre>
<p>I am new to Istio and have not been able to find this issue with a resolution in the documentation or other resources.</p>
<p>The only differences that I am aware of between my local cluster and our EKS, is that EKS is running in AWS is using TLS and has 3rd party JWT tokens enabled while my local version has 1st party JWT tokens.</p>
<p>Why is the token or values from the token not being passed to the istio-proxy sidecar on my application pod and how can I configure this to secure my app?</p>

gooeylewie
<p>I was able to resolve this by adding the following to my AuthorizationPolicy:</p>
<pre><code>  rules:
    - from:
      - source:
          requestPrincipals: [&quot;$ISS/$SUB&quot;]
</code></pre>
