<p>Auto inject is enabled on namespace and I am attempting to use <code>Auto mTLS</code>. Verified Istio pilot and citadel are running correctly. The Cert secret was properly created and mounted. The istio proxy fails to start with the following logs. The specific error is <code>Invalid path: ./var/run/secrets/istio/root-cert.pem</code></p>
<pre><code>kubectl logs   ratings-v1-85c656b747-czg8z -c istio-proxy -n bookinfo
2020-04-29T05:39:42.276131Z     info    FLAG: --binaryPath=&quot;/usr/local/bin/envoy&quot;
2020-04-29T05:39:42.276164Z     info    FLAG: --concurrency=&quot;2&quot;
2020-04-29T05:39:42.276172Z     info    FLAG: --configPath=&quot;/etc/istio/proxy&quot;
2020-04-29T05:39:42.276186Z     info    FLAG: --connectTimeout=&quot;10s&quot;
2020-04-29T05:39:42.276192Z     info    FLAG: --controlPlaneAuthPolicy=&quot;MUTUAL_TLS&quot;
2020-04-29T05:39:42.276202Z     info    FLAG: --controlPlaneBootstrap=&quot;true&quot;
2020-04-29T05:39:42.276208Z     info    FLAG: --customConfigFile=&quot;&quot;
2020-04-29T05:39:42.276224Z     info    FLAG: --datadogAgentAddress=&quot;&quot;
2020-04-29T05:39:42.276378Z     info    FLAG: --disableInternalTelemetry=&quot;false&quot;
2020-04-29T05:39:42.276389Z     info    FLAG: --discoveryAddress=&quot;istiod.istio-system.svc:15012&quot;
2020-04-29T05:39:42.276396Z     info    FLAG: --dnsRefreshRate=&quot;300s&quot;
2020-04-29T05:39:42.276406Z     info    FLAG: --domain=&quot;bookinfo.svc.cluster.local&quot;
2020-04-29T05:39:42.276413Z     info    FLAG: --drainDuration=&quot;45s&quot;
2020-04-29T05:39:42.276419Z     info    FLAG: --envoyAccessLogService=&quot;&quot;
2020-04-29T05:39:42.276428Z     info    FLAG: --envoyMetricsService=&quot;&quot;
2020-04-29T05:39:42.276434Z     info    FLAG: --help=&quot;false&quot;
2020-04-29T05:39:42.276440Z     info    FLAG: --id=&quot;&quot;
2020-04-29T05:39:42.276445Z     info    FLAG: --ip=&quot;&quot;
2020-04-29T05:39:42.276455Z     info    FLAG: --lightstepAccessToken=&quot;&quot;
2020-04-29T05:39:42.276514Z     info    FLAG: --lightstepAddress=&quot;&quot;
2020-04-29T05:39:42.276525Z     info    FLAG: --lightstepCacertPath=&quot;&quot;
2020-04-29T05:39:42.276531Z     info    FLAG: --lightstepSecure=&quot;false&quot;
2020-04-29T05:39:42.276541Z     info    FLAG: --log_as_json=&quot;false&quot;
2020-04-29T05:39:42.276581Z     info    FLAG: --log_caller=&quot;&quot;
2020-04-29T05:39:42.276588Z     info    FLAG: --log_output_level=&quot;default:info&quot;
2020-04-29T05:39:42.276593Z     info    FLAG: --log_rotate=&quot;&quot;
2020-04-29T05:39:42.276604Z     info    FLAG: --log_rotate_max_age=&quot;30&quot;
2020-04-29T05:39:42.276610Z     info    FLAG: --log_rotate_max_backups=&quot;1000&quot;
2020-04-29T05:39:42.276616Z     info    FLAG: --log_rotate_max_size=&quot;104857600&quot;
2020-04-29T05:39:42.276622Z     info    FLAG: --log_stacktrace_level=&quot;default:none&quot;
2020-04-29T05:39:42.276640Z     info    FLAG: --log_target=&quot;[stdout]&quot;
2020-04-29T05:39:42.276646Z     info    FLAG: --mixerIdentity=&quot;&quot;
2020-04-29T05:39:42.276652Z     info    FLAG: --outlierLogPath=&quot;&quot;
2020-04-29T05:39:42.276667Z     info    FLAG: --parentShutdownDuration=&quot;1m0s&quot;
2020-04-29T05:39:42.276704Z     info    FLAG: --pilotIdentity=&quot;&quot;
2020-04-29T05:39:42.276717Z     info    FLAG: --proxyAdminPort=&quot;15000&quot;
2020-04-29T05:39:42.276723Z     info    FLAG: --proxyComponentLogLevel=&quot;misc:error&quot;
2020-04-29T05:39:42.276734Z     info    FLAG: --proxyLogLevel=&quot;warning&quot;
2020-04-29T05:39:42.276740Z     info    FLAG: --serviceCluster=&quot;ratings.bookinfo&quot;
2020-04-29T05:39:42.276746Z     info    FLAG: --serviceregistry=&quot;Kubernetes&quot;
2020-04-29T05:39:42.276752Z     info    FLAG: --statsdUdpAddress=&quot;&quot;
2020-04-29T05:39:42.276762Z     info    FLAG: --statusPort=&quot;15020&quot;
2020-04-29T05:39:42.276767Z     info    FLAG: --stsPort=&quot;0&quot;
2020-04-29T05:39:42.276773Z     info    FLAG: --templateFile=&quot;&quot;
2020-04-29T05:39:42.276784Z     info    FLAG: --tokenManagerPlugin=&quot;GoogleTokenExchange&quot;
2020-04-29T05:39:42.276790Z     info    FLAG: --trust-domain=&quot;cluster.local&quot;
2020-04-29T05:39:42.276797Z     info    FLAG: --zipkinAddress=&quot;zipkin.istio-system:9411&quot;
2020-04-29T05:39:42.276837Z     info    Version 1.5.1-9d07e185b0dd50e6fb1418caa4b4d879788807e3-Clean
2020-04-29T05:39:42.277060Z     info    Obtained private IP [10.244.3.53 fe80::a81e:5bff:fe52:5e73]
2020-04-29T05:39:42.277157Z     info    Proxy role: &amp;model.Proxy{ClusterID:&quot;&quot;, Type:&quot;sidecar&quot;, IPAddresses:[]string{&quot;10.244.3.53&quot;, &quot;10.244.3.53&quot;, &quot;fe80::a81e:5bff:fe52:5e73&quot;}, ID:&quot;ratings-v1-85c656b747-czg8z.bookinfo&quot;, Locality:(*envoy_api_v2_core.Locality)(nil), DNSDoma
in:&quot;bookinfo.svc.cluster.local&quot;, ConfigNamespace:&quot;&quot;, Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:labels.Collection(nil), IstioVersi
on:(*model.IstioVersion)(nil)}
2020-04-29T05:39:42.277183Z     info    PilotSAN []string{&quot;spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account&quot;}
2020-04-29T05:39:42.277199Z     info    MixerSAN []string{&quot;spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account&quot;}
2020-04-29T05:39:42.278153Z     info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: /etc/istio/proxy
connectTimeout: 10s
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: ratings.bookinfo
statNameLength: 189
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2020-04-29T05:39:42.278214Z     info    JWT policy is first-party-jwt
2020-04-29T05:39:42.280069Z     info    Istio Agent uses default istiod CA
2020-04-29T05:39:42.280086Z     info    istiod uses self-issued certificate
2020-04-29T05:39:42.280207Z     warn    Failed to load root cert, assume IP secure network: open var/run/secrets/istio/root-cert.pem: no such file or directory
2020-04-29T05:39:42.384355Z     info    parsed scheme: &quot;&quot;
2020-04-29T05:39:42.384408Z     info    scheme &quot;&quot; not registered, fallback to default scheme
2020-04-29T05:39:42.384540Z     info    ccResolverWrapper: sending update to cc: {[{istiod.istio-system.svc:15010  &lt;nil&gt; 0 &lt;nil&gt;}] &lt;nil&gt; &lt;nil&gt;}
2020-04-29T05:39:42.385082Z     info    ClientConn switching balancer to &quot;pick_first&quot;
2020-04-29T05:39:42.391739Z     info    sds     SDS gRPC server for workload UDS starts, listening on &quot;/etc/istio/proxy/SDS&quot;

2020-04-29T05:39:42.392005Z     info    PilotSAN []string{&quot;spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account&quot;, &quot;istiod.istio-system.svc&quot;}
2020-04-29T05:39:42.392173Z     info    Starting proxy agent
2020-04-29T05:39:42.392458Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc000a6c020, {CONNECTING &lt;nil&gt;}
2020-04-29T05:39:42.393821Z     info    sds     Start SDS grpc server
2020-04-29T05:39:42.393911Z     info    Opening status port 15020

2020-04-29T05:39:42.394508Z     info    Received new config, creating new Envoy epoch 0
2020-04-29T05:39:42.394626Z     info    Epoch 0 starting
2020-04-29T05:39:42.398684Z     warn    failed to read pod labels: open ./etc/istio/pod/labels: no such file or directory
2020-04-29T05:39:42.404881Z     info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster ratings.bookinfo --service-node sidecar~10.244.3.53~ratings-v1-85c656b747-czg8z.bookinfo~bookinfo
.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error --concurrency 2]
[Envoy (Epoch 0)] [2020-04-29 05:39:42.450][15][critical][main] [external/envoy/source/server/server.cc:96] error initializing configuration '/etc/istio/proxy/envoy-rev0.json': Invalid path: ./var/run/secrets/istio/root-cert.pem
Invalid path: ./var/run/secrets/istio/root-cert.pem
2020-04-29T05:39:42.453430Z     error   Epoch 0 exited with error: exit status 1
2020-04-29T05:39:42.453665Z     info    No more active epochs, terminating`
</code></pre>
<p>Secret</p>
<pre><code>kubectl describe secret   istio.bookinfo-ratings -n bookinfo
Name:         istio.bookinfo-ratings
Namespace:    bookinfo
Labels:       &lt;none&gt;
Annotations:  istio.io/service-account.name: bookinfo-ratings

Type:  istio.io/key-and-cert

Data
====
cert-chain.pem:  1159 bytes
key.pem:         1679 bytes
root-cert.pem:   1054 bytes
 &quot;name&quot;: &quot;istio-certs&quot;,
 &quot;secret&quot;: {
   &quot;defaultMode&quot;: 420,
   &quot;optional&quot;: true,
   &quot;secretName&quot;: &quot;istio.bookinfo-ratings&quot;
</code></pre>
<p>Pod volumes</p>
<pre><code>Volumes:
  bookinfo-ratings-token-fs642:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  bookinfo-ratings-token-fs642
    Optional:    false
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  &lt;unset&gt;
  istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.bookinfo-ratings
    Optional:    true
    
</code></pre>
<p>UPDATE:</p>
<p>When looking at the istio injector config map and comparing with another, there is an if condition missing. Any one know how to get this condition back?</p>
<pre><code>
{{- if eq .Values.global.pilotCertProvider &quot;istiod&quot; }}
    - mountPath: /var/run/secrets/istio
      name: istiod-ca-cert
{{- end }}

</code></pre>

CodyK
<p>There are number of reasons that cause the error You have. It would be best to check if the certificate is actually there and if its valid.</p>
<p>According to istio <a href="https://istio.io/pt-br/docs/tasks/security/authentication/mutual-tls/#verify-keys-and-certificates-installation" rel="nofollow noreferrer">documentation</a> You can verify keys and certificates:</p>
<blockquote>
<h2>Verify keys and certificates installation<a href="https://istio.io/pt-br/docs/tasks/security/authentication/mutual-tls/#verify-keys-and-certificates-installation" rel="nofollow noreferrer"></a></h2>
<p>Istio automatically installs necessary keys and certificates for mutual TLS authentication in all sidecar containers. Run command below to confirm key and certificate files exist under  <code>/etc/certs</code>:</p>
</blockquote>
<pre><code>$ kubectl exec $(kubectl get pod -l app=httpbin -o jsonpath={.items..metadata.name}) -c istio-proxy -- ls /etc/certs

cert-chain.pem key.pem root-cert.pem
</code></pre>
<blockquote>
<p><code>cert-chain.pem</code>  is Envoy’s cert that needs to be presented to the other side.  <code>key.pem</code>  is Envoy’s private key paired with Envoy’s cert in  <code>cert-chain.pem</code>.  <code>root-cert.pem</code>  is the root cert to verify the peer’s cert. In this example, we only have one Citadel in a cluster, so all Envoys have the same  <code>root-cert.pem</code>.</p>
<p>Use the  <code>openssl</code>  tool to check if certificate is valid (current time should be in between  <code>Not Before</code>  and  <code>Not After</code>)</p>
</blockquote>
<pre><code>$ kubectl exec $(kubectl get pod -l app=httpbin -o jsonpath={.items..metadata.name}) -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout | grep Validity -A 2

Validity Not Before: May 17 23:02:11 2018 GMT Not After : Aug 15 23:02:11 2018 GMT
</code></pre>
<blockquote>
<p>You can also check the  <em>identity</em>  of the client certificate:</p>
</blockquote>
<pre><code>$ kubectl exec $(kubectl get pod -l app=httpbin -o jsonpath={.items..metadata.name}) -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout | grep 'Subject Alternative Name' -A 1

X509v3 Subject Alternative Name: URI:spiffe://cluster.local/ns/default/sa/default
</code></pre>
<blockquote>
<p>Please check  <a href="https://istio.io/pt-br/docs/concepts/security/#istio-identity" rel="nofollow noreferrer">Istio identity</a>  for more information about  <em>service identity</em>  in Istio.</p>
</blockquote>
<p>Hope it helps.</p>
