<p>I'm trying to install the prometheus operator and inject using the sidecar. Mutual TLS is turned on and works okay for Jaeger. For the operator though we get a failure on the oper-admission job (see image). <a href="https://i.stack.imgur.com/SFlG9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SFlG9.png" alt="Prometheus Operator Error"></a></p>

<p>I believe Istio is causing this as if I release prometheus-operator prior to istio or without istio it works okay, but then it isn't injected.</p>

<p>I've tried setting the following in the istio operator sidecar settings:</p>

<pre><code>rewriteapphttpprobe:true
</code></pre>

<p>I've also tried to extend the readinessInitialDelaySeconds to 10s but still get the error. Does anyone else have any ideas?</p>

HandsDown
<p>Fisrt of all according to istio <a href="https://istio.io/docs/concepts/observability/#service-level-metrics" rel="nofollow noreferrer">documentation</a> Prometheus is used as default observation operator in istio mesh by default:</p>

<blockquote>
  <p>The <a href="https://istio.io/docs/reference/config/policy-and-telemetry/metrics/" rel="nofollow noreferrer">default Istio metrics</a> are defined by a set of configuration artifacts that ship with Istio and are exported to <a href="https://istio.io/docs/reference/config/policy-and-telemetry/adapters/prometheus/" rel="nofollow noreferrer">Prometheus</a> by default. Operators are free to modify the shape and content of these metrics, as well as to change their collection mechanism, to meet their individual monitoring needs.</p>
</blockquote>

<p>So by having istio injected prometheus operator You end up with two Prometheus operators in Your istio mesh.</p>

<p>Secondly, when you enforce Mutual TLS in Your istio mesh every connection has to be secure (<code>TLS</code>). And as You mentioned it works when there is no istio injection.</p>

<p>So the most likely cause is that the readiness probe fails because it is using <code>HTTP</code> protocol which is insecure (plain text) and this is one of the reason why You would get <code>503</code> error.</p>

<p>If you really need prometheus operator within istio mesh, this could be fixed by creating <code>DestinationRule</code> to <code>Disable</code> tls mode just for the readiness probe.</p>

<p>Example:</p>

<pre><code>$ kubectl apply -f - &lt;&lt;EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
 name: "readiness-probe-dr"
 namespace: "prometheus-namespace"
spec:
 host: "prometheus-prometheus-oper-prometheus.svc.cluster.local"
 trafficPolicy:
   tls:
     mode: DISABLE
EOF
</code></pre>

<p>Note: Make sure to modify it so that it matches Your namespaces and hosts. Also there could be some other prometheus collisions within mesh.</p>

<hr>

<p>The other solution would be not to have prometheus istio injected in the first place. You can disable istio injection in prometheus namespace by using the following commands:</p>

<pre><code>$ kubectl get namespace -L istio-injection
NAME              STATUS   AGE     ISTIO-INJECTION
default           Active   4d22h   enabled
istio-system      Active   4d22h   disabled
kube-node-lease   Active   4d22h   
kube-public       Active   4d22h   
kube-system       Active   4d22h   
prometheus        Active   30s     enabled
</code></pre>

<pre><code>$ kubectl label namespace prometheus istio-injection=disabled --overwrite
namespace/prometheus labeled
</code></pre>

<pre><code>$ kubectl get namespace -L istio-injection
NAME              STATUS   AGE     ISTIO-INJECTION
default           Active   4d22h   enabled
istio-system      Active   4d22h   disabled
kube-node-lease   Active   4d22h   
kube-public       Active   4d22h   
kube-system       Active   4d22h   
prometheus        Active   73s     disabled
</code></pre>
