<p>I have k8s cluster with Istio v1.6.4. The sidecar injection is disabled by default.
I have Kafka cluster running on this k8s installed with strimzi kafka operator.
The Kafka cluster works without any problems when kafka as well as client pods doesn't have Istio-proxy injected.
My problem:
When I create a pod with kafka client and Istio-proxy injected I can't connect to Kafka cluster.
The logs on client side:</p>
<pre><code>java.io.IOException: Connection reset by peer
</code></pre>
<p>and on the server side:
<code>org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295616 larger than 104857600)</code></p>
<p>After some googling and checking the Istio-proxy logs it turns out the problem is that Istio-proxy connects to kafka plaintext endpoint with TLS.
I can workaround this by setting the default PeerAuthentication with <code>mtls.mode: DISABLED</code> but I don't want to set global setting for it.</p>
<p>What is strange if I create a simple k8s service and run the netcat &quot;server&quot; on pod running kafka server and netcat &quot;client&quot; on pod running kafka client - everything works fine.</p>
<p>I have 2 question:</p>
<ol>
<li>Why the kafka Istio-proxy behaves different when connecting to Kafka
cluster than other TCP connections (like using nc)?</li>
<li>How to disable mtls for one host only? I was playing with PeerAuthentication but no luck...</li>
</ol>

<p>With <a href="https://stackoverflow.com/users/11977760/jt97">jt97</a>'s  help I was able to solve this problem.</p>
<p>As I wrote I'm using <a href="https://strimzi.io" rel="noreferrer">Strimzi Operator</a> to install kafka cluster on k8s. It creates 2 services:</p>
<ul>
<li>kafka-bootstrap - which is a regular service with ClusterIP</li>
<li>kafka-brokers - a headless service.</li>
</ul>
<p>In my case the full name of the services are <code>kafka-kafka-operated-kafka-bootstrap</code> and <code>kafka-kafka-operated-kafka-brokers</code>, respectively.</p>
<p>I created a DestinationRule:</p>
<pre><code>apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: kafka-no-tls
spec:
  host: &quot;kafka-kafka-operated-kafka-brokers&quot;
  trafficPolicy:
    tls:
      mode: DISABLE
</code></pre>
<p>and used the headless service when connecting to kafka:</p>
<pre><code>kafka-topics --bootstrap-server kafka-kafka-operated-kafka-brokers:9092 --list
__consumer_offsets
_schemas
</code></pre>
<p>and it worked as expected.</p>
<p>BTW, setting <code>tls.mode</code> to <code>SIMPLE</code> didn't help.</p>
<p>To be honest I still don't understand why in this particular case Istio-proxy by default (without DestinationRule) tries to connect with TLS - according to <a href="https://istio.io/latest/docs/tasks/security/authentication/authn-policy/#auto-mutual-tls" rel="noreferrer">documentation</a>:</p>
<blockquote>
<p>By default, Istio tracks the server workloads migrated to Istio proxies, and configures client proxies to send mutual TLS traffic to those workloads automatically, and to send plain text traffic to workloads without sidecars.</p>
</blockquote>
