<p>I have a local k3s Kubernetes cluster created by multipass.</p>
<p>I am trying to set up Kafka using Ingress way based on <a href="https://strimzi.io/blog/2019/05/23/accessing-kafka-part-5/" rel="nofollow noreferrer">this tutorial</a> to make it accessible for clients running outside of Kubernetes.</p>
<p>Here are my steps:</p>
<p>First get my cluster IP by</p>
<pre><code>➜ kubectl get nodes
NAME          STATUS   ROLES                  AGE   VERSION
west-master   Ready    control-plane,master   15m   v1.26.3+k3s1

➜ kubectl get node west-master -o wide
NAME          STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
west-master   Ready    control-plane,master   16m   v1.26.3+k3s1   192.168.205.5   &lt;none&gt;        Ubuntu 22.04.2 LTS   5.15.0-67-generic   containerd://1.6.19-k3s1

➜ kubectl cluster-info
Kubernetes control plane is running at https://192.168.205.5:6443
CoreDNS is running at https://192.168.205.5:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://192.168.205.5:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
</code></pre>
<p>which is <code>192.168.205.5</code> (Is this correct IP address for the one I used in <strong>my-kafka-persistent.yaml</strong> I used in next step?).</p>
<p>Then I deploy my Kafka by:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl create namespace hm-kafka
kubectl apply --filename=&quot;https://strimzi.io/install/latest?namespace=hm-kafka&quot; --namespace=hm-kafka
kubectl apply --filename=my-kafka-persistent.yaml --namespace=hm-kafka
</code></pre>
<p><strong>my-kafka-persistent.yaml</strong> (based on <a href="https://strimzi.io/examples/latest/kafka/kafka-persistent.yaml" rel="nofollow noreferrer">kafka-persistent.yaml</a>):</p>
<pre class="lang-yaml prettyprint-override"><code>---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: hm-kafka
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          bootstrap:
            host: kafka-bootstrap.192.168.205.5.nip.io
          brokers:
          - broker: 0
            host: kafka-broker-0.192.168.205.5.nip.io
          - broker: 1
            host: kafka-broker-1.192.168.205.5.nip.io
          - broker: 2
            host: kafka-broker-2.192.168.205.5.nip.io
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: &quot;3.4&quot;
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 100Gi
          deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
</code></pre>
<p>After deployment:</p>
<p>Pods</p>
<p><a href="https://i.stack.imgur.com/KaFqX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KaFqX.png" alt="enter image description here" /></a></p>
<p>Services</p>
<p><a href="https://i.stack.imgur.com/FvlSO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/FvlSO.png" alt="enter image description here" /></a></p>
<p>Ingresses</p>
<p><a href="https://i.stack.imgur.com/HhtuF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HhtuF.png" alt="enter image description here" /></a></p>
<p>Also, for each Ingress, I can see SSL passthrough in the annotations:</p>
<p><a href="https://i.stack.imgur.com/sUuQR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sUuQR.png" alt="enter image description here" /></a></p>
<p>Then I succeed following the tutorial creating the truststore.</p>
<pre class="lang-bash prettyprint-override"><code>➜ kubectl get secret hm-kafka-cluster-ca-cert \
  --namespace=hm-kafka \
  --output=jsonpath=&quot;{.data.ca\.crt}&quot; \
  | base64 -d \
  &gt; ca.crt

➜ keytool -importcert \
  -trustcacerts \
  -alias root \
  -file ca.crt \
  -keystore kafka-truststore.jks \
  -storepass my_passw0rd \
  -noprompt
Certificate was added to keystore
</code></pre>
<p>However, when I try to produce data, I met this issue:</p>
<pre><code>➜ kafka-console-producer \
  --broker-list kafka-bootstrap.192.168.205.5.nip.io:443 \
  --producer-property security.protocol=SSL \
  --producer-property ssl.truststore.password=my_passw0rd \
  --producer-property ssl.truststore.location=kafka-truststore.jks \
  --topic my-topic
&gt;[2023-04-14 15:57:06,047] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.192.168.205.5.nip.io/192.168.205.5:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,047] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.192.168.205.5.nip.io:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,200] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.192.168.205.5.nip.io/192.168.205.5:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,201] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.192.168.205.5.nip.io:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,691] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.192.168.205.5.nip.io/192.168.205.5:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-04-14 15:57:06,691] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.192.168.205.5.nip.io:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
</code></pre>
<p>Any guide would be appreciate, thanks!</p>
<hr />
<h2>UPDATE 1</h2>
<p>Thanks @OneCricketeer pointing the issue!</p>
<p>As I am using <strong>multipass</strong> on macOS, I can provide <code>INSTALL_K3S_EXEC=&quot;server --disable traefik&quot;</code>, so the updated command to create k3s cluster is:</p>
<pre class="lang-bash prettyprint-override"><code>multipass launch --name=west-master --cpus=4 --memory=16g --disk=128g
multipass exec west-master -- \
  bash -c 'curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=&quot;server --disable traefik&quot; K3S_KUBECONFIG_MODE=&quot;644&quot; sh -'
</code></pre>
<p>I actually switched to Rancher Desktop as it is also using k3s and easy to disable Traefik which can be set up in the UI.</p>
<p><a href="https://i.stack.imgur.com/AyRC2.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AyRC2.png" alt="enter image description here" /></a></p>
<h2>UPDATE 2</h2>
<p>Regarding how to deploy ingress-nginx and how to resolve another issue &quot;ingress does not contain a valid IngressClass&quot; I met, I posted at <a href="https://stackoverflow.com/questions/76020537/strimzi-kafka-brokers-not-created-because-of-ingress-does-not-contain-a-valid-i">Strimzi Kafka brokers not be created because of &quot;ingress does not contain a valid IngressClass&quot;</a></p>

Hongbo Miao
<p>k3s uses traefik, not nginx, so those annotations aren't doing anything... The referenced blog assumes you are using nginx instead</p>
<p>Restart your k3s cluster, but provide <code>--no-deploy-traefik</code> option, and install nginx ingress controller</p>
<p>Otherwise, you will need to refer to Traefik ingress docs on what matching annotations it will use for SSL passthrough.</p>
<p>Keeping in mind, Kafka is not an HTTP/S service, so you should not be using ports 80/443 to communicate with it.</p>
