<p>We're using <code>sasl_tls</code> mechanism with bitnami/kafka helm chart. We're using Let's Encrypt and cert-manager for issuing the certificate. Created a secret out of the Let's Encrypt generated certificate and passed the secret to the <code>existingSecrets</code> parameter in the helm chart. Now when I'm using KafkaJS library to connect to the Kafka broker, with <code>ssl: true</code> it is throwing an error:</p>
<pre><code>KafkaJSConnectionError: Connection error: unable to verify the first certificate
</code></pre>
<p><strong>Detailed Steps/How to generate:</strong></p>
<ul>
<li>Enabled external access to kafka chart so that it gives us an IP at port 9094</li>
</ul>
<pre><code>externalAccess.enabled: true
externalAccess.autoDiscovery.enabled: true
externalAccess.service.type: LoadBalancer
externalAccess.service.ports.external: 9094
externalAccess.service.domain: &quot;&quot;
</code></pre>
<ul>
<li>Bound this IP to a domain <code>xyz.com</code></li>
<li>Bound this domain name to Let's Encrypt certificate issuer to issue certificate for this domain</li>
<li><code>tls.crt</code> and <code>tls.key</code> are generated</li>
<li>Renamed these files and used these to create a secret</li>
</ul>
<pre><code>kubectl create secret generic kafka-tls-0 --from-file=tls.crt=kafka-0.tls.crt --from-file=tls.key=kafka-0.tls.key
</code></pre>
<ul>
<li>Modified chart value to configure tls part</li>
</ul>
<pre><code>tls.type: pem
tls.pemChainIncluded: true
tls.existingSecrets: [&quot;kafka-tls-0&quot;]
</code></pre>
<ul>
<li>Applied the values of the chart (started broker)</li>
<li>Now in KafkaJS client setup, tried to pass value to the <code>brokers</code> parameter in either format <code>ip:9094</code> or <code>xyz.com:9094</code>, also passed <code>ssl:true</code></li>
</ul>
<p><strong>My Questions:</strong></p>
<ul>
<li><p>Is the flow correct? Or are we going to the wrong direction?</p>
</li>
<li><p>What is the reason behind the problem? Is this the certificate chain that is being being wrong? (seems like it is!)</p>
</li>
<li><p>Is there any other chart that I can use to achieve my goal?</p>
</li>
</ul>
<p><strong>Followup Question:</strong></p>
<ol>
<li>If we can make it work, what will be the next steps for ensuring auto-renewal of the certificates? Is it managed automatically? Or should we have to maintain a script for Lets' Encrypt certificate auto-renewal?</li>
</ol>

Azman Amin
<p>There could be multiple causes. I'll try to list what needs to be true for this to work:</p>
<ol>
<li>Your node.js KafkaJS client should have a certificate store that is able to verify the CA that signed the Let's Encrypt certificate. node.js has a built-in list of certificates, and you can add to it. I have not checked but I expect the Let's Encrypt root CAs to be there.</li>
</ol>
<p>A couple years ago Let's Encrypt switched root CAs, so if you have an old version of node.js that could be it.</p>
<ol start="2">
<li>The Kafka broker must present a certificate chain (not just your signed certificate) that includes a certificate that the node.js client can verify.
Depending on which CAs your client can verify, this could mean the chain needs to go as far as the root CA (as opposed to an intermediate CA).</li>
</ol>
<p>You should check which certificates are in your chain. You can do this with the OpenSSL CLI:
<code>openssl x509 -in cert.pem -text -noout</code></p>
<p>Specific advice for the Bitnami Kafka chart: we've had trouble with how the scripts included with the chart deal with PEM keys and cert chains, where it would sometimes not extract the entire chain as it processes the PEM, and then Kafka would only see a partial chain.</p>
<p>I would try to use the JKS format (Java keystore &amp; truststore) instead and see if that helps. You would create a JKS keystore with your key and a truststore with all the certificates in the chain.</p>
<p>Regarding auto-renewal of certificates - you should be able to achieve that with cert-manager, however that might be challenging with the Bitnami Kafka chart as it's not suited to renewing certificates periodically, and is less suited for short-lived certificates from a CA like Let's Encrypt. Normally, you'd use Let's Encrypt with a load balancer like NGINX, you'd usually have a Kubernetes ingress controller that handles noticing the new certificates and reloading the load balancer.</p>
<p>In your case, since you are trying to generate TLS certificates for use by your backend services to communicate with Kafka, you might have an easier time doing this with something that was intended for inter-service communication (which Let's Encrypt is not), like SPIRE and a matching Kubernetes operator.</p>
<p><a href="https://github.com/spiffe/spire" rel="nofollow noreferrer">SPIRE</a>, which is a CNCF project that deals with attesting workload identities and representing them cryptographically - in your case as a TLS keypair for the Kafka server. It takes care of things like renewing the certificates.</p>
<p>To make SPIRE easy to use in Kubernetes, deploy it together with <a href="https://github.com/otterize/spire-integration-operator" rel="nofollow noreferrer">Otterize SPIRE integration operator</a>, which uses SPIRE to generate TLS credentials, saves them in Kubernetes secrets, and takes care of refreshing the secrets as the certificates require renewal by SPIRE. You deploy it in your cluster than annotate pods with what you'd like the secret to be called that holds the certificates, and you can use other annotations to configure things like whether the format is PEM or JKS or what the TTL is. That set of configuration should make it easy to get it working with Bitnami. We use it with the Bitnami chart successfully, and <a href="https://docs.otterize.com/guides/ibac-for-k8s-kafka/" rel="nofollow noreferrer">even have a tutorial for getting it working with Bitnami</a> - stop at the section that configures ACLs if all you want is TLS.</p>
<p>Since you also mentioned you use SASL, you might want to just replace the username/password completely with certificates and switch to mTLS. If you also want to add Kafka ACLs into the mix and allow access to certain topics/operations only for certain workloads, you can also deploy the <a href="https://github.com/otterize/intents-operator" rel="nofollow noreferrer">Otterize intents operator</a>. It lets you declare which topics a workload needs access to, and works together with SPIRE and built-in Kafka ACLs so that workloads can only access what they've declared.</p>
