<p>I’m playing around with a Kafka pod (the <code>incubator/kafka</code> chart) I have running in a kubernetes cluster, trying to send and receive some test data. I have a consumer running that I kicked off using:</p>
<pre><code>kubectl exec kafka-0 \
  -n kafka-namespace \
  -- kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic test
</code></pre>
<p>I was able to successfully send data to the consumer using:</p>
<pre><code>kubectl exec -i kafka-0 \
  -n kafka-namespace \
  -- kafka-console-producer \
  --broker-list localhost:9092 \
  --topic test &lt; testdata.jsonl
</code></pre>
<p>I’d like to send data using port-forwarding, as that will make it easier for me to do some partitioning stuff. However, it keeps failing for me. I set up port forwarding by running:</p>
<pre><code>kubectl port-forward \
  -n kafka-namespace \
  svc/kafka 9092:9092
</code></pre>
<p>I’m trying to send data to Kafka by running from my local machine:</p>
<pre><code>kafka-console-producer \
  --broker-list localhost:9092 \
  --topic test &lt; testdata.jsonl
</code></pre>
<p>In my port-forwarding terminal, I see <code>Handling connection for 9092</code> appear. However, this fails with error:</p>
<pre><code>[2022-07-27 15:48:44,227] ERROR Error when sending message to topic test with key: null, value: 139 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 12 record(s) for test-0:120000 ms has passed since batch creation
</code></pre>
<p>If I’m doing port-forwarding, then I should just be able to use the <code>kafka-console-producer</code> command as if it’s running on localhost, right?</p>

James Kelleher
<p>Short answer: <a href="https://stackoverflow.com/questions/51630260/connect-to-kafka-running-in-docker">no, a port forward will not (directly) work</a>. Secondly, <code>helm/charts</code> repo in its entirety is deprecated and the Kafka Chart no longer updated past Nov 2020, and that's the only <code>incubator</code> repo I am familiar with...</p>
<p>In any case, specifically, you need <code>external.enabled=true</code> set in the values file, along with a few other values, and this will create a Service resource that is available to use externally from the k8s environment. This will expose the brokers on a NodePort so will not need <code>kubectl port-forward</code> and will instead use the exposed pods directly. Ref (scroll down to search for &quot;external&quot;) - <a href="https://github.com/helm/charts/blob/master/incubator/kafka/README.md" rel="nofollow noreferrer">https://github.com/helm/charts/blob/master/incubator/kafka/README.md</a></p>
<p>Or, you can simply use <code>kubectl exec</code> instead into a Kafka pod, and then you wouldn't need any networking setup or need to clutter your host with Kafka downloads, as the pod will have all the scripts you need.</p>
<p>Also worth mentioning, if you want to easily use Kafka in Kubernetes (in a way which has been updated in the meantime), use an Operator such as <a href="https://strimzi.io" rel="nofollow noreferrer">https://strimzi.io</a></p>
