<p>I install the incubator Kafka chart. Version <code>kafka-0.8.5</code> as of this writing.</p>

<pre><code>helm install --name kafka \
  --set replicas=1 \
  --set persistence.enabled=false \
  --set zookeeper.replicaCount=1 \
  incubator/kafka
</code></pre>

<p>To try this out I run a separate pod as just a bash shell with Kafka cli tools. I'm using the exact same Docker image <code>confluentinc/cp-kafka:4.1.1-2</code> that the <code>kafka-0</code> pod is using so that there is a perfect version match between the client and server:</p>

<pre><code>kubectl run shell --rm -i --tty --image confluentinc/cp-kafka:4.1.1-2 -- /bin/bash
</code></pre>

<p>Listing topics, publishing messages, getting topic offsets all works perfectly, as shown below. However, when I try to run <code>kafka-console-consumer</code> and see the test record in the topic, it hangs indefinitely. Why?</p>

<pre><code>root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-topics --zookeeper kafka-zookeeper:2181 --list

__confluent.support.metrics

root@shell-5c6ddf5d99-tbsvm:/# echo "abcxyz" | /usr/bin/kafka-console-producer --broker-list kafka:9092 --topic test-topic

&gt;[2018-08-07 16:43:26,110] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-topics --zookeeper kafka-zookeeper:2181 --list

__confluent.support.metrics
test-topic

root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-run-class kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic test-topic --time -1

test-topic:0:1

root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-console-consumer --bootstrap-server kafka:9092 --from-beginning --topic test-topic

&lt;hangs indefinitely&gt;
</code></pre>

<p>FYI, this is a local minikube development cluster with the latest minikube with Kubernetes 1.10.x server-side and 1.10.x kubectl client tools. This is a clean, new minikube, with nothing else running besides kafka, kafka-zookeeper, and my shell pod.</p>

<p>Also, writing a small Java client test app to consume gets a similar result of polling indefinitely with no messages. When my Java client subscribes to <code>test-topic</code>, it never gets notification callbacks of being assigned to the one topic partition.</p>

clay
<p>This cost me hours myself, there's a <a href="https://github.com/kubernetes/minikube/issues/1568" rel="nofollow noreferrer">bug in minikube</a> which prevents Kafka from working.</p>

<p>I'm not familiar with the Helm deployment, but you have to make sure of two things. First, the Kafka advertised host has to be the same as your Kubernetes service IP (or DNS name in kube dns), and second, you have to put minikube's network interface in promiscuous mode:</p>

<pre><code>minikube ssh sudo ip link set docker0 promisc on
</code></pre>

<p>If you don't do this workaround, kafka can't contant itself via the Kubernetes service, and its leader election fails. I've found it to be very fragile inside a container deployment environment.</p>
