<p>Using <a href="https://github.com/Yolean/kubernetes-kafka" rel="noreferrer">kubernetes-kafka</a> as a starting point with minikube.</p>

<p>This uses a StatefulSet and a <a href="https://github.com/Yolean/kubernetes-kafka/blob/master/20dns.yml" rel="noreferrer">headless service</a> for service discovery within the cluster.</p>

<p>The goal is to expose the individual Kafka Brokers externally which are internally addressed as:</p>

<pre><code>kafka-0.broker.kafka.svc.cluster.local:9092
kafka-1.broker.kafka.svc.cluster.local:9092 
kafka-2.broker.kafka.svc.cluster.local:9092
</code></pre>

<p>The constraint is that this external service be able to address the brokers specifically.</p>

<p>Whats the right (or one possible) way of going about this? Is it possible to expose a external service per <code>kafka-x.broker.kafka.svc.cluster.local:9092</code>?</p>

<p>We have solved this in 1.7 by changing the headless service to <code>Type=NodePort</code> and setting the <code>externalTrafficPolicy=Local</code>. This bypasses the internal load balancing of a Service and traffic destined to a specific node on that node port will only work if a Kafka pod is on that node.</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: broker
spec:
  externalTrafficPolicy: Local
  ports:
  - nodePort: 30000
    port: 30000
    protocol: TCP
    targetPort: 9092
  selector:
    app: broker
  type: NodePort
</code></pre>

<p>For example, we have two nodes nodeA and nodeB, nodeB is running a kafka pod. nodeA:30000 will not connect but nodeB:30000 will connect to the kafka pod running on nodeB.</p>

<p><a href="https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typenodeport" rel="noreferrer">https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typenodeport</a></p>

<p>Note this was also available in 1.5 and 1.6 as a beta annotation, more can be found here on feature availability: <a href="https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip" rel="noreferrer">https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip</a></p>

<p>Note also that while this ties a kafka pod to a specific external network identity, it does not guarantee that your storage volume will be tied to that network identity. If you are using the VolumeClaimTemplates in a StatefulSet then your volumes are tied to the pod while kafka expects the volume to be tied to the network identity.</p>

<p>For example, if the kafka-0 pod restarts and kafka-0 comes up on nodeC instead of nodeA, kafka-0's pvc (if using VolumeClaimTemplates) has data that it is for nodeA and the broker running on kafka-0 starts rejecting requests thinking that it is nodeA not nodeC. </p>

<p>To fix this, we are looking forward to Local Persistent Volumes but right now we have a single PVC for our kafka StatefulSet and data is stored under <code>$NODENAME</code> on that PVC to tie volume data to a particular node.</p>

<p><a href="https://github.com/kubernetes/features/issues/121" rel="noreferrer">https://github.com/kubernetes/features/issues/121</a>
<a href="https://kubernetes.io/docs/concepts/storage/volumes/#local" rel="noreferrer">https://kubernetes.io/docs/concepts/storage/volumes/#local</a></p>
