<p>I am running a KOPS Kubernetes cluster on AWS, trying to make <a href="https://github.com/Yolean/kubernetes-kafka" rel="nofollow noreferrer">kubernetes-kafka</a> example work with an Elastic Load Balancer. Here is the external services portion for two of the brokers:</p>

<pre><code>kind: Service
apiVersion: v1
metadata:
  name: outside-0
  namespace: kafka
spec:
  selector:
    app: kafka
    kafka-broker-id: "0"
  ports:
  - protocol: TCP
    targetPort: 9094
    port: 32400
    nodePort: 32400
  type: NodePort
---
kind: Service
apiVersion: v1
metadata:
  name: outside-1
  namespace: kafka
spec:
  selector:
    app: kafka
    kafka-broker-id: "1"
  ports:
  - protocol: TCP
    targetPort: 9094
    port: 32401
    nodePort: 32401
  type: NodePort
</code></pre>

<p>Here is my attempt to expose those brokers via an ELB (actual FQDN replaced with <code>my.copmany.com</code>).</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: kafka-0
  annotations:
    dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com
spec:
  type: LoadBalancer
  ports:
  - port: 32400
    name: outside
    targetPort: 32400
  selector:
    app: outside-0
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-1
  annotations:
    dns.alpha.kubernetes.io/external: kafka-1.kafka.my.company.com
spec:
  type: LoadBalancer
  ports:
  - port: 32401
    name: outside
    targetPort: 32401
  selector:
    app: outside-1
</code></pre>

<p>Looking at AWS ELB console shows 0 of 3 instances available for each of the Kafka ELB brokers and producing to <code>kafka-1.kafka.my.company.com:9094</code> using Kafka command line client times out. How can <code>outside-0</code> NodePort service be exposed via <code>kafka-0</code> LoadBalancer service? Or are there other approached to be considered?</p>

<p>Kakfa is very particular about clients needing direct access to the server who is the leader of a topic. To achieve this, I had done the following:</p>

<p>1) Setup the configmap to dynamically override values for the advertised.listeners based on the pod's ordinal value </p>

<pre><code>POD_ID=${HOSTNAME##*-}

kafka-server-start.sh server.properties \
      --override advertised.listeners=INSIDE://`hostname -f`:9092,OUTSIDE://kafka-${POD_ID}.kafka.my.company.com:9094 \
      --override broker.id=${POD_ID} \
      --override listeners=INSIDE://:9092,OUTSIDE://:9094
</code></pre>

<p>2) Create a LoadBalancer service for each Kafka pod. Change the selector to match your kafka-pod-id. </p>

<pre><code>    apiVersion: v1
    kind: Service
    metadata:
      name: kafka-0
      annotations:
        dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com
    spec:
      externalTrafficPolicy: Local
      type: LoadBalancer
      ports:
      - port: 9094
        name: outside
        targetPort: 9094
      selector:
        app: kafka
        kafka-pod-id: "0"
---     
apiVersion: v1
        kind: Service
        metadata:
          name: kafka-1
          annotations:
            dns.alpha.kubernetes.io/external: kafka-1.kafka.my.company.com
        spec:
          externalTrafficPolicy: Local
          type: LoadBalancer
          ports:
          - port: 9094
            name: outside
            targetPort: 9094
          selector:
            app: kafka
            kafka-pod-id: "1"
</code></pre>
