<p>I'm setting up the kafka and zookeeper cluster with high availability.I have 2 kafka brokers(pod1,pod2) and 3 zookeeper(pod1,pod2,pod3).The setup is working fine. When I enter into the one kafka broker(pod1) I'm able to produce and consume the message. But when I enter to the other kafka broker(pod2) I'm not able to get any messages though I have set replication factor to two.So I have added volumes in the container spec, now I'm not able to create any pod getting the crashloopbackoff. </p>

<p>When I checked the logs the information is as follows:
Bad request to server. Container is not able to create.</p>

<p>kafka_pod.yaml contains the kafka deployment.</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka
  labels:
    app: kafka
spec:
  replicas: 2
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      hostname: kafka
      containers:
      - name: kafka
        image: wurstmeister/kafka:2.11-1.0.2
        ports:
        - containerPort: 9092
          protocol: TCP
        env:
         - name: KAFKA_ADVERTISED_HOST_NAME
           value: kafka
         - name: KAFKA_ADVERTISED_PORT 
           value: "9092"
         - name: KAFKA_ZOOKEEPER_CONNECT
           value: zookeeper:2181
         - name: KAFKA_OFFSET_TOPIC_REPLICATION_FACTOR
           value: "2"
         - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
           value: "true"
         - name: KAFKA_LOG_DIRS
           value: /opt/kafka
        volumeMounts:
        - name: socket
          mountPath: /var/run/docker.sock
        - name: logdir
          mountPath: /opt/kafka
      volumes:
      - name: socket
        hostPath:
         path: /var/run/docker.sock
      - name: logdir
        hostPath:
         path: ~/datadir

zookeeper_pod.yaml contains the following.
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  ports:
  - port: 2181
  selector:
    app: zookeeper
  clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper
  labels:
    app: zookeeper
spec:
  replicas: 3
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      nodeName: akshatha-ha
      containers:
      - name: zookeeper
        image: wurstmeister/zookeeper
        ports:
        - containerPort: 2181
          protocol: TCP
</code></pre>

<p>I need to deploy the kafka with two broker and zookeeper with 3 server.When one of the server goes down the other should able to provide the data.</p>

<p>Use StatefulSets to deploy Kafka and zookeeper. There is a good tutorial on zookeeper StatefulSets on Kubernetes.io website. Follow that.</p>

<p>Avoid hostpath if you are not running single node cluster. Use persistent volume or ephemeral storage. If you are on version 1.14 then consider local persistent volumes for statefulsets</p>
