<p>I have one Zookeeper server (say <code>xx.xx.xx.xxx:2181</code>) running on one GCP Compute Instance VM separately.</p>

<p>I have 3 GKE clusters all in different regions on which I am trying to install Kafka broker nodes so that all nodes connect to one Zookeeper server(<code>xx.xx.xx.xxx:2181</code>).</p>

<p>I installed the Zookeeper server on the VM following <a href="https://codeforgeek.com/how-to-setup-zookeeper-cluster-for-kafka/" rel="nofollow noreferrer">this guide</a> with zookeeper properties looking like below:</p>

<pre><code>dataDir=/tmp/data                                                                                                                                                                                                                                                                                                                                                                                                 
clientPort=2181
maxClientCnxns=0
initLimit=5
syncLimit=2
tickTime=2000

# list of servers
server.1=0.0.0.0:2888:3888
</code></pre>

<p>I am using this <a href="https://github.com/helm/charts/tree/master/incubator/kafka" rel="nofollow noreferrer">Incubator Helm Chart</a> to deploy the brokers on GKE clusters.</p>

<p>As per the <a href="https://github.com/helm/charts/tree/master/incubator/kafka#installing-the-chart" rel="nofollow noreferrer">README.md</a> I am trying to install with the below command:</p>

<pre><code>helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator

helm install --name my-kafka \
  --set replicas=1,zookeeper.enabled=false,configurationOverrides."broker\.id"=1,configurationOverrides."zookeeper\.connect"="xx.xx.xx.xxx:2181" \
  incubator/kafka
</code></pre>

<h3>Error</h3>

<p>When I deploy using any of the above ways described above on all of the three GKE Clusters, only one of the brokers gets connected to the Zookeeper server and the other two pods just restart infinitely.</p>

<p>When I check the Zookeeper log (on the VM), it looks something like below:</p>

<pre><code>...
[2019-10-30 14:32:30,930] INFO Accepted socket connection from /xx.xx.xx.xxx:54978 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2019-10-30 14:32:30,936] INFO Client attempting to establish new session at /xx.xx.xx.xxx:54978 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-10-30 14:32:30,938] INFO Established session 0x100009621af0057 with negotiated timeout 6000 for client /xx.xx.xx.xxx:54978 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-10-30 14:32:32,335] INFO Got user-level KeeperException when processing sessionid:0x100009621af0057 type:create cxid:0xc zxid:0x422 txntype:-1 reqpath:n/a Error Path:/config/users Error:KeeperErrorCode = NodeExists for /config/users (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-10-30 14:32:34,472] INFO Got user-level KeeperException when processing sessionid:0x100009621af0057 type:create cxid:0x14 zxid:0x424 txntype:-1 reqpath:n/a Error Path:/brokers/ids/0 Error:KeeperErrorCode = NodeExists for /brokers/ids/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-10-30 14:32:35,126] INFO Processed session termination for sessionid: 0x100009621af0057 (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-10-30 14:32:35,127] INFO Closed socket connection for client /xx.xx.xx.xxx:54978 which had sessionid 0x100009621af0057 (org.apache.zookeeper.server.NIOServerCnxn)
[2019-10-30 14:36:49,123] INFO Expiring session 0x100009621af003b, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
...
</code></pre>

<p>I am sure I have created firewall rules to open necessary ports and that is not a problem because one of the broker nodes is able to connect (the one who reaches first).</p>

<p>To me, this seems like the <code>borkerID</code> are not getting changed for some reason and that is the reason why Zookeeper is rejecting the connections.
I say this because <code>kubectl logs pod/my-kafka-n</code> outputs something like below:</p>

<pre><code>...
[2019-10-30 19:56:24,614] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
...
[2019-10-30 19:56:24,627] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
...
</code></pre>

<p>As we can see above says <code>brokerId=0</code> for all of the pods in all the 3 clusters.</p>

<p>However, when I do <code>kubectl exec -ti pod/my-kafka-n -- env | grep BROKER</code>, I can see the environment variable <code>KAFKA_BROKER_ID</code> is changed to <code>1</code>, <code>2</code> and <code>3</code> for different brokers as I set.</p>

<p>What am I doing wrong? What is the correct way to change the kafka-broker id or to make all brokers connect to one Zookeeper instance?</p>

Amit Yadav
<blockquote>
  <p>make all brokers connect to one Zookeeper instance?</p>
</blockquote>

<p>Seems like you are doing that okay via the <code>configurationOverrides</code> option. That'll deploy all pods with the same configuration. </p>

<p>That being said, the broker ID should not be the same per pod. If you inspect the statefulset YAML, it appears that <a href="https://github.com/helm/charts/blob/master/incubator/kafka/templates/statefulset.yaml#L192-L201" rel="nofollow noreferrer">the broker ID is calculated based on the <code>POD_NAME</code> variable</a></p>

<hr>

<p><sub>Sidenote</sub></p>

<blockquote>
  <p>3 GKE clusters all in different regions on which I am trying to install Kafka broker nodes so that all nodes connect to one Zookeeper server</p>
</blockquote>

<p>It's not clear to me how you would able to deploy to 3 sepearate clusters in one API call. But, this architecture isn't recommended by Kafka, Zookeeper, or Kubernetes communities unless these regions are "geographically close"</p>
