<p>I have 3 GKE clusters sitting in 3 different regions on Google Cloud Platform.
I would like to create a Kafka cluster which has one Zookeeper and one Kafka node (broker) in every region (each GKE cluster).</p>

<p>This set-up is intended to survive regional failure (I know a whole GCP region going down is rare and highly unlikely).</p>

<p>I am trying this set-up using this <a href="https://github.com/helm/charts/tree/master/incubator/kafka" rel="nofollow noreferrer">Helm Chart</a> provided by Incubator.</p>

<p>I tried this setup manually on 3 GCP VMs following <a href="https://codeforgeek.com/how-to-setup-zookeeper-cluster-for-kafka/" rel="nofollow noreferrer">this guide</a> and I was able to do it without any issues.</p>

<p>However, setting up a Kafka cluster on Kubernetes seems complicated.</p>

<p>As we know we have to provide the IPs of all the zookeeper server in each zookeeper configuration file like below:</p>

<pre><code>...
# list of servers
server.1=0.0.0.0:2888:3888
server.2=&lt;Ip of second server&gt;:2888:3888
server.3=&lt;ip of third server&gt;:2888:3888
...
</code></pre>

<p>As I can see in the Helm chart <a href="https://github.com/helm/charts/blob/ca895761948d577df1cb37243b6afaf7b077bac3/incubator/zookeeper/templates/config-script.yaml#L82" rel="nofollow noreferrer">config-script.yaml</a> file has a script which creates the Zookeeper configuration file for every deployment.</p>

<p>The part of the script which <em>echos</em> the zookeeper servers looks something like below:</p>

<pre><code>...
for (( i=1; i&lt;=$ZK_REPLICAS; i++ ))
do
   echo "server.$i=$NAME-$((i-1)).$DOMAIN:$ZK_SERVER_PORT:$ZK_ELECTION_PORT" &gt;&gt; $ZK_CONFIG_FILE
done
...
</code></pre>

<p>As of now the configuration that this Helm chart creates has the below Zookeeper server in the configuration with one replica (<em>replica</em> here means <code>Kubernetes Pods replicas</code>).</p>

<pre><code>...
# "release-name" is the name of the Helm release
server.1=release-name-zookeeper-0.release-name-zookeeper-headless.default.svc.cluster.local:2888:3888
...
</code></pre>

<p>At this point, I am clueless and do not know what to do, so that all the Zookeeper servers get included in the configuration file?</p>

<p>How shall I modify the script?</p>

<p>I see you are trying to create 3 node <em>zookeeper</em> cluster on top of 3 different GKE clusters.</p>

<p>This is not an easy task and I am sure there are multiple ways to achieve it
but I will show you one way in which it can be done and I believe it should solve your problem.</p>

<p>The first thing you need to do is create a LoadBalancer service for every zookeeper instance.
After LoadBalancers are created, note down the ip addresses that got assigned
(remember that by default these ip addresses are ephemeral so you might want to change them later to static).</p>

<p>Next thing to do is to create an <a href="https://cloud.google.com/compute/docs/internal-dns" rel="nofollow noreferrer">private DNS zone</a>
on GCP and create A records for every zookeeper LoadBalancer endpoint e.g.:</p>

<pre><code>release-name-zookeeper-1.zookeeper.internal.
release-name-zookeeper-2.zookeeper.internal.
release-name-zookeeper-3.zookeeper.internal.
</code></pre>

<p>and in GCP it would look like this:<br><br>
<a href="https://i.stack.imgur.com/7xjRJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7xjRJ.png" alt="dns"></a></p>

<p>After it's done, just modify <a href="https://github.com/helm/charts/blob/ca895761948d577df1cb37243b6afaf7b077bac3/incubator/zookeeper/templates/config-script.yaml#L48" rel="nofollow noreferrer">this line</a>:</p>

<pre><code>...
DOMAIN=`hostname -d'
...
</code></pre>

<p>to something like this:</p>

<pre><code>...
DOMAIN={{ .Values.domain }}
...
</code></pre>

<p>and remember to set <code>domain</code> variable in <code>Values</code> file to <code>zookeeper.internal</code></p>

<p>so in the end it should look like this:</p>

<pre><code>DOMAIN=zookeeper.internal
</code></pre>

<p>and it should generate the folowing config:</p>

<pre><code>...
server.1=release-name-zookeeper-1.zookeeper.internal:2888:3888
server.2=release-name-zookeeper-2.zookeeper.internal:2888:3888
server.3=release-name-zookeeper-3.zookeeper.internal:2888:3888
...
</code></pre>

<p>Let me know if it is helpful</p>
