<p>I have a Kafka Custer on GKE, and I'm using Apache Spark on Dataproc to access the Kafka Cluster.
Dataproc cluster is a private cluster i.e. --no-address is specified when creating the Dataproc cluster, which means it does not have public IP
(required due to compliance).</p>
<p>Also the Kafka cluster  is SSL enabled.</p>
<p>Note : GKE &amp; Dataproc are in the same VPC/project &amp; region</p>
<p>We have a NAT created, which is allowing the Spark on Dataproc to access Kafka on GKE (using the public IP on Kafka brokers).
Without the NAT, Spark is not able to connect to Kafka on GKE - even though they are on the same VPC &amp; region.</p>
<p>We would like to get rid of the NAT to reduce costs, any ideas on how to architect/instrument this ?</p>
<p>Here is the script used for creating the Dataproc cluster :</p>
<pre><code># --no-address - enables it to be a private

TYPE=n1-highmem-4
CNAME=&lt;cluster-name&gt;
BUCKET=&lt;gcp-bucket&gt;
REGION=us-east1
ZONE=us-east1-b
IMG_VERSION=2.0-ubuntu18
PROJECT=versa-sml-googl
NUM_WORKER=3

gcloud beta dataproc clusters create $CNAME \
           --enable-component-gateway \
           --bucket $BUCKET \
           --region $REGION \
           --zone $ZONE \
           --no-address --master-machine-type $TYPE \
           --master-boot-disk-size 100 \
           --master-boot-disk-type pd-ssd \
           --num-workers $NUM_WORKER \
           --worker-machine-type $TYPE \
           --worker-boot-disk-type pd-ssd \
           --worker-boot-disk-size 500 \
           --image-version $IMG_VERSION \
           --scopes 'https://www.googleapis.com/auth/cloud-platform' \
           --project $PROJECT \
           --initialization-actions 'gs://dataproc-spark-configs/pip_install_versions.sh' \
           --metadata 'gcs-connector-version=2.0.0' \
           --properties 'dataproc:dataproc.logging.stackdriver.job.driver.enable=true,dataproc:dataproc.conscrypt.provider.enable=false,dataproc:job.history.to-gcs.enabled=true,spark:spark.dynamicAllocation.enabled=true,spark:spark.eventLog.enabled=true,spark:spark.eventLog.dir=gs://dataproc-spark-logs/eventlogs,spark:spark.history.fs.logDirectory=gs://dataproc-spark-logs/jobhistorylogs'



</code></pre>

<p>Not sure if you have <strong>whitelisted</strong> the <strong>IP</strong> of <strong>Spark cluster NAT</strong> to <strong>Kafka GKE</strong>, i think the request is routing outside to <strong>internet</strong> from <strong>NAT</strong> and then accessing the <strong>Kafka</strong>.</p>
<p>If both are in same <strong>VPC</strong> what you can do is expose the <strong>Kafka</strong> <strong>service</strong> on <strong>GKE</strong> with <strong>internal private load balancer</strong>, once <strong>SVC</strong> in <strong>GKE</strong> have <strong>private LB</strong> attached you will be able to access <strong>Kafka</strong> from anywhere in <strong>VPC</strong> without using the <strong>NAT</strong>.</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: ilb-svc
  annotations:
    networking.gke.io/load-balancer-type: &quot;Internal&quot;
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app: ilb-deployment
  ports:
  - name: tcp-port
    protocol: TCP
    port: 8080
    targetPort: 8080
</code></pre>
<p>If you are using any <strong>Ingress controller</strong> to expose the <strong>Kafka</strong> like <strong>Nginx</strong> or so with <strong>TCP port proxy (Static port)</strong> <a href="https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/" rel="nofollow noreferrer">Read More</a> you can attach the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing" rel="nofollow noreferrer">internal LB</a> to <strong>Nginx ingress</strong> instead of attaching it to <strong>Kafka</strong> <strong>service</strong>.</p>
