<p>I've deployed <a href="https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka" rel="nofollow noreferrer">https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka</a> on my on prem k8s cluster. 
I'm trying to expose it my using a TCP controller with nginx.</p>

<p>My TCP nginx configmap looks like </p>

<pre><code>data:
  "&lt;zookeper-tcp-port&gt;": &lt;namespace&gt;/cp-zookeeper:2181
  "&lt;kafka-tcp-port&gt;": &lt;namespace&gt;/cp-kafka:9092
</code></pre>

<p>And i've made the corresponding entry in my nginx ingress controller</p>

<pre><code>  - name: &lt;zookeper-tcp-port&gt;-tcp
    port: &lt;zookeper-tcp-port&gt;
    protocol: TCP
    targetPort: &lt;zookeper-tcp-port&gt;-tcp
  - name: &lt;kafka-tcp-port&gt;-tcp
    port: &lt;kafka-tcp-port&gt;
    protocol: TCP
    targetPort: &lt;kafka-tcp-port&gt;-tcp
</code></pre>

<p>Now I'm trying to connect to my kafka instance. 
When i just try to connect to the IP and port using kafka tools, I get the error message </p>

<pre><code>Unable to determine broker endpoints from Zookeeper.
One or more brokers have multiple endpoints for protocol PLAIN...
Please proved bootstrap.servers value in advanced settings
[&lt;cp-broker-address-0&gt;.cp-kafka-headless.&lt;namespace&gt;:&lt;port&gt;][&lt;ip&gt;]
</code></pre>

<p>When I enter, what I assume are the correct broker addresses (I've tried them all...) I get a time out. There are no logs coming from the nginx controler excep </p>

<pre><code>[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:14 +0000]TCP200000.001
</code></pre>

<p>From the pod <code>kafka-zookeeper-0</code> I'm gettting loads of </p>

<pre><code>[2020-04-08 15:52:02,415] INFO Accepted socket connection from /&lt;ip:port&gt; (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-08 15:52:02,415] WARN Unable to read additional data from client sessionid 0x0, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-04-08 15:52:02,415] INFO Closed socket connection for client /&lt;ip:port&gt;  (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
</code></pre>

<p>Though I'm not sure these have anything to do with it?</p>

<p>Any ideas on what I'm doing wrong? 
Thanks in advance. </p>

t3ng1l
<p><strong>TL;DR:</strong></p>

<ul>
<li>Change the value <code>nodeport.enabled</code> to <code>true</code> inside <code>cp-kafka/values.yaml</code> before deploying.</li>
<li>Change the service name and ports in you TCP NGINX Configmap and Ingress object.</li>
<li>Set <code>bootstrap-server</code> on your kafka tools to <code>&lt;Cluster_External_IP&gt;:31090</code></li>
</ul>

<hr>

<p><strong>Explanation:</strong></p>

<blockquote>
  <p>The <a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services" rel="nofollow noreferrer">Headless Service</a> was created alongside the StatefulSet. The created service will <strong>not</strong> be given a <code>clusterIP</code>, but will instead simply include a list of <code>Endpoints</code>.
   These <code>Endpoints</code> are then used to generate instance-specific DNS records in the form of:
    <code>&lt;StatefulSet&gt;-&lt;Ordinal&gt;.&lt;Service&gt;.&lt;Namespace&gt;.svc.cluster.local</code></p>
</blockquote>

<p>It creates a DNS name for each pod, e.g:</p>

<pre><code>[ root@curl:/ ]$ nslookup my-confluent-cp-kafka-headless
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      my-confluent-cp-kafka-headless
Address 1: 10.8.0.23 my-confluent-cp-kafka-1.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 2: 10.8.1.21 my-confluent-cp-kafka-0.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 3: 10.8.3.7 my-confluent-cp-kafka-2.my-confluent-cp-kafka-headless.default.svc.cluster.local
</code></pre>

<ul>
<li>This is what makes this services connect to each other inside the cluster.</li>
</ul>

<hr>

<p>I've gone through a lot of trial and error, until I realized how it was supposed to be working. Based your TCP Nginx Configmap I believe you faced the same issue.</p>

<ul>
<li>The <a href="https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/" rel="nofollow noreferrer">Nginx ConfigMap</a> asks for: <code>&lt;PortToExpose&gt;: "&lt;Namespace&gt;/&lt;Service&gt;:&lt;InternallyExposedPort&gt;"</code>.</li>
<li>I realized that you don't need to expose the Zookeeper, since it's a internal service and handled by kafka brokers.</li>
<li>I also realized that you are trying to expose <code>cp-kafka:9092</code> which is the headless service, also only used internally, as I explained above.</li>
<li>In order to get outside access <strong>you have to set the parameters <code>nodeport.enabled</code> to <code>true</code></strong> as stated here: <a href="https://github.com/confluentinc/cp-helm-charts/blob/master/charts/cp-kafka/README.md#external-access" rel="nofollow noreferrer">External Access Parameters</a>.</li>
<li>It adds one service to each kafka-N pod during chart deployment.</li>
<li>Then you change your configmap to map to one of them:</li>
</ul>

<pre><code>data:
"31090": default/demo-cp-kafka-0-nodeport:31090
</code></pre>

<p>Note that the service created has the selector <code>statefulset.kubernetes.io/pod-name: demo-cp-kafka-0</code> this is how the service identifies the pod it is intended to connect to.</p>

<ul>
<li>Edit the nginx-ingress-controller:</li>
</ul>

<pre><code>- containerPort: 31090
  hostPort: 31090
  protocol: TCP
</code></pre>

<ul>
<li>Set your kafka tools to <code>&lt;Cluster_External_IP&gt;:31090</code></li>
</ul>

<hr>

<p><strong>Reproduction:</strong>
- Snippet edited in <code>cp-kafka/values.yaml</code>:</p>

<pre><code>nodeport:
  enabled: true
  servicePort: 19092
  firstListenerPort: 31090
</code></pre>

<ul>
<li>Deploy the chart:</li>
</ul>

<pre><code>$ helm install demo cp-helm-charts
$ kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
demo-cp-control-center-6d79ddd776-ktggw    1/1     Running   3          113s
demo-cp-kafka-0                            2/2     Running   1          113s
demo-cp-kafka-1                            2/2     Running   0          94s
demo-cp-kafka-2                            2/2     Running   0          84s
demo-cp-kafka-connect-79689c5c6c-947c4     2/2     Running   2          113s
demo-cp-kafka-rest-56dfdd8d94-79kpx        2/2     Running   1          113s
demo-cp-ksql-server-c498c9755-jc6bt        2/2     Running   2          113s
demo-cp-schema-registry-5f45c498c4-dh965   2/2     Running   3          113s
demo-cp-zookeeper-0                        2/2     Running   0          112s
demo-cp-zookeeper-1                        2/2     Running   0          93s
demo-cp-zookeeper-2                        2/2     Running   0          74s

$ kubectl get svc
NAME                         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
demo-cp-control-center       ClusterIP   10.0.13.134   &lt;none&gt;        9021/TCP            50m
demo-cp-kafka                ClusterIP   10.0.15.71    &lt;none&gt;        9092/TCP            50m
demo-cp-kafka-0-nodeport     NodePort    10.0.7.101    &lt;none&gt;        19092:31090/TCP     50m
demo-cp-kafka-1-nodeport     NodePort    10.0.4.234    &lt;none&gt;        19092:31091/TCP     50m
demo-cp-kafka-2-nodeport     NodePort    10.0.3.194    &lt;none&gt;        19092:31092/TCP     50m
demo-cp-kafka-connect        ClusterIP   10.0.3.217    &lt;none&gt;        8083/TCP            50m
demo-cp-kafka-headless       ClusterIP   None          &lt;none&gt;        9092/TCP            50m
demo-cp-kafka-rest           ClusterIP   10.0.14.27    &lt;none&gt;        8082/TCP            50m
demo-cp-ksql-server          ClusterIP   10.0.7.150    &lt;none&gt;        8088/TCP            50m
demo-cp-schema-registry      ClusterIP   10.0.7.84     &lt;none&gt;        8081/TCP            50m
demo-cp-zookeeper            ClusterIP   10.0.9.119    &lt;none&gt;        2181/TCP            50m
demo-cp-zookeeper-headless   ClusterIP   None          &lt;none&gt;        2888/TCP,3888/TCP   50m
</code></pre>

<ul>
<li>Create the TCP configmap:</li>
</ul>

<pre><code>$ cat nginx-tcp-configmap.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: kube-system
data:
  31090: "default/demo-cp-kafka-0-nodeport:31090"

$ kubectl apply -f nginx-tcp.configmap.yaml
configmap/tcp-services created
</code></pre>

<ul>
<li>Edit the Nginx Ingress Controller:</li>
</ul>

<pre><code>$ kubectl edit deploy nginx-ingress-controller -n kube-system

$kubectl get deploy nginx-ingress-controller -n kube-system -o yaml
{{{suppressed output}}}
        ports:
        - containerPort: 31090
          hostPort: 31090
          protocol: TCP
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
</code></pre>

<ul>
<li>My ingress is on IP <code>35.226.189.123</code>, now let's try to connect from outside the cluster. For that I'll connect to another VM where I have a minikube, so I can use <code>kafka-client</code> pod to test:</li>
</ul>

<pre><code>user@minikube:~$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
kafka-client   1/1     Running   0          17h

user@minikube:~$ kubectl exec kafka-client -it -- bin/bash

root@kafka-client:/# kafka-console-consumer --bootstrap-server 35.226.189.123:31090 --topic demo-topic --from-beginning --timeout-ms 8000 --max-messages 1
Wed Apr 15 18:19:48 UTC 2020
Processed a total of 1 messages
root@kafka-client:/# 
</code></pre>

<p>As you can see, I was able to access the kafka from outside.</p>

<ul>
<li>If you need external access to Zookeeper as well I'll leave a service model for you:</li>
</ul>

<p><code>zookeeper-external-0.yaml</code></p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    app: cp-zookeeper
    pod: demo-cp-zookeeper-0
  name: demo-cp-zookeeper-0-nodeport
  namespace: default
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: external-broker
    nodePort: 31181
    port: 12181
    protocol: TCP
    targetPort: 31181
  selector:
    app: cp-zookeeper
    statefulset.kubernetes.io/pod-name: demo-cp-zookeeper-0
  sessionAffinity: None
  type: NodePort
</code></pre>

<ul>
<li>It will create a service for it:</li>
</ul>

<pre><code>NAME                           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
demo-cp-zookeeper-0-nodeport   NodePort    10.0.5.67     &lt;none&gt;        12181:31181/TCP     2s
</code></pre>

<ul>
<li>Patch your configmap:</li>
</ul>

<pre><code>data:
  "31090": default/demo-cp-kafka-0-nodeport:31090
  "31181": default/demo-cp-zookeeper-0-nodeport:31181
</code></pre>

<ul>
<li>Add the Ingress rule:</li>
</ul>

<pre><code>        ports:
        - containerPort: 31181
          hostPort: 31181
          protocol: TCP
</code></pre>

<ul>
<li>Test it with your external IP:</li>
</ul>

<pre><code>pod/zookeeper-client created
user@minikube:~$ kubectl exec -it zookeeper-client -- /bin/bash
root@zookeeper-client:/# zookeeper-shell 35.226.189.123:31181
Connecting to 35.226.189.123:31181
Welcome to ZooKeeper!
JLine support is disabled
</code></pre>

<p>If you have any doubts, let me know in the comments!</p>
