<p><strong>What I want to achieve:</strong><br />
We have an on premise Kafka cluster. I want to set up KSQLDB in OpenShift and connect it to the brokers of the on premise Kafka cluster.</p>
<p><strong>The problem:</strong><BR />
When I try to start the KSQLDB server with the command &quot;/usr/bin/ksql-server-start /etc/ksqldb/ksql-server.properties&quot; I get the error message:</p>
<pre><code>[2020-05-14 15:47:48,519] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:60)
io.confluent.ksql.util.KsqlServerException: Could not get Kafka cluster configuration!
        at io.confluent.ksql.services.KafkaClusterUtil.getConfig(KafkaClusterUtil.java:90)
        at io.confluent.ksql.security.KsqlAuthorizationValidatorFactory.isKafkaAuthorizerEnabled(KsqlAuthorizationValidatorFactory.java:81)
        at io.confluent.ksql.security.KsqlAuthorizationValidatorFactory.create(KsqlAuthorizationValidatorFactory.java:51)
        at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:624)
        at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:544)
        at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:98)
        at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:56)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1589471268517) timed out at 1589471268518 after 1 attempt(s)
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
        at io.confluent.ksql.services.KafkaClusterUtil.getConfig(KafkaClusterUtil.java:60)
        ... 6 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1589471268517) timed out at 1589471268518 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
</code></pre>
<p><strong>My configuration:</strong><br />
I set up my Dockerfile on the basis of this image: <a href="https://hub.docker.com/r/confluentinc/ksqldb-server" rel="nofollow noreferrer">https://hub.docker.com/r/confluentinc/ksqldb-server</a>, the ports 9092, 9093, 8080, 8082 and 443 are open.</p>
<p>My service-yaml looks like that:</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: social-media-dev
  namespace: abc
  selfLink: xyz
  uid: xyz
  resourceVersion: '1'
  creationTimestamp: '2020-05-14T09:47:15Z'
  labels:
    app: social-media-dev
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
spec:
  ports:
    - name: social-media-dev
      protocol: TCP
      port: 9092
      targetPort: 9092
      nodePort: 31364
  selector:
    app: social-media-dev
    deploymentconfig: social-media-dev
  clusterIP: XX.XX.XXX.XXX
  type: LoadBalancer
  externalIPs:
    - XXX.XX.XXX.XXX
  sessionAffinity: None
  externalTrafficPolicy: Cluster
status:
  loadBalancer:
    ingress:
      - ip: XX.XX.XXX.XXX
</code></pre>
<p>My ksql-server.properties file includes the following information:<br />
listeners: <a href="http://0.0.0.0:8082" rel="nofollow noreferrer">http://0.0.0.0:8082</a><br />
bootstrap.servers: X.X.X.X:9092, X.X.X.Y:9092, X.X.X.Z:9092</p>
<p><strong>What I have tried so far:</strong><br /></p>
<p>I tried to connect from within my pod to a broker and it worked: <br /><code>(timeout 1 bash -c '&lt;/dev/tcp/X.X.X.X/9092 &amp;&amp; echo PORT OPEN || echo PORT CLOSED') 2&gt;/dev/null</code><br />
result: PORT OPEN</p>
<p>I also played around with the listener but then the error message got shorter just with the information &quot;Could not get Kafka cluster configuration!&quot; and without the timeout error.</p>
<p>I tried to exchange LoadBalancer to Nodeport, but also without success.</p>
<p>Do you have any ideas what I could try next?</p>
<p>UPDATE: With an upgrade to Cloudera CDH6, the Cloudera Kafka cluster works now also with Kafka Streams. Hence I was able to connect from my KSQLDB Cluster in Openshift to the on-premise Kafka cluster now.</p>

<p>UPDATE:
With an upgrade to Cloudera CDH6, the Cloudera Kafka cluster works now also with Kafka Streams. Hence I was able to connect from my KSQLDB Cluster in Openshift to the on-premise Kafka cluster now.</p>
<p>I will also describe my final way of connecting to the kerberized Kafka-cluster here as I have been struggling a lot to get it running:</p>
<ol>
<li>Getting Kerberos-tickets and establish connections via SSL</li>
</ol>
<p>ksql-server.properties (the sasl_ssl part of it):</p>
<pre><code>security.protocol=SASL_SSL
sasl.mechanism=GSSAPI

ssl.truststore.location=truststore.jks
ssl.truststore.password=password
ssl.truststore.type=JKS

ssl.ca.location=cert

sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=&quot;my.keytab&quot; serviceName=&quot;kafka&quot;  principal=&quot;myprincipal&quot;;
serviceName=&quot;kafka&quot;

producer.ssl.endpoint.identification.algorithm=HTTPS
producer.security.protocol=SASL_SSL
producer.ssl.truststore.location=truststore.jks
producer.ssl.truststore.password=password
producer.sasl.mechanism=GSSAPI
producer.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=&quot;my.keytab&quot; serviceName=&quot;kafka&quot;  principal=&quot;myprincipal&quot;;

consumer.ssl.endpoint.identification.algorithm=HTTPS
consumer.security.protocol=SASL_SSL
consumer.ssl.truststore.location=truststore.jks
consumer.ssl.truststore.password=password
consumer.sasl.mechanism=GSSAPI
consumer.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=&quot;my.keytab&quot; serviceName=&quot;kafka&quot;  principal=&quot;myprincipal&quot;;`
</code></pre>
<ol start="2">
<li>Set up Sentry rules therefore</li>
</ol>
<p><code>HOST=[HOST]-&gt;CLUSTER=kafka-cluster-&gt;action=idempotentwrite</code></p>
<p><code>HOST=[HOST]-&gt;TRANSACTIONALID=[ID]-&gt;action=describe</code></p>
<p><code>HOST=[HOST]-&gt;TRANSACTIONALID=[ID]-&gt;action=write</code></p>
