<p>to start with - I am a sort of newbie to Kubernetes and I might omit some fundamentals.</p>
<p>I have a working containerized app that is orchestrated with docker-compose (and works alright) and I am rewriting it to deploy into Kubernetes. I've converted it to K8s .yaml files via Kompose and modified it to some degree. I am struggling to set up a connection between a Python app and Kafka that are running on separate pods. The Python app constantly returns NoBrokersAvailable() error no matter what I try to apply - it's quite obvious that it cannot connect to a broker. What am I missing? I've defined proper listeners and network policy. I am running it locally on Minikube with local Docker images registry.</p>
<p>The Python app connects to the following address:
<code>KafkaProducer(bootstrap_servers='kafka-service.default.svc.cluster.local:9092')</code></p>
<p>kafka-deployment.yaml (the Dockerfile image is based on confluentinc/cp-kafka:6.2.0 with a topics setup script added to it):</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: C:\ProgramData\chocolatey\lib\kubernetes-kompose\tools\kompose.exe convert
    kompose.version: 1.27.0 (b0ed6a2c9)
  creationTimestamp: null
  labels:
    io.kompose.service: kafka
  name: kafka-app
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: kafka
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: C:\ProgramData\chocolatey\lib\kubernetes-kompose\tools\kompose.exe convert
        kompose.version: 1.27.0 (b0ed6a2c9)
      creationTimestamp: null
      labels:
        io.kompose.network/pipeline-network: &quot;true&quot;
        io.kompose.service: kafka
    spec:
      containers:
        - env:
            - name: KAFKA_LISTENERS
              value: &quot;LISTENER_INTERNAL://0.0.0.0:29092,LISTENER_EXTERNAL://0.0.0.0:9092&quot;
            - name: KAFKA_ADVERTISED_LISTENERS
              value: &quot;LISTENER_INTERNAL://localhost:29092,LISTENER_EXTERNAL://kafka-service.default.svc.cluster.local:9092&quot;
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: &quot;LISTENER_EXTERNAL:PLAINTEXT,LISTENER_INTERNAL:PLAINTEXT&quot;
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: &quot;LISTENER_INTERNAL&quot;
            - name: KAFKA_BROKER_ID
              value: &quot;1&quot;
            - name: KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR
              value: &quot;1&quot;
            - name: KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR
              value: &quot;1&quot;
            - name: KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS
              value: &quot;0&quot;
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: &quot;1&quot;
            - name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
              value: &quot;1&quot;
            - name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
              value: &quot;1&quot;
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: zookeeper:2181
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          image: finnhub-streaming-data-pipeline-kafka:latest
          imagePullPolicy: Never
          lifecycle:
            postStart:
              exec: 
                command: [&quot;/bin/sh&quot;,&quot;-c&quot;,&quot;/kafka-setup-k8s.sh&quot;]
          name: kafka-app
          ports:
            - containerPort: 9092
            - containerPort: 29092
          resources: {}
      restartPolicy: Always

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
spec:
  selector:
    app: kafka
  ports:
    - protocol: TCP
      name: firstport
      port: 9092
      targetPort: 9092
    - protocol: TCP
      name: secondport
      port: 29092
      targetPort: 29092
</code></pre>
<p>finnhub-producer.yaml (aka my Python app deployment):</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: C:\ProgramData\chocolatey\lib\kubernetes-kompose\tools\kompose.exe convert
    kompose.version: 1.27.0 (b0ed6a2c9)
  creationTimestamp: null
  labels:
    io.kompose.service: finnhubproducer
  name: finnhubproducer
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: finnhubproducer
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: C:\ProgramData\chocolatey\lib\kubernetes-kompose\tools\kompose.exe convert
        kompose.version: 1.27.0 (b0ed6a2c9)
      creationTimestamp: null
      labels:
        io.kompose.network/pipeline-network: &quot;true&quot;
        io.kompose.service: finnhubproducer
    spec:
      containers:
        - env:
            - name: KAFKA_PORT
              value: &quot;9092&quot;
            - name: KAFKA_SERVER
              value: kafka-service.default.svc.cluster.local
            - name: KAFKA_TOPIC_NAME
              value: market
          image: docker.io/library/finnhub-streaming-data-pipeline-finnhubproducer:latest
          imagePullPolicy: Never
          name: finnhubproducer
          ports:
            - containerPort: 8001
          resources: {}
      restartPolicy: Always
status: {}

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: C:\ProgramData\chocolatey\lib\kubernetes-kompose\tools\kompose.exe convert
    kompose.version: 1.27.0 (b0ed6a2c9)
  creationTimestamp: null
  labels:
    io.kompose.service: finnhubproducer
  name: finnhubproducer
spec:
  ports:
    - name: &quot;8001&quot;
      port: 8001
      targetPort: 8001
  selector:
    io.kompose.service: finnhubproducer
status:
  loadBalancer: {}
</code></pre>
<p>pipeline-network-networkpolicy.yaml:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  creationTimestamp: null
  name: pipeline-network
spec:
  ingress:
    - from:
        - podSelector:
            matchLabels:
              io.kompose.network/pipeline-network: &quot;true&quot;
  podSelector:
    matchLabels:
      io.kompose.network/pipeline-network: &quot;true&quot;
</code></pre>
<p>EDIT:
Dockerfile for Kafka image:</p>
<pre><code>FROM confluentinc/cp-kafka:6.2.0

COPY ./scripts/kafka-setup-k8s.sh /kafka-setup-k8s.sh
</code></pre>
<p>kafka-setup-k8s.sh:</p>
<pre><code>
# blocks until kafka is reachable
kafka-topics --bootstrap-server localhost:29092 --list

echo -e 'Creating kafka topics'
kafka-topics --bootstrap-server localhost:29092 --create --if-not-exists --topic market --replication-factor 1 --partitions 1

echo -e 'Successfully created the following topics:'
kafka-topics --bootstrap-server localhost:29092 --list
</code></pre>

RSK RSK
<p>I have managed to make it work by deleting services from deployment and running <code>kubectl expose deployment kafka-app</code>. The issue comes from Kompose labeling.</p>
