<p>I have an application that relies on a kafka service.</p>
<p>With Kafka connect, I'm getting an error when trying to <code>curl localhost:8083</code>, on the Linux VM that's running the kubernetes pod for Kafka connect.</p>
<p><code>curl -v localhost:8083</code> gives:</p>
<ul>
<li>Rebuilt URL to: localhost:8083/</li>
<li>Trying 127.0.0.1...</li>
<li>connect to 127.0.0.1 port 8083 failed: Connection refused</li>
<li>Failed to connect to localhost port 8083: Connection refused</li>
<li>Closing connection 0
curl: (7) Failed to connect to localhost port 8083: Connection refused</li>
</ul>
<p><code>kubectl get po -o wide</code> for my kubernetes namespace gives:</p>
<p><a href="https://i.stack.imgur.com/5Wj11.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5Wj11.png" alt="enter image description here" /></a></p>
<p>When I check open ports using <code>sudo lsof -i -P -n | grep LISTEN</code> I don't see 8083 listed. The kafka connect pod is running and there's nothing suspicious in the logs for the pod.</p>
<p>There's a kubernetes manifest that I think was probably used to set up the Kafka connect service, these are the relevant parts. I'd really appreciate any advice about how to figure out why I can't curl localhost:8083</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
   name: kafka-connect
   namespace: my-namespace
spec:

   ...

   template:
     metadata:
       labels:
         app: connect
     spec:
        containers:
          - name: kafka-connect
            image: confluentinc/cp-kafka-connect:3.0.1
            ports:
              - containerPort: 8083

            env:
              
              - name: CONNECT_REST_PORT
                value: &quot;8083&quot;
              - name: CONNECT_REST_ADVERTISED_HOST_NAME
                value: &quot;kafka-connect&quot;

        volumes:
          - name: connect-plugins
            persistentVolumeClaim:
                claimName: pvc-connect-plugin
          - name: connect-helpers          
            secret:
              secretName: my-kafka-connect-config
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-connect
  namespace: my-namespace
  labels:
    app: connect  
spec:
  ports:
    - port: 8083
  selector:
    app: connect
</code></pre>

<p>You can't connect to a service running inside your cluster, from outside your cluster, without a little bit of tinkering.</p>
<p>You have three possible solutions:</p>
<ol>
<li><p>Use a service with type <code>NodePort</code> or <code>LoadBalancer</code> to make the service reachable outside the cluster.</p>
<p>See the <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="nofollow noreferrer">services</a> and <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#expose" rel="nofollow noreferrer">kubectl expose</a> documentation.<br />
<strong>Be aware, depending on your environment, this may expose the service to the internet.</strong></p>
</li>
<li><p>Access using <em>Proxy Verb:</em> (see <a href="https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/#manually-constructing-apiserver-proxy-urls" rel="nofollow noreferrer">here</a>)</p>
<p>This only works for HTTP/HTTPS. Use this if your service is not secure
enough to be exposed to the internet.</p>
</li>
<li><p>Access from pod running inside your cluster.</p>
<p>As you have noticed in the comments, you can <code>curl</code> from inside the pod. You can also do this from any other pod running in the same cluster. Pods can communicate with each other without any additional configuration.</p>
</li>
</ol>
<hr />
<blockquote>
<p>Why can I not curl 8083 when I ssh onto the VM?</p>
</blockquote>
<p>Pods/services are not reachable from outside the cluster, if not exposed using aforementioned methods (point 1 or 2).</p>
<blockquote>
<p>Why isn't the port exposed on the host VM that has the pods?</p>
</blockquote>
<p>It's not exposed on your VM, it's exposed inside your cluster.</p>
<hr />
<p>I would strongly recommend going through <a href="https://kubernetes.io/docs/concepts/cluster-administration/networking/" rel="nofollow noreferrer">Cluster Networking</a> documentation to learn more.</p>
