<p>I am new to Kubernetes and I'm working on deploying an application within a new Kubernetes cluster.</p>
<p>Currently, the service running has multiple pods that need to communicate with each other.  I'm looking for a general approach to go about debugging the issue, rather than getting into the specifies of the service as the question will become much too specific.</p>
<p>The pods within the cluster are throwing an error:
<code>err=&quot;Get \&quot;http://testpod.mynamespace.svc.cluster.local:8080/&quot;: dial tcp 10.10.80.100:8080: connect: connection refused&quot;</code>
Both pods are in the same cluster.</p>
<p>What are the best steps to take to debug this?</p>
<p>I have tried running:
<code>kubectl exec -it testpod --namespace mynamespace -- cat /etc/resolv.conf</code>
And this returns:
<code>search mynamespace.svc.cluster.local svc.cluster.local cluster.local us-east-2.compute.internal</code>
Which I found here: <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="noreferrer">https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/</a></p>

fuzzi
<p>First of all, the following pattern:</p>
<pre><code>my-svc.my-namespace.svc.cluster-domain.example
</code></pre>
<p>is applicable only to <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#services" rel="noreferrer">FQDNs of Services</a>, not <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pods" rel="noreferrer">Pods</a> which have the following form:</p>
<pre><code>pod-ip-address.my-namespace.pod.cluster-domain.example
</code></pre>
<p>e.g.:</p>
<pre><code>172-17-0-3.default.pod.cluster.local
</code></pre>
<p>So in fact you're querying cluster dns about FQDN of the <code>Service</code> named <code>testpod</code> and not about FQDN of the <code>Pod</code>. Judging by the fact that it's being resolved successfully, such <code>Service</code> already exists in your cluster but most probably is misconfigured. The fact that you're getting the error message <code>connection refused</code> can mean the following:</p>
<ol>
<li>your <code>Service</code> FQDN <code>testpod.mynamespace.svc.cluster.local</code> has been successfully resolved
(otherwise you would receive something like <code>curl: (6) Could not resolve host: testpod.default.svc.cluster.local</code>)</li>
<li>you've reached successfully your <code>testpod</code> <code>Service</code>
(otherwise, i.e. if it existed but wasn't listening on <code>8080</code> port, you're trying to connect to, you would receive <code>timeout</code> e.g. <code>curl: (7) Failed to connect to testpod.default.svc.cluster.local port 8080: Connection timed out</code>)</li>
<li>you've reached the <code>Pod</code>, exposed by <code>testpod</code> <code>Service</code> (you've been sussessfully redirected to it by the <code>testpod</code> <code>Service</code>)</li>
<li>but once reached the <code>Pod</code>, you're trying to connect to incorect port and that's why the connection is being refused by the server</li>
</ol>
<p>My best guess is that your <code>Pod</code> in fact listens on different port, like <code>80</code> but you exposed it via the <code>ClusterIP</code> <code>Service</code> by specifying only <code>--port</code> value e.g. by:</p>
<pre><code>kubectl expose pod testpod --port=8080
</code></pre>
<p>In such case both <code>--port</code> (port of the <code>Service</code>) and <code>--targetPort</code> (port of the <code>Pod</code>) will have the same value. In other words you've created a <code>Service</code> like the one below:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: testpod
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
</code></pre>
<p>And you probably should've exposed it either this way:</p>
<pre><code>kubectl expose pod testpod --port=8080 --targetPort=80
</code></pre>
<p>or with the following yaml manifest:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: testpod
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
</code></pre>
<p>Of course your <code>targetPort</code> may be different than <code>80</code>, but <code>connection refused</code> in such case can mean only one thing: target http server (running in a <code>Pod</code>) refuses connection to <code>8080</code> port (most probably because it isn't listening on it). You didn't specify what image you're using, whether it's a standard <code>nginx</code> webserver or something based on your custom image. But if it's <code>nginx</code> and wasn't configured differently it listens on port <code>80</code>.</p>
<p>For further debug, you can attach to your <code>Pod</code>:</p>
<pre><code>kubectl exec -it testpod --namespace mynamespace -- /bin/sh
</code></pre>
<p>and if <code>netstat</code> command is not present (the most likely scenario) run:</p>
<pre><code>apt update &amp;&amp; apt install net-tools
</code></pre>
<p>and then check with <code>netstat -ntlp</code> on which port your container listens on.</p>
<p>I hope this helps you solve your issue. In case of any doubts, don't hesitate to ask.</p>
