<p>I'm trying to report custom metrics to Prometheus by exposing an http &quot;metrics&quot; service (running on the same pod as my main service) as a k8s endpoint. But connection attempts from the prometheus' pod to the my metrics endpoint are refused (even though I can reach my main service from the prometheus pod using wget :8010). It seems I've exposed the main service port, but something is blocking traffic to my metrics port on the same pod? HELP!</p>
<pre><code>kubectl get svc mysvc
NAME       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                         AGE
mysvc      LoadBalancer   10.106.36.79   localhost     8767:31285/TCP,8010:30953/TCP   3m23s
</code></pre>
<pre><code>kubectl describe ep mysvc
Name:         mysvc
Namespace:    default
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-08-06T22:37:54Z
Subsets:
  Addresses:          10.1.18.170
  NotReadyAddresses:  &lt;none&gt;
  Ports:
    Name      Port  Protocol
    ----      ----  --------
    metrics   8767  TCP
    mysvcport 8010  TCP

Events:  &lt;none&gt;
</code></pre>
<p>Prometheus attempts to fetch metrics from the &quot;metrics&quot; endpoint, but reports:
&quot;Get &quot;http://10.1.18.170:8767/metrics&quot;: dial tcp 10.1.18.170:8767: connect: connection refused&quot;</p>
<p>I can confirm mysvc:8767 is not accessible from the prometheus pod, but mysvc:8010 is!</p>
<p>On mysvc's pod, I can reach my metrics service via localhost:8767 but not via mysvc:8767.</p>

Rod LN
<p>In that case, port 8767 is only exposed on the Pod's localhost interface (127.0.0.1) but not on the Pod's public network interface.</p>
<p>You can verify this by doing an <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec" rel="nofollow noreferrer">exec</a> into the Pod and running something like:</p>
<pre class="lang-sh prettyprint-override"><code>netstat -tulpn
</code></pre>
<p>If it says <code>127.0.0.1:8767</code>, the port is only exposed on the localhost interface and not accessible from outside the Pod.</p>
<p>To change this, you have to make sure in the code of your metrics container that the port is exposed as <code>0.0.0.0:8767</code> or <code>:8767</code> or a similar notation that exposes the port on the Pod's public network interface.</p>
