<p>I deployed Prometheus <a href="https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting" rel="nofollow noreferrer">with an Helm chart from Rancher</a>. Targets such as Alertmanager, Prometheus, Grafana, Node-exporter, Kubelet etc. are configured automatically. The endpoint from alertmanager refers to the IP address of the specific pod for example. I also configured multiple targets successfully like Jira and Confluence.</p>
<p>Since the service <a href="https://github.com/kubernetes-sigs/external-dns" rel="nofollow noreferrer">external-dns</a> is running in the namespace <code>kube-system</code>, it's also configured automatically. But only this service is getting the error <code>Context deadline exceeded</code>.</p>
<p>I checked in a random pod if those metrics are accessible by running the command <code>curl -s http://&lt;IP-ADDRESS-POD&gt;:7979/metrics</code>. Also did this with the service ip address (<code>kubectl get service external-dns</code> and <code>curl-s http://&lt;IP-ADDRESS-SVC&gt;:7979/metrics</code>).
Both of these curl commands returned the metrics within a second. So increasing the scrape timeout won't help.</p>
<p>But when I <code>exec</code> in the Prometheus container and use the <code>promtool debug metrics</code> command it shows the same behaviour like in my browser. The external-dns returns a timeout with both of the IP addresses and if I try this with another target it just returns the metrics.</p>
<p>I also don't think it's a SSL issue, because I already injected the correct CA bundle for the targets Jira and Confluence.</p>
<p>So anybody an idea? :)</p>

Lucas Scheepers
<p>I had to edit the NetworkPolicy in the <code>kube-system</code> namespace. The containers from the <code>cattle-monitoring-system</code> namespace are now allowed to access the containers from the <code>kube-system</code> namespace. <a href="https://editor.cilium.io" rel="nofollow noreferrer">You can upload your NetworkPolicies here</a> and it visualizes which resources has access or not. The NetworkPolicy looks like this now:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-network-policy
  namespace: kube-system
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: cattle-monitoring-system
  - from:
    - podSelector: {}
  podSelector: {}
  policyTypes:
  - Ingress
</code></pre>
