<p>I installed Prometheus (follow in this link: <a href="https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/" rel="nofollow noreferrer">https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/</a>)</p>

<p>But, when checking status of Targets, it shows "Down" for AlertManager service, every another endpoint are up, please see the attached file <a href="https://i.stack.imgur.com/qT5KF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qT5KF.png" alt="enter image description here"></a></p>

<p>Then, I check Service Discovery, the discovered labels shows:</p>

<blockquote>
  <p>"<strong>address</strong>="192.168.180.254:9093"
  __meta_kubernetes_endpoint_address_target_kind="Pod"
  __meta_kubernetes_endpoint_address_target_name="alertmanager-6c666985cc-54rjm"
  __meta_kubernetes_endpoint_node_name="worker-node1"
  __meta_kubernetes_endpoint_port_protocol="TCP"
  __meta_kubernetes_endpoint_ready="true"
  __meta_kubernetes_endpoints_name="alertmanager"
  __meta_kubernetes_namespace="monitoring"
  __meta_kubernetes_pod_annotation_cni_projectcalico_org_podIP="192.168.180.254/32"
  __meta_kubernetes_pod_annotationpresent_cni_projectcalico_org_podIP="true"
  __meta_kubernetes_pod_container_name="alertmanager"
  __meta_kubernetes_pod_container_port_name="alertmanager"
  __meta_kubernetes_pod_container_port_number="9093""</p>
</blockquote>

<p>But Target Labels show another port (<strong>8080</strong>), I don't know why:</p>

<blockquote>
<pre><code>instance="192.168.180.254:8080"
job="kubernetes-service-endpoints"
kubernetes_name="alertmanager"
kubernetes_namespace="monitoring"
</code></pre>
</blockquote>

<p><a href="https://i.stack.imgur.com/n3iSf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/n3iSf.png" alt="enter image description here"></a></p>

<p>First, if you want to install prometheus and grafana without getting sick, you need to do it though helm.</p>

<p><a href="https://helm.sh/docs/intro/install/" rel="nofollow noreferrer">First install helm</a></p>

<p>And then</p>

<pre class="lang-sh prettyprint-override"><code>helm install installationWhatEverName stable/prometheus-operator
</code></pre>
