<p>I installed Prometheus (follow in this link: <a href="https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/" rel="nofollow noreferrer">https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/</a>)</p>

<p>But, when checking status of Targets, it shows "Down" for AlertManager service, every another endpoint are up, please see the attached file <a href="https://i.stack.imgur.com/qT5KF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qT5KF.png" alt="enter image description here"></a></p>

<p>Then, I check Service Discovery, the discovered labels shows:</p>

<blockquote>
  <p>"<strong>address</strong>="192.168.180.254:9093"
  __meta_kubernetes_endpoint_address_target_kind="Pod"
  __meta_kubernetes_endpoint_address_target_name="alertmanager-6c666985cc-54rjm"
  __meta_kubernetes_endpoint_node_name="worker-node1"
  __meta_kubernetes_endpoint_port_protocol="TCP"
  __meta_kubernetes_endpoint_ready="true"
  __meta_kubernetes_endpoints_name="alertmanager"
  __meta_kubernetes_namespace="monitoring"
  __meta_kubernetes_pod_annotation_cni_projectcalico_org_podIP="192.168.180.254/32"
  __meta_kubernetes_pod_annotationpresent_cni_projectcalico_org_podIP="true"
  __meta_kubernetes_pod_container_name="alertmanager"
  __meta_kubernetes_pod_container_port_name="alertmanager"
  __meta_kubernetes_pod_container_port_number="9093""</p>
</blockquote>

<p>But Target Labels show another port (<strong>8080</strong>), I don't know why:</p>

<blockquote>
<pre><code>instance="192.168.180.254:8080"
job="kubernetes-service-endpoints"
kubernetes_name="alertmanager"
kubernetes_namespace="monitoring"
</code></pre>
</blockquote>

<p><a href="https://i.stack.imgur.com/n3iSf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/n3iSf.png" alt="enter image description here"></a></p>

<p>I've reproduced your issue on GCE.</p>

<p>If you are using version <code>1.16+</code> you have probably changed <code>apiVersion</code> as in tutorial you have <code>Deployment</code> in <code>extensions/v1beta1</code>. Since K8s <code>1.16+</code> you need to change it to <code>apiVersion: apps/v1</code>. Otherwise you will get error like:</p>

<pre><code>error: unable to recognize "STDIN": no matches for kind "Deployment" in version "extensions/v1beta1"
</code></pre>

<p>Second thing, in 1.16+ you need to specify <code>selector</code>. If you will not do it you will receive another error:</p>

<pre><code>`error: error validating "STDIN": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false`
</code></pre>

<p>It would look like:</p>

<pre><code>...
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
...
</code></pre>

<p>Regarding port <code>8080</code> please check <a href="https://vitalflux.com/kubernetes-port-targetport-and-nodeport/" rel="nofollow noreferrer">this article</a> with example.</p>

<blockquote>
  <p><strong>Port:</strong> Port is the port number which makes a service visible to
  other services running within the same K8s cluster.  In other words,
  in case a service wants to invoke another service running within the
  same Kubernetes cluster, it will be able to do so using port specified
  against “port” in the service spec file.</p>
</blockquote>

<p>It worked for my environment in GCE. Did you configure firewall for your endpoints?</p>

<p>In addition. In <code>Helm 3</code> some <code>hooks</code> were deprecated. You can find this information <a href="https://github.com/datawire/ambassador/issues/2057#issuecomment-560007192" rel="nofollow noreferrer">here</a>.</p>

<p>If you still have issue please provide your YAMLs witch applied changes to version 1.16+.</p>
