<p>I have a golang webapp pod running in kubernetes cluster, and I tried to deploy a prometheus pod to monitor the golang webapp pod.</p>
<p>I specified <code>prometheus.io/port: to 2112</code> in the service.yaml file, which is the port that the golang webapp is listening on, but when I go to the Prometheus dashboard, it says that the <code>2112</code> endpoint is down.</p>
<p>I'm following <a href="https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/" rel="nofollow noreferrer">this guide</a>, tried this thread's solution <a href="https://stackoverflow.com/questions/53365191/monitor-custom-kubernetes-pod-metrics-using-prometheus">thread</a>, but still getting result saying <code>2112</code> endpoint is down.</p>
<p>Below is the my service.yaml and deployment.yaml</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path: '/metrics'
      prometheus.io/port:   '2112'
spec:
  selector: 
    app: prometheus-server
  type: NodePort  
  ports:
    - port: 8080
      targetPort: 9090 
      nodePort: 30000
---
apiVersion: v1
kind: Service
metadata: 
  namespace: monitoring
  name: goapp
spec:
  type: NodePort
  selector:
    app: golang
  ports:
    - name: main
      protocol: TCP
      port: 80
      targetPort: 2112
      nodePort: 30001






apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
  labels:
    app: prometheus-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          args:
            - &quot;--config.file=/etc/prometheus/prometheus.yml&quot;
            - &quot;--storage.tsdb.path=/prometheus/&quot;
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf
  
        - name: prometheus-storage-volume
          emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: monitoring
  name: golang
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: golang
    spec:
      containers:
        - name: gogo
          image: effy77/gogo2
          ports: 
            - containerPort: 2112
  selector:
    matchLabels:
      app: golang
</code></pre>
<p>I will try add <code>prometheus.io/port: 2112</code> to the prometheus deployment part, as I suspect that might be the cause.</p>

<p>I was confused with where to put the annotations,got my clarifications from this <a href="https://stackoverflow.com/questions/57957614/prometheus-only-scrapes-one-pod">thread</a>, I needed to put it under the service's metadata that needs to be scraped by prothemeus. So in my case it needs to be in goapp's metadata.</p>
<pre><code>apiVersion: v1
kind: Service
metadata: 
  namespace: monitoring
  name: goapp
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path: '/metrics'
      prometheus.io/port:   '2112'
</code></pre>
