<p>I am running my services on EKS clusters. In order to collect the application metrics [API response times, status and number of calls], I came across Prometheus. There are following steps that I think needs to be done:</p>
<ol>
<li>Cluster role, Service account and role binding: this will allow my prometheus service to talk to the cluster nods, pods and services [defined in the resources section].</li>
<li>Configmap: this allows the scraping process and defines different roles.</li>
<li>Service and ingress: to establish the endpoints [e.g.: 9090] and routes the traffic from internet.</li>
</ol>
<p>I came across <a href="https://docs.aws.amazon.com/eks/latest/userguide/prometheus.html" rel="nofollow noreferrer">prometheus using helm</a> which describes how we can make use of helm predefined prometheus charts in order to get the raw metrics from kubernetes.</p>
<p>I followed the steps:</p>
<pre><code>kubectl create namespace prometheus

helm install prometheus stable/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass=&quot;gp2&quot;,server.persistentVolume.storageClass=&quot;gp2&quot;

kubectl get pods -n prometheus
</code></pre>
<p>I can see the pods running with that namespace. Now, I have two questions,</p>
<ol>
<li>I am having multiple services (For example, service A and service B)
running on the cluster. So, how can I distinguish the metrics on
Prometheus.</li>
<li>Do I need to run <code>kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090</code> everytime to see the results? I see
<a href="https://github.com/helm/charts/blob/master/stable/prometheus/templates/server-service.yaml#L37" rel="nofollow noreferrer">targetPort</a> is defined as <code>9090</code> then why do I need to run the
command? Can I just <code>values.yaml</code> instead?</li>
</ol>

<p>Well to answer your questions</p>
<ol>
<li><p>Yes, you can distinguish the service metrics by using label just use like this in your configMap of prometheus</p>
<pre><code> static_configs:
   - targets:
       - &quot;&lt;yourfirstservicename&gt;.&lt;namespace&gt;.svc.cluster.local:&lt;yourservice1portnumber&gt;&quot;
     labels:
       instance: 'service1'
   - targets:
       - &quot;&lt;yourservice2name&gt;.&lt;namespace&gt;.svc.cluster.local:&lt;yourservice2port&gt;&quot;
     labels:
       instance: 'service2'
</code></pre>
</li>
<li><p>Yes you have to do that port-forward but if you are planning to use grafana for visualization then new grafana version provide built in query run functionality.</p>
</li>
</ol>
<p>I hope this will help !!</p>
