<p>I have two Kubernetes clusters representing dev and staging environments.</p>

<p>Separately, I am also deploying a custom DevOps dashboard which will be used to monitor these two clusters. On this dashboard I will need to show information such as:</p>

<ul>
<li>RAM/HD Space/CPU usage of each deployed Pod in each environment</li>
<li>Pod health (as in if it has too many container restarts etc)</li>
<li>Pod uptime</li>
</ul>

<p>All these stats have to be at a cluster level and also per namespace, preferably. As in, if I query a for a particular namespace, I have to get all the resource usages of that namespace.</p>

<p>So the webservice layer of my dashboard will send a service request to the master node of my respective cluster in order to fetch this information.</p>

<p>Another thing I need is to implement real time notifications in my DevOps dashboard. Every time a container  fails, I need to catch that event and notify relevant personnel.</p>

<p>I have been reading around and two things that pop up a lot are Prometheus and Metric Server. Do I need both or will one do? I set up Prometheus on a local cluster but I can't find any endpoints it exposes which could be called by my dashboard service. I'm also trying to set up Prometheus AlertManager but so far it hasn't worked as expected. Trying to fix it now. Just wanted to check if these technologies have the capabilities to meet my requirements.</p>

<p>Thanks!</p>

<p><a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a> + <a href="https://grafana.com/" rel="nofollow noreferrer">Grafana</a> are a pretty standard setup. </p>

<p>Installing <a href="https://github.com/coreos/kube-prometheus" rel="nofollow noreferrer">kube-prometheus</a> or <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">prometheus-operator</a> via <a href="https://helm.sh/" rel="nofollow noreferrer">helm</a> will give you 
 Grafana, Alertmanager, <code>node-exporter</code> and <code>kube-state-metrics</code> by default and all be setup for kubernetes metrics. </p>

<p>Configure alertmanager to <a href="https://prometheus.io/docs/alerting/configuration/" rel="nofollow noreferrer">do something with the alerts</a>. SMTP is usually the first thing setup but I would recommend some sort of event manager if this is a service people need to rely on. </p>

<p>Although a dashboard isn't part of your requirements, this will inform how you can connect into prometheus as a data source. <a href="https://prometheus.io/docs/visualization/grafana/#creating-a-prometheus-data-source" rel="nofollow noreferrer">There is docco on adding prometheus data source for grafana</a>.</p>

<p>There are a number of <a href="https://grafana.com/grafana/dashboards?dataSource=prometheus&amp;page=1&amp;search=kubernetes" rel="nofollow noreferrer">prebuilt charts available to add to Grafana</a>. There are some <a href="https://grafana.com/grafana/dashboards/9578" rel="nofollow noreferrer">charts to visualise alertmanager</a> too.</p>

<p>Your external service won't be querying the metrics directly with prometheus, in will be querying the collected data in prometheus stored inside your cluster. To access the API externally you will need to setup an external path to the prometheus service. This can be configured via an ingress controller in the helm deployment:</p>

<pre><code>prometheus.ingress.enabled: true
</code></pre>

<p>You can do the same for the alertmanager API and grafana if needed. </p>

<pre><code>alertmanager.ingress.enabled: true
grafana.ingress.enabled: true
</code></pre>

<p>You could use Grafana outside the cluster as your dashboard via the same prometheus ingress if it proves useful.</p>
