<p>I have two Kubernetes clusters representing dev and staging environments.</p>

<p>Separately, I am also deploying a custom DevOps dashboard which will be used to monitor these two clusters. On this dashboard I will need to show information such as:</p>

<ul>
<li>RAM/HD Space/CPU usage of each deployed Pod in each environment</li>
<li>Pod health (as in if it has too many container restarts etc)</li>
<li>Pod uptime</li>
</ul>

<p>All these stats have to be at a cluster level and also per namespace, preferably. As in, if I query a for a particular namespace, I have to get all the resource usages of that namespace.</p>

<p>So the webservice layer of my dashboard will send a service request to the master node of my respective cluster in order to fetch this information.</p>

<p>Another thing I need is to implement real time notifications in my DevOps dashboard. Every time a container  fails, I need to catch that event and notify relevant personnel.</p>

<p>I have been reading around and two things that pop up a lot are Prometheus and Metric Server. Do I need both or will one do? I set up Prometheus on a local cluster but I can't find any endpoints it exposes which could be called by my dashboard service. I'm also trying to set up Prometheus AlertManager but so far it hasn't worked as expected. Trying to fix it now. Just wanted to check if these technologies have the capabilities to meet my requirements.</p>

<p>Thanks!</p>

<p>I don't know why you are considering your own custom monitoring system. <a href="https://github.com/coreos/prometheus-operator" rel="nofollow noreferrer">Prometheus operator</a> provides all the functionality that you mentioned.
You will end up only with your own grafana dashboard with all required information. </p>

<p>If you need custom notification you can set it up in <a href="https://prometheus.io/docs/alerting/configuration/#configuration" rel="nofollow noreferrer">Alertmanager</a> creating correct <code>prometheusrules.monitoring.coreos.com</code>, you can find a lot of preconfigured prometheusrules in <a href="https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/runbook.md#group-name-kubernetes-apps" rel="nofollow noreferrer">kubernetes-mixin
</a>.
Using labels and namespaces in Alertmanager you can setup a correct route to notify person responsible for a given deployment.</p>

<p><code>Do I need both or will one do?</code>, yes, you need both - <code>Prometheus</code> collects and aggregates metric when <code>Metrick server</code> exposes metrics from your cluster node for your Prometheus to scrape it.</p>

<p>If you have problems with Prometheus, Alertmanger and so on consider using <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">helm chart</a> as entrypoint.</p>
