<p>I am trying to set up a k8s monitoring stack for my on-premises cluster. What I want to set up is:</p>

<ul>
<li>Prometheus</li>
<li>Grafana</li>
<li>Kube-state-metrics</li>
<li>Alertmanager</li>
<li>Loki</li>
</ul>

<p>I can find a lot of resources to do that like:</p>

<ul>
<li><p>This configures the monitoring stack except Loki using their own CRD files:
<a href="https://medium.com/faun/production-grade-kubernetes-monitoring-using-prometheus-78144b835b60" rel="nofollow noreferrer">https://medium.com/faun/production-grade-kubernetes-monitoring-using-prometheus-78144b835b60</a></p></li>
<li><p>Configure Prometheus and Grafana in different namespaces using separate helm charts:
<a href="https://github.com/helm/charts/tree/master/stable/prometheus" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/prometheus</a>
<a href="https://github.com/helm/charts/tree/master/stable/grafana" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/grafana</a></p></li>
<li>Configure Prometheus-operator helm chart into a single namespace:
<a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/prometheus-operator</a></li>
</ul>

<p>I have doubts regarding the configuration of the alert notifications.</p>

<ul>
<li><p>All three setups mentioned above have Grafana UI. So, there is an option to configure alert rules and notification channels via that UI.</p></li>
<li><p>But in the first option, Prometheus rules are configured with Prometheus setup and notification channels are configured with the alert-manager setup using configMap CRDs.</p></li>
</ul>

<p><strong>Which is the better configuration option?</strong></p>

<p>What is the difference in setting up alerts via Grafana UI &amp; Prometheus rules and channels via such configMap CRDs?</p>

<p>What are the advantages and disadvantages of both methods?</p>

AnjK
<p>I chose the third option to setup prometheus-operator in a namespace. Because this chart configures prometheus, grafana, and alertmanager. Prometheus is added as a datasource in grafana by default. It allows adding additional alert rules for Prometheus , datasources, and dashboards for grafana using chart's values file.</p>

<p>Then Configured Loki in the same namespace and added it as a datasource in grafana.
Also configured a webhook to redirect the notifications from alertmanager to MS teams.</p>
