<p>I recently learned about <code>helm</code> and how easy it is to deploy the whole <code>prometheus</code> stack for monitoring a Kubernetes cluster, so I decided to try it out on a staging cluster at my work.</p>
<p>I started by creating a dedicates namespace on the cluster for monitoring with:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl create namespace monitoring
</code></pre>
<p>Then, with <code>helm</code>, I added the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">prometheus-community repo</a> with:</p>
<pre class="lang-sh prettyprint-override"><code>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
</code></pre>
<p>Next, I installed the chart with a <code>prometheus</code> release name:</p>
<pre class="lang-sh prettyprint-override"><code>helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring
</code></pre>
<p>At this time I didn't pass any custom configuration because I'm still trying it out.</p>
<p>After the install is finished, it all looks good. I can access the prometheus dashboard with:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090 -n monitoring
</code></pre>
<p>There, I see a bunch of pre-defined alerts and rules that are monitoring but the problem is that I don't quite understand how to create new rules to check the pods in the <code>default</code> namespace, where I actually have my services deployed.</p>
<p>I am looking at <code>http://localhost:9090/graph</code> to play around with the queries and I can't seem to use any that will give me metrics on my pods in the <code>default</code> namespace.</p>
<p>I am a bit overwhelmed with the amount of information so I would like to know what did I miss or what am I doing wrong here?</p>

everspader
<p>The Prometheus Operator includes several Custom Resource Definitions (CRDs) including <code>ServiceMonitor</code> (and <code>PodMonitor</code>). <code>ServiceMonitor</code>'s are used to define services to the Operator to be monitored.</p>
<p>I'm familiar with the Operator although not the Helm deployment but I suspect you'll want to create <code>ServiceMonitors</code> to generate metrics for your apps in any (including <code>default</code>) namespace.</p>
<p>See: <a href="https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions" rel="nofollow noreferrer">https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions</a></p>
