<p>Hy there,</p>

<p>I'm trying to configure Kubernetes Cronjobs monitoring &amp; alerts with Prometheus. I found this helpful <a href="https://medium.com/@tristan_96324/prometheus-k8s-cronjob-alerts-94bee7b90511" rel="nofollow noreferrer">guide</a></p>

<p>But I always get a <strong>many-to-many matching not allowed: matching labels must be unique on one side</strong> error. </p>

<p>For example, this is the PromQL query which triggers this error:</p>

<p><code>max(
            kube_job_status_start_time
            * ON(job_name) GROUP_RIGHT()
            kube_job_labels{label_cronjob!=""}
          ) BY (job_name, label_cronjob)
</code></p>

<p>The queries by itself result in e.g. these metrics</p>

<p><strong>kube_job_status_start_time</strong>:
<code>
kube_job_status_start_time{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",namespace="test-develop",release="kube-state-metrics"}
</code></p>

<p><strong>kube_job_labels{label_cronjob!=""}</strong>:
<code>
kube_job_labels{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",label_cronjob="test",label_environment="test-develop",namespace="test-develop",release="kube-state-metrics"}
</code></p>

<p>Is there something I'm missing here? The same many-to-many error happens for every query I tried from the guide.
Even constructing it by myself from ground up resulted in the same error.
Hope you can help me out here :)</p>

<blockquote>
  <p>In my case I don't get this extra label from Prometheus when installed via helm (stable/prometheus-operator).</p>
</blockquote>

<p>You need to configure it in Prometheus. It calls: honor_labels: false</p>

<pre><code># If honor_labels is set to "false", label conflicts are resolved by renaming
# conflicting labels in the scraped data to "exported_&lt;original-label&gt;" (for
# example "exported_instance", "exported_job") and then attaching server-side
# labels.
</code></pre>

<p>So you have to configure your prometheus.yaml file - config with option honor_labels: false</p>

<pre><code># Setting honor_labels to "true" is useful for use cases such as federation and
# scraping the Pushgateway, where all labels specified in the target should be
# preserved
</code></pre>

<p>Anyway if I have it like this (I have now exported_jobs), still can't do proper query, but I guess is still because of my LHS.</p>

<pre><code>Error executing query: found duplicate series for the match group 
{exported_job="kube-state-metrics"} on the left hand-side of the operation: 
[{__name__=
</code></pre>
