<p>Hy there,</p>

<p>I'm trying to configure Kubernetes Cronjobs monitoring &amp; alerts with Prometheus. I found this helpful <a href="https://medium.com/@tristan_96324/prometheus-k8s-cronjob-alerts-94bee7b90511" rel="nofollow noreferrer">guide</a></p>

<p>But I always get a <strong>many-to-many matching not allowed: matching labels must be unique on one side</strong> error. </p>

<p>For example, this is the PromQL query which triggers this error:</p>

<p><code>max(
            kube_job_status_start_time
            * ON(job_name) GROUP_RIGHT()
            kube_job_labels{label_cronjob!=""}
          ) BY (job_name, label_cronjob)
</code></p>

<p>The queries by itself result in e.g. these metrics</p>

<p><strong>kube_job_status_start_time</strong>:
<code>
kube_job_status_start_time{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",namespace="test-develop",release="kube-state-metrics"}
</code></p>

<p><strong>kube_job_labels{label_cronjob!=""}</strong>:
<code>
kube_job_labels{app="kube-state-metrics",chart="kube-state-metrics-0.12.1",heritage="Tiller",instance="REDACTED",job="kubernetes-service-endpoints",job_name="test-1546295400",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",kubernetes_node="REDACTED",label_cronjob="test",label_environment="test-develop",namespace="test-develop",release="kube-state-metrics"}
</code></p>

<p>Is there something I'm missing here? The same many-to-many error happens for every query I tried from the guide.
Even constructing it by myself from ground up resulted in the same error.
Hope you can help me out here :)</p>

hajowieland
<p>I ran into the same issue when I followed that article, but for me, I actually get duplicate job names but in different namespaces.</p>
<p>Ex. When running <code>kube_job_status_start_time</code>:</p>
<pre><code>kube_job_status_start_time{instance=&quot;REDACTED&quot;,job=&quot;kube-state-metrics&quot;,job_name=&quot;job-abc-123&quot;,namespace=&quot;us&quot;}
kube_job_status_start_time{instance=&quot;REDACTED&quot;,job=&quot;kube-state-metrics&quot;,job_name=&quot;job-abc-123&quot;,namespace=&quot;ca&quot;}
</code></pre>
<p>So I had to either add a filter for the namespace or add namespace into the ON/BY clauses to get it to be unique.</p>
<p>e.g. for one of the subqueries I had to do this:</p>
<pre><code>   max(
      kube_job_status_start_time
      * ON(namespace, job_name) GROUP_RIGHT()
      kube_job_labels{label_cronjob!=&quot;&quot;}
    ) BY (namespace, label_cronjob)

</code></pre>
<p>Essentially had to apply that principle to all the rest of the queries for it to work for me. Not sure if that applies in your case.</p>
