<p>I have installed the kube prometheus stack helm chart. The chart appears to install on my 3 node Kubernetes cluster (1.27.3) without issues. All the deployments and pods seems to enter a ready state. I port forwarded the prometheus pod and attempted to connect but was not able to.</p>
<p>When I review the logs from the prometheus pod it seems that there is an error or warning that something is wrong with the rule set it generated:</p>
<pre><code>ts=2023-07-15T22:26:26.110Z caller=manager.go:663 level=warn component=&quot;rule manager&quot; file=/etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0/default-prometheus-kube-prometheus-kubernetes-system-kubelet-e76e1f61-e704-4f1c-a9f8-87d91012dd7c.yaml group=kubernetes-system-kubelet name=KubeletPodStartUpLatencyHigh index=5 msg=&quot;Evaluating rule failed&quot; rule=&quot;alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by (cluster, instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}[5m])))\n  * on (cluster, instance) group_left (node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}\n  &gt; 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n    on node {{ $labels.node }}.\n  runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeletpodstartuplatencyhigh\n  summary: Kubelet Pod startup latency is too high.\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;192.168.2.10:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;192.168.2.10:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;kubenode03\&quot;, service=\&quot;prometheus-prime-kube-prom-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;192.168.2.10:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;kubenode03\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side
</code></pre>
<p>I am a newbie at prometheus so am unsure what to look for or how to troubleshoot this. Can someone help me understand:</p>
<ol>
<li>What is this error complaining about?</li>
<li>How would I fix this error?</li>
</ol>
<p>It appears that it is something in the cube-system namespace but I only have 3 pods in that namespace all with unique names:</p>
<p>CoreDNS
Local Path Provisioner
metrics-server</p>
<p>I appreciate any help or advice on how to troubleshoot this.</p>

Edward S.
<blockquote>
<p>What is this error complaining about?</p>
</blockquote>
<p>Problem lies within your expression:</p>
<pre><code>histogram_quantile(0.99, 
 sum by (cluster, instance, le) (
  rate(
   kubelet_pod_worker_duration_seconds_bucket{job=&quot;kubelet&quot;,metrics_path=&quot;/metrics&quot;}
   [5m])))
* on (cluster, instance) group_left (node) kubelet_node_name{job=&quot;kubelet&quot;,metrics_path=&quot;/metrics&quot;}
&gt; 60
</code></pre>
<p>It seems like there are multiple metrics on the both sides of <code>on</code> with the same values of labels <code>cluster</code> and <code>instance</code>. It constitutes many-to-many relation, and Prometheus doesn't allow those.</p>
<blockquote>
<p>How would I fix this error?</p>
</blockquote>
<p>Go to Prometheus' web ui, execute both parts of you on clause. You'll see labelsets produced by them. Correct your on clause, so that no many-to-many relations occur.</p>
<p>I'm not familiar with this exact metrics, so it's hard to say more specific what to look on.</p>
