<p>I got this alert:</p>
<pre><code>Alert:  PrometheusRuleFailures  - critical Description:  Prometheus monitoring/prometheus-prometheus-kube-prometheus-prometheus-0 has failed to evaluate 30 rules in the last 5m. Details:
  • alertname: PrometheusRuleFailures
  • container: prometheus
  • endpoint: web
  • instance: 10.244.0.159:9090
  • job: prometheus-kube-prometheus-prometheus
  • namespace: monitoring
  • pod: prometheus-prometheus-kube-prometheus-prometheus-0
  • prometheus: monitoring/prometheus-kube-prometheus-prometheus
  • rule_group: /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0/monitoring-prometheus-kube-prometheus-kubelet.rules.yaml;kubelet.rules
  • service: prometheus-kube-prometheus-prometheus
  • severity: critical
</code></pre>
<p>But when I try to get logs from the pod, it shows no related error (only warn and info)</p>
<pre><code>level=warn ts=2021-05-04T13:36:57.986Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.5\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:02.027Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubernetes-system-kubelet msg=&quot;Evaluating rule failed&quot; rule=&quot;alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}[5m])))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}\n &gt; 60\nfor: 15m\nlabels:\n severity: warning\nannotations:\n description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n on node {{ $labels.node }}.\n runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n summary: Kubelet Pod startup latency is too high.\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:27.985Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.99\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:27.986Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.9\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:27.986Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.5\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:32.026Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubernetes-system-kubelet msg=&quot;Evaluating rule failed&quot; rule=&quot;alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}[5m])))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}\n &gt; 60\nfor: 15m\nlabels:\n severity: warning\nannotations:\n description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n on node {{ $labels.node }}.\n runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n summary: Kubelet Pod startup latency is too high.\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:57.985Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.99\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:57.986Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.9\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:37:57.987Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.5\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:38:02.028Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubernetes-system-kubelet msg=&quot;Evaluating rule failed&quot; rule=&quot;alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}[5m])))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;}\n &gt; 60\nfor: 15m\nlabels:\n severity: warning\nannotations:\n description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n on node {{ $labels.node }}.\n runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n summary: Kubelet Pod startup latency is too high.\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:38:27.985Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.99\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:38:27.986Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.9\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
level=warn ts=2021-05-04T13:38:27.987Z caller=manager.go:601 component=&quot;rule manager&quot; group=kubelet.rules msg=&quot;Evaluating rule failed&quot; rule=&quot;record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n * on(instance) group_left(node) kubelet_node_name{job=\&quot;kubelet\&quot;,metrics_path=\&quot;/metrics\&quot;})\nlabels:\n quantile: \&quot;0.5\&quot;\n&quot; err=&quot;found duplicate series for the match group {instance=\&quot;209.151.158.125:10250\&quot;} on the right hand-side of the operation: [{__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-operator-kubelet\&quot;}, {__name__=\&quot;kubelet_node_name\&quot;, endpoint=\&quot;https-metrics\&quot;, instance=\&quot;209.151.158.125:10250\&quot;, job=\&quot;kubelet\&quot;, metrics_path=\&quot;/metrics\&quot;, namespace=\&quot;kube-system\&quot;, node=\&quot;cyza-node6\&quot;, service=\&quot;prometheus-kube-prometheus-kubelet\&quot;}];many-to-many matching not allowed: matching labels must be unique on one side&quot;
</code></pre>
<p>where could I get which (those 30) rules that are failed?
(I'm using <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="noreferrer">prometheus-kube-stack</a>)</p>

Kokizzu
<p>Your best starting point is the rules page of the Prometheus UI (:9090/rules).<br />
It will show the error on specific rule(s).</p>
