<p>I am currently having issues trying to get Prometheus to scrape the metrics for my Minikube cluster. Prometheus is installed via the <code>kube-prometheus-stack</code></p>
<pre class="lang-bash prettyprint-override"><code>kubectl create namespace monitoring &amp;&amp; \
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts &amp;&amp; \
helm repo update &amp;&amp; \
helm install -n monitoring prometheus-stack prometheus-community/kube-prometheus-stack
</code></pre>
<p>I am currently accessing Prometheus from an Ingress with a locally signed TLS certificate and it appears it's leading to conflicts as connection keeps getting refused by the cluster.</p>
<p>TLS is set up via Minikube ingress <a href="https://minikube.sigs.k8s.io/docs/tutorials/custom_cert_ingress/" rel="nofollow noreferrer">add-on</a>:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl create secret -n kube-system tls mkcert-tls-secret --cert=cert.pem --key=key.pem
minikube addons configure ingress &lt;&lt;&lt; &quot;kube-system/mkcert-tls-secret&quot; &amp;&amp; \
minikube addons disable ingress &amp;&amp; \
minikube addons enable ingress
</code></pre>
<p>It seems Prometheus can't get access to <code>http-metrics</code> as a target. I installed Prometheus via a helm chart:</p>
<pre class="lang-bash prettyprint-override"><code>kubectl create namespace monitoring &amp;&amp; \
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts &amp;&amp; \
helm repo update &amp;&amp; \
helm install -n monitoring prometheus-stack prometheus-community/kube-prometheus-stack
</code></pre>
<p>Here is my Prometheus configuration:</p>
<pre class="lang-yaml prettyprint-override"><code>global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: monitoring/prometheus-stack-kube-prom-prometheus
    prometheus_replica: prometheus-prometheus-stack-kube-prom-prometheus-0
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - follow_redirects: true
    enable_http2: true
    scheme: http
    path_prefix: /
    timeout: 10s
    api_version: v2
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: prometheus-stack-kube-prom-alertmanager
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: http-web
      replacement: $1
      action: keep
    kubernetes_sd_configs:
    - role: endpoints
      kubeconfig_file: &quot;&quot;
      follow_redirects: true
      enable_http2: true
      namespaces:
        own_namespace: false
        names:
        - monitoring
rule_files:
- /etc/prometheus/rules/prometheus-prometheus-stack-kube-prom-prometheus-rulefiles-0/*.yaml
scrape_configs:
- job_name: serviceMonitor/monitoring/prometheus-stack-kube-prom-kube-controller-manager/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-controller-manager);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus-stack);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: &quot;0&quot;
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: &quot;&quot;
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/monitoring/prometheus-stack-kube-prom-kube-etcd/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-etcd);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus-stack);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: &quot;0&quot;
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: &quot;&quot;
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system

</code></pre>
<p>I am also currently accessing (works just fine) the Prometheus instance outside of the cluster with an Ingress using the TLS certificate:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheusdashboard-ingress
  namespace: monitoring
  labels:
    name: prometheusdashboard-ingress
spec:
  tls:
  - hosts:
    - prometheus.demo
    secretName: mkcert-tls-secret
  rules:
  - host: prometheus.demo
    http:
      paths:
      - pathType: Prefix
        path: &quot;/&quot;
        backend:
          service:
            name: prometheus-stack-kube-prom-prometheus
            port:
              number: 9090

</code></pre>
<p>Here's the output in the target page of Prometheus:</p>
<p><a href="https://i.stack.imgur.com/iOd8O.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iOd8O.png" alt="Prometheus targets" /></a></p>
<p>What do I get the stack access to this TLS certificate which I assume is the main issue here?</p>

<h1>Solution to the problem</h1>
<p>After some thorough analysis of my various kubernetes clusters, I have found out the following errors exist:</p>
<h3>Docker Desktop Environment throws this:</h3>
<p><code>Warning  Failed     6s (x3 over 30s)   kubelet            Error: failed to start container &quot;node-exporter&quot;: Error response from daemon: path / is mounted on / but it is not a shared or slave mount</code></p>
<p>The <a href="https://stackoverflow.com/questions/70556984/kubernetes-node-exporter-container-is-not-working-it-shows-this-error-message">solution</a> tries solving it but it didn't work out for me.</p>
<h3>Minikube Environment</h3>
<p>The same error was replicated there too, I opened the web UI for minikube and found that these services were related to these ports
<a href="https://i.stack.imgur.com/X7fyw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/X7fyw.png" alt="Web UI" /></a></p>
<p>Through this image, we can understand that these services are relevant to these ports. You can try port-forwarding them but even that doesn't help much.</p>
<p>The only way I can see this working is to apply the configuration through <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus" rel="nofollow noreferrer">prometheus chart</a>.</p>
