<ul>
<li>We have kubernetes cluster where I have service account "kube", namespace "monitoring" with cluster role binding created to monitor cluster </li>
<li>We have prometheus installed on a linux system (on prem) outside the cluster and is installed using "root"</li>
<li>When I try to connect to the k8 cluster with the https api using <code>ca.crt</code> and user <code>token</code> (given by kubernetes admin), it throws multiple errors. </li>
</ul>

<p>Error messages:</p>

<pre><code>component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig

component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig
</code></pre>

<p>Prometheus configuration:</p>

<pre><code>
  - job_name: 'kubernetes-apiservers'
    scheme: https
    tls_config:
      ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/prometheus-2.12.0.linux-amd64/ca.crt
    relabel_configs:
    - source_labels: [monitoring, monitoring-sa, 6443]
      action: keep
      regex: default;kubernetes;https

  - job_name: 'kubernetes-nodes'
    scheme: https
    tls_config:
        ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token

    kubernetes_sd_configs:
    - role: node
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/ca.crt
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: https://example.com:1234
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics
</code></pre>

<p>The main problem you're facing is: <code>"unable to use specified CA cert /root/prometheus/ca.crt"</code></p>

<p>Someone recently faced the same problem:
<a href="https://github.com/prometheus/prometheus/issues/6015#issuecomment-532058465" rel="nofollow noreferrer">https://github.com/prometheus/prometheus/issues/6015#issuecomment-532058465</a></p>

<p>He solved it by reinstalling the new version.</p>

<p>Version <code>2.13.1</code> is out. Try installing the latest version, it might solve your problem too.</p>
