<ul>
<li>We have kubernetes cluster where I have service account "kube", namespace "monitoring" with cluster role binding created to monitor cluster </li>
<li>We have prometheus installed on a linux system (on prem) outside the cluster and is installed using "root"</li>
<li>When I try to connect to the k8 cluster with the https api using <code>ca.crt</code> and user <code>token</code> (given by kubernetes admin), it throws multiple errors. </li>
</ul>

<p>Error messages:</p>

<pre><code>component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig

component="discovery manager scrape" msg="Cannot create service discovery" err="unable to use specified CA cert /root/prometheus/ca.crt" type=*kubernetes.SDConfig
</code></pre>

<p>Prometheus configuration:</p>

<pre><code>
  - job_name: 'kubernetes-apiservers'
    scheme: https
    tls_config:
      ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/prometheus-2.12.0.linux-amd64/ca.crt
    relabel_configs:
    - source_labels: [monitoring, monitoring-sa, 6443]
      action: keep
      regex: default;kubernetes;https

  - job_name: 'kubernetes-nodes'
    scheme: https
    tls_config:
        ca_file: /root/prometheus/ca.crt
    bearer_token_file: /root/prometheus/user_token

    kubernetes_sd_configs:
    - role: node
      api_server: https://example.com:1234
      bearer_token_file: /root/prometheus/user_token
      tls_config:
        ca_file: /root/prometheus/ca.crt
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: https://example.com:1234
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics
</code></pre>

<p>Your <code>ca.crt</code> is most probably still in <code>base64</code> format since secrets are encoded that way when describing them, as explained <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/#service-account-tokens" rel="nofollow noreferrer">here</a>.</p>
