<p>Can someone guide the configuration for auto discover for K8s. The Prometheus server is outside of the cluster. I tried <a href="https://movio.co/en/blog/prometheus-service-discovery-kubernetes/" rel="noreferrer">Service Discovery With Kubernetes</a> and someone mentioned in this <a href="https://groups.google.com/forum/#!topic/prometheus-developers/kll2itGFkVg" rel="noreferrer">discussion</a> </p>

<blockquote>
  <p>I'm not yet a K8s expert enough to explain all the details here, but
  fundamentally it's perfectly possible to run Prometheus outside of the
  cluster (and required for things like redundant cross-cluster
  meta-monitoring). Cf. the <code>in_cluster</code> config option in
  <a href="http://prometheus.io/docs/operating/configuration/#kubernetes-sd-configurations-kubernetes_sd_config" rel="noreferrer">http://prometheus.io/docs/operating/configuration/#kubernetes-sd-configurations-kubernetes_sd_config</a>
  . You need to jump through certificate hoops if you run it outside.</p>
</blockquote>

<p>So, I made a simple configuration</p>

<pre><code>  - job_name: 'kubernetes'

    kubernetes_sd_configs:
      -
        # The API server addresses. In a cluster this will normally be
        # `https://kubernetes.default.svc`. Supports multiple HA API servers.
        api_servers:
          - https://xxx.xx.xx.xx

        # Run in cluster. This will use the automounted CA certificate and bearer
        # token file at /var/run/secrets/kubernetes.io/serviceaccount/ in the pod.
        in_cluster: false

        # Optional HTTP basic authentication information.
        basic_auth:
        username: prometheus
        password: secret

        # Retry interval between watches if they disconnect.
        retry_interval: 5s
</code></pre>

<p>Getting <code>unknown fields in kubernetes_sd_config: api_servers, in_cluster, retry_interval"</code> or some other indentation errors</p>

<p>In <a href="https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml" rel="noreferrer">sample configuration</a>, they mentioned <code>ca_file:</code>. How to get that certificate file from K8s or is there any way to specify K8s <code>config</code> file(~/.kube/config)</p>

<p>By digging though the source code I figured out, that Prometheus always uses the in cluster config, if no <code>api_server</code> is provided in the config  (<a href="https://github.com/prometheus/prometheus/blob/099df0c/discovery/kubernetes/kubernetes.go#L90-L96" rel="noreferrer"><code>discovery/kubernetes/kubernetes.go#L90-L96</code></a>).</p>

<p>Somehow the <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#&lt;kubernetes_sd_config" rel="noreferrer">docs</a> don't say anything about the Kubernetes configuration parameters, but the source code does (<a href="https://github.com/prometheus/prometheus/blob/099df0c5f00c45c007a9779a2e4ab51cf4d076bf/config/config.go#L1026-L1037" rel="noreferrer"><code>config/config.go#L1026-L1037</code></a>). Therefore there is not list named <code>api_servers</code>, but a single parameter named <code>api_server</code>.</p>

<p>So your config should look like this (untested):</p>

<pre><code>  - job_name: 'kubernetes'

    kubernetes_sd_configs:
      -
        # The API server addresses. In a cluster this will normally be
        # `https://kubernetes.default.svc`. Supports multiple HA API servers.
        api_server: https://xxx.xx.xx.xx

        # Optional HTTP basic authentication information.
        basic_auth:
          username: prometheus
          password: secret

        # specify the CA
        tls_config:
          ca_file: /path/to/ca.crt
          ## If the actual CA file isn't available you need to disable verification:
          # insecure_skip_verify: true
</code></pre>

<p>I don't know where the <code>retry_interval</code> parameter comes from, but AFAIK this isn't a Kubernetes config parameter and it's also not part of the Prometheus config.</p>
