<p>I have been deploy a prometheus outsite of kubernetes Cluser. And I want to monitoring kubernetes with it. unfortunately, I encounter a lot of problem.</p>
<p>Such as:</p>
<ul>
<li><strong>server returned HTTP status 403 Forbidden</strong></li>
<li>and **Get &quot;https://kubernetes.default.svc:443/api/v1/nodes/company/proxy/metrics&quot;: dial tcp: lookup kubernetes.default.svc on 192.168.0.1:53: no such host **</li>
</ul>
<p><a href="https://i.stack.imgur.com/rYF0U.png" rel="nofollow noreferrer">Here is the Screenshot</a></p>
<p>Here is my deploy script:</p>
<pre><code>docker run -it -d --name prometheus -p 9090:9090 \
  --user 1000 \
  -v /home/prometheus/prometheus:/etc/prometheus/ \
  -v /home/prometheus/data:/prometheus \
  quay.io/prometheus/prometheus:v2.20.1
</code></pre>
<p>and here is my prometheus.yml</p>
<pre class="lang-yaml prettyprint-override"><code># my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - &quot;first_rules.yml&quot;
  # - &quot;second_rules.yml&quot;

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=&lt;job_name&gt;` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']


  - job_name: kubernetes-apiservers
    metrics_path: /metrics
    # metrics_path: /

    scrape_interval: 10s
    # scrape_interval: 1m
    scrape_timeout: 10s
    scheme: https

    tls_config:
      insecure_skip_verify: true 
      # ca_file: /etc/prometheus/ca.crt

    kubernetes_sd_configs:
    - api_server: https://192.168.0.146:6443
      role: endpoints
      bearer_token_file: /etc/prometheus/prome.token
      tls_config:
        insecure_skip_verify: true  
      # namespaces:
      #   names: []

    relabel_configs:
    - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: default;kubernetes;https
      replacement: $1
      action: keep


  - job_name: kubernetes-nodes
    metrics_path: /metrics

    scrape_interval: 10s
    scrape_timeout: 10s
    scheme: https

    tls_config:
      insecure_skip_verify: true
      # ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

    kubernetes_sd_configs:
    - api_server: https://192.168.0.146:6443
      role: node
      bearer_token_file: /etc/prometheus/prome.token
      tls_config:
        insecure_skip_verify: true
        # ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      namespaces:
        names: []

    relabel_configs:
    - separator: ;
      regex: __meta_kubernetes_node_label_(.+)
      replacement: $1
      action: labelmap
    - separator: ;
      regex: (.*)
      target_label: __address__
      replacement: kubernetes.default.svc:443
      action: replace
    - source_labels: [__meta_kubernetes_node_name]
      separator: ;
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics
      action: replace

</code></pre>
<p>Also, I check the prometheus log, and found nothing suspect:</p>
<pre><code>[root@company-server-121 prometheus]# docker logs -f prometheus --tail 100
level=info ts=2020-09-04T02:55:49.571Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599170400000 maxt=1599177600000 ulid=01EHBA1FG79CSJGAEFKEBKX8WA
level=info ts=2020-09-04T02:55:49.577Z caller=head.go:641 component=tsdb msg=&quot;Replaying on-disk memory mappable chunks if any&quot;
level=info ts=2020-09-04T02:55:49.578Z caller=head.go:655 component=tsdb msg=&quot;On-disk memory mappable chunks replay completed&quot; duration=1.465365ms
level=info ts=2020-09-04T02:55:49.579Z caller=head.go:661 component=tsdb msg=&quot;Replaying WAL, this may take a while&quot;
level=info ts=2020-09-04T02:55:49.583Z caller=head.go:687 component=tsdb msg=&quot;WAL checkpoint loaded&quot;
level=info ts=2020-09-04T02:55:49.613Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=35 maxSegment=39
level=info ts=2020-09-04T02:55:49.644Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=36 maxSegment=39
level=info ts=2020-09-04T02:55:49.676Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=37 maxSegment=39
level=info ts=2020-09-04T02:55:49.703Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=38 maxSegment=39
level=info ts=2020-09-04T02:55:49.704Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=39 maxSegment=39
level=info ts=2020-09-04T02:55:49.704Z caller=head.go:716 component=tsdb msg=&quot;WAL replay completed&quot; checkpoint_replay_duration=4.494795ms wal_replay_duration=121.056294ms total_replay_duration=127.049792ms
level=info ts=2020-09-04T02:55:49.707Z caller=main.go:700 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-09-04T02:55:49.707Z caller=main.go:701 msg=&quot;TSDB started&quot;
level=info ts=2020-09-04T02:55:49.707Z caller=main.go:805 msg=&quot;Loading configuration file&quot; filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-04T02:55:49.708Z caller=main.go:833 msg=&quot;Completed loading of configuration file&quot; filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-04T02:55:49.708Z caller=main.go:652 msg=&quot;Server is ready to receive web requests.&quot;
level=info ts=2020-09-04T03:00:01.264Z caller=compact.go:495 component=tsdb msg=&quot;write block&quot; mint=1599177600000 maxt=1599184800000 ulid=01EHBGWZ2SH6X5N3JK6T4NRM2Z duration=23.544803ms
level=info ts=2020-09-04T03:00:01.269Z caller=head.go:804 component=tsdb msg=&quot;Head GC completed&quot; duration=1.195693ms
level=info ts=2020-09-04T03:00:01.269Z caller=checkpoint.go:96 component=tsdb msg=&quot;Creating checkpoint&quot; from_segment=35 to_segment=37 mint=1599184800000
level=info ts=2020-09-04T03:00:01.302Z caller=head.go:884 component=tsdb msg=&quot;WAL checkpoint complete&quot; first=35 last=37 duration=32.653413ms
level=info ts=2020-09-04T03:00:01.328Z caller=compact.go:441 component=tsdb msg=&quot;compact blocks&quot; count=3 mint=1599156000000 maxt=1599177600000 ulid=01EHBGWZ4S5F3GJ259C7070CYK sources=&quot;[01EHAWA1070CVB7X1BZ8CE9TXH 01EHB35R87CQVVJYNHBB2T76G6 01EHBA1FG79CSJGAEFKEBKX8WA]&quot; duration=23.473664ms
level=warn ts=2020-09-04T03:38:34.452Z caller=main.go:530 msg=&quot;Received SIGTERM, exiting gracefully...&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=main.go:553 msg=&quot;Stopping scrape discovery manager...&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=main.go:567 msg=&quot;Stopping notify discovery manager...&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=main.go:589 msg=&quot;Stopping scrape manager...&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=main.go:549 msg=&quot;Scrape discovery manager stopped&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=main.go:563 msg=&quot;Notify discovery manager stopped&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=manager.go:888 component=&quot;rule manager&quot; msg=&quot;Stopping rule manager...&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=manager.go:898 component=&quot;rule manager&quot; msg=&quot;Rule manager stopped&quot;
level=info ts=2020-09-04T03:38:34.453Z caller=main.go:583 msg=&quot;Scrape manager stopped&quot;
level=info ts=2020-09-04T03:38:34.454Z caller=notifier.go:601 component=notifier msg=&quot;Stopping notification manager...&quot;
level=info ts=2020-09-04T03:38:34.454Z caller=main.go:755 msg=&quot;Notifier manager stopped&quot;
level=info ts=2020-09-04T03:38:34.454Z caller=main.go:767 msg=&quot;See you next time!&quot;
level=info ts=2020-09-04T03:38:34.952Z caller=main.go:308 msg=&quot;No time or size retention was set so using the default time retention&quot; duration=15d
level=info ts=2020-09-04T03:38:34.952Z caller=main.go:343 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.20.1, branch=HEAD, revision=983ebb4a513302315a8117932ab832815f85e3d2)&quot;
level=info ts=2020-09-04T03:38:34.952Z caller=main.go:344 build_context=&quot;(go=go1.14.6, user=root@7cbd4d1c15e0, date=20200805-17:26:58)&quot;
level=info ts=2020-09-04T03:38:34.952Z caller=main.go:345 host_details=&quot;(Linux 4.14.15-1.el7.elrepo.x86_64 #1 SMP Tue Jan 23 20:28:26 EST 2018 x86_64 34ba7bdc34ce (none))&quot;
level=info ts=2020-09-04T03:38:34.952Z caller=main.go:346 fd_limits=&quot;(soft=1048576, hard=1048576)&quot;
level=info ts=2020-09-04T03:38:34.952Z caller=main.go:347 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
level=info ts=2020-09-04T03:38:34.954Z caller=web.go:524 component=web msg=&quot;Start listening for connections&quot; address=0.0.0.0:9090
level=info ts=2020-09-04T03:38:34.954Z caller=main.go:684 msg=&quot;Starting TSDB ...&quot;
level=info ts=2020-09-04T03:38:34.955Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599023110806 maxt=1599069600000 ulid=01EH8GS198RCSZBAPC1Z7P629X
level=info ts=2020-09-04T03:38:34.956Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599069600000 maxt=1599134400000 ulid=01EHA7PVAGDKSJTEJE1572FC7J
level=info ts=2020-09-04T03:38:34.956Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599134400000 maxt=1599156000000 ulid=01EHAWA116KSFTR969PJ5MKAQ2
level=info ts=2020-09-04T03:38:34.956Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599177600000 maxt=1599184800000 ulid=01EHBGWZ2SH6X5N3JK6T4NRM2Z
level=info ts=2020-09-04T03:38:34.956Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599156000000 maxt=1599177600000 ulid=01EHBGWZ4S5F3GJ259C7070CYK
level=info ts=2020-09-04T03:38:34.963Z caller=head.go:641 component=tsdb msg=&quot;Replaying on-disk memory mappable chunks if any&quot;
level=info ts=2020-09-04T03:38:34.964Z caller=head.go:655 component=tsdb msg=&quot;On-disk memory mappable chunks replay completed&quot; duration=899.254µs
level=info ts=2020-09-04T03:38:34.964Z caller=head.go:661 component=tsdb msg=&quot;Replaying WAL, this may take a while&quot;
level=info ts=2020-09-04T03:38:34.967Z caller=head.go:687 component=tsdb msg=&quot;WAL checkpoint loaded&quot;
level=info ts=2020-09-04T03:38:34.997Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=38 maxSegment=41
level=info ts=2020-09-04T03:38:35.002Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=39 maxSegment=41
level=info ts=2020-09-04T03:38:35.029Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=40 maxSegment=41
level=info ts=2020-09-04T03:38:35.029Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=41 maxSegment=41
level=info ts=2020-09-04T03:38:35.029Z caller=head.go:716 component=tsdb msg=&quot;WAL replay completed&quot; checkpoint_replay_duration=2.805909ms wal_replay_duration=62.637276ms total_replay_duration=66.40127ms
level=info ts=2020-09-04T03:38:35.031Z caller=main.go:700 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-09-04T03:38:35.031Z caller=main.go:701 msg=&quot;TSDB started&quot;
level=info ts=2020-09-04T03:38:35.031Z caller=main.go:805 msg=&quot;Loading configuration file&quot; filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-04T03:38:35.032Z caller=main.go:833 msg=&quot;Completed loading of configuration file&quot; filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-04T03:38:35.032Z caller=main.go:652 msg=&quot;Server is ready to receive web requests.&quot;
level=warn ts=2020-09-04T03:47:13.411Z caller=main.go:530 msg=&quot;Received SIGTERM, exiting gracefully...&quot;
level=info ts=2020-09-04T03:47:13.411Z caller=main.go:553 msg=&quot;Stopping scrape discovery manager...&quot;
level=info ts=2020-09-04T03:47:13.411Z caller=main.go:567 msg=&quot;Stopping notify discovery manager...&quot;
level=info ts=2020-09-04T03:47:13.411Z caller=main.go:589 msg=&quot;Stopping scrape manager...&quot;
level=info ts=2020-09-04T03:47:13.411Z caller=main.go:549 msg=&quot;Scrape discovery manager stopped&quot;
level=info ts=2020-09-04T03:47:13.411Z caller=main.go:563 msg=&quot;Notify discovery manager stopped&quot;
level=info ts=2020-09-04T03:47:13.412Z caller=manager.go:888 component=&quot;rule manager&quot; msg=&quot;Stopping rule manager...&quot;
level=info ts=2020-09-04T03:47:13.412Z caller=manager.go:898 component=&quot;rule manager&quot; msg=&quot;Rule manager stopped&quot;
level=info ts=2020-09-04T03:47:13.412Z caller=main.go:583 msg=&quot;Scrape manager stopped&quot;
level=info ts=2020-09-04T03:47:13.412Z caller=notifier.go:601 component=notifier msg=&quot;Stopping notification manager...&quot;
level=info ts=2020-09-04T03:47:13.412Z caller=main.go:755 msg=&quot;Notifier manager stopped&quot;
level=info ts=2020-09-04T03:47:13.412Z caller=main.go:767 msg=&quot;See you next time!&quot;
level=info ts=2020-09-04T03:47:13.939Z caller=main.go:308 msg=&quot;No time or size retention was set so using the default time retention&quot; duration=15d
level=info ts=2020-09-04T03:47:13.939Z caller=main.go:343 msg=&quot;Starting Prometheus&quot; version=&quot;(version=2.20.1, branch=HEAD, revision=983ebb4a513302315a8117932ab832815f85e3d2)&quot;
level=info ts=2020-09-04T03:47:13.939Z caller=main.go:344 build_context=&quot;(go=go1.14.6, user=root@7cbd4d1c15e0, date=20200805-17:26:58)&quot;
level=info ts=2020-09-04T03:47:13.939Z caller=main.go:345 host_details=&quot;(Linux 4.14.15-1.el7.elrepo.x86_64 #1 SMP Tue Jan 23 20:28:26 EST 2018 x86_64 34ba7bdc34ce (none))&quot;
level=info ts=2020-09-04T03:47:13.939Z caller=main.go:346 fd_limits=&quot;(soft=1048576, hard=1048576)&quot;
level=info ts=2020-09-04T03:47:13.939Z caller=main.go:347 vm_limits=&quot;(soft=unlimited, hard=unlimited)&quot;
level=info ts=2020-09-04T03:47:13.941Z caller=web.go:524 component=web msg=&quot;Start listening for connections&quot; address=0.0.0.0:9090
level=info ts=2020-09-04T03:47:13.941Z caller=main.go:684 msg=&quot;Starting TSDB ...&quot;
level=info ts=2020-09-04T03:47:13.942Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599023110806 maxt=1599069600000 ulid=01EH8GS198RCSZBAPC1Z7P629X
level=info ts=2020-09-04T03:47:13.942Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599069600000 maxt=1599134400000 ulid=01EHA7PVAGDKSJTEJE1572FC7J
level=info ts=2020-09-04T03:47:13.942Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599134400000 maxt=1599156000000 ulid=01EHAWA116KSFTR969PJ5MKAQ2
level=info ts=2020-09-04T03:47:13.942Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599177600000 maxt=1599184800000 ulid=01EHBGWZ2SH6X5N3JK6T4NRM2Z
level=info ts=2020-09-04T03:47:13.942Z caller=repair.go:59 component=tsdb msg=&quot;Found healthy block&quot; mint=1599156000000 maxt=1599177600000 ulid=01EHBGWZ4S5F3GJ259C7070CYK
level=info ts=2020-09-04T03:47:13.948Z caller=head.go:641 component=tsdb msg=&quot;Replaying on-disk memory mappable chunks if any&quot;
level=info ts=2020-09-04T03:47:13.949Z caller=head.go:655 component=tsdb msg=&quot;On-disk memory mappable chunks replay completed&quot; duration=727.383µs
level=info ts=2020-09-04T03:47:13.949Z caller=head.go:661 component=tsdb msg=&quot;Replaying WAL, this may take a while&quot;
level=info ts=2020-09-04T03:47:13.952Z caller=head.go:687 component=tsdb msg=&quot;WAL checkpoint loaded&quot;
level=info ts=2020-09-04T03:47:13.983Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=38 maxSegment=42
level=info ts=2020-09-04T03:47:13.987Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=39 maxSegment=42
level=info ts=2020-09-04T03:47:14.017Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=40 maxSegment=42
level=info ts=2020-09-04T03:47:14.023Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=41 maxSegment=42
level=info ts=2020-09-04T03:47:14.023Z caller=head.go:713 component=tsdb msg=&quot;WAL segment loaded&quot; segment=42 maxSegment=42
level=info ts=2020-09-04T03:47:14.023Z caller=head.go:716 component=tsdb msg=&quot;WAL replay completed&quot; checkpoint_replay_duration=2.967015ms wal_replay_duration=71.121903ms total_replay_duration=74.856822ms
level=info ts=2020-09-04T03:47:14.025Z caller=main.go:700 fs_type=EXT4_SUPER_MAGIC
level=info ts=2020-09-04T03:47:14.025Z caller=main.go:701 msg=&quot;TSDB started&quot;
level=info ts=2020-09-04T03:47:14.025Z caller=main.go:805 msg=&quot;Loading configuration file&quot; filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-04T03:47:14.026Z caller=main.go:833 msg=&quot;Completed loading of configuration file&quot; filename=/etc/prometheus/prometheus.yml
level=info ts=2020-09-04T03:47:14.026Z caller=main.go:652 msg=&quot;Server is ready to receive web requests.&quot;

</code></pre>
<p>Hope to get your help， thanks.</p>

zmhuang
<p>Yes, you can monitor Kubernetes from outside, but it is strongly advised against. Prometheus best works when in k8s cluster. A cleaner solution is to have Prometheus + thanos/cortana in the cluster and use a secondary, central Prometheus to monitor all.</p>
<p>To resolve the issue you have you need to give the Cert also, have a look at :
<a href="https://github.com/prometheus/prometheus/blob/release-2.20/documentation/examples/prometheus-kubernetes.yml" rel="nofollow noreferrer">https://github.com/prometheus/prometheus/blob/release-2.20/documentation/examples/prometheus-kubernetes.yml</a></p>
