<p>First, My kubernetes cluster is based on Bare-metal environment.</p>
<p>cluster info:</p>
<pre><code>k get no -o wide

NAME       STATUS   ROLES    AGE    VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
k-master   Ready    master   142d   v1.18.10   192.168.6.211   &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://18.9.9
k-node-1   Ready    &lt;none&gt;   142d   v1.18.10   192.168.6.212   &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://18.9.9
k-node-2   Ready    &lt;none&gt;   142d   v1.18.10   192.168.6.213   &lt;none&gt;        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://18.9.9
</code></pre>
<p>When i install Prometheus in cluster to monitor k8s, My step:</p>
<ol>
<li>Create RBAC</li>
</ol>
<pre><code>apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [&quot;&quot;]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- nonResourceURLs: [&quot;/metrics&quot;]
  verbs: [&quot;get&quot;]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitor
</code></pre>
<ol start="2">
<li>Create ConfigMaps</li>
</ol>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-conf
  namespace: monitor
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s #

    scrape_configs:

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    - job_name: 'kubernetes-service'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: service

    - job_name: 'kubernetes-endpoints'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: endpoints

    - job_name: 'kubernetes-ingress'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: ingress

    - job_name: 'kubernetes-pods'
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: pod
</code></pre>
<ol start="3">
<li>Create Prometheus Server</li>
</ol>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: monitor
spec:
  selector:
    matchLabels:
      app: prometheus-server
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
      - name: prometheus-server
        image: prom/prometheus
        volumeMounts:
        - name: conf
          mountPath: &quot;/etc/prometheus&quot;
          readOnly: true
        ports:
        - containerPort: 9090
      serviceAccountName: prometheus
      volumes:
      - name: conf
        configMap:
          name: prometheus-conf
</code></pre>
<p>Anyway, service and ingress of prometheus related has already created, and i can access prometheus webUI, but so many Unhealthy Targets in Service Discovery page.
<a href="https://i.stack.imgur.com/8oUSO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8oUSO.png" alt="enter image description here" /></a></p>
<p>For kubernetes-nodes details:
server returned HTTP status 403 Forbidden</p>
<p><a href="https://i.stack.imgur.com/1irwK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1irwK.png" alt="enter image description here" /></a></p>
<p>I don't know how to fix it and other more. Can anybody teach me? thanks!</p>

user7665869
<p>You can check the server side certification file for kubelet:</p>
<pre><code># openssl x509 -text -noout -in /var/lib/kubelet/pki/kubelet.crt
</code></pre>
<p>Maybe another file name. and check the SAN part of the output, like this:</p>
<pre><code> X509v3 Subject Alternative Name: 
                DNS:host.yourdomain.com 
</code></pre>
<p>If there is only the node name get displayed, then you access it through ip will not succeed. In this way, either probe the node through api-server, or regenerate the certification files with ip in the SAN field.</p>
