<p>After deploying Prometheus -operator according to the documentation, I find that <code>kubectl top Nodes</code> cannot run properly.</p>
<pre><code>$ kubectl get apiService v1beta1.metrics.k8s.io 
v1beta1.metrics.k8s.io                  monitoring/prometheus-adapter   False (FailedDiscoveryCheck)   44m


$ kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

$ kubectl get  --raw &quot;/apis/metrics.k8s.io/v1beta1&quot;
Error from server (ServiceUnavailable): the server is currently unable to handle the request
</code></pre>
<blockquote>
<p>prometheus-adapter.yaml</p>
</blockquote>
<pre><code>...
      - args:
        - --cert-dir=/var/run/serving-cert
        - --config=/etc/adapter/config.yaml
        - --logtostderr=true
        - --metrics-relist-interval=1m
        - --prometheus-url=http://prometheus-k8s.monitoring.svc.cluster.local:9090/prometheus
        - --secure-port=6443
...
</code></pre>
<p>When I was looking for a problem, I found a solution (<a href="https://github.com/banzaicloud/banzai-charts/issues/1060" rel="nofollow noreferrer">#1060</a>) by adding <code>hostNetwork: true</code> to the configuration file.</p>
<p>When I thought the solution was successful, I found that <code>kubectl top nodes</code> still does not work.</p>
<pre><code>$ kubectl get apiService v1beta1.metrics.k8s.io
v1beta1.metrics.k8s.io   monitoring/prometheus-adapter   True        64m

$ kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

$ kubectl get  --raw &quot;/apis/metrics.k8s.io/v1beta1&quot;
{&quot;kind&quot;:&quot;APIResourceList&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;groupVersion&quot;:&quot;metrics.k8s.io/v1beta1&quot;,&quot;resources&quot;:[{&quot;name&quot;:&quot;nodes&quot;,&quot;singularName&quot;:&quot;&quot;,&quot;namespaced&quot;:false,&quot;kind&quot;:&quot;NodeMetrics&quot;,&quot;verbs&quot;:[&quot;get&quot;,&quot;list&quot;]},{&quot;name&quot;:&quot;pods&quot;,&quot;singularName&quot;:&quot;&quot;,&quot;namespaced&quot;:true,&quot;kind&quot;:&quot;PodMetrics&quot;,&quot;verbs&quot;:[&quot;get&quot;,&quot;list&quot;]}]}
</code></pre>
<p>View logs of Prometheus-adapter</p>
<pre><code>E0812 10:03:02.469561       1 provider.go:265] failed querying node metrics: unable to fetch node CPU metrics: unable to execute query: Get &quot;http://prometheus-k8s.monitoring.svc.cluster.local:9090/prometheus/api/v1/query?query=sum+by+%28node%29+%28%0A++1+-+irate%28%0A++++node_cpu_seconds_total%7Bmode%3D%22idle%22%7D%5B60s%5D%0A++%29%0A++%2A+on%28namespace%2C+pod%29+group_left%28node%29+%28%0A++++node_namespace_pod%3Akube_pod_info%3A%7Bnode%3D~%22node02.whisper-tech.net%7Cnode03.whisper-tech.net%22%7D%0A++%29%0A%29%0Aor+sum+by+%28node%29+%28%0A++1+-+irate%28%0A++++windows_cpu_time_total%7Bmode%3D%22idle%22%2C+job%3D%22windows-exporter%22%2Cnode%3D~%22node02.whisper-tech.net%7Cnode03.whisper-tech.net%22%7D%5B4m%5D%0A++%29%0A%29%0A&amp;time=1628762582.467&quot;: dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 100.100.2.136:53: no such host

</code></pre>
<p>The cause of the problem was that <code>hostNetwork: true</code> was added to the <code>Prometheus-Adapter</code>, which prevented pod from accessing <code>Prometheus-K8s</code> in the cluster through <code>coreDNS</code>.</p>
<p>One idea I've come up with is to have <code>Kubernetes nodes</code> access the inner part of the cluster through <code>coreDNS</code></p>
<p>Is there a better way to solve the current problem? What should I do?</p>

<p>Your Pods are running with <code>hostNetwork</code>, so you should explicitly set its DNS policy &quot;ClusterFirstWithHostNet&quot; as described in the <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy" rel="nofollow noreferrer">Pod's DNS Policy</a> documentation:</p>
<blockquote>
<p>&quot;ClusterFirstWithHostNet&quot;: For Pods running with hostNetwork, you should explicitly set its DNS policy &quot;ClusterFirstWithHostNet&quot;.</p>
</blockquote>
<p>I've created a simple example to illustrate how it works.</p>
<hr />
<p>First, I created the <code>app-1</code> Pod with <code>hostNetwork: true</code>:</p>
<pre><code>$ cat app-1.yml
kind: Pod
apiVersion: v1
metadata:
  name: app-1
spec:
  hostNetwork: true
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - &quot;3600&quot;

$ kubectl apply -f app-1.yml
pod/app-1 created
</code></pre>
<p>We can test that the <code>app-1</code> cannot resolve e.g. <code>kubernetes.default.svc</code>:</p>
<pre><code>$ kubectl exec -it app-1 -- sh

/ # nslookup kubernetes.default.svc
Server:         169.254.169.254
Address:        169.254.169.254#53

** server can't find kubernetes.default.svc: NXDOMAIN
</code></pre>
<p>Let's add the <code>dnsPolicy: ClusterFirstWithHostNet</code> to the <code>app-1</code> Pod and recreate it:</p>
<pre><code>$ cat app-1.yml
kind: Pod
apiVersion: v1
metadata:
  name: app-1
spec:
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - &quot;3600&quot;

$ kubectl delete pod app-1 &amp;&amp; kubectl apply -f app-1.yml
pod &quot;app-1&quot; deleted
pod/app-1 created
</code></pre>
<p>Finally, we can check if the <code>app-1</code> Pod is able to resolve <code>kubernetes.default.svc</code>:</p>
<pre><code>$ kubectl exec -it app-1 -- sh
/ # nslookup kubernetes.default.svc
Server:         10.8.0.10
Address:        10.8.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.8.0.1
</code></pre>
<p>As you can see in the example above, everything works as expected with the <code>ClusterFirstWithHostNet</code> dnsPolicy.</p>
<p>For more information, see the <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="nofollow noreferrer">DNS for Services and Pods</a> documentation.</p>
