<p>I have installed kube-prometheus-stack as a <strong>dependency</strong> in my helm chart on a local Docker for Mac Kubernetes cluster v1.19.7.</p>
<p>The <strong>myrelease-name-prometheus-node-exporter</strong> service is failing with errors received from the node-exporter daemonset after installation of the helm chart for <a href="https://hub.kubeapps.com/charts/prometheus-community/kube-prometheus-stack#!" rel="nofollow noreferrer">kube-prometheus-stack</a> is installed. This is installed in a Docker Desktop for Mac Kubernetes Cluster environment.</p>
<p><strong>release-name-prometheus-node-exporter daemonset error log</strong></p>
<pre class="lang-sh prettyprint-override"><code>MountVolume.SetUp failed for volume &quot;flaskapi-prometheus-node-exporter-token-zft28&quot; : failed to sync secret cache: timed out waiting for the condition
Error: failed to start container &quot;node-exporter&quot;: Error response from daemon: path / is mounted on / but it is not a shared or slave mount
Back-off restarting failed container
</code></pre>
<p>The scrape targets for <code>kube-scheduler:http://192.168.65.4:10251/metrics</code>, <code>kube-proxy:http://192.168.65.4:10249/metrics</code>, <code>kube-etcd:http://192.168.65.4:2379/metrics</code>, <code>kube-controller-manager:http://192.168.65.4:10252/metrics</code> and <code>node-exporter:http://192.168.65.4:9100/metrics</code> are marked as unhealthy. All show as <code>connection refused</code>, except for <code>kube-etcd</code> which displays <code>connection reset by peer</code>.</p>
<p><strong>Chart.yaml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v2
appVersion: &quot;0.0.1&quot;
description: A Helm chart for flaskapi deployment
name: flaskapi
version: 0.0.1
dependencies:
- name: kube-prometheus-stack
  version: &quot;14.4.0&quot;
  repository: &quot;https://prometheus-community.github.io/helm-charts&quot;
- name: ingress-nginx
  version: &quot;3.25.0&quot;
  repository: &quot;https://kubernetes.github.io/ingress-nginx&quot;
- name: redis
  version: &quot;12.9.0&quot;
  repository: &quot;https://charts.bitnami.com/bitnami&quot;
</code></pre>
<p><strong>Values.yaml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>hostname: flaskapi-service
redis_host: flaskapi-redis-master.default.svc.cluster.local 
redis_port: &quot;6379&quot;
</code></pre>
<p><strong>Environment</strong>
Mac OS Catalina 10.15.7
Docker Desktop For Mac 3.2.2(61853) with docker engine v20.10.5
Local Kubernetes 1.19.7 Cluster provided by Docker Desktop For Mac</p>
<ul>
<li><p>Prometheus Operator version:</p>
<p><a href="https://hub.kubeapps.com/charts/prometheus-community/kube-prometheus-stack#!" rel="nofollow noreferrer">kube-prometheus-stack</a> 14.4.0</p>
</li>
<li><p>Kubernetes version information:</p>
<p><code>kubectl version</code></p>
</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>    Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;19&quot;, GitVersion:&quot;v1.19.7&quot;, GitCommit:&quot;1dd5338295409edcfff11505e7bb246f0d325d15&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-01-13T13:23:52Z&quot;, GoVersion:&quot;go1.15.5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;19&quot;, GitVersion:&quot;v1.19.7&quot;, GitCommit:&quot;1dd5338295409edcfff11505e7bb246f0d325d15&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-01-13T13:15:20Z&quot;, GoVersion:&quot;go1.15.5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<p><strong>kubectl get all</strong></p>
<pre class="lang-sh prettyprint-override"><code>NAME                                                         READY   STATUS             RESTARTS   AGE
pod/alertmanager-flaskapi-kube-prometheus-s-alertmanager-0   2/2     Running            0          16m
pod/flask-deployment-775fcf8ff-2hp9s                         1/1     Running            0          16m
pod/flask-deployment-775fcf8ff-4qdjn                         1/1     Running            0          16m
pod/flask-deployment-775fcf8ff-6bvmv                         1/1     Running            0          16m
pod/flaskapi-grafana-6cb58f6656-77rqk                        2/2     Running            0          16m
pod/flaskapi-ingress-nginx-controller-ccfc7b6df-qvl7d        1/1     Running            0          16m
pod/flaskapi-kube-prometheus-s-operator-69f4bcf865-tq4q2     1/1     Running            0          16m
pod/flaskapi-kube-state-metrics-67c7f5f854-hbr27             1/1     Running            0          16m
pod/flaskapi-prometheus-node-exporter-7hgnm                  0/1     CrashLoopBackOff   8          16m
pod/flaskapi-redis-master-0                                  1/1     Running            0          16m
pod/flaskapi-redis-slave-0                                   1/1     Running            0          16m
pod/flaskapi-redis-slave-1                                   1/1     Running            0          15m
pod/prometheus-flaskapi-kube-prometheus-s-prometheus-0       2/2     Running            0          16m

NAME                                                  TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated                         ClusterIP      None             &lt;none&gt;        9093/TCP,9094/TCP,9094/UDP   16m
service/flask-api-service                             ClusterIP      10.108.242.86    &lt;none&gt;        4444/TCP                     16m
service/flaskapi-grafana                              ClusterIP      10.98.186.112    &lt;none&gt;        80/TCP                       16m
service/flaskapi-ingress-nginx-controller             LoadBalancer   10.102.217.51    localhost     80:30347/TCP,443:31422/TCP   16m
service/flaskapi-ingress-nginx-controller-admission   ClusterIP      10.99.21.136     &lt;none&gt;        443/TCP                      16m
service/flaskapi-kube-prometheus-s-alertmanager       ClusterIP      10.107.215.73    &lt;none&gt;        9093/TCP                     16m
service/flaskapi-kube-prometheus-s-operator           ClusterIP      10.107.162.227   &lt;none&gt;        443/TCP                      16m
service/flaskapi-kube-prometheus-s-prometheus         ClusterIP      10.96.168.75     &lt;none&gt;        9090/TCP                     16m
service/flaskapi-kube-state-metrics                   ClusterIP      10.100.118.21    &lt;none&gt;        8080/TCP                     16m
service/flaskapi-prometheus-node-exporter             ClusterIP      10.97.61.162     &lt;none&gt;        9100/TCP                     16m
service/flaskapi-redis-headless                       ClusterIP      None             &lt;none&gt;        6379/TCP                     16m
service/flaskapi-redis-master                         ClusterIP      10.96.192.160    &lt;none&gt;        6379/TCP                     16m
service/flaskapi-redis-slave                          ClusterIP      10.107.119.108   &lt;none&gt;        6379/TCP                     16m
service/kubernetes                                    ClusterIP      10.96.0.1        &lt;none&gt;        443/TCP                      5d1h
service/prometheus-operated                           ClusterIP      None             &lt;none&gt;        9090/TCP                     16m

NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/flaskapi-prometheus-node-exporter   1         1         0       1            0           &lt;none&gt;          16m

NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/flask-deployment                      3/3     3            3           16m
deployment.apps/flaskapi-grafana                      1/1     1            1           16m
deployment.apps/flaskapi-ingress-nginx-controller     1/1     1            1           16m
deployment.apps/flaskapi-kube-prometheus-s-operator   1/1     1            1           16m
deployment.apps/flaskapi-kube-state-metrics           1/1     1            1           16m

NAME                                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/flask-deployment-775fcf8ff                       3         3         3       16m
replicaset.apps/flaskapi-grafana-6cb58f6656                      1         1         1       16m
replicaset.apps/flaskapi-ingress-nginx-controller-ccfc7b6df      1         1         1       16m
replicaset.apps/flaskapi-kube-prometheus-s-operator-69f4bcf865   1         1         1       16m
replicaset.apps/flaskapi-kube-state-metrics-67c7f5f854           1         1         1       16m

NAME                                                                    READY   AGE
statefulset.apps/alertmanager-flaskapi-kube-prometheus-s-alertmanager   1/1     16m
statefulset.apps/flaskapi-redis-master                                  1/1     16m
statefulset.apps/flaskapi-redis-slave                                   2/2     16m
statefulset.apps/prometheus-flaskapi-kube-prometheus-s-prometheus       1/1     16m
</code></pre>
<p><strong>kubectl get svc -n kube-system</strong></p>
<pre class="lang-sh prettyprint-override"><code>flaskapi-kube-prometheus-s-coredns                   ClusterIP   None         &lt;none&gt;        9153/TCP                       29s
flaskapi-kube-prometheus-s-kube-controller-manager   ClusterIP   None         &lt;none&gt;        10252/TCP                      29s
flaskapi-kube-prometheus-s-kube-etcd                 ClusterIP   None         &lt;none&gt;        2379/TCP                       29s
flaskapi-kube-prometheus-s-kube-proxy                ClusterIP   None         &lt;none&gt;        10249/TCP                      29s
flaskapi-kube-prometheus-s-kube-scheduler            ClusterIP   None         &lt;none&gt;        10251/TCP                      29s
flaskapi-kube-prometheus-s-kubelet                   ClusterIP   None         &lt;none&gt;        10250/TCP,10255/TCP,4194/TCP   2d18h
kube-dns                                             ClusterIP   10.96.0.10   &lt;none&gt;        53/UDP,53/TCP,9153/TCP         5d18h
</code></pre>
<p>Tried updating values.yaml to include this:</p>
<p><strong>Updated values.yaml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>prometheus-node-exporter:
  hostRootFsMount: false
</code></pre>
<p>and this:</p>
<pre class="lang-yaml prettyprint-override"><code>prometheus:
  prometheus-node-exporter:
    hostRootFsMount: false
</code></pre>
<p>However, the issue described, remains, except the log for node-exporter daemonset now gives:</p>
<pre class="lang-sh prettyprint-override"><code>failed to try resolving symlinks in path &quot;/var/log/pods/default_flaskapi-prometheus-node-exporter-p5cc8_54c20fc6-c914-4cc6-b441-07b68cda140e/node-exporter/3.log&quot;: lstat /var/log/pods/default_flaskapi-prometheus-node-exporter-p5cc8_54c20fc6-c914-4cc6-b441-07b68cda140e/node-exporter/3.log: no such file or directory
</code></pre>
<p><strong>Updated Information From Comment Suggestions</strong></p>
<p><code>kubectl get pod flaskapi-prometheus-node-exporter-p5cc8</code>
No args available since node-exporter crashing...</p>
<pre class="lang-sh prettyprint-override"><code>NAME                                      READY   STATUS             RESTARTS   AGE
flaskapi-prometheus-node-exporter-p5cc8   0/1     CrashLoopBackOff   7          14m
</code></pre>
<p>The Args from the yaml output of <code>kubectl describe pod flaskapi-prometheus-node-exporter-p5cc8</code> gives:</p>
<pre class="lang-yaml prettyprint-override"><code> Args:
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --path.rootfs=/host/root
      --web.listen-address=$(HOST_IP):9100
      --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
      --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
</code></pre>
<p>After updating the values.yaml to include root <code>kube-prometheus-stack</code> as suggested in comments of answer allows the prometheus-node-exporter daemonset to start successfully. However, the scrape targets mentioned above are still unavailable....</p>
<pre class="lang-yaml prettyprint-override"><code>kube-prometheus-stack:
  prometheus-node-exporter:
    hostRootFsMount: false
</code></pre>
<p>How do I get the node-exporter working and make the associated scrape targets healthy?</p>
<p>Is the node-exporter of the <a href="https://hub.kubeapps.com/charts/prometheus-community/kube-prometheus-stack#!" rel="nofollow noreferrer">kube-prometheus-stack</a> helm chart incompatible with Docker Desktop for Mac Kubernetes clusters?</p>
<p>I have raised this as an <a href="https://github.com/prometheus-operator/kube-prometheus/issues/1073" rel="nofollow noreferrer">issue</a> at kube-prometheus-stack with log output included for scrape targets for <code>docker-desktop</code> and <code>minikube</code> clusters.</p>
<p><strong>Conclusion</strong>
It looks as though the unavailable scrape targets is a problem/bug with kube-prometheus-stack. I searched and found similar issues on their GitHub page: <a href="https://github.com/prometheus-operator/kube-prometheus/issues/713" rel="nofollow noreferrer">713</a> and <a href="https://github.com/prometheus-operator/kube-prometheus/issues/718" rel="nofollow noreferrer">718</a>. Tried on a minikube cluster with hyperkit vm-driver. On minikube the node-exporter functions out of the box, but the scrape targets issue still occurs. Not sure what a safe solution is?</p>
<p>I may investigate an alternative helm chart dependency for prometheus and grafana...</p>

anon_dcs3spp
<p>This issue was solved recently. Here is more information: <a href="https://github.com/prometheus-community/helm-charts/issues/467" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts/issues/467</a> and here: <a href="https://github.com/prometheus-community/helm-charts/pull/757" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts/pull/757</a></p>
<p>Here is the solution (<a href="https://github.com/prometheus-community/helm-charts/issues/467#issuecomment-802642666" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts/issues/467#issuecomment-802642666</a>):</p>
<blockquote>
<p>[you need to] opt-out the rootfs host mount (preventing the crash). In order to do that you need to specify the following value in values.yaml file:</p>
<pre><code>prometheus-node-exporter:
    hostRootFsMount: false
</code></pre>
</blockquote>
