<p>I'm fairly new to Kubernetes and Prometheus.  I've been following <a href="https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/" rel="nofollow noreferrer">these directions</a> step by step.</p>
<p>I get to the section where I do port forwarding with:</p>
<pre><code>kubectl port-forward &lt;prometheus pod&gt; 8080:9090 -n monitoring
</code></pre>
<p>At this point I should be able to go to http://localhost:8080 to get the Prometheus dashboard.  Instead I get:</p>
<pre><code># curl http://localhost:8080
curl: (52) Empty reply from server
</code></pre>
<p>And on the port forwarding process I get permission denied:</p>
<pre><code>Handling connection for 8080
E0723 19:45:40.532119   18108 portforward.go:400] an error occurred forwarding 8080 -&gt; 9090: error forwarding port 9090 to pod 3a04bc19fcc2f23f2c1b095df8ba472b455f91d6453a72575078b858df8d87a1, uid : exit status 1: 2021/07/23 19:45:40 socat[22259] E connect(5, AF=2 127.0.0.1:9090, 16): Permission denied
</code></pre>
<p>If I look at the prometheus pod logs, it looks happy:</p>
<pre><code>level=info ts=2021-07-23T19:14:38.691Z caller=kubernetes.go:282 component=&quot;discovery manager scrape&quot; discovery=kubernetes msg=&quot;Using pod service account via in-cluster config&quot;
level=info ts=2021-07-23T19:14:38.692Z caller=kubernetes.go:282 component=&quot;discovery manager scrape&quot; discovery=kubernetes msg=&quot;Using pod service account via in-cluster config&quot;
level=info ts=2021-07-23T19:14:38.693Z caller=kubernetes.go:282 component=&quot;discovery manager scrape&quot; discovery=kubernetes msg=&quot;Using pod service account via in-cluster config&quot;
level=info ts=2021-07-23T19:14:38.695Z caller=main.go:1012 msg=&quot;Completed loading of configuration file&quot; filename=/etc/prometheus/prometheus.yml totalDuration=10.027206ms remote_storage=8.294µs web_handler=307ns query_engine=2.796µs scrape=801.585µs scrape_sd=3.181796ms notify=23.042µs notify_sd=10.213µs rules=550.517µs
level=info ts=2021-07-23T19:14:38.695Z caller=main.go:796 msg=&quot;Server is ready to receive web requests.&quot;
</code></pre>
<p>I tried google first and haven't found any matches that includes all the above.</p>
<p>I'm not sure if it matters running on localhost, but I do have the firewall open on that port</p>
<pre><code># iptables -xvn -L|grep 8080
      21     1166 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080

</code></pre>
<p>I suspect that the issue might have something to do with the root user running the Prometheus pod.  If I <code>kubectl exec</code> in to the Prometheus pod, everything is owned and running as root also.  I've looked at manual installations of Prometheus and those instructions always have you create a Prometheus user and group and run as the Prometheus user.  But the Prometheus container is coming right from dockerhub so I don't know how I would change the user inside the docker image.  But this is just all a guess at this point and have no idea really.</p>
<p>Any help would be appreciated.</p>
<p>This is a Centos 7 server running Kubernetes 1.18.
Prometheus is the latest dockerhub image <code>prom/prometheus</code></p>
<p>TIA</p>
<p>-Jon</p>

Jon Hayden
<p>Issue has been resolved by switching centos to <a href="https://wiki.gentoo.org/wiki/SELinux/Tutorials/Permissive_versus_enforcing" rel="nofollow noreferrer">permissive mode</a>.</p>
<p>During kubernetes installation on SELinux-enabled systems you should disable  it so all containers can access host filesystem.</p>
<p>Please note that disabling SELinux by <code>setenforce 0</code> is only a temporary solution (as like turning off swap by <code>swapoff -a</code></p>
<p>To completely disable them:</p>
<ul>
<li>SELinux: edit <code>/etc/selinux/config</code> and set <code>SELINUX=disabled</code></li>
<li>Swap: edit <code>/etc/fstab</code> and comment(#) swap entry</li>
</ul>
