<p>I want to add a wireguard sidecar container to prometheus stack installed via helm chart such that I can fetch client connected via vpn. I'm using the helm chart from <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">prometheus-community/kube-prometheus-stack</a> with modified values.yml. To integrate wireguard I added a wireguard container in the values.yml like this:</p>
<pre><code>...
containers:
    - name: &quot;wireguard&quot;
      image: &quot;lscr.io/linuxserver/wireguard:latest&quot;
      volumeMounts:
        - name: wireguard-config
          mountPath: /config
          readOnly: true
        - name: wireguard-run
          mountPath: /run
      securityContext:
        runAsGroup: 0
        runAsUser: 0
        privileged: true
        capabilities:
          add:
            - NET_ADMIN
            - SYS_MODULE
</code></pre>
<p>but when I start the containers, I get the following error:</p>
<pre><code>Normal   Pulled     4s               kubelet            Successfully pulled image &quot;lscr.io/linuxserver/wireguard:latest&quot; in 500.578587ms
Warning  Failed     3s (x3 over 4s)  kubelet            Error: container's runAsUser breaks non-root policy (pod: &quot;XX&quot;, container: wireguard)
Normal   Pulled     3s               kubelet            Successfully pulled image &quot;lscr.io/linuxserver/wireguard:latest&quot; in 456.879479ms
</code></pre>
<p>As wireguard needs to be able to change network interfaces, it needs the root privileges. If I don't run the container with root privileges I get the following:</p>
<pre><code>...
SOME OTHER PERMISSION ERROS
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise coredns: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise wireguard: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise coredns: warning: unable to spawn ./run - waiting 10 seconds
</code></pre>
<p><strong>What I tried</strong> is modifying the <em>podSecurityPolicy</em> to allow running containers as root in the prometheus values.yml. I was hoping that I could simply run the container as root (for testing at least):</p>
<pre><code>podSecurityPolicy:
    allowedCapabilities: 
       - runAsUser: RunAsAny
       - NET_ADMIN
       - SYS_MODULE
    allowedHostPaths: []
    volumes: []
</code></pre>
<p>This didn't change anything (am I doing this correctly?)</p>
<p>How would I allow running a sidecar container as root? Or is there a way to run wireguard without root priv?</p>

<ol>
<li><p>there is no psp in the newer kube-prometheus-stack. see <a href="https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md</a> - search for PodSecurityPolicies - &quot;From 27.x to 28.x This version disables PodSecurityPolicies by default because they are deprecated in Kubernetes 1.21 and will be removed in Kubernetes 1.25.&quot;</p>
</li>
<li><p>we need to play with securityContext. it seems that you cannot have a pod running as non-root with a container (or init container) running as root. this is the error &quot;kubelet Error: container's runAsUser breaks non-root policy &quot;.</p>
</li>
</ol>
<p>so we need to specify for the whole pod the securityContext run as root. for this, you may use a values.yaml for helm chart like below. the effect is the pod prometheus-prometheus-stack-kube-prom-prometheus-0  will run all its containers as root. not ideal but it works.</p>
<pre><code>prometheus:
  prometheusSpec:

    securityContext:
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
      fsGroup: 0

#    initContainers:
#      - name: &quot;chmod&quot;
#        image: alpine:3.16.0
#        command:
#        - &quot;/bin/sh&quot;
#        - &quot;-c&quot;
#        - &quot;chmod 777 /prometheus&quot;
#        volumeMounts:
#        - name: prometheus-prometheus-stack-kube-prom-prometheus-db
#          mountPath: /prometheus

[...]
</code></pre>
