<p>I have a cluster w/ 3 nodes.  Hurray!  The cluster doesn't autoscale nodes.</p>

<p>These nodes run an amazing web app, yet most of the time do almost nothing.</p>

<p>I also have a background process that could use an infinite amount of CPU (the usefulness drops rapidly but remains useful).</p>

<p>I want these background pods to run on each Node and slowed down to leave a 20% CPU headroom on the Node.  Or similar.</p>

<p>That's the shape of a DaemonSet.</p>

<ul>
<li><p>Can I tell Kubernetes to deprioritize the DaemonSet Pods w/ a 20% headroom? </p></li>
<li><p>Can the DaemonSet Pods detect the Nodes CPU usage and deprioritize themselves (risky if buggy)?</p></li>
</ul>

<p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/" rel="nofollow noreferrer">QoS</a> looks like it's for scheduling and evicting pods to make room for other pods, but they don't get 'niced'.</p>

<p><a href="https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/" rel="nofollow noreferrer">Priority</a> also looks like it's for eviction.</p>

Michael Cole
<p>You may achieve what you're looking for in many ways.</p>

<p>I imagine that you've already read <a href="https://docs.docker.com/config/containers/resource_constraints/" rel="nofollow noreferrer">this</a> and <a href="https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-cpu-time-9eff74d3161b" rel="nofollow noreferrer">that</a>, based on the theory of <a href="https://engineering.squarespace.com/blog/2017/understanding-linux-container-scheduling" rel="nofollow noreferrer">this other</a>.</p>

<p>Also RedHat has nice <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01" rel="nofollow noreferrer">documentation</a> about setting hardware limits via softwarre.</p>

<p><a href="https://scoutapm.com/blog/restricting-process-cpu-usage-using-nice-cpulimit-and-cgroups" rel="nofollow noreferrer">Here</a> you can find how to restrict cpu usage, which may be set inside a container to achieve what you're looking for.</p>

<p>So, to recap: with K8S you can set requests and limits, and inside the container you can set even further restrictive limits.</p>

<p>Hope this gives you the solution or at least the path to follow in order to achieve what you want.</p>
