<p>I have some openshift nodes, some  nodes have some Tesla P40s and should be dedicated to ML use through nvidia device plugin. But i don't want to let users add some taints or node affinity in their original DeploymentConfigs which could lead to a messy. How could i achieve this implicitly?</p>

<p>what i want to achieve is: </p>

<ol>
<li>only ML pods could stay on these nodes which has GPU </li>
<li>ML users doesn't need to change their dc except the resource limit of "nvidia.com/gpu".</li>
</ol>

<p>if a scheduler is the only way, then how to write it ? Thanks</p>

<p>My understanding of your problem is that you're trying to do two things:</p>

<ol>
<li>Schedule GPU workloads correctly onto nodes that have GPUs available</li>
<li>Make sure pods that <em>don't</em> need GPUs are not scheduled onto nodes that <em>have</em> GPUs.</li>
</ol>

<p>You're doing (1) With the NVidia device plugin, which seems to be the correct way, since it uses the concept of <a href="https://kubernetes.io/docs/tasks/administer-cluster/extended-resource-node/" rel="nofollow noreferrer">Extended Resources</a>.</p>

<p>To do (2), Taints and Tolerations are indeed the recommended way. The docs even talk explicitly about the GPU use case -
<a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#example-use-cases" rel="nofollow noreferrer">Quoting the documentation:</a></p>

<blockquote>
  <p><strong>Nodes with Special Hardware:</strong> In a cluster where a small subset of
  nodes have specialized hardware (for example GPUs), it is desirable to
  keep pods that don’t need the specialized hardware off of those nodes,
  thus leaving room for later-arriving pods that do need the specialized
  hardware. This can be done by tainting the nodes that have the
  specialized hardware (e.g. kubectl taint nodes nodename
  special=true:NoSchedule or kubectl taint nodes nodename
  special=true:PreferNoSchedule) and adding a corresponding toleration
  to pods that use the special hardware. As in the dedicated nodes use
  case, it is probably easiest to apply the tolerations using a custom
  admission controller). For example, it is recommended to use Extended
  Resources to represent the special hardware, taint your special
  hardware nodes with the extended resource name and run the
  ExtendedResourceToleration admission controller. Now, because the
  nodes are tainted, no pods without the toleration will schedule on
  them. But when you submit a pod that requests the extended resource,
  the ExtendedResourceToleration admission controller will automatically
  add the correct toleration to the pod and that pod will schedule on
  the special hardware nodes. This will make sure that these special
  hardware nodes are dedicated for pods requesting such hardware and you
  don’t have to manually add tolerations to your pods.</p>
</blockquote>

<p>Only those users that explicitly <em>need</em> the GPUs need to add a toleration for it in their pod spec, and it's fairly straightforward to do so. It looks like this (ref: <a href="https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes" rel="nofollow noreferrer">Advanced Scheduling in Kubernetes</a>):</p>

<pre><code>tolerations:
- key: "gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"
</code></pre>

<p>So usually this is an acceptable trade-off.</p>

<p>However if you absolutely <strong><em>do not</em></strong> want to let users have to add that toleration. What you need is an <a href="https://kubernetes.io/docs/admin/admission-controllers/" rel="nofollow noreferrer">Admission Controller</a>.</p>

<blockquote>
  <p>An admission controller is a piece of code that intercepts requests to
  the Kubernetes API server prior to persistence of the object, but
  after the request is authenticated and authorized.</p>
</blockquote>

<p>In particular you want the special AdmissionController known as <code>MutatingAdmissionWebhook</code>.</p>

<p>Your custom <code>MutatingAdmissionWebhook</code> can take a look at the pod spec, look for:</p>

<pre><code>resources:
        limits:
          nvidia.com/gpu: 2
</code></pre>

<p>and then automatically add the required "Toleration" to the pod spec, all without letting the user know. You still end up using Taints and Tolerations, the users just don't see them anymore. You <strong>don't</strong> need to write a new scheduler for this.</p>

<p><a href="https://github.com/kubernetes/kubernetes/blob/v1.10.0-beta.1/test/images/webhook/main.go" rel="nofollow noreferrer">Here's an example</a> of how to <em>write</em> an admission controller webhook, is available in the official kubernetes repository as part of the e2e tests.</p>
