<p>I am trying have kubernetes create new pods on the most requested nodes instead of pods spreading the load across available nodes. The rationale is that this simplifies scale down scenarios and relaunching of application if pods gets moved and a node gets killed during autoscaling. </p>

<p>The preferred strategy for descaling is - 1) Never kill a node if there is any running pod 2) New pods are created preferentially on the  most requested Nodes 3) The pods will self destruct after job completion. This should, over time, result in free nodes after the tasks are completed and thus descaling will be safe and I don't need to worry about resilience of the running jobs.</p>

<p>For this, is there any way I can specify the NodeAffinity in the pod spec, something like:</p>

<pre><code>    spec:
      affinity:
        nodeAffinity:
          RequiredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              nodeAffinityTerm: {MostRequestedPriority}
</code></pre>

<p>The above code has no effect. The documentation for <code>NodeAffinity</code> doesn't specify if I can use <code>MostRequestedPriority</code> in this context. <code>MostRequestedPriority</code> is an option in the kubernetes scheduler service spec. But I am trying to see if I can directly put t in the pod spec, instead of creating a new custom kubernetes scheduler.</p>

Abdul Muneer
<p>Unfortunately there is no option to pass <code>MostRequestedPriority</code> to nodeAffinity field. However you can create <a href="https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/" rel="nofollow noreferrer">simple scheduler</a> to manage pod scheduling. Following configuration will be just enough. </p>

<p>First, you have to create <code>Service Account</code> and <code>ClusterRoleBinding</code> for this scheduler: </p>

<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: own-scheduler
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: own-scheduler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: own-scheduler
  namespace: kube-system
</code></pre>

<p>Then create config map with desired <code>policy</code> field including <code>MostRequestedPriority</code>. Each field in <a href="https://kubernetes.io/docs/concepts/scheduling/#filtering" rel="nofollow noreferrer">predicates</a> can be modified to suit your needs best and basically what it does is it filters the nodes to find where a pod can be placed, for example, the <code>PodFitsResources</code> filter checks whether a Node has enough available resource to meet a Pod’s specific resource requests:</p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  labels:
  k8s-addon: scheduler.addons.k8s.io
  name: own-scheduler
  namespace: kube-system
data:
  policy.cfg: |-
  {
  "kind" : "Policy",
  "apiVersion" : "v1",
  "predicates" : [
  {"name" : "PodFitsHostPorts"},
  {"name" : "PodFitsResources"},
  {"name" : "NoDiskConflict"},
  {"name" : "PodMatchNodeSelector"},
  {"name" : "PodFitsHost"}
  ],
  "priorities" : [
  {"name" : "MostRequestedPriority", "weight" : 1},
  {"name" : "EqualPriorityMap", "weight" : 1}
  ]
  }
</code></pre>

<p>Then wrap it up in Deployment:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
    tier: control-plane
  name: own-scheduler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      component: scheduler
      tier: control-plane
  replicas: 1
  template:
    metadata:
      labels:
        component: scheduler
        tier: control-plane
        version: second
    spec:
      serviceAccountName: own-scheduler
      containers:
      - command:
        - /usr/local/bin/kube-scheduler
        - --address=0.0.0.0
        - --leader-elect=false
        - --scheduler-name=own-scheduler
        - --policy-configmap=own-scheduler
        image: k8s.gcr.io/kube-scheduler:v1.15.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10251
          initialDelaySeconds: 15
        name: kube-second-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10251
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts: []
      hostNetwork: false
      hostPID: false
      volumes: []
</code></pre>
