<p>I'm using k8 v1.7 and <a href="https://github.com/kubernetes-client/python" rel="nofollow noreferrer">Python Client</a> v2.0. My custom scheduler detects a pending pod and schedules it successfully. However, after assigning a pod to a node, it complains that the pod is already assigned to a node though it is just assigned by the scheduler itself. Is this something to be concerned? Or how can I fix this?</p>

<p>Error Message  </p>

<pre><code>create_namespaced_binding: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 19 Jun 2018 16:14:57 GMT', 'Content-Length': '289', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on pods/binding \"ps0-16-r935x\": pod ps0-16-r935x is already assigned to node \"blipp65\"","reason":"Conflict","details":{"name":"ps0-16-r935x","kind":"pods/binding"},"code":409}
</code></pre>

<p>scheduler.py</p>

<pre><code>from kubernetes import client, config, watch
from kubernetes.client.rest import ApiException
config.load_kube_config()
v1 = client.CoreV1Api()

scheduler_name = 'my-custom-scheduler-v1'

def nodes_available():
    ready_nodes = []
    for n in v1.list_node().items:
        for status in n.status.conditions:
            if status.status == 'True' and status.type == 'Ready':
                ready_nodes.append(n.metadata.name)
    return ready_nodes


def scheduler(name, node, namespace='default'):
    body = client.V1Binding()

    target = client.V1ObjectReference()
    target.kind = 'Node'
    target.apiVersion = 'v1'
    target.name = node

    meta = client.V1ObjectMeta()
    meta.name = name

    body.target = target
    body.metadata = meta

    return v1.create_namespaced_binding_binding(name, namespace, body)


def main():
    w = watch.Watch()
    for event in w.stream(v1.list_namespaced_pod, 'default'):
        if event['object'].status.phase == 'Pending' and event['object'].spec.scheduler_name == scheduler_name:
            print "Pending Found"
            try:
                res = scheduler(event['object'].metadata.name,random.choice(nodes_available()))
                print "success"
            except Exception as a:
                print ("Exception when calling CoreV1Api-&gt;create_namespaced_binding: %s\n" % a)
</code></pre>

<p>POD YML file</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: shoeb-pod
spec:
  schedulerName: my-custom-scheduler-v1
  containers:
  - name: redis
    image: redis
</code></pre>

<p><strong>Updated on 2019-06-03</strong></p>

<p>I just added the updated main method (according to @VAS's answer, thanks) to find the right <code>PENDING</code> pod that has not been scheduled yet. Please see my answer.   </p>

Abu Shoeb
<p>When a pod is created, the scheduler gets three "Pending" events:</p>

<ol>
<li>Pod is not scheduled yet (<code>'node_name': None, 'status': {'conditions': None,...}</code>)</li>
<li>Pod is scheduled (<code>'node_name': 'some_node_name','status': {'conditions': [...,'status': True, 'type':'PodScheduled'],...}</code> )</li>
<li>Pod is initialized but not ready yet (<code>'node_name': 'minikube','status': {'conditions': [...,'status': True, 'type':'Initialized'], ... ,'status': False, 'type':'Ready']}</code> )</li>
</ol>

<p>Therefore, your custom scheduler should bind pod to the node on the first event, check the status of the pod and make sure it is scheduled when the second event appears, and then check if the pod is initialized when the third event appears.</p>

<p>If something goes wrong, the scheduler may need to take into account the previous errors and probably try to schedule the pod to different nodes.</p>

<p>In your case, your scheduler threats all three events like the first one and tries to schedule the pod again and again. That's why you see that "<code>pod xxx is already assigned to node yyy</code>" error.</p>
