<p>We are using client-go to create kubernetes jobs and deployments. Today in one of our cluster (kubernetes v1.18.19), I encounter below weird problem.</p>
<p>Pods of kubernetes Job are always stuck in Pending status, without any reasons. <code>kubectl describe pod</code> shows there are no events. Creating Jobs from host (via kubectl) are normal and pods became running eventually.</p>
<p>What surprises me is Creating Deployments is ok, pods get running eventually!! It won't work only for Kubernetes Jobs. Why? How to fix that?? What I can do?? I have taken hours here but got no progress.</p>
<p>kubeconfig by client-go:</p>
<pre><code>Mount from host machine, path: /root/.kube/config
</code></pre>
<p>kubectl describe job shows:</p>
<pre><code>Name:           unittest
Namespace:      default
Selector:       controller-uid=f3cec901-c0f4-4098-86d7-f9a7d1fe6cd1
Labels:         job-id=unittest
Annotations:    &lt;none&gt;
Parallelism:    1
Completions:    1
Start Time:     Sat, 19 Jun 2021 00:20:12 +0800
Pods Statuses:  1 Running / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  controller-uid=f3cec901-c0f4-4098-86d7-f9a7d1fe6cd1
           job-name=unittest
  Containers:
   unittest:
    Image:      ubuntu:18.04
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      echo hello
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
  Volumes:        &lt;none&gt;
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  21m   job-controller  Created pod: unittest-tt5b2
</code></pre>
<p>Kubectl describe on target pod shows:</p>
<pre><code>Name:           unittest-tt5b2
Namespace:      default
Priority:       0
Node:           &lt;none&gt;
Labels:         controller-uid=f3cec901-c0f4-4098-86d7-f9a7d1fe6cd1
               job-name=unittest
Annotations:    &lt;none&gt;
Status:         Pending
IP:             
IPs:            &lt;none&gt;
Controlled By:  Job/unittest
Containers:
 unittest:
   Image:      ubuntu:18.04
   Port:       &lt;none&gt;
   Host Port:  &lt;none&gt;
   Command:
     echo hello
   Environment:  &lt;none&gt;
   Mounts:
     /var/run/secrets/kubernetes.io/serviceaccount from default-token-72g27 (ro)
Volumes:
 default-token-72g27:
   Type:        Secret (a volume populated by a Secret)
   SecretName:  default-token-72g27
   Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt; 
</code></pre>
<p>kubectl get events shows:</p>
<pre><code>55m         Normal    ScalingReplicaSet   deployment/job-scheduler              Scaled up replica set job-scheduler-76b7465d74 to 1
19m         Normal    ScalingReplicaSet   deployment/job-scheduler              Scaled up replica set job-scheduler-74f8896f48 to 1
58m         Normal    SuccessfulCreate    job/unittest                          Created pod: unittest-pp665
49m         Normal    SuccessfulCreate    job/unittest                          Created pod: unittest-xm6ck
17m         Normal    SuccessfulCreate    job/unittest                          Created pod: unittest-tt5b2
</code></pre>

<p>I fixed the issue.</p>
<p>We use a custom scheduler for NPU devices and default scheduler for GPU devices. For GPU devices, the scheduler name is &quot;default-scheduler&quot; other than &quot;default&quot;. I passed &quot;default&quot; for those kube Jobs, this causes the pods to stuck in pending.</p>
