<p>When I trying to follow the <a href="https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/" rel="nofollow noreferrer">Kubernetes Configure Multiple Schedulers guide</a>, I cannot successfully get the my-scheduler pod running.</p>

<p>I succeed to build the image and push it to a registry, and the image shows in my docker and Google Could Platform.
But when I apply the deployment file to set the pods, the status is <code>CrashLoopBackOff</code>.</p>

<pre class="lang-sh prettyprint-override"><code>my-scheduler-64576b7897-8b549            0/1     CrashLoopBackOff   14         84m
</code></pre>

<p>Here is the output for <code>kuberctl describe pod my-scheduler-64576b7897-8b549 -n kube-system</code>:</p>

<pre class="lang-sh prettyprint-override"><code>Name:               my-scheduler-64576b7897-8b549
Namespace:          kube-system
Priority:           0
PriorityClassName:  &lt;none&gt;
Node:               minikube/10.0.2.15
Start Time:         Tue, 07 May 2019 15:06:59 -0500
Labels:             component=scheduler
                    pod-template-hash=64576b7897
                    tier=control-plane
                    version=second
Annotations:        &lt;none&gt;
Status:             Running
IP:                 172.17.0.15
Controlled By:      ReplicaSet/my-scheduler-64576b7897
Containers:
  kube-second-scheduler:
    Container ID:  docker://a0cadf03dd5ffd11cbb6f1f8e669fc04cd6aa00aa6f361550f06f32785a4d63c
    Image:         gcr.io/my-gcp-project-239900/my-kube-scheduler:1.0
    Image ID:      docker-pullable://gcr.io/my-gcp-project-239900/my-kube-scheduler@sha256:0895687b7646d159438f086e8f8b23b253332930e169cf29d8b9951ebf3ba653
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Command:
      /usr/local/bin/kube-scheduler
      --address=1.1.1.1
      --leader-elect=false
      --scheduler-name=my-scheduler
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 07 May 2019 15:07:45 -0500
      Finished:     Tue, 07 May 2019 15:07:45 -0500
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 07 May 2019 15:07:24 -0500
      Finished:     Tue, 07 May 2019 15:07:24 -0500
    Ready:          False
    Restart Count:  3
    Requests:
      cpu:        100m
    Liveness:     http-get http://:10251/healthz delay=15s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:10251/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from my-scheduler-token-lzwvk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  my-scheduler-token-lzwvk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-scheduler-token-lzwvk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  48s               default-scheduler  Successfully assigned kube-system/my-scheduler-64576b7897-8b549 to minikube
  Normal   Pulled     2s (x4 over 47s)  kubelet, minikube  Container image "gcr.io/my-gcp-project-239900/my-kube-scheduler:1.0" already present on machine
  Normal   Created    2s (x4 over 47s)  kubelet, minikube  Created container kube-second-scheduler
  Normal   Started    2s (x4 over 47s)  kubelet, minikube  Started container kube-second-scheduler
  Warning  BackOff    1s (x6 over 45s)  kubelet, minikube  Back-off restarting failed container
</code></pre>

<p>And when I try to see the log of this pod, this error message appears:</p>

<pre class="lang-sh prettyprint-override"><code> standard_init_linux.go:207: exec user process caused "exec format error"
</code></pre>

<p>And my <code>Dockerfile</code> is bit different from the guide, it looks like:</p>

<pre><code>FROM busybox
ADD ./_output/local/bin/darwin/amd64/kube-scheduler /usr/local/bin/kube-scheduler
</code></pre>

<p>I changed</p>

<pre><code>ADD ./_output/dockerized/bin/linux/amd64/kube-scheduler /usr/local/bin/kube-scheduler
</code></pre>

<p>into</p>

<pre><code>ADD ./_output/local/bin/darwin/amd64/kube-scheduler /usr/local/bin/kube-scheduler
</code></pre>

<p>because <code>./_output/local/bin/darwin/amd64/kube-scheduler</code>is where the scheduler file acctually located. I would also appreciate it if anyone could tell me why would this happen.</p>

<hr>

<p><strong>Some Updates</strong></p>

<h2>Different file location</h2>

<p>Following the guide, I typed these command in my shell</p>

<pre class="lang-sh prettyprint-override"><code>git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
make
</code></pre>

<p>by executing <code>make</code>, the <code>Makefile</code>file will generate the kubernetes exec files in certain path (including the kube-scheduler we want). And the generation path is related to your system. Since I'm using <strong>minikube</strong> to run kubernetes and docker, and it runs on <strong>darwin(amd64)</strong>. I think that's the reason why my path is different from the guide. </p>

<h2>The "exec format error"</h2>

<p>For this error</p>

<pre class="lang-sh prettyprint-override"><code> standard_init_linux.go:207: exec user process caused "exec format error"
</code></pre>

<p>I find <a href="https://github.com/moby/moby/issues/23865" rel="nofollow noreferrer">this issue</a> suggesting add the <code>#!/bin/bash</code> before you try to run commands in the container. This tells the system to treat the commands as bash commands. Here in busybox it should be <code>/bin/sh</code>. (I don't know much about shell language mechanism, so correct me if I'm wrong)</p>

<p>So I changed these lines in <code>my-sheduler.yaml</code>:</p>

<pre><code>- command:
 - /usr/local/bin/kube-scheduler
 - --address=0.0.0.0
 - --leader-elect=false
 - --scheduler-name=my-scheduler
</code></pre>

<p>into</p>

<pre><code>- command:
 - /bin/sh
 - /usr/local/bin/kube-scheduler
 - --address=0.0.0.0
 - --leader-elect=false
 - --scheduler-name=my-schedule
</code></pre>

<p>Now the <code>my-sheduler</code> pods still fail and get the <code>CrashLoopBackOff</code> error. 
Good news is I now have the log when executing <code>kubectl logs</code>.
Bad news is the logs are unreadable like this</p>

<pre class="lang-sh prettyprint-override"><code>/usr/local/bin/kube-scheduler: line 1: ����: not found
/usr/local/bin/kube-scheduler: line 2: syntax error: unexpected ")"
</code></pre>

<p>I assume this happened because of something happened in the docker image. But since I'm using the scrips right from the <a href="https://github.com/kubernetes/kubernetes" rel="nofollow noreferrer">kubernetes github page</a>, there shouldn't be any bug.</p>

<p>The same log shows up when I run <code>docker run -it {my-image-id} /bin/sh</code> and go into my image and run the exec kube-scheduler file.</p>

<p>I can't figure it out what's in the GO script causes this problem. </p>

<hr>

<h1>Another alternative</h1>

<p>I ended up trying another way to build my custom scheduler.</p>

<p>I followed this tutorial: <a href="https://www.youtube.com/watch?v=IYcL0Un1io0&amp;t=1807s" rel="nofollow noreferrer">https://www.youtube.com/watch?v=IYcL0Un1io0&amp;t=1807s</a></p>

<p>And you can find the source code here: <a href="https://github.com/kelseyhightower/scheduler" rel="nofollow noreferrer">https://github.com/kelseyhightower/scheduler</a></p>

<p>Comparing to the official guide, this one is more clear light (in in official guide you have to build the whole thing to get the scheduler file). </p>

<p>Hope this helps if someone is running into the same problem. </p>

Brenda Wang
<p>This totally looks like that you are using the <code>darwin</code> <code>kube-scheduler</code> go binary to run in a <a href="https://hub.docker.com/_/busybox" rel="nofollow noreferrer">Linux container</a>. </p>

<p>You probably ran this on your mac:</p>

<pre><code>git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
make
</code></pre>

<p>Which created the <code>darwin</code> binaries and not the Linux binaries. I would try to run that from a Linux box. Or change the make target to use <a href="https://gist.github.com/asukakenji/f15ba7e588ac42795f421b48b8aede63" rel="nofollow noreferrer"><code>GOOS=linux</code> and <code>GOARCH=amd64</code></a>.</p>

<p>Note that K8s mainly supports Linux and only Windows nodes as of this writing.</p>
