<p>I used calico as CNI in my k8s, I'm trying to deploy a single master cluster in 3 servers. I'm using <code>kubeadm</code>, follow the official <a href="https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/" rel="nofollow noreferrer">setup guide</a>. But some error occurred, <code>kube-controller-manager</code> and <code>kube-scheduler</code> go in CrashLoopBackOff error and cannot run well.</p>

<p>I have tried <code>kubeadm reset</code> at every server, and also restart the server, downgrade docker.</p>

<p>I use <code>kubeadm init --apiserver-advertise-address=192.168.213.128 --pod-network-cidr=192.168.0.0/16</code> to init the master, and run <code>kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml</code> and <code>kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml</code> to start calico.</p>

<pre class="lang-sh prettyprint-override"><code>[root@k8s-master ~]# docker info
Containers: 20
 Running: 18
 Paused: 0
 Stopped: 2
Images: 10
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.12.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 972.6MiB
Name: k8s-master
ID: RN6I:PP52:4WTU:UP7E:T3LF:MXVZ:EDBX:RSII:BIRW:36O2:CYJ3:FRV2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://i70c3eqq.mirror.aliyuncs.com/
 https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: false
Product License: Community Engine
</code></pre>

<pre class="lang-sh prettyprint-override"><code>[root@k8s-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master ~]# kubeadm version
kubeadm version: &amp;version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:08:49Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master ~]# kubelet --version
Kubernetes v1.14.1
</code></pre>

<pre class="lang-sh prettyprint-override"><code>[root@k8s-master ~]# kubectl get no -A
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   49m   v1.14.1
</code></pre>

<pre class="lang-sh prettyprint-override"><code>[root@k8s-master ~]# kubectl get pods -A
NAMESPACE     NAME                                 READY   STATUS             RESTARTS   AGE
kube-system   calico-node-xmc5t                    2/2     Running            0          27m
kube-system   coredns-6765558d84-945mt             1/1     Running            0          28m
kube-system   coredns-6765558d84-xz7lw             1/1     Running            0          28m
kube-system   coredns-fb8b8dccf-z87sl              1/1     Running            0          31m
kube-system   etcd-k8s-master                      1/1     Running            0          30m
kube-system   kube-apiserver-k8s-master            1/1     Running            0          29m
kube-system   kube-controller-manager-k8s-master   0/1     CrashLoopBackOff   8          30m
kube-system   kube-proxy-wp7n9                     1/1     Running            0          31m
kube-system   kube-scheduler-k8s-master            1/1     Running            7          29m

</code></pre>

<pre class="lang-sh prettyprint-override"><code>[root@k8s-master ~]# kubectl logs -n kube-system kube-controller-manager-k8s-master
I0513 13:49:51.836448       1 serving.go:319] Generated self-signed cert in-memory
I0513 13:49:52.988794       1 controllermanager.go:155] Version: v1.14.1
I0513 13:49:53.003873       1 secure_serving.go:116] Serving securely on 127.0.0.1:10257
I0513 13:49:53.005146       1 deprecated_insecure_serving.go:51] Serving insecurely on [::]:10252
I0513 13:49:53.008661       1 leaderelection.go:217] attempting to acquire leader lease  kube-system/kube-controller-manager...
I0513 13:50:12.687383       1 leaderelection.go:227] successfully acquired lease kube-system/kube-controller-manager
I0513 13:50:12.700344       1 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"39adc911-7582-11e9-a70e-000c2908c796", APIVersion:"v1", ResourceVersion:"1706", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s-master_fbfa0502-7585-11e9-9939-000c2908c796 became leader
I0513 13:50:13.131264       1 plugins.go:103] No cloud provider specified.
I0513 13:50:13.166088       1 controller_utils.go:1027] Waiting for caches to sync for tokens controller
I0513 13:50:13.368381       1 controllermanager.go:497] Started "podgc"
I0513 13:50:13.368666       1 gc_controller.go:76] Starting GC controller
I0513 13:50:13.368697       1 controller_utils.go:1027] Waiting for caches to sync for GC controller
I0513 13:50:13.368717       1 controller_utils.go:1034] Caches are synced for tokens controller
I0513 13:50:13.453276       1 controllermanager.go:497] Started "attachdetach"
I0513 13:50:13.453534       1 attach_detach_controller.go:323] Starting attach detach controller
I0513 13:50:13.453545       1 controller_utils.go:1027] Waiting for caches to sync for attach detach controller
I0513 13:50:13.461756       1 controllermanager.go:497] Started "clusterrole-aggregation"
I0513 13:50:13.461833       1 clusterroleaggregation_controller.go:148] Starting ClusterRoleAggregator
I0513 13:50:13.461849       1 controller_utils.go:1027] Waiting for caches to sync for ClusterRoleAggregator controller
I0513 13:50:13.517257       1 controllermanager.go:497] Started "endpoint"
I0513 13:50:13.525394       1 endpoints_controller.go:166] Starting endpoint controller
I0513 13:50:13.525425       1 controller_utils.go:1027] Waiting for caches to sync for endpoint controller
I0513 13:50:14.151371       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for rolebindings.rbac.authorization.k8s.io
I0513 13:50:14.151463       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for leases.coordination.k8s.io
I0513 13:50:14.151489       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for limitranges
I0513 13:50:14.163632       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.extensions
I0513 13:50:14.163695       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.apps
I0513 13:50:14.163721       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for ingresses.networking.k8s.io
I0513 13:50:14.163742       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for poddisruptionbudgets.policy
W0513 13:50:14.163757       1 shared_informer.go:311] resyncPeriod 67689210101997 is smaller than resyncCheckPeriod 86008177281797 and the informer has already started. Changing it to 86008177281797
I0513 13:50:14.163840       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for networkpolicies.networking.k8s.io
W0513 13:50:14.163848       1 shared_informer.go:311] resyncPeriod 64017623179979 is smaller than resyncCheckPeriod 86008177281797 and the informer has already started. Changing it to 86008177281797
I0513 13:50:14.163867       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for serviceaccounts
I0513 13:50:14.163885       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.extensions
I0513 13:50:14.163911       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for daemonsets.extensions
I0513 13:50:14.163925       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for controllerrevisions.apps
I0513 13:50:14.163942       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for roles.rbac.authorization.k8s.io
I0513 13:50:14.163965       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for podtemplates
I0513 13:50:14.163994       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for cronjobs.batch
I0513 13:50:14.164004       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for endpoints
I0513 13:50:14.164019       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.extensions
I0513 13:50:14.164030       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for replicasets.apps
I0513 13:50:14.164039       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for deployments.apps
I0513 13:50:14.164054       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for jobs.batch
I0513 13:50:14.164079       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for statefulsets.apps
I0513 13:50:14.164097       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for events.events.k8s.io
I0513 13:50:14.164115       1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for horizontalpodautoscalers.autoscaling
E0513 13:50:14.164139       1 resource_quota_controller.go:171] initial monitor sync has error: [couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "crd.projectcalico.org/v1, Resource=networkpolicies": unable to monitor quota for resource "crd.projectcalico.org/v1, Resource=networkpolicies"]
I0513 13:50:14.164154       1 controllermanager.go:497] Started "resourcequota"
I0513 13:50:14.171002       1 resource_quota_controller.go:276] Starting resource quota controller
I0513 13:50:14.171096       1 controller_utils.go:1027] Waiting for caches to sync for resource quota controller
I0513 13:50:14.171138       1 resource_quota_monitor.go:301] QuotaMonitor running
I0513 13:50:15.776814       1 controllermanager.go:497] Started "job"
I0513 13:50:15.771658       1 job_controller.go:143] Starting job controller
I0513 13:50:15.807719       1 controller_utils.go:1027] Waiting for caches to sync for job controller
I0513 13:50:23.065972       1 controllermanager.go:497] Started "csrcleaner"
I0513 13:50:23.047495       1 cleaner.go:81] Starting CSR cleaner controller
I0513 13:50:25.019036       1 event.go:209] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"39adc911-7582-11e9-a70e-000c2908c796", APIVersion:"v1", ResourceVersion:"1706", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' k8s-master_fbfa0502-7585-11e9-9939-000c2908c796 stopped leading
I0513 13:50:25.125784       1 leaderelection.go:263] failed to renew lease kube-system/kube-controller-manager: failed to tryAcquireOrRenew context deadline exceeded
F0513 13:50:25.189307       1 controllermanager.go:260] leaderelection lost

</code></pre>

<pre class="lang-sh prettyprint-override"><code>[root@k8s-master ~]# kubectl logs -n kube-system kube-scheduler-k8s-master
I0513 14:16:04.350818       1 serving.go:319] Generated self-signed cert in-memory
W0513 14:16:06.203477       1 authentication.go:387] failed to read in-cluster kubeconfig for delegated authentication: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0513 14:16:06.215933       1 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W0513 14:16:06.215947       1 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W0513 14:16:06.218951       1 authorization.go:177] failed to read in-cluster kubeconfig for delegated authorization: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0513 14:16:06.218983       1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I0513 14:16:06.961417       1 server.go:142] Version: v1.14.1
I0513 14:16:06.974064       1 defaults.go:87] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
W0513 14:16:06.997875       1 authorization.go:47] Authorization is disabled
W0513 14:16:06.997889       1 authentication.go:55] Authentication is disabled
I0513 14:16:06.997908       1 deprecated_insecure_serving.go:49] Serving healthz insecurely on [::]:10251
I0513 14:16:06.998196       1 secure_serving.go:116] Serving securely on 127.0.0.1:10259
I0513 14:16:08.872649       1 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
I0513 14:16:08.973148       1 controller_utils.go:1034] Caches are synced for scheduler controller
I0513 14:16:09.003227       1 leaderelection.go:217] attempting to acquire leader lease  kube-system/kube-scheduler...
I0513 14:16:25.814160       1 leaderelection.go:227] successfully acquired lease kube-system/kube-scheduler
</code></pre>

<p>What is the reason for <code>kube-controller-manager</code> and <code>kube-scheduler</code> going into CrashLoopBackoff? And how can I make <code>kube-controller-manager</code> and <code>kube-scheduler</code> run well?</p>

mio leon
<p>I have reproduced the steps you listed on a cloud VM and managed to make it work fine.</p>

<p>Got few ideas that might help:</p>

<ol>
<li><p>Be sure to meet <strong>all</strong> the prerequisites listed <a href="https://kubernetes.io/docs/setup/independent/install-kubeadm/" rel="nofollow noreferrer">here</a> </p></li>
<li><p>Install the most recent version of Docker following the guide from <a href="https://docs.docker.com/install/linux/docker-ce/debian/" rel="nofollow noreferrer">here</a>
(chose the proper OS that you use)</p></li>
<li><p>Install kubeadm useing the commands below:</p>

<pre><code> apt-get update &amp;&amp; apt-get install -y apt-transport-https curl
 curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
 cat &lt;&lt;EOF &gt;/etc/apt/sources.list.d/kubernetes.list
 deb https://apt.kubernetes.io/ kubernetes-xenial main
 EOF
 apt-get update
 apt-get install -y kubelet kubeadm kubectl
 apt-mark hold kubelet kubeadm kubectl
</code></pre></li>
<li><p>Make sure you got the latest version of kubeadm by executing: <code>apt-get update &amp;&amp; apt-get upgrade</code></p></li>
<li><p>Make sure you use the proper arguments alongside <code>kubeadm init</code></p></li>
<li><p>Don't forget to run:</p>

<ul>
<li><p><code>mkdir -p $HOME/.kube</code></p></li>
<li><p><code>sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config</code></p></li>
<li><p><code>sudo chown $(id -u):$(id -g) $HOME/.kube/config</code></p></li>
</ul></li>
</ol>

<p>after <code>kubeadm init</code> finishes (these commands are also part of the kubeadm init output).</p>

<ol start="7">
<li>Finally apply the.yaml files you listed in your question.</li>
</ol>

<p>Notice that by following above steps you will have <code>kubectl version</code>, <code>kubelet --version</code> and <code>kubectl get no -A</code> all in v1.14.3 and not v1.14.1 like you showed, which might be the case.</p>

<p>I hope it helps.</p>
