<p>I am trying to deploy a Kubernetes cluster, my master node is UP and running but some pods are stuck in pending state. Below is the output of get pods.</p>

<pre><code>NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE   IP       NODE                NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-65b4876956-29tj9    0/1     Pending   0          9h    &lt;none&gt;   &lt;none&gt;              &lt;none&gt;           &lt;none&gt;
kube-system   calico-node-bf25l                           2/2     Running   2          9h    &lt;none&gt;   master-0-eccdtest   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-7d6cf57b54-b55zw                    0/1     Pending   0          9h    &lt;none&gt;   &lt;none&gt;              &lt;none&gt;           &lt;none&gt;
kube-system   coredns-7d6cf57b54-bk6j5                    0/1     Pending   0          12m   &lt;none&gt;   &lt;none&gt;              &lt;none&gt;           &lt;none&gt;
kube-system   kube-apiserver-master-0-eccdtest            1/1     Running   1          9h    &lt;none&gt;   master-0-eccdtest   &lt;none&gt;           &lt;none&gt;
kube-system   kube-controller-manager-master-0-eccdtest   1/1     Running   1          9h    &lt;none&gt;   master-0-eccdtest   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-jhfjj                            1/1     Running   1          9h    &lt;none&gt;   master-0-eccdtest   &lt;none&gt;           &lt;none&gt;
kube-system   kube-scheduler-master-0-eccdtest            1/1     Running   1          9h    &lt;none&gt;   master-0-eccdtest   &lt;none&gt;           &lt;none&gt;
kube-system   openstack-cloud-controller-manager-tlp4m    1/1     CrashLoopBackOff   114        9h    &lt;none&gt;   master-0-eccdtest   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>When I am trying to check the pod logs I am getting the below error.</p>

<pre><code>Error from server: no preferred addresses found; known addresses: []
</code></pre>

<p>Kubectl get events has lot of warnings.</p>

<pre><code>NAMESPACE     LAST SEEN   TYPE      REASON                    KIND   MESSAGE
default       23m         Normal    Starting                  Node   Starting kubelet.
default       23m         Normal    NodeHasSufficientMemory   Node   Node master-0-eccdtest status is now: NodeHasSufficientMemory
default       23m         Normal    NodeHasNoDiskPressure     Node   Node master-0-eccdtest status is now: NodeHasNoDiskPressure
default       23m         Normal    NodeHasSufficientPID      Node   Node master-0-eccdtest status is now: NodeHasSufficientPID
default       23m         Normal    NodeAllocatableEnforced   Node   Updated Node Allocatable limit across pods
default       23m         Normal    Starting                  Node   Starting kube-proxy.
default       23m         Normal    RegisteredNode            Node   Node master-0-eccdtest event: Registered Node master-0-eccdtest in Controller
kube-system   26m         Warning   FailedScheduling          Pod    0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
kube-system   3m15s       Warning   FailedScheduling          Pod    0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
kube-system   25m         Warning   DNSConfigForming          Pod    Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.96.0.10 10.51.40.100 10.51.40.103
kube-system   23m         Normal    SandboxChanged            Pod    Pod sandbox changed, it will be killed and re-created.
kube-system   23m         Normal    Pulled                    Pod    Container image "registry.eccd.local:5000/node:v3.6.1-26684321" already present on machine
kube-system   23m         Normal    Created                   Pod    Created container
kube-system   23m         Normal    Started                   Pod    Started container
kube-system   23m         Normal    Pulled                    Pod    Container image "registry.eccd.local:5000/cni:v3.6.1-26684321" already present on machine
kube-system   23m         Normal    Created                   Pod    Created container
kube-system   23m         Normal    Started                   Pod    Started container
kube-system   23m         Warning   Unhealthy                 Pod    Readiness probe failed: Threshold time for bird readiness check:  30s
calico/node is not ready: felix is not ready: Get http://localhost:9099/readiness: dial tcp [::1]:9099: connect: connection refused
kube-system   23m     Warning   Unhealthy          Pod          Liveness probe failed: Get http://localhost:9099/liveness: dial tcp [::1]:9099: connect: connection refused
kube-system   26m     Warning   FailedScheduling   Pod          0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
kube-system   3m15s   Warning   FailedScheduling   Pod          0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
kube-system   105s    Warning   FailedScheduling   Pod          0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
kube-system   26m     Warning   FailedScheduling   Pod          0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
kube-system   22m     Warning   FailedScheduling   Pod          0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
kube-system   21m     Warning   FailedScheduling   Pod          skip schedule deleting pod: kube-system/coredns-7d6cf57b54-w95g4
kube-system   21m     Normal    SuccessfulCreate   ReplicaSet   Created pod: coredns-7d6cf57b54-bk6j5
kube-system   26m     Warning   DNSConfigForming   Pod          Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.96.0.10 10.51.40.100 10.51.40.103
kube-system   23m     Normal    SandboxChanged     Pod          Pod sandbox changed, it will be killed and re-created.
kube-system   23m     Normal    Pulled             Pod          Container image "registry.eccd.local:5000/kube-apiserver:v1.13.5-1-80cc0db3" already present on machine
kube-system   23m     Normal    Created            Pod          Created container
kube-system   23m     Normal    Started            Pod          Started container
kube-system   26m     Warning   DNSConfigForming   Pod          Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.96.0.10 10.51.40.100 10.51.40.103
kube-system   23m     Normal    SandboxChanged     Pod          Pod sandbox changed, it will be killed and re-created.
kube-system   23m     Normal    Pulled             Pod          Container image "registry.eccd.local:5000/kube-controller-manager:v1.13.5-1-80cc0db3" already present on machine
kube-system   23m     Normal    Created            Pod          Created container
kube-system   23m     Normal    Started            Pod          Started container
kube-system   23m     Normal    LeaderElection     Endpoints    master-0-eccdtest_ed8f0ece-a6cd-11e9-9dd7-fa163e182aab became leader
kube-system   26m     Warning   DNSConfigForming   Pod          Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.96.0.10 10.51.40.100 10.51.40.103
kube-system   23m     Normal    SandboxChanged     Pod          Pod sandbox changed, it will be killed and re-created.
kube-system   23m     Normal    Pulled             Pod          Container image "registry.eccd.local:5000/kube-proxy:v1.13.5-1-80cc0db3" already present on machine
kube-system   23m     Normal    Created            Pod          Created container
kube-system   23m     Normal    Started            Pod          Started container
kube-system   26m     Warning   DNSConfigForming   Pod          Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.96.0.10 10.51.40.100 10.51.40.103
kube-system   23m     Normal    SandboxChanged     Pod          Pod sandbox changed, it will be killed and re-created.
kube-system   23m     Normal    Pulled             Pod          Container image "registry.eccd.local:5000/kube-scheduler:v1.13.5-1-80cc0db3" already present on machine
kube-system   23m     Normal    Created            Pod          Created container
kube-system   23m     Normal    Started            Pod          Started container
kube-system   23m     Normal    LeaderElection     Endpoints    master-0-eccdtest_ee2520c1-a6cd-11e9-96a3-fa163e182aab became leader
kube-system   26m     Warning   DNSConfigForming   Pod          Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.96.0.10 10.51.40.100 10.51.40.103
kube-system   36m     Warning   BackOff            Pod          Back-off restarting failed container
kube-system   23m     Normal    SandboxChanged     Pod          Pod sandbox changed, it will be killed and re-created.
kube-system   20m     Normal    Pulled             Pod          Container image "registry.eccd.local:5000/openstack-cloud-controller-manager:v1.14.0-1-11023d82" already present on machine
kube-system   20m     Normal    Created            Pod          Created container
kube-system   20m     Normal    Started            Pod          Started container
kube-system   3m20s   Warning   BackOff            Pod          Back-off restarting failed container
</code></pre>

<p>The only nameserver in reslov.conf is</p>

<pre><code>nameserver 10.96.0.10
</code></pre>

<p>I have extensively used google for these problems but did not get any working solution. Any suggestions would be appreciated.</p>

<p>TIA</p>

<p>Your main issue here is  <code>0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate</code> warning message.
You are getting this due to the <code>node-role.kubernetes.io/master:NoSchedule</code> and <code>node.kubernetes.io/not-ready:NoSchedule</code> <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="noreferrer">taints</a></p>

<p>This taint prevents scheduling pods on current node.</p>

<p>If you want to be able to schedule pods on the control-plane node, e.g. for a single-machine Kubernetes cluster for development, run:</p>

<pre><code>kubectl taint nodes instance-1 node-role.kubernetes.io/master-
kubectl taint nodes instance-1 node.kubernetes.io/not-ready:NoSchedule-
</code></pre>

<p>But from my POW it is better to:</p>

<p>-<a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/" rel="noreferrer">initiate cluster using kubeadm</a></p>

<p>-<a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network" rel="noreferrer">apply CNI</a></p>

<p>-<a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#join-nodes" rel="noreferrer">add new worker node</a></p>

<p>-and let all your new pods be scheduled on worker node.</p>

<pre><code>sudo kubeadm init --pod-network-cidr=192.168.0.0/16
[init] Using Kubernetes version: v1.15.0
...

Your Kubernetes control-plane has initialized successfully!

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config


$ kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.extensions/calico-node created
serviceaccount/calico-node created
deployment.extensions/calico-kube-controllers created
serviceaccount/calico-kube-controllers created


-ADD worker node using kubeadm join string on slave node

$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
instance-1   Ready    master   21m   v1.15.0
instance-2   Ready    &lt;none&gt;   34s   v1.15.0

    $ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE    IP               NODE         NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-658558ddf8-v2rqx   1/1     Running   0          11m    192.168.23.129   instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   calico-node-c2tkt                          1/1     Running   0          11m    10.132.0.36      instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   calico-node-dhc66                          1/1     Running   0          107s   10.132.0.38      instance-2   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-5c98db65d4-dqjm7                   1/1     Running   0          22m    192.168.23.130   instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   coredns-5c98db65d4-hh7vd                   1/1     Running   0          22m    192.168.23.131   instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   etcd-instance-1                            1/1     Running   0          21m    10.132.0.36      instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-apiserver-instance-1                  1/1     Running   0          21m    10.132.0.36      instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-controller-manager-instance-1         1/1     Running   0          21m    10.132.0.36      instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-qwvkq                           1/1     Running   0          107s   10.132.0.38      instance-2   &lt;none&gt;           &lt;none&gt;
kube-system   kube-proxy-s9gng                           1/1     Running   0          22m    10.132.0.36      instance-1   &lt;none&gt;           &lt;none&gt;
kube-system   kube-scheduler-instance-1                  1/1     Running   0          21m    10.132.0.36      instance-1   &lt;none&gt;           &lt;n
</code></pre>
