<p>I got Kubernetes Cluster with 1 master and 3 workers nodes.</p>
<p>calico v3.7.3 kubernetes v1.16.0 installed via kubespray <a href="https://github.com/kubernetes-sigs/kubespray" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/kubespray</a></p>
<p>Before that, I normally deployed all the pods without any problems.</p>
<p>I can't start a few pod (Ceph):</p>
<pre><code>kubectl get all --namespace=ceph
NAME                                 READY   STATUS             RESTARTS   AGE
pod/ceph-cephfs-test                 0/1     Pending            0          162m
pod/ceph-mds-665d849f4f-fzzwb        0/1     Pending            0          162m
pod/ceph-mon-744f6dc9d6-jtbgk        0/1     CrashLoopBackOff   24         162m
pod/ceph-mon-744f6dc9d6-mqwgb        0/1     CrashLoopBackOff   24         162m
pod/ceph-mon-744f6dc9d6-zthpv        0/1     CrashLoopBackOff   24         162m
pod/ceph-mon-check-6f474c97f-gjr9f   1/1     Running            0          162m


NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/ceph-mon   ClusterIP   None         &lt;none&gt;        6789/TCP   162m

NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR       AGE
daemonset.apps/ceph-osd   0         0         0       0            0           node-type=storage   162m

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ceph-mds         0/1     1            0           162m
deployment.apps/ceph-mon         0/3     3            0           162m
deployment.apps/ceph-mon-check   1/1     1            1           162m

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/ceph-mds-665d849f4f        1         1         0       162m
replicaset.apps/ceph-mon-744f6dc9d6        3         3         0       162m
replicaset.apps/ceph-mon-check-6f474c97f   1         1         1       162m

</code></pre>
<p>But another obe is ok:</p>
<pre><code>kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6d57b44787-xlj89   1/1     Running   19         24d
calico-node-dwm47                          1/1     Running   310        19d
calico-node-hhgzk                          1/1     Running   15         24d
calico-node-tk4mp                          1/1     Running   309        19d
calico-node-w7zvs                          1/1     Running   312        19d
coredns-74c9d4d795-jrxjn                   1/1     Running   0          2d23h
coredns-74c9d4d795-psf2v                   1/1     Running   2          18d
dns-autoscaler-7d95989447-7kqsn            1/1     Running   10         24d
kube-apiserver-master                      1/1     Running   4          24d
kube-controller-manager-master             1/1     Running   3          24d
kube-proxy-9bt8m                           1/1     Running   2          19d
kube-proxy-cbrcl                           1/1     Running   4          19d
kube-proxy-stj5g                           1/1     Running   0          19d
kube-proxy-zql86                           1/1     Running   0          19d
kube-scheduler-master                      1/1     Running   3          24d
kubernetes-dashboard-7c547b4c64-6skc7      1/1     Running   591        24d
nginx-proxy-worker1                        1/1     Running   2          19d
nginx-proxy-worker2                        1/1     Running   0          19d
nginx-proxy-worker3                        1/1     Running   0          19d
nodelocaldns-6t92x                         1/1     Running   2          19d
nodelocaldns-kgm4t                         1/1     Running   0          19d
nodelocaldns-xl8zg                         1/1     Running   0          19d
nodelocaldns-xwlwk                         1/1     Running   12         24d
tiller-deploy-8557598fbc-7f2w6             1/1     Running   0          131m

</code></pre>
<p>I use Centos 7:</p>
<pre><code>NAME=&quot;CentOS Linux&quot;
VERSION=&quot;7 (Core)&quot;
ID=&quot;centos&quot;
ID_LIKE=&quot;rhel fedora&quot;
VERSION_ID=&quot;7&quot;
PRETTY_NAME=&quot;CentOS Linux 7 (Core)&quot;
ANSI_COLOR=&quot;0;31&quot;
CPE_NAME=&quot;cpe:/o:centos:centos:7&quot;
HOME_URL=&quot;https://www.centos.org/&quot;
BUG_REPORT_URL=&quot;https://bugs.centos.org/&quot;

CENTOS_MANTISBT_PROJECT=&quot;CentOS-7&quot;
CENTOS_MANTISBT_PROJECT_VERSION=&quot;7&quot;
REDHAT_SUPPORT_PRODUCT=&quot;centos&quot;
REDHAT_SUPPORT_PRODUCT_VERSION=&quot;7&quot;
</code></pre>
<p>The error log:</p>
<pre><code>Get https://10.2.67.203:10250/containerLogs/ceph/ceph-mon-744f6dc9d6-mqwgb/ceph-mon?tailLines=5000&amp;timestamps=true: dial tcp 10.2.67.203:10250: connect: no route to host
</code></pre>
<p>Maybe someone came across this and can help me? I will provide any additional information</p>
<p>logs from pending pods:</p>
<p>Warning  FailedScheduling  98s (x125 over 3h1m)  default-scheduler  0/4 nodes are available: 4 node(s) didn't match node selector.</p>

<p>It seems that a firewall is blocking ingress traffic from port <code>10250</code> on the <code>10.2.67.203</code> node.</p>
<p>You can open it by running the commands below (I'm assuming <a href="https://firewalld.org/" rel="noreferrer">firewalld</a> is installed or you can run the commands of the equivalent firewall module):</p>
<pre><code>sudo firewall-cmd --add-port=10250/tcp --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --list-all  # you should see that port `10250` is updated
</code></pre>
