<p>I am setting up a small Kubernetes cluster using a VM (master) and 3 bare metal servers (all running Ubuntu 14.04). I followed the Kubernetes <a href="https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/getting-started-guides/ubuntu.md" rel="nofollow">install tutorial for Ubuntu</a>. Each bare metal server also has 2T of disk space exported using <a href="http://docs.ceph.com/docs/v0.94.5/start/" rel="nofollow">Ceph 0.94.5</a>. Everything was working fine, but when one node failed to start (it wasn't able to mount a partition) the only service the cluster was providing also stopped working. I run some commands:</p>

<pre><code>$ kubectl get nodes
NAME        LABELS                             STATUS
10.70.2.1   kubernetes.io/hostname=10.70.2.1   Ready,SchedulingDisabled
10.70.2.2   kubernetes.io/hostname=10.70.2.2   Ready
10.70.2.3   kubernetes.io/hostname=10.70.2.3   NotReady
10.70.2.4   kubernetes.io/hostname=10.70.2.4   Ready
</code></pre>

<p>It just showed that I had a node down.</p>

<pre><code>$ kubectl get pods
NAME               READY     STATUS    RESTARTS   AGE
java-mysql-5v7iu   1/1       Running   1          5d
java-site-vboaq    1/1       Running   0          4d

$ kubectl get services
NAME         LABELS                                    SELECTOR          IP(S)          PORT(S)
java-mysql   name=java-mysql                           name=java-mysql   ***.***.3.12   3306/TCP
java-site    name=java-site                            name=java-site    ***.***.3.11   80/TCP
kubernetes   component=apiserver,provider=kubernetes   &lt;none&gt;            ***.***.3.1    443/TCP
</code></pre>

<p>It showed all pods and services working fine. However, I could not connect to one of the pods (<code>java-site-vboaq</code>):</p>

<pre><code>$ kubectl exec java-site-vboaq -i -t -- bash
error: Error executing remote command: Error executing command in container: container not found ("java-site")
</code></pre>

<p>But, the pods weren't even running on the downed node:</p>

<pre><code>$ kubectl describe pod java-mysql-5v7iu
Image(s):           mysql:5
Node:               10.70.2.2/10.70.2.2
Status:             Running

$ kubectl describe pod java-site-vboaq
Image(s):           javasite-img
Node:               10.70.2.2/10.70.2.2
Status:             Running
</code></pre>

<p>After the downed node (<code>10.70.2.3</code>) was back, everything went back to normal.</p>

<p>How do I fix this problem? If a node is out, I want Kubernetes to migrate pods accordingly and keep the services working. Does it have to do with the fact that the downed node was stuck in the boot (waiting for a partition to mount) and not 100% down? </p>

<p>A few potential problems here:
1) Ceph needs its nodes to be up and running to be accessible: did you say the nodes were mounting disk from a different Ceph cluster, or is the Ceph cluster running on the same nodes? If the same nodes, then it makes sense that the drive not being accessible paralyzes K8s.</p>

<p>2) There is a bug (at least it was there on 1.0.6 and not sure if it was resolved) about pods not being able to start when trying to mount a disk that was already mounted on a different node, because it was never unmounted. This is a K8s issue to follow (sorry can't seem to find the link right now)</p>

<p>3) etcd may also get stuck waiting for node 3 if it only has 2 nodes, as it needs a majority to elect a master.</p>
