<p>I am setting up a small Kubernetes cluster using a VM (master) and 3 bare metal servers (all running Ubuntu 14.04). I followed the Kubernetes <a href="https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/getting-started-guides/ubuntu.md" rel="noreferrer">install tutorial for Ubuntu</a>. Each bare metal server also has 2T of disk space exported using <a href="http://docs.ceph.com/docs/v0.94.5/start/" rel="noreferrer">Ceph 0.94.5</a>. Everything is working fine, but when I try to start a Replication Controller I get the following (kubectl get pods):</p>

<pre><code>NAME          READY     STATUS                                         RESTARTS   AGE
site2-zecnf   0/1       Image: site-img is ready, container is creating    0      12m
</code></pre>

<p>The pod will be in this Not Ready state forever, but, if I kill it and start it again, it will run fine (sometimes I have to repeat this operation a few times though). Once the pod is running, everything works just fine.</p>

<p>If, for some reason, the pod dies, it's restarted by Kubernetes, but can enter in this Not Ready state again. Running:</p>

<pre><code>kubectl describe pod java-site2-crctv
</code></pre>

<p>I get (some fields deleted):</p>

<pre><code>Namespace:          default
Status:             Pending
Replication Controllers:    java-site2 (1/1 replicas created)
Containers:
  java-site:
    Image:      javasite-img
    State:      Waiting
      Reason:       Image: javasite-img is ready, container is creating
    Ready:      False
    Restart Count:  0
Conditions:
  Type      Status
  Ready     False 
Events:
  FirstSeen             LastSeen            Count   From            SubobjectPath   Reason      Message
  Sat, 14 Nov 2015 12:37:56 -0200   Sat, 14 Nov 2015 12:37:56 -0200 1   {scheduler }                scheduled   Successfully assigned java-site2-crctv to 10.70.2.3
  Sat, 14 Nov 2015 12:37:57 -0200   Sat, 14 Nov 2015 12:45:29 -0200 46  {kubelet 10.70.2.3}         failedMount Unable to mount volumes for pod "java-site2-crctv_default": exit status 22
  Sat, 14 Nov 2015 12:37:57 -0200   Sat, 14 Nov 2015 12:45:29 -0200 46  {kubelet 10.70.2.3}         failedSync  Error syncing pod, skipping: exit status 22
</code></pre>

<p>The pod cannot mount the volume. But, if I mount the volumes (rdb blocks) by hand in a local folder in all nodes, the problem is gone (pods start without problems). </p>

<p>It seems to me that Kubernetes isn't able to map them (<code>sudo rbd map java-site-vol</code>), only to mount them (<code>sudo mount /dev/rbd/rbd/java-site-vol /...</code>).</p>

<p>Should I map all Ceph volumes that I use or should Kubernetes do that?   </p>

<p>I finally solved the problem. In the yaml files describing the Replication Controllers, I was using <code>keyring:</code> in the volume section:</p>

<pre><code>keyring: "ceph.client.admin.keyring" 
</code></pre>

<p>After I <a href="https://ceph.com/planet/bring-persistent-storage-for-your-containers-with-krbd-on-kubernetes/" rel="noreferrer">generated a Ceph secret</a> and changed the yaml files to use <code>secretRef</code>:</p>

<pre><code>secretRef:
  name: "ceph-secret"
</code></pre>

<p>Kubernetes was able to map and mount the Ceph volumes and the pods began to start normally. I don't know why using <code>keyring:</code> doesn't work in this case.</p>
