<p>Using Rook 0.9.3 I was able to bring up a Ceph-based directory for a MySQL-database on a three-node Kubernetes cluster (1 master, two workers) simply as follows:</p>

<pre><code>kubectl create -f cluster/examples/kubernetes/ceph/operator.yaml
kubeclt create -f cluster/examples/kubernetes/ceph/cluster.yaml
vim cluster/examples/kubernetes/ceph/storageclass.yaml # change xfs to ext4
kubectl create -f cluster/examples/kubernetes/ceph/storageclass.yaml
kubectl create -f cluster/examples/kubernetes/mysql.yaml
</code></pre>

<p>When I now <code>bash</code> into the pod <code>wordpress-mysql-*</code> I can see that <code>/var/lib/mysql</code> is mounted from <code>/dev/rbd1</code>. If I create a random file in this directory and then delete the pod, the file has persisted when a new instance of the pod comes up.</p>

<p>My first worker contains these directories in <code>/var/lib/rook</code>: <code>mon-a</code> <code>mon-c</code> <code>mon-d</code> <code>osd0</code> <code>rook-ceph</code>. My second worker contains only one directory in <code>/var/lib/rook</code>: <code>mon-b</code>. This and other evidence (from <code>df</code>) suggest that Rook (and by extension Ceph) stores all of its file data (e.g. all blocks that constitute the mounted <code>/var/lib/mysql</code>) in <code>/var/lib/rook/osd0</code>, i.e. once on a single node. </p>

<p>I would have expected that blocks are distributed across several nodes so that when one node (the first worker, in my case) fails, data access is still available. Is this a naive expectation? If not, how can I configure Rook accordingly? Also, I have second, unformatted disks on both worker nodes, and I would prefer for Rook/Ceph to use those. How can this be accomplished?</p>

<p>for using other partition as osd 
you should change cluster.yml and add</p>

<pre><code>nodes:
- name: "kube-node1"
  devices:
  - name: "sdb"
- name: "kube-node2"
  devices:
  - name: "sdb"
- name: "kube-node3"
  devices:
  - name: "sdb"
</code></pre>
