title: Able to get basic Rook/Ceph example to work, but all data apparently sits on a single node
body: <p>Using Rook 0.9.3 I was able to bring up a Ceph-based directory for a MySQL-database on a three-node Kubernetes cluster (1 master, two workers) simply as follows:</p>
  
  <pre><code>kubectl create -f cluster/examples/kubernetes/ceph/operator.yaml
  kubeclt create -f cluster/examples/kubernetes/ceph/cluster.yaml
  vim cluster/examples/kubernetes/ceph/storageclass.yaml # change xfs to ext4
  kubectl create -f cluster/examples/kubernetes/ceph/storageclass.yaml
  kubectl create -f cluster/examples/kubernetes/mysql.yaml
  </code></pre>
  
  <p>When I now <code>bash</code> into the pod <code>wordpress-mysql-*</code> I can see that <code>/var/lib/mysql</code> is mounted from <code>/dev/rbd1</code>. If I create a random file in this directory and then delete the pod, the file has persisted when a new instance of the pod comes up.</p>
  
  <p>My first worker contains these directories in <code>/var/lib/rook</code>: <code>mon-a</code> <code>mon-c</code> <code>mon-d</code> <code>osd0</code> <code>rook-ceph</code>. My second worker contains only one directory in <code>/var/lib/rook</code>: <code>mon-b</code>. This and other evidence (from <code>df</code>) suggest that Rook (and by extension Ceph) stores all of its file data (e.g. all blocks that constitute the mounted <code>/var/lib/mysql</code>) in <code>/var/lib/rook/osd0</code>, i.e. once on a single node. </p>
  
  <p>I would have expected that blocks are distributed across several nodes so that when one node (the first worker, in my case) fails, data access is still available. Is this a naive expectation? If not, how can I configure Rook accordingly? Also, I have second, unformatted disks on both worker nodes, and I would prefer for Rook/Ceph to use those. How can this be accomplished?</p>