<p>I'm setting up rethinkdb cluster inside kubernetes, but it doesn't work as expected for high availability requirement. Because when a pod is down, kubernetes will creates another pod, which runs another container of the same image, old mounted data (which is already persisted on host disk) will be erased and the new pod will join the cluster as a brand new instance. I'm running k8s in CoreOS v773.1.0 stable.</p>

<p>Please correct me if i'm wrong, but that way it seems impossible to setup a database cluster inside k8s.</p>

<p>Update: As documented here <a href="http://kubernetes.io/v1.0/docs/user-guide/pod-states.html#restartpolicy" rel="nofollow">http://kubernetes.io/v1.0/docs/user-guide/pod-states.html#restartpolicy</a>, if <code>RestartPolicy: Always</code> it will restart the container if exits failure. It means by "restart" that it brings up the same container, or create another one? Or maybe because I stop the pod via command <code>kubectl stop po</code> so it doesn't restart the same container?</p>

<p>That's how Kubernetes works, and other solution works probably same way. When a machine is dead, the container on it will be rescheduled to run on another machine. That other machine has no state of container. Event when   it is the same machine, the container on it is created as a new one instead of restarting the exited container(with data inside it).  </p>

<p>To persistent data, you need some kind of external storage(NFS, EBS, EFS,...). In case of k8s, you may want to look into this <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/design/persistent-storage.md" rel="nofollow">https://github.com/kubernetes/kubernetes/blob/master/docs/design/persistent-storage.md</a> This Github issue also has many information <a href="https://github.com/kubernetes/kubernetes/issues/6893" rel="nofollow">https://github.com/kubernetes/kubernetes/issues/6893</a></p>

<p>And in deed, that's the way to achieve HA in my opinion. Container are all stateless, they don't hold anything inside them. Any configuration needs for them should be store outside such as using thing like Consul or Etcd. By separating this like this, it's easier to restart a container</p>
