<p>I want to run a flink job on kubernetes, using a (persistent) state backend it seems like crashing taskmanagers are no issue as they can ask the jobmanager which checkpoint they need to recover from, if I understand correctly.</p>

<p>A crashing jobmanager seems to be a bit more difficult. On this <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077" rel="noreferrer">flip-6 page</a> I read zookeeper is needed to be able to know what checkpoint the jobmanager needs to use to recover and for leader election.</p>

<p>Seeing as kubernetes will restart the jobmanager whenever it crashes is there a way for the new jobmanager to resume the job without having to setup a zookeeper cluster?</p>

<p>The current solution we are looking at is: when kubernetes wants to kill the jobmanager (because it want to move it to another vm for example) and then create a savepoint, but this would only work for graceful shutdowns.</p>

<p>Edit:
<a href="http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-HA-with-Kubernetes-without-Zookeeper-td15033.html" rel="noreferrer">http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-HA-with-Kubernetes-without-Zookeeper-td15033.html</a> seems to be interesting but has no follow-up</p>

<p>Out of the box, Flink requires a ZooKeeper cluster to recover from JobManager crashes. However, I think you can have a lightweight implementation of the <code>HighAvailabilityServices</code>, <code>CompletedCheckpointStore</code>, <code>CheckpointIDCounter</code> and <code>SubmittedJobGraphStore</code> which can bring you quite far.</p>

<p>Given that you have only one JobManager running at all times (not entirely sure whether K8s can guarantee this) and that you have a persistent storage location, you could implement a <code>CompletedCheckpointStore</code> which retrieves the completed checkpoints from the persistent storage system (e.g. reading all stored checkpoint files). Additionally, you would have a file which contains the current checkpoint id counter for <code>CheckpointIDCounter</code> and all the submitted job graphs for the <code>SubmittedJobGraphStore</code>. So the basic idea is to store everything on a persistent volume which is accessible by the single JobManager.</p>
