<p>What exactly are the practical consequences of missing consensus on a Kubernetes cluster? Or in other words: which functions on a Kubernetes cluster require consensus? What will work, what won't work? </p>

<p>For example (and really only for example):</p>

<ul>
<li>will existing pods keep running?</li>
<li>can pods still be scaled horizontally?</li>
</ul>

<p>Example scenario: A cluster with two nodes loses one node. No consensus possible.</p>

<p>Consensus is fundamental to etcd - the distributed database that Kubernetes is built upon. Without consensus you can <em>read</em> but not <em>write</em> from the database. E.g. if only 1 of 3 nodes is available.</p>

<blockquote>
  <p>When you lose quorum etcd goes into a <strong>read only</strong> state where it can respond with data, but no new actions can take place since it will be unable to decide if the action is allowed.</p>
</blockquote>

<p><a href="https://blog.containership.io/etcd/" rel="nofollow noreferrer">Understanding Etcd Consensus and How to Recover from Failure
</a></p>

<p>Kubernetes is designed so pods only need kubernetes for changes, e.g. deployment. After that they run independent of kubernetes in a loosely coupled fashion.</p>

<p>Kubernetes is contstructed for keeping <em>desired state</em> in the etcd database. Then controllers watch etcd for changes and act upon change. This means that you can not scale or change any configuration of pods if etcd doesn't have consensus. Kubernetes does many self-healing operations, but they will not work if etcd is not available since all operations is done through the ApiServer and etcd.</p>

<blockquote>
  <p>Loosing quorum means that <strong>no new actions</strong> can take place. Everything that is running will continue to run until there is a failure.</p>
</blockquote>

<p><a href="https://www.youtube.com/watch?v=n9VKAKwBj_0" rel="nofollow noreferrer">Understanding Distributed Consensus in etcd and Kubernetes</a></p>
