<h2><strong>Background</strong></h2>
<p>I had planned to go with Service Fabric (on premises) for my service and container orchestration.  But, due to internal discussions, I am giving Kubernetes a look.  Mostly because it is so very popular.</p>
<p>Service Fabric has concepts called Upgrade Domains and Failure Domains.  A &quot;domain&quot; is a grouping of host nodes.</p>
<p>Upgrade Domains are used when pushing out an application service or container update.  Service Fabric makes sure that the upgrading service/container is still available by only taking down one Upgrade Domain at at a time.  (These are also used when updating the Service Fabric cluster software itself.)</p>
<p>Failure Domains work in a similar way.  The idea is that the Failure Domains are created in alignment with hardware failure groups.  Service Fabric makes sure that there are service/container instances running in each failure domain. (To allow for up time during a hardware failure.)</p>
<h2><strong>Question</strong></h2>
<p>As I look at docs and listen to podcasts on Kubernetes I don't see any of these concepts.  It seems it just hosts containers (Pods).  I have heard a bit about &quot;scheduling&quot; and &quot;tags&quot;.  But it seems it is just the way to manually configure pods.</p>
<p><strong>Are application upgrades and failure tolerance things that are done manually in Kubernetes? (via scheduling and/or tags perhaps)</strong></p>
<p>Or is there a feature I am missing?</p>

<blockquote>
  <p>A "domain" is a grouping of host nodes.</p>
</blockquote>

<p>Is not that simple, it would be more accurate if you said <strong><em>"A 'domain' is a logical grouping of resources".</em></strong></p>

<p>To understand it correctly, you have to first understand most components in isolation. I recommend these readings first:</p>

<ul>
<li>Virtual Machines domains in <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/windows/manage-availability" rel="nofollow noreferrer">this docs</a>.</li>
<li>Service Fabric domains in <a href="https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-resource-manager-cluster-description#fault-domains" rel="nofollow noreferrer">this docs</a></li>
</ul>

<p><strong>Then, we can take some points out of it:</strong></p>

<ul>
<li><p>Nodes are not Virtual Machines, Nodes runs on top of azure virtual machines. </p>

<p>They often have a 1:1 mapping, but in some cases you can have 5:1 node/VM <em>mapping, an example is when you install a local development cluster.</em></p></li>
<li><p>Azure Virtual machines has Update Domains and Fault Domains, Service fabric nodes has Upgrade Domains and Fault Domains</p>

<p>As much they look the same, they have their differences:</p>

<p><strong>Fault Domains:</strong>   </p>

<ul>
<li>VM Fault Domains are isolated slots for physical deployment, that means: Power Supply, Network, Disks, and so on... They are limited by region.</li>
<li>SF Fault Domains are logical node slots for application deployment, that means, when SF deploy an application to nodes it will distribute on different fault domains, to make them reliable, most of the time FD will be mapped to VM Fault Domains, but in complex scenarios, you can map this to anything, for example, you could map an entire region to single SF FD.</li>
</ul>

<p>.</p>

<p><strong>Update\Upgrade Domains:</strong></p>

<ul>
<li>VM Update Domains are about OS\Hardware patches and updates, separate update domains will be handled in isolation and not updated at the same time, so that, when an OS update is required to bring down your VM, they will update domain by domain. Lower number of update domains means more machines will be put down during updates.</li>
<li>SF Upgrade Domains use a similar approach of VM update domains, but focused on the services and the cluster upgrade itself, bringing down UD per UD at a time and moving to the next one when the previous UD succeed.</li>
<li>In both cases, you adjust the Update\Upgrade to how many instances (%) of your vm\nodes\services can be put down during upgrades. So for example, if your service has 100 instances on a cluster with 5 UD, SF will update 20 services at time, if you increase the number of UD to 10 for example, the number of instances down will reduce to 10 instances at time, but the time to deploy your application will increase at same proportion.</li>
</ul></li>
</ul>

<p>Based on that, you can see FD &amp; UP as a matrix of reliable deployment slots, as much as you have, more the reliability will increase (with trade offs, like the update time required). Example below taken from SF docs:</p>

<p><a href="https://i.stack.imgur.com/Aa7LK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Aa7LK.png" alt="enter image description here"></a></p>

<p>Service Fabric, out of the box tries to place your services instances on different FD\UD on best effort, that means, if possible they will be on different FD\UD otherwise it will find another one with least number of instances of the service being deployed.</p>

<p><strong>And about Kubernetes:</strong></p>

<p>On Kubernetes, these features are not out of the box, k8s have the concept of <a href="https://kubernetes.io/docs/admin/multiple-zones/" rel="nofollow noreferrer">zones</a>, but according to the docs, they are limited by regions, they cannot span across regions. </p>

<blockquote>
  <p>Kubernetes will automatically spread the pods in a replication controller or service across nodes in a single-zone cluster (to reduce the impact of failures). With multiple-zone clusters, this spreading behaviour is extended across zones (to reduce the impact of zone failures). This is achieved via SelectorSpreadPriority.</p>
  
  <p>This is a best-effort placement, and so if the zones in your cluster are heterogeneous (e.g. different numbers of nodes, different types of nodes, or different pod resource requirements), this might prevent equal spreading of your pods across zones. If desired, you can use homogenous zones (same number and types of nodes) to reduce the probability of unequal spreading.</p>
</blockquote>

<p>Is not the same as as FD but is a very similar concept.</p>

<p>To achieve a similar result as SF, will be required to deploy your cluster across zones or map the nodes to VM FD\UD, so that they behave as nodes on SF. Add the labels to the nodes to identify these domains. You also would need to create NodeType labels on the nodes over different FD, so that you can use for deploying your pods on delimited nodes.</p>

<p>For example:</p>

<ul>
<li><strong>Node01</strong>: FD01 : NodeType=<strong>FrontEnd</strong> </li>
<li><strong>Node02</strong>: FD02 : NodeType=<strong>FrontEnd</strong> </li>
<li><strong>Node03</strong>: FD03 : NodeType=<strong>FrontEnd</strong> </li>
<li><strong>Node04</strong>: FD01 : NodeType=BackEnd </li>
<li><strong>Node05</strong>: FD02 : NodeType=BackEnd </li>
</ul>

<p>When you deploy you application, you should make use of the <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/" rel="nofollow noreferrer">affinity</a> feature to assign PODs to a node, and in this case your service would have:</p>

<ul>
<li><strong>Required Affinity</strong> to a <em>NodeType=FrontEnd</em></li>
<li><strong>Prefered AntiAfinity</strong> to <em>ContainerName=[itself]</em></li>
</ul>

<p>With these settings, using  of the affinity and anti-affinity k8s will try to place replicas\instances of your container on separate nodes, and the nodes will be already separate by FD\zone delimited by the NoteType labels, then k8s will handle the rolling updates as SF does. </p>

<p>Because the anti-affinity rules are prefered, k8s will try to balance across these nodes on best effort, if no valid nodes are available it will start adding more instances on node that already contains instances of same container, </p>

<p><strong>Conclusion</strong></p>

<p>It is a bit of extra work, but not much different to what is currently used on other solutions. 
The major concern here will be configuring the Nodes on FD\Zones, after you place your nodes on the right FD, the rest will work smoothly. </p>

<p>On SF you don't have to worry about this when you deploy a cluster on Azure, but if you do it from scratch, is a big work to do, even bigger than k8s. </p>

<p>NOTE: If you use <a href="https://learn.microsoft.com/en-us/azure/aks/" rel="nofollow noreferrer">AKS</a>, it will distribute the nodes across availability sets (set which specifies VM fault domains and update domains). Currently, according to <a href="https://stackoverflow.com/questions/49747221/default-number-of-availability-zones-for-an-aks-kubernetes-cluster">this post</a>, AKS does not provide the zone distribution for you, so you would have to do it from scratch if you need it this level of distribution.</p>
