<p>We have configured Kubernetes cluster on EC2 machines in our AWS account using kops tool (<a href="https://github.com/kubernetes/kops" rel="nofollow noreferrer">https://github.com/kubernetes/kops</a>) and based on AWS posts (<a href="https://aws.amazon.com/blogs/compute/kubernetes-clusters-aws-kops/" rel="nofollow noreferrer">https://aws.amazon.com/blogs/compute/kubernetes-clusters-aws-kops/</a>) as well as other resources. </p>

<p>We want to setup a K8s cluster of master and slaves  such that:</p>

<ol>
<li>It will automatically resize (both masters as well as nodes/slaves) based on system load.</li>
<li>Runs in Multi-AZ mode i.e. at least one master and one slave in every AZ (availability zone) in the same region for e.g. us-east-1a, us-east-1b, us-east-1c and so on. </li>
</ol>

<p>We tried to configure the cluster in the following ways to achieve the above.</p>

<ol>
<li><p>Created K8s cluster on AWS EC2 machines using kops this below configuration: node count=3, master count=3, zones=us-east-1c, us-east-1b, us-east-1a. We observed that a K8s cluster was created with 3 Master &amp; 3 Slave Nodes. Each of the master and slave server was in each of the 3 AZ’s.</p></li>
<li><p>Then we tried to resize the Nodes/slaves in the cluster using (<a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml</a>). We set the node_asg_min to 3 and node_asg_max to 5. When we increased the workload on the slaves such that auto scale policy was triggered, we saw that additional (after the default 3 created during setup) slave nodes were spawned, and they did join the cluster in various AZ’s. This worked as expected. There is no question here.</p></li>
<li><p>We also wanted to set up the cluster such that the number of masters increases based on system load. Is there some way to achieve this? We tried a couple of approaches and results are shared below:</p></li>
</ol>

<p>A) We were not sure if the cluster-auto scaler helps here, but nevertheless tried to resize the Masters in the cluster using (<a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-master.yaml</a>). This is useful while creating a new cluster but was not useful to resize the number of masters in an existing cluster. We did not find a parameter to specify node_asg_min, node_asg_max for Master the way it is present for slave Nodes. Is there some way to achieve this?</p>

<p>B) We increased the count MIN from 1 to 3 in ASG (auto-scaling group), associated with one the three IG (instance group) for each master. We found that new instances were created. However, they did not join the master cluster. Is there some way to achieve this?</p>

<p>Could you please point us to steps, resources on how to do this correctly so that we could configure the number of masters to automatically resize based on system load and is in Multi-AZ mode?</p>

<p>Kind regards,
Shashi</p>

<p>There is no need to scale <code>Master</code> nodes.</p>
<blockquote>
<p>Master components provide the cluster’s control plane. Master components make global decisions about the cluster (for example, scheduling), and detecting and responding to cluster events (starting up a new pod when a replication controller’s ‘replicas’ field is unsatisfied).</p>
<p>Master components can be run on any machine in the cluster. However, for simplicity, set up scripts typically start all master components on the same machine, and do not run user containers on this machine. See  <a href="https://kubernetes.io/docs/admin/high-availability/" rel="nofollow noreferrer">Building High-Availability Clusters</a>  for an example multi-master-VM setup.</p>
</blockquote>
<h2>Master node consists of the following components:</h2>
<p><strong>kube-apiserver</strong></p>
<blockquote>
<p>Component on the master that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane.</p>
</blockquote>
<p><strong>etcd</strong></p>
<blockquote>
<p>Consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.</p>
</blockquote>
<p><strong>kube-scheduler</strong></p>
<blockquote>
<p>Component on the master that watches newly created pods that have no node assigned, and selects a node for them to run on.</p>
</blockquote>
<p><strong>kube-controller-manager</strong></p>
<blockquote>
<p>Component on the master that runs  <a href="https://kubernetes.io/docs/admin/kube-controller-manager/" rel="nofollow noreferrer">controllers</a>.</p>
</blockquote>
<p><strong>cloud-controller-manager</strong></p>
<blockquote>
<p>runs controllers that interact with the underlying cloud providers. The cloud-controller-manager binary is an alpha feature introduced in Kubernetes release 1.6.</p>
</blockquote>
<p>For more detailed explanation please read the <a href="https://kubernetes.io/docs/concepts/overview/components/" rel="nofollow noreferrer">Kubernetes Components</a> docs.
Also if You are thinking about HA, you can read about <a href="https://kubernetes.io/docs/setup/independent/high-availability/" rel="nofollow noreferrer">Creating Highly Available Clusters with kubeadm</a></p>
