<p>I'm experimenting Kops on AWS. My cluster is composed by 1 master node and 3 worker nodes. All works fine and in order to test a Master node failure i terminated the corresponding EC2 instance and of course the AutoScaling group handled that problem and created a new instance and it became the the new Master node. So is fine.</p>

<p>My question is how the AutoScaling group did to configure the new EC2 instance to properly be configured as a Master Kubernetes node ? Is there any predefined AMI created when i setup KOPS ? Or is there any user data script that is launched each time a new instance is created ?</p>

<p>Thanks.</p>

<p>This is because kops has the concept of <a href="https://github.com/kubernetes/kops/blob/master/docs/instance_groups.md" rel="nofollow noreferrer">instance groups</a>. On AWS they are directly mapped to AutoScalingGroup - which is a similar concept. You can check your instance groups by running <code>kops get ig</code> and also you can edit and descale your master and nodes to 0 and then relaunch them by <code>kops edit ig nodes/nameofthemaster</code>. The second part is kops <a href="https://github.com/kubernetes/kops/blob/master/docs/state.md" rel="nofollow noreferrer">State Store</a>. Which is a location where the cluster configuration is located. This maps to the most of the Kubernetes configuration, except some of the resources and for example deployments (so internal state) which are stored in <a href="https://kubernetes.io/docs/concepts/overview/components/#etcd" rel="nofollow noreferrer">etcd</a>. </p>

<p>So in your case when you delete a master node, the AWS will see that the state of your AutoScalingGroup is 0 instead of 1 so it will recreate the EC2 machine.</p>

<pre><code>Description:DescriptionLaunching a new EC2 instance: i-0e06f8fbb78aca2e6
Cause:CauseAt 2019-04-10T12:54:31Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.
</code></pre>

<p>After that Kubernetes will take it's configuration from the S3 bucket and the internal state from etcd. Next question would be how does the etcd survive the deletion of the master. You can check it in your Volumes, as etcd has two separate volumes (just as etcd pods one for events and one main) After you delete the master, volumes go into <code>avalivable</code> state and after the new master EC2 instance is spawned this volumes will be mounted to the new master, and you will restore the internal state (not sure, but I think <a href="https://github.com/kubernetes/kops/tree/master/protokube#protokube" rel="nofollow noreferrer">protokube</a> is also somewhere in the picture). </p>

<p>This is also the reason why you can restore your kops cluster from just s3 bucket as there is all the configuration that kops need to run. Except the internal state, which is in etcd for which you would need some separate backup. </p>
