<p>What I am trying to do:</p>

<p>I have setup kubernete cluster using documentation available on Kubernetes website (http_kubernetes.io/v1.1/docs/getting-started-guides/aws.html). Using kube-up.sh, i was able to bring kubernete cluster up with 1 master and 3 minions (as highlighted in blue rectangle in the diagram below). From the documentation as far as i know we can add minions as and when required, So from my point of view k8s master instance is single point of failure when it comes to high availability.</p>

<p><a href="http://i.stack.imgur.com/ZrtZI.png" rel="nofollow">Kubernetes Master HA on AWS</a></p>

<p>So I am trying to setup HA k8s master layer with the three master nodes as shown above in the diagram. For accomplishing this I am following kubernetes high availability cluster guide, http_kubernetes.io/v1.1/docs/admin/high-availability.html#establishing-a-redundant-reliable-data-storage-layer
What I have done:</p>

<p>Setup k8s cluster using kube-up.sh and provider aws (master1 and minion1, minion2, and minion3)
Setup two fresh master instance’s (master2 and master3)
I then started configuring etcd cluster on master1, master 2 and master 3 by following below mentioned link: 
http_kubernetes.io/v1.1/docs/admin/high-availability.html#establishing-a-redundant-reliable-data-storage-layer
    So in short i have copied etcd.yaml from the kubernetes website (http_kubernetes.io/v1.1/docs/admin/high-availability/etcd.yaml) and updated Node_IP, Node_Name and Discovery Token on all the three nodes as shown below. </p>

<blockquote>
  <p>NODE_NAME NODE_IP DISCOVERY_TOKEN</p>
  
  <p>Master1
  172.20.3.150 https_discovery.etcd.io/5d84f4e97f6e47b07bf81be243805bed</p>
  
  <p>Master2
  172.20.3.200 https_discovery.etcd.io/5d84f4e97f6e47b07bf81be243805bed</p>
  
  <p>Master3
  172.20.3.250 https_discovery.etcd.io/5d84f4e97f6e47b07bf81be243805bed</p>
</blockquote>

<p>And on running etcdctl member list on all the three nodes, I am getting:</p>

<pre><code>$ docker exec &lt;container-id&gt; etcdctl member list
ce2a822cea30bfca: name=default peerURLs=http_localhost:2380,http_localhost:7001 clientURLs=http_127.0.0.1:4001
</code></pre>

<p>As per documentation we need to keep etcd.yaml in /etc/kubernete/manifest, this directory already contains etcd.manifest and etcd-event.manifest files. For testing I modified etcd.manifest file with etcd parameters.</p>

<p>After making above changes I forcefully terminated docker container, container was existing after few seconds and I was getting below mentioned error on running kubectl get nodes:
error: couldn't read version from server: Get httplocalhost:8080/api: dial tcp 127.0.0.1:8080: connection refused</p>

<p>So please kindly suggest how can I setup k8s master highly available setup on AWS.</p>

<p>Setting up HA controllers for kubernetes is not trivial and I can't provide all the details here but I'll outline what was successful for me.</p>

<ol>
<li>Use kube-aws to set up a single-controller cluster: <a href="https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html" rel="nofollow">https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html</a>.  This will create CloudFormation stack templates and cloud-config templates that you can use as a starting point.</li>
<li>Go the AWS CloudFormation Management Console, click the "Template" tab and copy out the complete stack configuration. Alternatively, use <code>$ kube-aws up --export</code> to generate the cloudformation stack file.</li>
<li>User the userdata cloud-config templates generated by kube-aws and replace the variables with actual values.  This guide will help you determine what those values should be: <a href="https://coreos.com/kubernetes/docs/latest/getting-started.html" rel="nofollow">https://coreos.com/kubernetes/docs/latest/getting-started.html</a>.  In my case I ended up with four cloud-configs:

<ul>
<li>cloud-config-controller-0</li>
<li>cloud-config-controller-1</li>
<li>cloud-config-controller-2</li>
<li>cloud-config-worker</li>
</ul></li>
<li>Validate your new cloud-configs here: <a href="https://coreos.com/validate/" rel="nofollow">https://coreos.com/validate/</a></li>
<li><p>Insert your cloud-configs into the CloudFormation stack config.  First compress and encode your cloud config:</p>

<pre><code>$ gzip -k cloud-config-controller-0
$ cat cloud-config-controller-0.gz | base64 &gt; cloud-config-controller-0.enc
</code></pre>

<p>Now copy the content into your encoded cloud-config into the CloudFormation config.  Look for the UserData key for the appropriate InstanceController.  (I added additional InstanceController objects for the additional controllers.)</p></li>
<li><p>Update the stack at the AWS CloudFormation Management Console using your newly created CloudFormation config.</p></li>
</ol>

<p>You will also need to generate TLS asssets: <a href="https://coreos.com/kubernetes/docs/latest/openssl.html" rel="nofollow">https://coreos.com/kubernetes/docs/latest/openssl.html</a>.  These assets will have to be compressed and encoded (same gzip and base64 as above), then inserted into your userdata cloud-configs.</p>

<p>When debugging on the server, journalctl is your friend:</p>

<pre><code>$ journalctl -u oem-cloudinit  # to debug problems with your cloud-config
$ journalctl -u etcd2
$ journalctl -u kubelet
</code></pre>

<p>Hope that helps.</p>
