<p>I'm wondering how people are deploying a production-caliber Kubernetes cluster in AWS and, more importantly, how they chose their approach.</p>

<p>The <a href="https://kubernetes.io/docs/getting-started-guides/aws/#supported-production-grade-tools-with-high-availability-options" rel="nofollow noreferrer">k8s documentation</a> points towards <a href="https://github.com/kubernetes/kops" rel="nofollow noreferrer">kops</a> for Debian, Ubuntu, CentOS, and RHEL or <a href="https://github.com/coreos/kube-aws/" rel="nofollow noreferrer">kube-aws</a> for CoreOS/Container Linux. Among these choices it's not clear how to pick one over the others. CoreOS seems like the most compelling option since it's designed for container workloads.</p>

<p>But wait, there's more.</p>

<p><a href="https://github.com/kubernetes-incubator/bootkube" rel="nofollow noreferrer">bootkube</a> seems to be next iteration of the CoreOS deployment technology and is on the <a href="https://github.com/coreos/kube-aws/blob/master/ROADMAP.md#v09x" rel="nofollow noreferrer">roadmap</a> for inclusion within kube-aws. Should I wait until kube-aws uses bootkube?</p>

<p><a href="https://www.heptio.com/" rel="nofollow noreferrer">Heptio</a> recently announced a <a href="https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/" rel="nofollow noreferrer">Quickstart architecture</a> for deploying k8s in AWS. This is the newest approach and so probably the least mature approach but it does seem to have gained traction from within AWS.</p>

<p>Lastly <a href="https://kubernetes.io/docs/getting-started-guides/kubeadm/" rel="nofollow noreferrer">kubeadm</a> is a thing and I'm not really sure where it fits into all of this.</p>

<p>There are probably more approaches that I'm missing too.</p>

<p>Given the number of options with overlapping intent it's very difficult to choose a path forward. I'm not interested in a proof-of-concept. I want to be able to deploy a secure, highly-available cluster for production use and be able to upgrade the cluster (host OS, etcd, and k8s system components) over time.</p>

<p>What did you choose and how did you decide? </p>

<p>I'd say pick anything which fit's your needs (see also <a href="https://kubernetes.io/docs/getting-started-guides/#table-of-solutions" rel="nofollow noreferrer">Picking the right solution</a>)...</p>

<p>Which could be:</p>

<ul>
<li>Speed of the cluster setup</li>
<li>Integration in your existing toolchain

<ul>
<li>e.g. kops integrates with Terraform which might be a good fit for some prople</li>
</ul></li>
<li>Experience within your team/company/...

<ul>
<li>e.g. how comfortable are you with the related Linux distribution</li>
</ul></li>
<li>Required maturity of the tool itself

<ul>
<li>some tools are very alpha, are you willing to play to role of an early adaptor?</li>
</ul></li>
<li>Ability to upgrade between Kubernetes versions

<ul>
<li>kubeadm has this on their agenda, some others prefer to throw away clusters instead of upgrading</li>
</ul></li>
<li>Required integration into external tools (monitoring, logging, auth, ...)</li>
<li>Supported cloud providers</li>
</ul>

<p>With your specific requirements I'd pick the Heptio or kubeadm approach.</p>

<ul>
<li><a href="https://www.heptio.com/" rel="nofollow noreferrer">Heptio</a> if you can live with the given constraints (e.g. predefined OS)</li>
<li><a href="https://kubernetes.io/docs/getting-started-guides/kubeadm/" rel="nofollow noreferrer">kubeadm</a> if you need more flexibility, everything done with kubeadm can be transferred to other cloud providers</li>
</ul>

<p>Other options for AWS lower on my list:</p>

<ul>
<li><a href="https://github.com/kelseyhightower/kubernetes-the-hard-way" rel="nofollow noreferrer">Kubernetes the hard way</a> - using this might be the only true way to setup a production cluster as this is the only way you can fully understand each moving part of the system. Lower on the list, because often the result from any of the tools might just be more than enough, even for production.</li>
<li><a href="https://kubernetes.io/docs/getting-started-guides/aws/#kube-up-bash-script" rel="nofollow noreferrer">kube-up.sh</a> - is deprecated by the community, so I'd not use it for new projects</li>
<li><a href="https://github.com/kubernetes/kops" rel="nofollow noreferrer">kops</a> - my team had some strange experiences with it which seemed due to our (custom) needs back then (existing VPC), that's why it's lower on my list - it would be #1 for an environment where Terraform is used too.</li>
<li><a href="https://github.com/kubernetes-incubator/bootkube" rel="nofollow noreferrer">bootkube</a> - lower on my list, because it's limitation to CoreOS</li>
<li><a href="http://rancher.com/" rel="nofollow noreferrer">Rancher</a> - interesting toolchain, seems to be too much for a single cluster</li>
</ul>

<hr>

<p>Offtopic: If you don't <em>have</em> to run on AWS, I'd also always consider to rather run on GCE for production workloads, as this is a well managed platform rather than something you've to build yourself.</p>
