<p>What is considered a good practice with K8S for managing multiple environments (QA, Staging, Production, Dev, etc)?</p>

<p>As an example, say that a team is working on a product which requires deploying a few APIs, along with a front-end application. Usually, this will require at least 2 environments:</p>

<ul>
<li>Staging: For iterations/testing and validation before releasing to the client </li>
<li>Production: This the environment the client has access to. Should contain stable and well-tested features.</li>
</ul>

<p>So, assuming the team is using Kubernetes, what would be a good practice to host these environments? This far we've considered two options:</p>

<ol>
<li>Use a K8s cluster for each environment</li>
<li>Use only one K8s cluster and keep them in different namespaces.</li>
</ol>

<p>(1) Seems the safest options since it minimizes the risks of potential human mistake and machine failures, that could put the production environment in danger. However, this comes with the cost of more master machines and also the cost of more infrastructure management. </p>

<p>(2) Looks like it simplifies infrastructure and deployment management because there is one single cluster but it raises a few questions like:</p>

<ul>
<li>How does one make sure that a human mistake might impact the production environment?</li>
<li>How does one make sure that a high load in the staging environment won't cause a loss of performance in the production environment?</li>
</ul>

<p>There might be some other concerns, so I'm reaching out to the K8s community on StackOverflow to have a better understanding of how people are dealing with these sort of challenges.</p>

<h2>Multiple Clusters Considerations</h2>
<p>Take a look at this blog post from Vadim Eisenberg (<em>IBM / Istio</em>): <a href="http://vadimeisenberg.blogspot.com/2019/03/multicluster-pros-and-cons.html" rel="noreferrer">Checklist: pros and cons of using multiple Kubernetes clusters, and how to distribute workloads between them</a>.</p>
<p>I'd like to highlight some of the pros/cons:</p>
<blockquote>
<p><strong>Reasons to have multiple clusters</strong></p>
<ul>
<li>Separation of production/development/test: especially for testing a    new version of Kubernetes, of a service mesh, of other cluster software</li>
<li>Compliance: according to some regulations some applications must run in separate clusters/separate VPNs</li>
<li>Better isolation for security</li>
<li>Cloud/on-prem: to split the load between on-premise services</li>
</ul>
<p><strong>Reasons to have a single cluster</strong></p>
<ul>
<li>Reduce setup, maintenance and administration overhead</li>
<li>Improve utilization</li>
<li>Cost reduction</li>
</ul>
</blockquote>
<p>Considering a not too expensive environment, with average maintenance, and yet still ensuring security isolation for production applications, I would recommend:</p>
<ul>
<li>1 cluster for DEV and STAGING (separated by namespaces, <em>maybe even isolated, using Network Policies, like in <a href="https://www.projectcalico.org/calico-network-policy-comes-to-kubernetes/" rel="noreferrer">Calico</a></em>)</li>
<li>1 cluster for PROD</li>
</ul>
<h2>Environment Parity</h2>
<p>It's a <a href="https://12factor.net/dev-prod-parity" rel="noreferrer">good practice</a> to keep development, staging, and production as similar as possible:</p>
<blockquote>
<p>Differences between backing services mean that tiny incompatibilities
crop up, causing code that worked and passed tests in development or
staging to fail in production. These types of errors create friction
that disincentivizes continuous deployment.</p>
</blockquote>
<p><strong>Combine a powerful CI/CD tool with <a href="https://github.com/helm/helm" rel="noreferrer">helm</a></strong>. You can use the flexibility of <a href="https://helm.sh/docs/chart_template_guide/values_files/#helm" rel="noreferrer">helm values</a> to set default configurations, just overriding the configs that differ from an environment to another.</p>
<p><a href="https://docs.gitlab.com/ee/user/project/clusters/" rel="noreferrer">GitLab CI/CD with AutoDevops</a> has a powerful integration with Kubernetes, which allows you to manage multiple Kubernetes clusters already with helm support.</p>
<h2><a href="https://medium.com/@eduardobaitello/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b" rel="noreferrer">Managing multiple clusters</a> <em>(<code>kubectl</code> interactions)</em></h2>
<blockquote>
<p>When you are working with multiple Kubernetes clusters, it’s easy to
mess up with contexts and run <code>kubectl</code> in the wrong cluster. Beyond
that, Kubernetes has <a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#supported-releases-and-component-skew" rel="noreferrer">restrictions</a> for versioning mismatch between the
client (<code>kubectl</code>) and server (kubernetes master), so running commands
in the right context does not mean running the right client version.</p>
</blockquote>
<p>To overcome this:</p>
<ul>
<li>Use <a href="https://github.com/asdf-vm/asdf" rel="noreferrer"><code>asdf</code></a> to manage multiple <code>kubectl</code> versions</li>
<li><a href="https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable" rel="noreferrer">Set the <code>KUBECONFIG</code></a> env var to change between multiple <code>kubeconfig</code> files</li>
<li>Use <a href="https://github.com/jonmosco/kube-ps1" rel="noreferrer"><code>kube-ps1</code></a> to keep track of your current context/namespace</li>
<li>Use <a href="https://github.com/ahmetb/kubectx" rel="noreferrer"><code>kubectx</code> and <code>kubens</code></a> to change fast between clusters/namespaces</li>
<li>Use aliases to combine them all together</li>
</ul>
<p>I have an article that exemplifies how to accomplish this: <a href="https://medium.com/@eduardobaitello/using-different-kubectl-versions-with-multiple-kubernetes-clusters-a3ad8707b87b" rel="noreferrer">Using different kubectl versions with multiple Kubernetes clusters</a></p>
<p>I also recommend the following reads:</p>
<ul>
<li><a href="https://medium.com/@ahmetb/mastering-kubeconfig-4e447aa32c75" rel="noreferrer">Mastering the KUBECONFIG file</a> by Ahmet Alp Balkan (<em>Google Engineer</em>)</li>
<li><a href="https://srcco.de/posts/how-zalando-manages-140-kubernetes-clusters.html" rel="noreferrer">How Zalando Manages 140+ Kubernetes Clusters</a> by Henning Jacobs (<em>Zalando Tech</em>)</li>
</ul>
