<p>When you use minikube, it automatically creates the local configurations, so it's ready to use. And it appears there is support for multiple clusters in the kubectl command based on the reference for <a href="https://kubernetes.io/docs/user-guide/kubectl/v1.7/#config" rel="noreferrer">kubectl config</a>.</p>

<p>In the docs for setting up clusters, there's <a href="https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#optional-controlling-your-cluster-from-machines-other-than-the-master" rel="noreferrer">a reference to copying the relevant files to your local machine</a> to access the cluster. I also found an <a href="https://stackoverflow.com/questions/36306904/configure-kubectl-command-to-access-remote-kubernetes-cluster-on-azure">SO Q&amp;A about editing your .kube/config to leverage azure remotely</a> that talked to editing the kube/config file.</p>

<p>It looks like the environment variable <code>$KUBECONFIG</code> can reference multiple locations of these configuration files, with the built-in default being <code>~/.kube/config</code> (which is what minikube creates).</p>

<p>If I want to be able to use kubectl to invoke commands to multiple clusters, should I download the relevant config file into a new location (for example into <code>~/gcloud/config</code>, set the <code>KUBECONFIG</code> environment variable to reference both locations?</p>

<p>Or is it better to just explicitly use the <code>--kubeconfig</code> option when invoking kubectl to specify a configuration for the cluster?</p>

<p>I wasn't sure if there was some way of merging the configuration files that would be better, and leverage the <code>kubectl config set-context</code> or <code>kubectl config set-cluster</code> commands instead. The documentation at Kubernetes on "<a href="https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/" rel="noreferrer">Configure Access to Multiple Clusters"</a> seems to imply a different means of using <code>--kubeconfig</code> along with these <code>kubectl config</code> commands.</p>

<p>In short, what's the best way to interact with multiple separate kubernetes clusters and what are the tradeoffs?</p>

<blockquote>
  <p>If I want to be able to use <code>kubectl</code> to invoke commands to multiple
  clusters, should I download the relevant config file into a new
  location (for example into <code>~/gcloud/config</code>, set the <code>KUBECONFIG</code>
  environment variable to reference both locations?</p>
  
  <p>Or is it better to just explicitly use the --kubeconfig option when
  invoking kubectl to specify a configuration for the cluster?</p>
</blockquote>

<p>That would probably depend on the approach you find simpler and more convenient, and if having security and access management concerns in mind is needed. </p>

<ul>
<li><p>From our experience merging various <code>kubeconfig</code> files is very useful for multi-cluster operations, in order to carry out maintenance tasks, and incident management over a group of clusters (contexts &amp; namespaces) simplifying troubleshooting issues based on the possibility to compare configs, manifests, resources and states of K8s services, pods, volumes, namespaces, rs, etc.</p></li>
<li><p>However, when automation and deployment (w/ tools like Jenkins, Spinnaker or Helm) are involved most likely having separate <code>kubeconfig</code> files could be a good idea. A hybrid approach can be merging <code>kubeconfig</code> files based on a division by <strong>Service tier</strong> -> Using files to partition development landscapes (dev, qa, stg, prod) clusters or <strong>for Teams</strong> -> Roles and Responsibilities in an Enterprise (teamA, teamB, …, teamN) can be understood also within good alternatives.</p></li>
<li><p>For multi-cluster merged <code>kubeconfig</code> files scenarios consider <a href="https://github.com/ahmetb/kubectx" rel="nofollow noreferrer">kubectx</a> + <a href="https://github.com/ahmetb/kubectx" rel="nofollow noreferrer">kubens</a>, which are very powerful tools for <code>kubectlt</code> that let you see the current context (cluster) and namespace, likewise to switch between them. </p></li>
</ul>

<blockquote>
  <p>In short, what's the best way to interact with multiple separate
  kubernetes clusters and what are the trade offs?</p>
</blockquote>

<ul>
<li><p>The trade offs should possibly be analyzed considering the most important factors for your project. Having a single merged <code>kubeconfig</code> file seems simpler, even simple if you merge it with <code>~/.kube/config</code> to be used by default by <code>kubectl</code> and just switching between cluster/namespaces with <code>--context kubectl</code> flag. On the other hand if limiting the scope of the <code>kubeconfig</code> is a must, having them segregated and using <code>--kubeconfig=file1</code> sounds like the best way to go.    </p></li>
<li><p>Probably there is NOT a best way for every case and scenario, knowing how to configure <code>kubeconfig</code> file knowing its precedence will help though. </p></li>
<li><p>In this article -> <a href="https://www.nrmitchi.com/2019/01/managing-kubeconfig-files/" rel="nofollow noreferrer">https://www.nrmitchi.com/2019/01/managing-kubeconfig-files/</a> you'll find a complementary and valuable opinion: </p>

<ul>
<li><p><em>While having all of the contexts you may need in one file is nice, it
is difficult to maintain, and seldom the default case. Multiple tools
which provide you with access credentials will provide a fresh
<code>kubeconfig</code> to use. While you can merge the configs together into
<code>~/.kube/config</code>, it is manual, and makes removing contexts more
difficult (having to explicitly remove the context, cluster, and
user). There is an open <a href="https://github.com/kubernetes/kubernetes/issues/46381" rel="nofollow noreferrer">issue</a> in Kubernetes tracking this. However by
keeping each provided config file separate, and just loading all of
them, removal is much easier (just remove the file). To me, this
seems like a much more manageable approach.</em></p></li>
<li><p><em>I prefer to keep all individual config files under ~/.kube/configs, and by taking advantage of the multiple-path aspect of the $KUBECONFIG environment variable option, we can make this happen.</em></p></li>
</ul></li>
</ul>

<p>If you’re using <code>kubectl</code>, here’s the preference that takes effect while determining which <strong>kubeconfig</strong> file is used.</p>

<ol>
<li>use <code>--kubeconfig</code> flag, if specified</li>
<li>use <code>KUBECONFIG</code> environment variable, if specified</li>
<li>use <code>$HOME/.kube/config</code> file</li>
</ol>

<p>With this, you can easily override <strong>kubeconfig</strong> file you use per the <code>kubectl</code> command:</p>

<pre class="lang-sh prettyprint-override"><code>#
# using --kubeconfig flag
#
kubectl get pods --kubeconfig=file1
kubectl get pods --kubeconfig=file2

#
# or 
# using `KUBECONFIG` environment variable
#
KUBECONFIG=file1 kubectl get pods
KUBECONFIG=file2 kubectl get pods

#
# or 
# merging your kubeconfig file w/ $HOME/.kube/config (w/ cp backup)
#
cp $HOME/.kube/config $HOME/.kube/config.backup.$(date +%Y-%m-%d.%H:%M:%S)
KUBECONFIG= $HOME/.kube/config:file2:file3 kubectl config view --merge --flatten &gt; \
~/.kube/merged_kubeconfig &amp;&amp; mv ~/.kube/merged_kubeconfig ~/.kube/config
kubectl get pods --context=cluster-1
kubectl get pods --context=cluster-2
</code></pre>

<p><strong>NOTE:</strong> The <code>--minify</code> flag allows us to extract only info about that context, and the <code>--flatten</code> flag allows us to keep the credentials unredacted.</p>

<hr>

<h2>Bonus (extra points!)</h2>

<h3>Using multiple kubeconfigs at once</h3>

<p>You can save AKS (Azure Container Service), or AWS EKS (Elastic Container Service for K8s) or GKE (Google Container Engine) cluster contexts to separate files and set the <code>KUBECONFIG</code> env var to reference both file locations. </p>

<p>For instance, when you create a GKE cluster (or retrieve its credentials) through the <code>gcloud</code> command, it normally modifies your default <code>~/.kube/config</code> file. However, you can set <code>$KUBECONFIG</code> for <code>gcloud</code> to save cluster credentials to a file:</p>

<pre><code>KUBECONFIG=c1.yaml gcloud container clusters get-credentials "cluster-1"
</code></pre>

<p>Then as we mentioned before <strong>using multiple <code>kubeconfigs</code> at once</strong> can be very useful to work with multiple contexts at the same time.</p>

<p>To do that, you need a “merged” <strong>kubeconfig</strong> file. In the section <strong><em>"Merging kubeconfig files"</em></strong> below, we explain how you can merge the <strong>kubeconfigs</strong> into a single file, but you can also merge them <em>in-memory</em>.</p>

<p>By specifying multiple files in <code>KUBECONFIG</code> environment variable, you can temporarily stitch <strong>kubeconfig</strong> files together and use them all in <code>kubectl</code> .</p>

<pre class="lang-sh prettyprint-override"><code>#
# Kubeconfig in-memory merge
#
export KUBECONFIG=file1:file2
kubectl get pods --context=cluster-1
kubectl get pods --context=cluster-2

#
# For your example
# merging your kubeconfig file w/ $HOME/.kube/config (w/ cp backup)
#
cp $HOME/.kube/config $HOME/.kube/config.backup.$(date +%Y-%m-%d.%H:%M:%S)
KUBECONFIG= $HOME/.kube/config:file2: kubectl config view --merge --flatten &gt; \
~/.kube/merged_kubeconfig &amp;&amp; mv ~/.kube/merged_kubeconfig ~/.kube/config
kubectl get pods --context=cluster-1
kubectl get pods --context=cluster-2
</code></pre>

<h3>Merging kubeconfig files</h3>

<p>Since <strong>kubeconfig</strong> files are structured YAML files, you can’t just append them to get one big <strong>kubeconfig</strong> file, but <code>kubectl</code> can help you merge these files:</p>

<pre class="lang-sh prettyprint-override"><code>#
# Merging your kubeconfig file w/ $HOME/.kube/config (w/ cp backup)
#
cp $HOME/.kube/config $HOME/.kube/config.backup.$(date +%Y-%m-%d.%H:%M:%S)
KUBECONFIG=$HOME/.kube/config:file2:file3 kubectl config view --merge --flatten &gt; \
~/.kube/merged_kubeconfig &amp;&amp; mv ~/.kube/merged_kubeconfig ~/.kube/config
kubectl get pods --context=cluster-1
kubectl get pods --context=cluster-2
</code></pre>

<ul>
<li><strong>ref article 1:</strong> <a href="https://ahmet.im/blog/mastering-kubeconfig/" rel="nofollow noreferrer">https://ahmet.im/blog/mastering-kubeconfig/</a></li>
<li><strong>ref article 2:</strong> <a href="https://github.com/kubernetes/kubernetes/issues/46381" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/issues/46381</a></li>
</ul>
