<p>I want to have my Cloud Composer environment (Google Cloud's managed Apache Airflow service) start pods on a <strong>different</strong> kubernetes cluster. How should I do this?</p>

<p><em>Note that Cloud composer runs airflow on a kubernetes cluster. That cluster is considered to be the composer "environment". Using the default values for the <code>KubernetesPodOperator</code>, composer will schedule pods on its own cluster. However in this case, I have a different kubernetes cluster on which I want to run the pods.</em></p>

<p>I can connect to the worker pods and run a <code>gcloud container clusters get-credentials CLUSTERNAME</code> there, but every now and then the pods get recycled so this is not a durable solution.</p>

<p>I noticed that the <a href="https://airflow.apache.org/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html#airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator" rel="nofollow noreferrer"><code>KubernetesPodOperator</code></a> has both an <code>in_cluster</code> and a <code>cluster_context</code> argument, which seem useful. I would expect that this would work:</p>

<pre class="lang-py prettyprint-override"><code>pod = kubernetes_pod_operator.KubernetesPodOperator(
    task_id='my-task',
    name='name',
    in_cluster=False,
    cluster_context='my_cluster_context',
    image='gcr.io/my/image:version'
)
</code></pre>

<p>But this results in <code>kubernetes.config.config_exception.ConfigException: Invalid kube-config file. Expected object with name CONTEXTNAME in kube-config/contexts list</code></p>

<p>Although if I run <code>kubectl config get-contexts</code> in the worker pods, I can see the cluster config listed.</p>

<p>So what I fail to figure out is:</p>

<ul>
<li>how to make sure that the context for my other kubernetes cluster is available on the worker pods (or should that be on the nodes?) of my composer environment?</li>
<li>if the context is set (as I did manually for testing purposes), how can I tell airflow to use that context?</li>
</ul>

bartaelterman
<p>Check out the <a href="https://airflow.apache.org/_api/airflow/contrib/operators/gcp_container_operator/index.html" rel="noreferrer">GKEPodOperator</a> for this.</p>

<p>Example usage from the docs : </p>

<pre><code>operator = GKEPodOperator(task_id='pod_op',
                          project_id='my-project',
                          location='us-central1-a',
                          cluster_name='my-cluster-name',
                          name='task-name',
                          namespace='default',
                          image='perl')
</code></pre>
