<p>I am using Google cloud composer ,and created composer environment.Composer environment is ready(has green tick), now I am trying to set variables used in DAG python code using google cloud shell.</p>

<p>command to set variables:</p>

<pre><code>     gcloud composer environments run test-environment \
       --location us-central1 variables -- \
       --set gcp_project xxx-gcp
</code></pre>

<p><strong>Exact error message:</strong></p>

<pre><code>  ERROR: (gcloud.composer.environments.run) Desired GKE pod not found. If the environment was recently started, please wait and retry.
</code></pre>

<p>I tried following things as part of investigation, but got same error each time.
I have created a new environment using UI and not google shell commands.
I checked pods in kubernetes engine and all are green , did not see any issue.
I verified composer API, Billing kubernetes, all required API's are enabled.</p>

<p>I have 'Editor' role assigned.</p>

<p><strong>added screenshot I saw first time some failures</strong></p>

<p><a href="https://i.stack.imgur.com/bkF1M.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bkF1M.png" alt="enter image description here"></a></p>

<p><a href="https://i.stack.imgur.com/SLfll.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SLfll.png" alt="enter image description here"></a></p>

<p><strong>Error with exit code 1</strong>
google troubleshooting guide describe: If the exit code is 1, the container crashed because the application crashed.</p>

<p>This is a side effect of Composer version 1.6.0 if you are using a <a href="https://github.com/google-cloud-sdk/google-cloud-sdk" rel="noreferrer">google-cloud-sdk</a> that is too old, because it now launches pods in namespaces other than <code>default</code>. The error you see is a result of <a href="https://github.com/google-cloud-sdk/google-cloud-sdk/blob/636d6c1b48f97c844aaa4cc2ddc8a5c2bb1be55e/lib/googlecloudsdk/command_lib/composer/util.py#L314" rel="noreferrer">looking for Kubernetes pods in the default namespace</a> and <a href="https://github.com/google-cloud-sdk/google-cloud-sdk/blob/636d6c1b48f97c844aaa4cc2ddc8a5c2bb1be55e/lib/googlecloudsdk/command_lib/composer/util.py#L275" rel="noreferrer">failing to find them</a>.</p>

<p>To fix this, run <code>gcloud components update</code>. If you cannot yet update, a workaround to execute Airflow commands is to manually SSH to a pod yourself and run <code>airflow</code>. To start, obtain GKE cluster credentials:</p>

<pre><code>$ gcloud container clusters get-credentials $COMPOSER_GKE_CLUSTER_NAME
</code></pre>

<p>Once you have the credentials, you should find which namespace the pods are running in (which you can also find using Cloud Console):</p>

<pre><code>$ kubectl get namespaces
NAME                                    STATUS   AGE
composer-1-6-0-airflow-1-9-0-6f89fdb7   Active   17h
default                                 Active   17h
kube-public                             Active   17h
kube-system                             Active   17h
</code></pre>

<p>You can then SSH into any scheduler/worker pod, and run commands:</p>

<pre><code>$ kubectl exec \
    --namespace=$NAMESPACE \
    -it airflow-worker-569bc59df5-x6jhl airflow list_dags -r
</code></pre>

<p>You can also open a shell if you prefer:</p>

<pre><code>$ kubectl exec \
    --namespace=$NAMESPACE \
    -it airflow-worker-569bc59df5-x6jhl bash

airflow@airflow-worker-569bc59df5-x6jhl:~$ airflow list_dags -r
</code></pre>

<p>The failed <code>airflow-database-init-job</code> jobs are unrelated and will not cause problems in your Composer environment.</p>
