<p>I'm working with Airflow DAG and Kubernetes for the first time.
I have a Python script that connects to AWS S3 and reads some files. This works fine if I run it in a Docker container/image using bash. But when I try to run this docker from an airflow task using a K8s pod, I get the following error (I replaced some sensitive values with XXXXX)</p>
<pre><code>    [2022-02-08 22:48:55,795] {kubernetes_pod.py:365} INFO - creating pod with labels {'dag_id': 'ECO_CELLS_POLYGON_STORES', 'task_id': 'process_AR', 'execution_date': '2022-02-08T224216.4628350000-e866f2011', 'try_number': '1'} and launcher &lt;airflow.providers.cncf.kubernetes.utils.pod_launcher.PodLauncher object at 0x7f649be71410&gt;
[2022-02-08 22:48:55,812] {pod_launcher.py:93} ERROR - Exception when attempting to create Namespaced Pod: {
  &quot;apiVersion&quot;: &quot;v1&quot;,
  &quot;kind&quot;: &quot;Pod&quot;,
  &quot;metadata&quot;: {
    &quot;annotations&quot;: {},
    &quot;labels&quot;: {
      &quot;airflow_version&quot;: &quot;2.0.0-astro.8&quot;,
      &quot;kubernetes_pod_operator&quot;: &quot;True&quot;,
      &quot;dag_id&quot;: &quot;ECO_CELLS_POLYGON_STORES&quot;,
      &quot;task_id&quot;: &quot;process_AR&quot;,
      &quot;execution_date&quot;: &quot;2022-02-08T224216.4628350000-e866f2011&quot;,
      &quot;try_number&quot;: &quot;1&quot;
    },
    &quot;name&quot;: &quot;k8s-pod-ml-operator.3aada8ada8df491ea63e9319bf779d10&quot;,
    &quot;namespace&quot;: &quot;default&quot;
  },
  &quot;spec&quot;: {
    &quot;affinity&quot;: {},
    &quot;containers&quot;: [
      {
        &quot;args&quot;: [],
        &quot;command&quot;: [
          &quot;python&quot;,
          &quot;main.py&quot;
        ],
        &quot;env&quot;: {
          &quot;AWS_ACCESS_KEY_ID&quot;: &quot;XXXXXX&quot;,
          &quot;AWS_SECRET_ACCESS_KEY&quot;: &quot;***&quot;,
          &quot;AWS_BUCKET_NAME&quot;: &quot;XXXXXX-dev&quot;,
          &quot;SNOWFLAKE_SERVER&quot;: &quot;XXXXXX&quot;,
          &quot;SNOWFLAKE_LOGIN&quot;: &quot;XXXXXX&quot;,
          &quot;SNOWFLAKE_PASSWORD&quot;: &quot;***&quot;,
          &quot;SNOWFLAKE_ACCOUNT&quot;: &quot;XXXXXX&quot;,
          &quot;SNOWFLAKE_DATABASE&quot;: &quot;XXXXXX&quot;,
          &quot;SNOWFLAKE_WAREHOUSE&quot;: &quot;XXXXXX&quot;,
          &quot;COUNTRY&quot;: &quot;AR&quot;,
          &quot;S3_PROJECT&quot;: &quot;ecom_polygon_stores&quot;,
          &quot;S3_TEAM_VERTICAL&quot;: &quot;ecommerce&quot;
        },
        &quot;envFrom&quot;: [],
        &quot;image&quot;: &quot;ecom_polygon_stores:v1.0.7&quot;,
        &quot;imagePullPolicy&quot;: &quot;Never&quot;,
        &quot;name&quot;: &quot;base&quot;,
        &quot;ports&quot;: [],
        &quot;resources&quot;: {},
        &quot;volumeMounts&quot;: []
      }
    ],
    &quot;hostNetwork&quot;: false,
    &quot;imagePullSecrets&quot;: [],
    &quot;initContainers&quot;: [],
    &quot;restartPolicy&quot;: &quot;Never&quot;,
    &quot;securityContext&quot;: {},
    &quot;serviceAccountName&quot;: &quot;default&quot;,
    &quot;tolerations&quot;: [],
    &quot;volumes&quot;: []
  }
}
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py&quot;, line 89, in run_pod_async
    body=sanitized_pod, namespace=pod.metadata.namespace, **kwargs
  File &quot;/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py&quot;, line 6174, in create_namespaced_pod
    (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File &quot;/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py&quot;, line 6265, in create_namespaced_pod_with_http_info
    collection_formats=collection_formats)
  File &quot;/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py&quot;, line 345, in call_api
    _preload_content, _request_timeout)
  File &quot;/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py&quot;, line 176, in __call_api
    _request_timeout=_request_timeout)
  File &quot;/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py&quot;, line 388, in request
    body=body)
  File &quot;/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py&quot;, line 278, in POST
    body=body)
  File &quot;/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py&quot;, line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '319700db-6333-4a4d-885c-1f45a0cd13a3', 'X-Kubernetes-Pf-Prioritylevel-Uid': '4d5b12e4-65e9-4ab9-ad63-de6f29ca0b6d', 'Date': 'Tue, 08 Feb 2022 22:48:55 GMT', 'Content-Length': '487'})
HTTP response body: {&quot;kind&quot;:&quot;Status&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;metadata&quot;:{},&quot;status&quot;:&quot;Failure&quot;,&quot;message&quot;:&quot;Pod in version \&quot;v1\&quot; cannot be handled as a Pod: v1.Pod.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: decode slice: expect [ or n, but found {, error found in #10 byte of ...|, \&quot;env\&quot;: {\&quot;AWS_ACCES|..., bigger context ...|s\&quot;: [], \&quot;command\&quot;: [\&quot;python\&quot;, \&quot;main.py\&quot;], \&quot;env\&quot;: {\&quot;AWS_ACCESS_KEY_ID\&quot;: \&quot;AXXXXXXXXXXXXXXXXXX6\&quot;, \&quot;AWS_|...&quot;,&quot;reason&quot;:&quot;BadRequest&quot;,&quot;code&quot;:400}
</code></pre>
<p>I'm not sure where to go from here... From what I'm reading the error says it expected a [ instead of a { in &quot;env&quot;: {&quot;AWS_ACCESS_KEY_ID&quot;... But I'm not sure how to correct that since I pass those parameters like this:</p>
<pre><code>        self.env_vars = {
            'AWS_ACCESS_KEY_ID': s3_connection.login,
            'AWS_SECRET_ACCESS_KEY': s3_connection.password,
            'AWS_BUCKET_NAME': bucket_name,
            'SNOWFLAKE_SERVER': str(snowflake_connection.host),
            'SNOWFLAKE_LOGIN': str(snowflake_connection.login),
            'SNOWFLAKE_PASSWORD': str(snowflake_connection.password),
            'SNOWFLAKE_ACCOUNT': str(snowflake_connection.extra_dejson['account']),
            'SNOWFLAKE_DATABASE': str(snowflake_connection.extra_dejson['database']),
            'SNOWFLAKE_WAREHOUSE': str(snowflake_connection.extra_dejson['warehouse']),
            'COUNTRY': code,
            'S3_PROYECT': s3_project,
            'S3_TEAM_VERTICAL': s3_team_vertical
        }
</code></pre>
<p>Any suggestions?</p>

Alain
<p>Your <code>env:</code> is malformed; one can see this in two different ways: (1) <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#container-v1-core" rel="nofollow noreferrer"><code>env:</code> in the <code>PodSpec</code></a> is a <strong>list</strong> of <code>{name: &quot;&quot;, value: &quot;&quot;}</code> items (2) the structure emitted in the error message is malformed regardless: <code>&quot;env&quot;: {&quot;AWS_ACCESS_KEY_ID&quot;: &quot;ASIAWCMTKGYGDU6KEOD6&quot;, &quot;AWS_|...</code> as there is no such  shape of data as <code>{&quot;&quot;:&quot;&quot;,&quot;&quot;</code></p>
<p>I don't have any Airflow reference documentation links handy, but you'd want to check them to ensure <code>self.env_vars</code> is what Airflow expects it to be, since python places the entire burden of correctness upon the programmer</p>
