<p>airflow 1.10.10  <br />
minikube 1.22.0  <br />
amazon emr</p>
<p>I am running airflow on kubernetes(minikube).
Dags are synced from github.
spark-submit on Amazon EMR as a CLI mode.</p>
<p>In order to do that, I attach EMR pem key.
So, I get pem key from AWS S3 while ExtraInitContainer is getting image awscli and mount the volume at airlfow/sshpem</p>
<p>error is reported when I make a connection from airflow WebUI as
&quot;con_type&quot;: &quot;ssh&quot;
&quot;key_file&quot;: &quot;/opt/sshepm/emr.pem&quot;</p>
<pre><code>SSH operator error: [Errno 2] No such file or directory: '/opt/airflow/sshpem/emr.pem'
</code></pre>
<p>it is there. I think it is related to some PATH or permission issue since I get emr.pem on ExtraInitContainer and it's permission was root. Although I temporarily changed a user as 1000:1000 there is some issue airflow WebUI can't get this directory while getting a key.</p>
<p>Full log is below</p>
<pre><code>&gt; Traceback (most recent call last):   File
    &gt; &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/operators/ssh_operator.py&quot;,
    &gt; line 108, in execute
    &gt;     with self.ssh_hook.get_conn() as ssh_client:   File &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/hooks/ssh_hook.py&quot;,
    &gt; line 194, in get_conn
    &gt;     client.connect(**connect_kwargs)   File &quot;/home/airflow/.local/lib/python3.6/site-packages/paramiko/client.py&quot;,
    &gt; line 446, in connect
    &gt;     passphrase,   File &quot;/home/airflow/.local/lib/python3.6/site-packages/paramiko/client.py&quot;,
    &gt; line 677, in _auth
    &gt;     key_filename, pkey_class, passphrase   File &quot;/home/airflow/.local/lib/python3.6/site-packages/paramiko/client.py&quot;,
    &gt; line 586, in _key_from_filepath
    &gt;     key = klass.from_private_key_file(key_path, password)   File &quot;/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py&quot;,
    &gt; line 235, in from_private_key_file
    &gt;     key = cls(filename=filename, password=password)   File &quot;/home/airflow/.local/lib/python3.6/site-packages/paramiko/rsakey.py&quot;,
    &gt; line 55, in __init__
    &gt;     self._from_private_key_file(filename, password)   File &quot;/home/airflow/.local/lib/python3.6/site-packages/paramiko/rsakey.py&quot;,
    &gt; line 175, in _from_private_key_file
    &gt;     data = self._read_private_key_file(&quot;RSA&quot;, filename, password)   File
    &gt; &quot;/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py&quot;,
    &gt; line 307, in _read_private_key_file
    &gt;     with open(filename, &quot;r&quot;) as f: FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/sshpem/emr-pa.pem'
    &gt; 
    &gt; During handling of the above exception, another exception occurred:
    &gt; 
    &gt; Traceback (most recent call last):   File
    &gt; &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py&quot;,
    &gt; line 979, in _run_raw_task
    &gt;     result = task_copy.execute(context=context)   File &quot;/opt/airflow/class101-airflow/plugins/operators/emr_ssh_operator.py&quot;,
    &gt; line 107, in execute
    &gt;     super().execute(context)   File &quot;/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/operators/ssh_operator.py&quot;,
    &gt; line 177, in execute
    &gt;     raise AirflowException(&quot;SSH operator error: {0}&quot;.format(str(e))) airflow.exceptions.AirflowException: SSH operator error: [Errno 2] No
    &gt; such file or directory: '/opt/airflow/sshpem/emr-pa.pem' [2021-07-14
    &gt; 05:40:31,624] Marking task as UP_FOR_RETRY. dag_id=test_staging,
    &gt; task_id=extract_categories_from_mongo, execution_date=20210712T190000,
    &gt; start_date=20210714T054031, end_date=20210714T054031 [2021-07-14
    &gt; 05:40:36,303] Task exited with return code 1
</code></pre>
<p>airflow home: /opt/airflow <br />
dags : /opt/airflow//dags <br />
pemkey : /opt/sshpem/ <br />
airflow.cfg: /opt/airflow <br />
airflow_env:
export PATH=&quot;/home/airflow/.local/bin:$PATH&quot;</p>
<p>my yaml file</p>
<pre><code>airflow: 
   image: 
     repository: airflow
   executor: KubernetesExecutor
   extraVolumeMounts:
    - name: sshpem
      mountPath: /opt/airflow/sshpem
   extraVolumes:
    - name: sshpem
      emptyDir: {}
scheduler:
    extraInitContainers:
    - name: emr-key-file-download
      image: amazon/aws-cli
      command: [
        &quot;sh&quot;,
        &quot;-c&quot;,
        &quot;aws s3 cp s3://mykeyfile/path.my.pem&amp;&amp; \
        chown -R 1000:1000 /opt/airflow/sshpem/&quot;
      volumeMounts:
        - mountPath: /opt/airflow/sshpem
          name: sshpem
</code></pre>

Yong Rhee
<p>Are you using KubernetesExecutor or CeleryExecutor?</p>
<p>If the former, then you have to make sure the extra init container is added to the pod_template you are using (tasks in KubernetesExecutor) run as separate PODs.</p>
<p>If the latter, you should make sure the extra init container is also added for workers, not only for scheduler).</p>
<p>BTW. Airflow 1.10 reached end-of-life on June 17th, 2021 and it will not receive even critical security fixes. You can watch our talk from the recent Airflow Summit &quot;Keep your Airflow Secure&quot; - <a href="https://airflowsummit.org/sessions/2021/panel-airflow-security/" rel="nofollow noreferrer">https://airflowsummit.org/sessions/2021/panel-airflow-security/</a> to learn why it is important to upgrade to Airflow 2.</p>
