<p>How can Apache Airflow's <code>KubernetesPodOperator</code> pull docker images from a private repository? </p>

<p>The <code>KubernetesPodOperator</code> has an <code>image_pull_secrets</code> which you can pass a <code>Secrets</code> object to authenticate with the private repository. But the secrets object can only represent an environment variable, or a volume - neither of which fit my understanding of how Kubernetes uses secrets to authenticate with private repos. </p>

<p>Using <code>kubectl</code> you can create the required secret with something like </p>

<pre><code>$ kubectl create secret docker-registry $SECRET_NAME \
              --docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
              --docker-username=AWS \
              --docker-password="${TOKEN}" \
              --docker-email="${EMAIL}"
</code></pre>

<p>But how can you create the authentication secret in Airflow? </p>

danodonovan
<p>There is <code>secret</code> object with <code>docker-registry</code> type according to kubernetes <a href="https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials" rel="nofollow noreferrer">documentation</a> which can be used to authenticate to private repository.</p>
<p>As You mentioned in Your question; You can use <code>kubectl</code> to <strong>create</strong> secret of <code>docker-registry</code> <strong>type</strong> that you can then try to pass with <code>image_pull_secrets</code>.</p>
<p>However depending on platform You are using this might have <strong>limited</strong> or <strong>no use at all</strong> according to <a href="https://kubernetes.io/docs/concepts/containers/images/#configuring-nodes-to-authenticate-to-a-private-registry" rel="nofollow noreferrer">kubernetes documentation</a>:</p>
<blockquote>
<h3>Configuring Nodes to Authenticate to a Private Registry</h3>
<p><strong>Note:</strong> If you are running on Google Kubernetes Engine, there will already be a <code>.dockercfg</code> on each node with credentials for Google Container Registry. You cannot use this approach.</p>
<p><strong>Note:</strong> If you are running on AWS EC2 and are using the EC2 Container Registry (ECR), the kubelet on each node will manage and update the ECR login credentials. You cannot use this approach.</p>
<p><strong>Note:</strong> This approach is suitable if you can control node configuration. It will not work reliably on GCE, and any other cloud provider that does automatic node replacement.</p>
<p><strong>Note:</strong> Kubernetes as of now only supports the <code>auths</code> and <code>HttpHeaders</code> section of docker config. This means credential helpers (<code>credHelpers</code> or <code>credsStore</code>) are not supported.</p>
</blockquote>
<p>Making this work on mentioned platforms is possible but it would require automated scripts and third party tools.</p>
<p>Like in Amazon ECR example: <a href="https://github.com/awslabs/amazon-ecr-credential-helper" rel="nofollow noreferrer">Amazon ECR Docker Credential Helper</a> would be needed to periodically pull AWS credentials to docker registry configuration and then have another script to update kubernetes docker-registry secrets.</p>
<p>As for Airflow itself I don't think it has functionality to create its own docker-repository secrets.
You can request functionality like that in <a href="https://issues.apache.org/jira/projects/AIRFLOW/issues/" rel="nofollow noreferrer">Apache Airflow JIRA</a>.</p>
<p>P.S.</p>
<p>If You still have issues with Your K8s cluster you might want to create new question on stack addressing them.</p>
