<p>We are having trouble giving a container within a pipeline uploaded to Kubeflow access to a private custom docker image stored in a google container registry. We are running kubeflow on top of a kubernetes cluster run on minikube. Can someone help us understand how to add the access token/service account to the Kubeflow deployment? We have read a couple of docs that achieve this on a custom Kubernetes deployment but not on a Kubeflow deployment.</p>

<p>The error we get when running the pipeline on Kubeflow is:
<strong>This step is in Pending state with this message: ImagePullBackOff: Back-off pulling image</strong></p>

<p>This is the pipeline code that calls the image.
<a href="https://i.stack.imgur.com/LLud9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LLud9.png" alt="enter image description here"></a></p>

<p>Thank you!!</p>

Federico K
<p>This is issues can occur in some scenarios like:</p>

<ul>
<li><p>Your kubeflow setup (Kubernetes cluster) and GCR are in different project</p></li>
<li><p>No GCR secret for the ml-pipeline service account which is responsible to run the pipeline. (you can see this kubectl --namespace=kubeflow get serviceaccount)</p></li>
</ul>

<p>In your case, I think it is the second scenario. Though the following path will work on both scenarios.</p>

<ol>
<li>Create service_account.json with sufficient permission (GCR needs storage permission so give 'Storage admin') using the GCP console</li>
</ol>

<pre><code>Select “API &amp; Services” &gt; “Credentials”Select “Create credentials” &gt; “Services Account Key” &gt; “Create New Services Account”

</code></pre>

<ol start="2">
<li>Add a Kubernetes Secret in Kubernetes Cluster to access GCR</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>kubectl create secret docker-registry $SECRETNAME \       
--docker-server=https://gcr.io \                          
--docker-username=_json_key \                             
--docker-email=user@example.com \                          
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key
</code></pre>

<ul>
<li>Above method is for default service account. But patch this in Kufelow namespace</li>
</ul>

<pre class="lang-sh prettyprint-override"><code>kubectl --namespace=kubeflow create secret docker-registry $SECRETNAME \  
--docker-server=https://gcr.io \                          
--docker-username=_json_key \                             
--docker-email=user@example.com \                          
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key
</code></pre>

<ol start="3">
<li>Patching GCR secret with respective service account</li>
</ol>

<pre class="lang-sh prettyprint-override"><code># For Kubeflow specific problem path pipeline-runner serviceaccount
kubectl --namespace=kubeflow patch serviceaccount pipeline-runner -p '{"imagePullSecrets": [{"name": "$SECRETNAME"}]}'
</code></pre>
