<p>Been learning kubernetes in the past several weeks. I've recently built a bare-metal kubernetes cluster with (3) master nodes and (3) worker nodes (containerd runtime). Installed an another stand-alone bare-metal gitlab server with container registry enabled.</p>
<p>I was successful in building a simple nginx container with a custom index.html using docker build and pushed it to the registry; up until this point everything works great.</p>
<p>Now I wanted to create a simple pod using the image built above.<br />
So, did the following steps.</p>
<ol>
<li>Created a deploy token with read_registry access</li>
<li>Created a secret in kubernetes with username and the token as the password</li>
<li>Inserted <code>imagePullSecrets</code> to the deployment yaml file.</li>
<li>kubectl apply -f nginx.yaml.</li>
</ol>
<p>Kubernetes pod status stays in <code>ImagePullBackOff</code>.</p>
<pre><code>Failed to pull image &quot;&lt;gitlab-host&gt;:5050/&lt;user&gt;/&lt;project&gt;/nginx:v1&quot;: rpc error: code = FailedPrecondition desc = failed to pull and unpack image  
&quot;&lt;gitlab-host&gt;:5050/&lt;user&gt;/&lt;project&gt;/nginx:v1&quot;: failed commit on ref &quot;unknown-sha256:4ca40a571e91ac4c425500a504490a65852ce49c1f56d7e642c0ec44d13be252&quot;: unexpected commit digest sha256:0d899af03c0398a85e36d5cd7ee9a8828e5618db255770a4a96331785ff26d9c, expected sha256:4ca40a571e91ac4c425500a504490a65852ce49c1f56d7e642c0ec44d13be252: failed precondition. 
</code></pre>
<p>Troubleshooting steps followed.</p>
<ol>
<li><code>docker login</code> from another server works.</li>
<li><code>docker pull</code> works</li>
<li>In one of the worker nodes where kubernetes was scheduling the pod, I did <code>ctr image pull</code> which works</li>
</ol>
<p>Did some googling but couldn't find any solutions.  So, here I am as a last resort to figure this out.<br />
Appreciate any help that I get.</p>
<p>My Deployment nginx.yml file</p>
<pre><code>---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: &lt;gitlab-host&gt;:5050/&lt;username&gt;/&lt;project&gt;/nginx:v1
        imagePullPolicy: IfNotPresent
        name: nginx
      imagePullSecrets:
      - name: regcred
</code></pre>

Bhargav
<p>I found the problem. I made a silly mistake in <code>/etc/containerd/config.toml</code> in the registry section and not mentioning the endpoint with port number <code>&lt;gitlab-host&gt;:5050</code>.<br />
And also adding the private registries in config.toml is not necessary unless you want to run ctr command on the k8s nodes.</p>
