<p>I'm having a hard time with pulling image from private repository. Here's the drill down:</p>
<p>The pod:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
[...]
spec:
  containers:
  - image: gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main
    imagePullPolicy: Always
[...]
  imagePullSecrets:
    - name: gitlab-dc-cdndemo-2
</code></pre>
<p>And the pull secret:</p>
<pre><code>$ base64 -d &lt;(kubectl -n test-cdndemo get secret gitlab-dc-cdndemo-2 -o json | jq -r '.data.&quot;.dockerconfigjson&quot;') | jq
{
  &quot;auths&quot;: {
    &quot;https://gitlab.private:31443&quot;: {
      &quot;username&quot;: &quot;gitlab+deploy-token-22&quot;,
      &quot;password&quot;: &quot;EGDLqGKJwBtfYYf9cDFg&quot;,
      &quot;email&quot;: &quot;example@example.com&quot;,
      &quot;auth&quot;: &quot;Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw==&quot;
    }
  }
}
</code></pre>
<p>It's a playbook example of how it should be done. But when I deploy this I get:</p>
<pre><code>Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned test-cdndemo/appcdnmanagerui-68c8f8c6dd-qcxr5 to node-waw107
  Normal   Pulling    13m (x4 over 14m)     kubelet            Pulling image &quot;gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main&quot;
  Warning  Failed     13m (x4 over 14m)     kubelet            Failed to pull image &quot;gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main&quot;: rpc error: code = Unknown desc = failed to pull and unpack image &quot;gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest&quot;: failed to resolve referen
ce &quot;gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest&quot;: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     13m (x4 over 14m)     kubelet            Error: ErrImagePull
  Warning  Failed     12m (x6 over 14m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m41s (x43 over 14m)  kubelet            Back-off pulling image &quot;gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main&quot;
                                                                                       gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main
</code></pre>
<p>Notice the error, it's 403 Forbidden, not 401 Unauthorized so the credentials do work. Despite this, the image cannot be pulled from my private repo. But when I do this manually on a worker node everything goes smoothly:</p>
<pre><code>$ crictl --debug pull --creds gitlab+deploy-token-22:EGDLqGKJwBtfYYf9cDFg gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &amp;PullImageRequest{Image:&amp;ImageSpec{Image:gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest,Annotations:map[string]string{},},Auth:&amp;AuthConfig{Username:gitlab+deploy-token-22,Password:EGDLqGKJwBtfYYf9cDFg,Auth:,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,}
DEBU[0006] PullImageResponse: &amp;PullImageResponse{ImageRef:sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260,}
Image is up to date for sha256:0c3b5d355c164d02aaa7b6cbe91bbfa12bd35826566472317efac63cb467d260

$ crictl image
IMAGE                                                   TAG                  IMAGE ID            SIZE
[...]
gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main   latest               0c3b5d355c164       105MB```
</code></pre>
<p>It alwo works when I use auth data instead (as expected):</p>
<pre><code>crictl --debug pull --auth Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw== gitlab.private:31443/cdn/cdndemo/appcdnmanagerui/main:latest
</code></pre>
<p>Now I'm stuck. The only thing that comes to my mind is some kind of weird bug that's in the k3s release I'm using:</p>
<pre><code># k3s -v
k3s version v1.25.4+k3s1 (0dc63334)
go version go1.19.3
# crictl -v
crictl version v1.25.0-k3s1
</code></pre>
<p>The bug hypothesis is based on previous experience with older k8s deployment, where this approach is used and it works. But on a fresh cluster all I get is 403 error from k3s despite crictl pulling the image with no problems.</p>
<p>Anyone had this kind of problem before and solved it?</p>

<p>OK, I've found the problem. Instead of:</p>
<pre><code>{
  &quot;auths&quot;: {
    &quot;https://gitlab.private:31443&quot;: {
      &quot;username&quot;: &quot;gitlab+deploy-token-22&quot;,
      &quot;password&quot;: &quot;EGDLqGKJwBtfYYf9cDFg&quot;,
      &quot;email&quot;: &quot;example@example.com&quot;,
      &quot;auth&quot;: &quot;Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw==&quot;
    }
  }
}
</code></pre>
<p>It should have been:</p>
<pre><code>    {
      &quot;auths&quot;: {
        &quot;gitlab.private:31443&quot;: {
          &quot;username&quot;: &quot;gitlab+deploy-token-22&quot;,
          &quot;password&quot;: &quot;EGDLqGKJwBtfYYf9cDFg&quot;,
          &quot;email&quot;: &quot;example@example.com&quot;,
          &quot;auth&quot;: &quot;Z2l0bGFiK2RlcGxveS10b2tlbi0yMjpFR0RMcUdLSndCdGZZWWY5Y0RGZw==&quot;
        }
      }
    }
</code></pre>
<p>Apparently the documentation at <a href="https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/</a> is a bit misleading since the example there does contain https:// in the URL.</p>
<pre><code>{
    &quot;auths&quot;: {
        &quot;https://index.docker.io/v1/&quot;: {
            &quot;auth&quot;: &quot;c3R...zE2&quot;
        }
    }
}
</code></pre>
<p>And yes, my private repo does work over HTTPS connections.</p>
