<p>I am getting ErrImagePull when trying to create a deployment from an image hosted on my private helm docker registry. Based on the "server gave HTTP response to HTTPS client" error, I tried adding the --insecure-registry option in systemd drop-in as well as the daemon.json file on all worker nodes and master but it's still not working.  What else should I try to troubleshoot?</p>

<p>edit: Perhaps it's because docker is using containerd?</p>

<pre class="lang-sh prettyprint-override"><code>    $ sudo systemctl status docker
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/docker.service.d
               └─docker.conf
       Active: active (running) since Fri 2020-02-07 10:00:25 UTC; 4min 44s ago
         Docs: https://docs.docker.com
     Main PID: 27700 (dockerd)
        Tasks: 14
       Memory: 41.8M
       CGroup: /system.slice/docker.service
               └─27700 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
</code></pre>

<pre class="lang-sh prettyprint-override"><code>    $ cat /etc/systemd/system/docker.service.d/docker.conf
    DOCKER_OPTS="--insecure-registry 10.10.30.200:5000"
</code></pre>

<pre class="lang-sh prettyprint-override"><code>    $ cat /etc/docker/daemon.json
</code></pre>

<pre class="lang-sh prettyprint-override"><code>    {
      "exec-opts": ["native.cgroupdriver=systemd"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2",
      "insecure-registries": ["10.10.30.200:5000"]
    }    
</code></pre>

<pre class="lang-sh prettyprint-override"><code>    $ curl 10.10.30.200:5000/v2/mybuild/tags/list
        {"name":"mybuild","tags":["v1"]}
</code></pre>

<pre class="lang-sh prettyprint-override"><code>    $ kubectl describe pod myweb-769d57d99-lz6xs
    ...
    Normal   Pulling    1s (x2 over 13s)   kubelet, k8s-node2  Pulling image "10.10.30.200:5000/mybuild:v1"
    Warning  Failed     1s (x2 over 13s)   kubelet, k8s-node2  Failed to pull image "10.10.30.200:5000/mybuild:v1": rpc error: code = Unknown desc = failed to resolve image "10.10.30.200:5000/mybuild:v1": no available registry endpoint: failed to do request: Head https://10.10.30.200:5000/v2/mybuild/manifests/v1: http: server gave HTTP response to HTTPS client
    Warning  Failed     1s (x2 over 13s)   kubelet, k8s-node2  Error: ErrImagePull
</code></pre>

<pre class="lang-sh prettyprint-override"><code>    $ cat deployment.yaml
</code></pre>

<pre class="lang-sh prettyprint-override"><code>    ---
    apiVersion: apps/v1
    kind: Deployment                                                                                                                                         metadata:
      labels:
        app: myweb
      name: myweb
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: myweb
      template:
        metadata:
          labels:
            app: myweb
        spec:
          containers:
          - image: 10.10.30.200:5000/mybuild:v1
            imagePullPolicy: Always
            name: myweb

    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: myweb
      name: myweb
    spec:
      ports:
      - nodePort: 32223
        port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: myweb
      type: NodePort
</code></pre>

<p>I reproduced your issue and the solution is to add </p>

<pre><code>{
  "insecure-registry" : ["10.10.30.200:5000"]
}
</code></pre>

<p>on <strong>every node (workers and master)</strong> in <code>/etc/docker/daemon.json</code> in the cluster and restarting docker <code>sudo systemctl restart docker</code> to load new configuration.</p>

<p>You can follow <a href="https://guide.opencord.org/cord-6.0/prereqs/docker-registry.html" rel="nofollow noreferrer">this guide</a> on how to set up insecure private docker registry using helm. </p>

<p>Once the chart was successfully installed, add entry in <code>daemon.json</code> on every node and restart docker. After that you can tag and push your image to the repository.</p>

<p>To check if the image was successfully pushed to your registry you can run:</p>

<p><code>curl -X GET http://10.10.30.200:5000/v2/_catalog</code> </p>

<p>and the output should be similar to this: </p>

<p><code>{"repositories":["ubuntu/myimage"]}</code></p>
