<p>I'm using a bitnami kubernetes image on a AWS EC2 Instance.</p>
<p>kubectl cluster-info says</p>
<pre><code>Kubernetes master is running at https://172.30.0.120:6443
Heapster is running at https://172.30.0.120:6443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://172.30.0.120:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
monitoring-grafana is running at https://172.30.0.120:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
monitoring-influxdb is running at https://172.30.0.120:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
</code></pre>
<p>so I'm asuming i only have a master and no node?</p>
<p>I deployed a private registry via</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
        id: &quot;registry&quot;
    spec:
      containers:
      - name: registry
        image: registry:2.7.0
        ports:
        - name: registry-port
          containerPort: 5000
        volumeMounts:
        - mountPath: /var/lib/registry
          name: images
        - mountPath: /certs
          name: certs
        - mountPath: /auth
          name: auth
        env:
        - name: REGISTRY_AUTH
          value: &quot;htpasswd&quot;
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: &quot;Registry Realm&quot;
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: /auth/htpasswd
        - name: REGISTRY_HTTP_TLS_CERTIFICATE
          value: /certs/domain.crt
        - name: REGISTRY_HTTP_TLS_KEY
          value: /certs/domain.key
      volumes:
      - name: images
        hostPath:
          path: /home/bitnami/registry/images
      - name: certs
        hostPath:
          path: /home/bitnami/docker_reg_certs
      - name: auth
        hostPath:
          path: /home/bitnami/registry/auth
</code></pre>
<p>I generate a self-signed ssl-ca via openssl and copied it to /etc/docker/certs.d/[ec2-insta-domain:port] and to /usr/local/share/ca-certificates/ca.crt and do update-ca-certificates and sudo service docker restart.
After that i login to my registry with docker login [domainname]:5000 and enter user and pw. This work.
Then i create a docker-registry secret with kubectl create secret generic my-secret --from-file=.dockerconfigjson=~/.docker/config.json --type=kubernetes.io/dockerconfigjson</p>
<p>After that i create my deployment</p>
<pre><code>kind: Deployment
apiVersion: apps/v1
metadata:
  name: app-config-service
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: app-config-service
  template:
    metadata:
      labels:
        app: app-config-service
    spec:
      containers:
      - name: app-config-service
        image: [AWS-EC2-DomainName]:5000/cloud/app-config-service:latest
        ports:
        - containerPort: 48000
        env:
        - name: SPRING_PROFILE
          value: docker
        - name: SPRING_BOOT_PORT
          value: &quot;48000&quot;
        - name: KAFKA_BINDER_BROKERS_URL
          value: xxxxxxxxxxx:29095
      imagePullSecrets:
        - name: my-secret
</code></pre>
<p>But when i try to run this deployment with kubectl create -f deployment.yml and describe the created pod i get the error failed to do request: Head https://[domainname]:5000/v2/cloud/app-config-service/manifests/latest: x509: certificate signed by unknown authority</p>
<p>The log of the registry says 2020/07/14 15:54:01 http: TLS handshake error from 10.32.0.1:37943: remote error: tls: bad certificate</p>
<p>If i try to connect manually to the private repositry with curl -X GET https://[domainname]:5000/v2/_catalog it works.</p>
<p>Anyone know what i'm missing or doing wrong here?</p>
<p>Greetings</p>
EDIT
<p>I found out something interesting. For my kubernetes i use the container runtime containerd github.com/containerd/containerd v1.3.3 d76c121f76a5fc8a462dc64594aea72fe18e1178</p>
<p>I edit the /etc/containered/config.toml like this</p>
<pre><code>version = 2
root = &quot;/var/lib/containerd&quot;
state = &quot;/run/containerd&quot;
plugin_dir = &quot;&quot;
disabled_plugins = []
required_plugins = []
oom_score = 0

[grpc]
  address = &quot;/run/containerd/containerd.sock&quot;
  tcp_address = &quot;&quot;
  tcp_tls_cert = &quot;&quot;
  tcp_tls_key = &quot;&quot;
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[ttrpc]
  address = &quot;&quot;
  uid = 0
  gid = 0

[debug]
  address = &quot;&quot;
  uid = 0
  gid = 0
  level = &quot;&quot;

[metrics]
  address = &quot;&quot;
  grpc_histogram = false

[cgroup]
  path = &quot;&quot;

[timeouts]
  &quot;io.containerd.timeout.shim.cleanup&quot; = &quot;5s&quot;
  &quot;io.containerd.timeout.shim.load&quot; = &quot;5s&quot;
  &quot;io.containerd.timeout.shim.shutdown&quot; = &quot;3s&quot;
  &quot;io.containerd.timeout.task.state&quot; = &quot;2s&quot;

[plugins]
  [plugins.&quot;io.containerd.gc.v1.scheduler&quot;]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = &quot;0s&quot;
    startup_delay = &quot;100ms&quot;
  [plugins.&quot;io.containerd.grpc.v1.cri&quot;]
    disable_tcp_service = true
    stream_server_address = &quot;127.0.0.1&quot;
    stream_server_port = &quot;0&quot;
    stream_idle_timeout = &quot;4h0m0s&quot;
    enable_selinux = false
    sandbox_image = &quot;k8s.gcr.io/pause:3.1&quot;
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    disable_cgroup = false
    disable_apparmor = false
    restrict_oom_score_adj = false
    max_concurrent_downloads = 3
    disable_proc_mount = false
    [plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd]
      snapshotter = &quot;overlayfs&quot;
      default_runtime_name = &quot;runc&quot;
      no_pivot = false
      [plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.default_runtime]
        runtime_type = &quot;&quot;
        runtime_engine = &quot;&quot;
        runtime_root = &quot;&quot;
        privileged_without_host_devices = false
      [plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.untrusted_workload_runtime]
        runtime_type = &quot;&quot;
        runtime_engine = &quot;&quot;
        runtime_root = &quot;&quot;
        privileged_without_host_devices = false
      [plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.runtimes]
        [plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.runtimes.runc]
          runtime_type = &quot;io.containerd.runc.v1&quot;
          runtime_engine = &quot;&quot;
          runtime_root = &quot;&quot;
          privileged_without_host_devices = false
    [plugins.&quot;io.containerd.grpc.v1.cri&quot;.cni]
      bin_dir = &quot;/opt/cni/bin&quot;
      conf_dir = &quot;/etc/cni/net.d&quot;
      max_conf_num = 1
      conf_template = &quot;&quot;
    [plugins.&quot;io.containerd.grpc.v1.cri&quot;.registry]
    [plugins.&quot;io.containerd.grpc.v1.cri&quot;.registry.mirrors]
      [plugins.&quot;io.containerd.grpc.va.cri&quot;.registry.mirrors.&quot;ip:5000&quot;]
        endpoint = [&quot;http://ip:5000&quot;]
        [plugins.&quot;io.containerd.gprc.v1.cri&quot;.registry.auths]
      [plugins.&quot;io.containerd.grpc.va.cri&quot;.registry.mirrors.&quot;172.30.0.120:5000&quot;]
        endpoint = [&quot;http://172.30.0.120:5000&quot;]
      [plugins.&quot;io.containerd.grpc.va.cri&quot;.registry.mirrors.&quot;domainname:5000&quot;]
        endpoint = [&quot;http://domainame:5000&quot;]
      [plugins.&quot;io.containerd.grpc.v1.cri&quot;.registry.mirrors.&quot;docker.io&quot;]
        endpoint = [&quot;https://registry-1.docker.io&quot;]
    [plugins.&quot;io.containerd.gprc.v1.cri&quot;.registry.auths.&quot;http://ip:5000&quot;]
      username = &quot;admin&quot;
      password = &quot;admin&quot;
    [plugins.&quot;io.containerd.grpc.v1.cri&quot;.x509_key_pair_streaming]
      tls_cert_file = &quot;&quot;
      tls_key_file = &quot;&quot;
  [plugins.&quot;io.containerd.internal.v1.opt&quot;]
    path = &quot;/opt/containerd&quot;
  [plugins.&quot;io.containerd.internal.v1.restart&quot;]
    interval = &quot;10s&quot;
  [plugins.&quot;io.containerd.metadata.v1.bolt&quot;]
    content_sharing_policy = &quot;shared&quot;
  [plugins.&quot;io.containerd.monitor.v1.cgroups&quot;]
    no_prometheus = false
  [plugins.&quot;io.containerd.runtime.v1.linux&quot;]
    shim = &quot;containerd-shim&quot;
    runtime = &quot;runc&quot;
    runtime_root = &quot;&quot;
    no_shim = false
    shim_debug = false
  [plugins.&quot;io.containerd.runtime.v2.task&quot;]
    platforms = [&quot;linux/amd64&quot;]
  [plugins.&quot;io.containerd.service.v1.diff-service&quot;]
    default = [&quot;walking&quot;]
  [plugins.&quot;io.containerd.snapshotter.v1.devmapper&quot;]
    root_path = &quot;&quot;
    pool_name = &quot;&quot;
    base_image_size = &quot;&quot;

</code></pre>
<p>but if i call crictl info the output show me the following</p>
<pre><code>...
...
 },
    &quot;registry&quot;: {
      &quot;mirrors&quot;: {
        &quot;docker.io&quot;: {
          &quot;endpoint&quot;: [
            &quot;https://registry-1.docker.io&quot;
          ]
        }
      },
...
...
</code></pre>
<p>So the <code>config.toml</code> has no effect of the used registry of my kubernetes? Any ideas?</p>

JensV
<p>Looks like you created the secret <code>my-secret</code> in default namespace but the deployment <code>app-config-service</code> is running in <code>kube-system</code> namespace and hence can not refer to <code>my-secret</code> using <code>imagePullSecrets</code>. Creating the secret <code>my-secret</code> in <code>kube-system</code> namespace should solve the issue.</p>
