<p>I have a Kubernetes Cluster in an on-premise server, I also have a server on <a href="https://www.ncloud.com/" rel="nofollow noreferrer">Naver Cloud</a> lets call it <code>server A</code>, I want to join my <code>server A</code> to my Kubernetes Cluster, the server can join normally, but the <code>kube-proxy</code> and <code>kube-flannel</code> pods spawned from daemonset are constantly in <code>CrashLoopBackOff</code> status</p>
<p>here is the log from <code>kube-proxy</code></p>
<pre><code>I0405 03:13:48.566285       1 node.go:163] Successfully retrieved node IP: 10.1.0.2
I0405 03:13:48.566382       1 server_others.go:109] &quot;Detected node IP&quot; address=&quot;10.1.0.2&quot;
I0405 03:13:48.566420       1 server_others.go:535] &quot;Using iptables proxy&quot;
I0405 03:13:48.616989       1 server_others.go:176] &quot;Using iptables Proxier&quot;
I0405 03:13:48.617021       1 server_others.go:183] &quot;kube-proxy running in dual-stack mode&quot; ipFamily=IPv4
I0405 03:13:48.617040       1 server_others.go:184] &quot;Creating dualStackProxier for iptables&quot;
I0405 03:13:48.617063       1 server_others.go:465] &quot;Detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6&quot;
I0405 03:13:48.617093       1 proxier.go:242] &quot;Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses&quot;
I0405 03:13:48.617420       1 server.go:655] &quot;Version info&quot; version=&quot;v1.26.0&quot;
I0405 03:13:48.617435       1 server.go:657] &quot;Golang settings&quot; GOGC=&quot;&quot; GOMAXPROCS=&quot;&quot; GOTRACEBACK=&quot;&quot;
I0405 03:13:48.618790       1 conntrack.go:52] &quot;Setting nf_conntrack_max&quot; nf_conntrack_max=131072
</code></pre>
<p>there is no log from <code>kube-flannel</code>, <code>kube-flannel</code> pods failed on its Init containers named <code>install-cni-plugin</code>, when I try <code>kubectl -n kube-flannel logs kube-flannel-ds-d2l4q -c install-cni-plugin</code> it returns</p>
<pre><code>unable to retrieve container logs for docker://47e4c8c580474b384b128c8e4d74297a0e891b5f227c6313146908b06ee7b376
</code></pre>
<p>I have no other clue that I can think of, please tell me if I need to attach more info</p>
<p>Please help, I've been stuck for so long T.T</p>
<p>More info:</p>
<p><code>kubectl get nodes</code></p>
<pre><code>NAME                      STATUS     ROLES           AGE   VERSION
accio-randi-ed05937533    Ready      &lt;none&gt;          8d    v1.26.3
accio-test-1-b3fb4331ee   NotReady   &lt;none&gt;          89m   v1.26.3
master                    Ready      control-plane   48d   v1.26.1
</code></pre>
<p><code>kubectl -n kube-system get pods</code></p>
<pre><code>NAME                             READY   STATUS             RESTARTS         AGE
coredns-787d4945fb-rms6t         1/1     Running            0                30d
coredns-787d4945fb-t6g8s         1/1     Running            0                33d
etcd-master                      1/1     Running            168 (36d ago)    48d
kube-apiserver-master            1/1     Running            158 (36d ago)    48d
kube-controller-manager-master   1/1     Running            27 (6d17h ago)   48d
kube-proxy-2r8tn                 1/1     Running            6 (36d ago)      48d
kube-proxy-f997t                 0/1     CrashLoopBackOff   39 (90s ago)     87m
kube-proxy-wc9x5                 1/1     Running            0                8d
kube-scheduler-master            1/1     Running            27 (6d17h ago)   48d
</code></pre>
<p><code>kubectl -n kube-system get events</code></p>
<pre><code>LAST SEEN   TYPE      REASON             OBJECT                               MESSAGE
42s         Warning   DNSConfigForming   pod/coredns-787d4945fb-rms6t         Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
54s         Warning   DNSConfigForming   pod/coredns-787d4945fb-t6g8s         Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
3m10s       Warning   DNSConfigForming   pod/etcd-master                      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
2m48s       Warning   DNSConfigForming   pod/kube-apiserver-master            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
3m33s       Warning   DNSConfigForming   pod/kube-controller-manager-master   Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
3m7s        Warning   DNSConfigForming   pod/kube-proxy-2r8tn                 Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
15s         Normal    SandboxChanged     pod/kube-proxy-f997t                 Pod sandbox changed, it will be killed and re-created.
5m15s       Warning   BackOff            pod/kube-proxy-f997t                 Back-off restarting failed container kube-proxy in pod kube-proxy-f997t_kube-system(7652a1c4-9517-4a8a-a736-1f746f36c7ab)
3m30s       Warning   DNSConfigForming   pod/kube-scheduler-master            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
</code></pre>
<p><code>kubectl -n kube-flannel get pods</code></p>
<pre><code>NAME                    READY   STATUS                  RESTARTS      AGE
kube-flannel-ds-2xgbw   1/1     Running                 0             8d
kube-flannel-ds-htgts   0/1     Init:CrashLoopBackOff   0 (2s ago)    88m
kube-flannel-ds-sznbq   1/1     Running                 6 (36d ago)   48d
</code></pre>
<p><code>kubectl -n kube-flannel get events</code></p>
<pre><code>LAST SEEN   TYPE      REASON             OBJECT                      MESSAGE
100s        Normal    SandboxChanged     pod/kube-flannel-ds-htgts   Pod sandbox changed, it will be killed and re-created.
26m         Normal    Pulled             pod/kube-flannel-ds-htgts   Container image &quot;docker.io/flannel/flannel-cni-plugin:v1.1.2&quot; already present on machine
46m         Warning   BackOff            pod/kube-flannel-ds-htgts   Back-off restarting failed container install-cni-plugin in pod kube-flannel-ds-htgts_kube-flannel(4f602997-5502-4dcf-8fca-23eba01325dd)
5m          Warning   DNSConfigForming   pod/kube-flannel-ds-sznbq   Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.8.0.1 192.168.18.1 fe80::1%3
</code></pre>
<p><code>kubectl -n kube-flannel describe pod kube-flannel-ds-htgts</code></p>
<pre><code>Name:                 kube-flannel-ds-htgts
Namespace:            kube-flannel
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      flannel
Node:                 accio-test-1-b3fb4331ee/10.1.0.2
Start Time:           Thu, 06 Apr 2023 09:25:12 +0900
Labels:               app=flannel
                      controller-revision-hash=6b7b59d784
                      k8s-app=flannel
                      pod-template-generation=1
                      tier=node
Annotations:          &lt;none&gt;
Status:               Pending
IP:                   10.1.0.2
IPs:
  IP:           10.1.0.2
Controlled By:  DaemonSet/kube-flannel-ds
Init Containers:
  install-cni-plugin:
    Container ID:  docker://0fed30cc41f305203bf5d6fb7668f92f449a65f722faf1360e61231e9107ef66
    Image:         docker.io/flannel/flannel-cni-plugin:v1.1.2
    Image ID:      docker-pullable://flannel/flannel-cni-plugin@sha256:bf4b62b131666d040f35a327d906ee5a3418280b68a88d9b9c7e828057210443
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Command:
      cp
    Args:
      -f
      /flannel
      /opt/cni/bin/flannel
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 06 Apr 2023 15:11:34 +0900
      Finished:     Thu, 06 Apr 2023 15:11:34 +0900
    Ready:          True
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /opt/cni/bin from cni-plugin (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gbk6z (ro)
  install-cni:
    Container ID:
    Image:         docker.io/flannel/flannel:v0.21.0
    Image ID:
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Command:
      cp
    Args:
      -f
      /etc/kube-flannel/cni-conf.json
      /etc/cni/net.d/10-flannel.conflist
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /etc/cni/net.d from cni (rw)
      /etc/kube-flannel/ from flannel-cfg (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gbk6z (ro)
Containers:
  kube-flannel:
    Container ID:
    Image:         docker.io/flannel/flannel:v0.21.0
    Image ID:
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Command:
      /opt/bin/flanneld
    Args:
      --ip-masq
      --kube-subnet-mgr
      --iface=accio-k8s-net
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:                 kube-flannel-ds-htgts (v1:metadata.name)
      POD_NAMESPACE:            kube-flannel (v1:metadata.namespace)
      KUBERNETES_SERVICE_HOST:  10.1.0.1
      KUBERNETES_SERVICE_PORT:  6443
      EVENT_QUEUE_DEPTH:        5000
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/flannel from run (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gbk6z (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  run:
    Type:          HostPath (bare host directory volume)
    Path:          /run/flannel
    HostPathType:
  cni-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-flannel-cfg
    Optional:  false
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  kube-api-access-gbk6z:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              &lt;none&gt;
Tolerations:                 :NoSchedule op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason          Age                      From     Message
  ----     ------          ----                     ----     -------
  Warning  BackOff         31m (x8482 over 5h46m)   kubelet  Back-off restarting failed container install-cni-plugin in pod kube-flannel-ds-htgts_kube-flannel(4f602997-5502-4dcf-8fca-23eba01325dd)
  Normal   Created         21m (x8783 over 5h46m)   kubelet  Created container install-cni-plugin
  Normal   Pulled          11m (x9051 over 5h46m)   kubelet  Container image &quot;docker.io/flannel/flannel-cni-plugin:v1.1.2&quot; already present on machine
  Normal   SandboxChanged  81s (x18656 over 5h46m)  kubelet  Pod sandbox changed, it will be killed and re-created.
</code></pre>

Rahandi Noor Pasha
<p>I had a similar issue in one of my nodes due to the container runtime being incorrectly configured. Please check the <code>containerd</code> configuration located at <code>/etc/containerd/config.toml</code> for specifying daemon level options.
The default configuration can be generated by running</p>
<pre><code>containerd config default &gt; /etc/containerd/config.toml
</code></pre>
<p>To use the systemd cgroup driver in <code>/etc/containerd/config.toml</code> with <code>runc</code>, set</p>
<pre><code>[plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.runtimes.runc]
  ...
 [plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.runtimes.runc.options]
    SystemdCgroup = true
</code></pre>
<p>If the cgroup driver is incorrect, that might lead to the pod in that node always being in a <code>CrashLoopBackOff</code>.</p>
