<p>I have a two node cluster in AWS which I set up using K3S v1.21.5+k3s1. I just did a default installation, so I have Traefik as my IngressController which should be binding to ports 80 and 443 on each node. However, checking <code>netstat</code> shows that nothing is listening on those ports for either node, and any attempt to connect to running pods from my browser fails.</p>
<p>The interesting thing is, I've tried rebooting the nodes to see if that fixes the issue (it doesn't), and I see that in addition to the normal <code>svclb-traefik</code> Daemonset, a second Daemonset gets created called <code>svclb-traefik-######</code>:</p>
<pre><code>$ kubectl -n kube-system get ds
NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
svclb-traefik                    2         2         2       2            2           &lt;none&gt;          21d
svclb-traefik-23fcfc42           2         2         0       2            0           &lt;none&gt;          8m45s
</code></pre>
<p>Checking that second Daemonset, I see that the pods it's trying to start up are failing because the nodes they are allocated to do not have free ports. This is as expected, since the &quot;real&quot; svclb-traefik appears to have already bound them (at least as far as kubernetes knows). Deleting this second svclb-traefik Daemonset does not solve the problem, and the Daemonset returns with the next reboot.</p>
<p>I've checked the traefik daemonset, the traefik LoadBalancer Service, and all the traefik pods to see what could be wrong, (output from <code>kubectl describe</code> is below) but everything looks healthy. What other things should I be looking at?</p>
<p>The Daemonset:</p>
<pre><code>$ kubectl -n kube-system describe ds svclb-traefik
Name:           svclb-traefik
Selector:       app=svclb-traefik
Node-Selector:  &lt;none&gt;
Labels:         objectset.rio.cattle.io/hash=f31475152fbf70655d3c016d368e90118938f6ea
                svccontroller.k3s.cattle.io/nodeselector=false
Annotations:    deprecated.daemonset.template.generation: 1
                objectset.rio.cattle.io/applied:
                  H4sIAAAAAAAA/8xUTW/jNhD9K8WcKUWy7EQW0MMiySFoNzFsby+BEVDkKGZNkQI5UmMY+u8FZWftbL6KbQ89ejjz/Oa90dvBRhkJBVxxrK1ZIAED3qg/0HllDRTAm8afdSkwqJG45M...
                objectset.rio.cattle.io/id: svccontroller
                objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
                objectset.rio.cattle.io/owner-name: traefik
                objectset.rio.cattle.io/owner-namespace: kube-system
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 2
Number of Nodes Misscheduled: 0
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=svclb-traefik
           svccontroller.k3s.cattle.io/svcname=traefik
  Containers:
   lb-port-80:
    Image:      rancher/klipper-lb:v0.2.0
    Port:       80/TCP
    Host Port:  80/TCP
    Environment:
      SRC_PORT:    80
      DEST_PROTO:  TCP
      DEST_PORT:   80
      DEST_IP:     10.43.82.221
    Mounts:        &lt;none&gt;
   lb-port-443:
    Image:      rancher/klipper-lb:v0.2.0
    Port:       443/TCP
    Host Port:  443/TCP
    Environment:
      SRC_PORT:    443
      DEST_PROTO:  TCP
      DEST_PORT:   443
      DEST_IP:     10.43.82.221
    Mounts:        &lt;none&gt;
  Volumes:         &lt;none&gt;
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  36m   daemonset-controller  Created pod: svclb-traefik-pjffb
</code></pre>
<p>The Service:</p>
<pre><code>$ kubectl -n kube-system describe svc traefik
Name:                     traefik
Namespace:                kube-system
Labels:                   app.kubernetes.io/instance=traefik
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-9.18.2
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.82.221
IPs:                      10.43.82.221
LoadBalancer Ingress:     &lt;node 1 IP&gt;, &lt;node 2 IP&gt;
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  30234/TCP
Endpoints:                10.42.1.116:8000
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  32623/TCP
Endpoints:                10.42.1.116:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  16m   service-controller  Ensuring load balancer
  Normal  AppliedDaemonSet      16m   service-controller  Applied LoadBalancer DaemonSet kube-system/svclb-traefik-23fcfc42
</code></pre>
<p>One of the &quot;good&quot; pods (from svclb-traefik):</p>
<pre><code>Name:         svclb-traefik-pjffb
Namespace:    kube-system
Priority:     0
Node:         &lt;node 1&gt;
Start Time:   Fri, 03 Feb 2023 08:25:09 -0500
Labels:       app=svclb-traefik
              controller-revision-hash=56b6bf6489
              pod-template-generation=1
              svccontroller.k3s.cattle.io/svcname=traefik
Annotations:  &lt;none&gt;
Status:       Running
IP:           10.42.1.96
IPs:
  IP:           10.42.1.96
Controlled By:  DaemonSet/svclb-traefik
Containers:
  lb-port-80:
    Container ID:   containerd://6ae25fd4dea39238f3d222dce1a25e3b01a7fb159cecd3e2684257e91dbfd4d7
    Image:          rancher/klipper-lb:v0.2.0
    Image ID:       docker.io/rancher/klipper-lb@sha256:5ea5f7904c404085ff24541a0e7a2267637af4bcf30fae9b747d871bfcd8a6f7
    Port:           80/TCP
    Host Port:      80/TCP
    State:          Running
      Started:      Fri, 03 Feb 2023 08:46:33 -0500
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Fri, 03 Feb 2023 08:25:10 -0500
      Finished:     Fri, 03 Feb 2023 08:46:05 -0500
    Ready:          True
    Restart Count:  1
    Environment:
      SRC_PORT:    80
      DEST_PROTO:  TCP
      DEST_PORT:   80
      DEST_IP:     10.43.82.221
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-72smb (ro)
  lb-port-443:
    Container ID:   containerd://b9ba3ec21cbd249f1e03d0f3230af9774ff7208ca56a5623a2b45b595a76889e
    Image:          rancher/klipper-lb:v0.2.0
    Image ID:       docker.io/rancher/klipper-lb@sha256:5ea5f7904c404085ff24541a0e7a2267637af4bcf30fae9b747d871bfcd8a6f7
    Port:           443/TCP
    Host Port:      443/TCP
    State:          Running
      Started:      Fri, 03 Feb 2023 08:46:33 -0500
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Fri, 03 Feb 2023 08:25:10 -0500
      Finished:     Fri, 03 Feb 2023 08:46:06 -0500
    Ready:          True
    Restart Count:  1
    Environment:
      SRC_PORT:    443
      DEST_PROTO:  TCP
      DEST_PORT:   443
      DEST_IP:     10.43.82.221
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-72smb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-72smb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              &lt;none&gt;
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                             node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Created         17m   kubelet            Created container lb-port-80
  Normal  Started         17m   kubelet            Started container lb-port-80
  Normal  Created         17m   kubelet            Created container lb-port-443
  Normal  Started         17m   kubelet            Started container lb-port-443
</code></pre>
<p>One of the &quot;bad&quot; pods (from the svclb-traefik-XXXXX Daemonset):</p>
<pre><code>Name:           svclb-traefik-23fcfc42-t6jx7
Namespace:      kube-system
Priority:       0
Node:           &lt;none&gt;
Labels:         app=svclb-traefik-23fcfc42
                controller-revision-hash=74f5f855c9
                pod-template-generation=1
                svccontroller.k3s.cattle.io/svcname=traefik
                svccontroller.k3s.cattle.io/svcnamespace=kube-system
Annotations:    &lt;none&gt;
Status:         Pending
IP:
IPs:            &lt;none&gt;
Controlled By:  DaemonSet/svclb-traefik-23fcfc42
Containers:
  lb-tcp-80:
    Image:      rancher/klipper-lb:v0.4.0
    Port:       80/TCP
    Host Port:  80/TCP
    Environment:
      SRC_PORT:    80
      SRC_RANGES:  0.0.0.0/0
      DEST_PROTO:  TCP
      DEST_PORT:   80
      DEST_IPS:    10.43.82.221
    Mounts:        &lt;none&gt;
  lb-tcp-443:
    Image:      rancher/klipper-lb:v0.4.0
    Port:       443/TCP
    Host Port:  443/TCP
    Environment:
      SRC_PORT:    443
      SRC_RANGES:  0.0.0.0/0
      DEST_PROTO:  TCP
      DEST_PORT:   443
      DEST_IPS:    10.43.82.221
    Mounts:        &lt;none&gt;
Conditions:
  Type           Status
  PodScheduled   False
Volumes:         &lt;none&gt;
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     CriticalAddonsOnly op=Exists
                 node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                 node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  19m   default-scheduler  0/2 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.
  Warning  FailedScheduling  19m   default-scheduler  0/2 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.
</code></pre>

redsoxfantom
<p>Extremely unsatisfying answer here, but I ended up deleting <em>both</em> the &quot;real&quot; <code>svclb-traefik</code> and the <code>svclb-traefik-XXXXX</code> Daemonsets and rebooting the nodes. On startup, both Daemonsets were recreated (and the <code>-XXXXX</code> one was still failing to start it's pods), but I can at least connect to my pods from my browser. I'll update this answer if the problem recurs.</p>
