<p>Let me preface this by saying this is running on a production cluster, so any 'destructive' solution that will cause downtime will not be an option (unless absolutely necessary).</p>
<p><strong>My environment</strong></p>
<p>I have a Kubernetes cluster (11 nodes, 3 of which are master nodes) running v1.13.1 on AWS. This cluster was created via kOps like so:</p>
<pre class="lang-sh prettyprint-override"><code>kops create cluster \
    --yes \
    --authorization RBAC \
    --cloud aws \
    --networking calico \
    ...
</code></pre>
<p>I don't think this is relevant, but everything on the cluster has been installed via helm3.</p>
<p>Here are my exact versions:</p>
<pre><code>$ helm version
version.BuildInfo{Version:&quot;v3.4.1&quot;, GitCommit:&quot;c4e74854886b2efe3321e185578e6db9be0a6e29&quot;, GitTreeState:&quot;dirty&quot;, GoVersion:&quot;go1.15.5&quot;}
</code></pre>
<pre><code>$ kubectl version
Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.1&quot;, GitCommit:&quot;c4d752765b3bbac2237bf87cf0b1c2e307844666&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2020-12-19T08:38:20Z&quot;, GoVersion:&quot;go1.15.5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;13&quot;, GitVersion:&quot;v1.13.1&quot;, GitCommit:&quot;eec55b9ba98609a46fee712359c7b5b365bdd920&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2018-12-13T10:31:33Z&quot;, GoVersion:&quot;go1.11.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
</code></pre>
<pre><code>$ kops version
Version 1.18.2
</code></pre>
<pre><code>$ kubectl get nodes                                                                                                                                                                            
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-2-147-44.ec2.internal    Ready    node     47h   v1.13.1
ip-10-2-149-115.ec2.internal   Ready    node     47h   v1.13.1
ip-10-2-150-124.ec2.internal   Ready    master   2d    v1.13.1
ip-10-2-151-33.ec2.internal    Ready    node     47h   v1.13.1
ip-10-2-167-145.ec2.internal   Ready    master   43h   v1.18.14
ip-10-2-167-162.ec2.internal   Ready    node     2d    v1.13.1
ip-10-2-172-248.ec2.internal   Ready    node     47h   v1.13.1
ip-10-2-173-134.ec2.internal   Ready    node     47h   v1.13.1
ip-10-2-177-100.ec2.internal   Ready    master   2d    v1.13.1
ip-10-2-181-235.ec2.internal   Ready    node     47h   v1.13.1
ip-10-2-182-14.ec2.internal    Ready    node     47h   v1.13.1
</code></pre>
<p><strong>What I am attempting to do</strong></p>
<p>I am trying to update the cluster from <code>v1.13.1</code> -&gt; <code>v1.18.14</code></p>
<p>I edited the config by</p>
<pre><code>$ kops edit cluster
</code></pre>
<p>and changed</p>
<pre><code>kubernetesVersion: 1.18.14
</code></pre>
<p>then I ran</p>
<pre class="lang-sh prettyprint-override"><code>kops update cluster --yes
kops rolling-update cluster --yes
</code></pre>
<p>Which then starting the rolling-update process.</p>
<pre><code>NAME                STATUS        NEEDUPDATE    READY   MIN   TARGET   MAX   NODES
master-us-east-1a   NeedsUpdate   1             0       1     1        1     1
master-us-east-1b   NeedsUpdate   1             0       1     1        1     1
master-us-east-1c   NeedsUpdate   1             0       1     1        1     1
nodes               NeedsUpdate   8             0       8     8        8     8
</code></pre>
<p><strong>The problem:</strong></p>
<p>The process gets stuck on the first node upgrade with this error</p>
<pre><code>I0108 10:48:40.137256   59317 instancegroups.go:440] Cluster did not pass validation, will retry in &quot;30s&quot;: master &quot;ip-10-2-167-145.ec2.internal&quot; is not ready, system-node-critical pod &quot;calico-node-m255f&quot; is not ready (calico-node).
I0108 10:49:12.474458   59317 instancegroups.go:440] Cluster did not pass validation, will retry in &quot;30s&quot;: system-node-critical pod &quot;calico-node-m255f&quot; is not ready (calico-node).
</code></pre>
<p><code>calico-node-m255f</code> is the only calico node in the cluster (I'm pretty sure there should be one for each k8s node?)</p>
<p>Info on that pod:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl get pods -n kube-system -o wide | grep calico-node
calico-node-m255f                                            0/1     Running             0          35m   10.2.167.145      ip-10-2-167-145.ec2.internal   &lt;none&gt;           &lt;none&gt;

$ kubectl describe pod calico-node-m255f -n kube-system

Name:                 calico-node-m255f
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 ip-10-2-167-145.ec2.internal/10.2.167.145
Start Time:           Fri, 08 Jan 2021 10:18:05 -0800
Labels:               controller-revision-hash=59875785d9
                      k8s-app=calico-node
                      pod-template-generation=5
                      role.kubernetes.io/networking=1
Annotations:          &lt;none&gt;
Status:               Running
IP:                   10.2.167.145
IPs:                  &lt;none&gt;
Controlled By:        DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  docker://9a6d035ee4a9d881574f45075e033597a33118e1ed2c964204cc2a5b175fbc60
    Image:         calico/cni:v3.15.3
    Image ID:      docker-pullable://calico/cni@sha256:519e5c74c3c801ee337ca49b95b47153e01fd02b7d2797c601aeda48dc6367ff
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 08 Jan 2021 10:18:06 -0800
      Finished:     Fri, 08 Jan 2021 10:18:06 -0800
    Ready:          True
    Restart Count:  0
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  &lt;set to the key 'calico_backend' of config map 'calico-config'&gt;  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-mnnrd (ro)
  install-cni:
    Container ID:  docker://5788e3519a2b1c1b77824dbfa090ad387e27d5bb16b751c3cf7637a7154ac576
    Image:         calico/cni:v3.15.3
    Image ID:      docker-pullable://calico/cni@sha256:519e5c74c3c801ee337ca49b95b47153e01fd02b7d2797c601aeda48dc6367ff
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Command:
      /install-cni.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 08 Jan 2021 10:18:07 -0800
      Finished:     Fri, 08 Jan 2021 10:18:08 -0800
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    &lt;set to the key 'cni_network_config' of config map 'calico-config'&gt;  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               &lt;set to the key 'veth_mtu' of config map 'calico-config'&gt;  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-mnnrd (ro)
  flexvol-driver:
    Container ID:   docker://bc8ad32a2dd0eb5bbb21843d4d248171bc117d2eede9e1efa9512026d9205888
    Image:          calico/pod2daemon-flexvol:v3.15.3
    Image ID:       docker-pullable://calico/pod2daemon-flexvol@sha256:cec7a31b08ab5f9b1ed14053b91fd08be83f58ddba0577e9dabd8b150a51233f
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 08 Jan 2021 10:18:08 -0800
      Finished:     Fri, 08 Jan 2021 10:18:08 -0800
    Ready:          True
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-mnnrd (ro)
Containers:
  calico-node:
    Container ID:   docker://8911e4bdc0e60aa5f6c553c0e0d0e5f7aa981d62884141120d8f7cc5bc079884
    Image:          calico/node:v3.15.3
    Image ID:       docker-pullable://calico/node@sha256:1d674438fd05bd63162d9c7b732d51ed201ee7f6331458074e3639f4437e34b1
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Running
      Started:      Fri, 08 Jan 2021 10:18:09 -0800
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                         kubernetes
      WAIT_FOR_DATASTORE:                     true
      NODENAME:                                (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:              &lt;set to the key 'calico_backend' of config map 'calico-config'&gt;  Optional: false
      CLUSTER_TYPE:                           kops,bgp
      IP:                                     autodetect
      CALICO_IPV4POOL_IPIP:                   Always
      CALICO_IPV4POOL_VXLAN:                  Never
      FELIX_IPINIPMTU:                        &lt;set to the key 'veth_mtu' of config map 'calico-config'&gt;  Optional: false
      FELIX_VXLANMTU:                         &lt;set to the key 'veth_mtu' of config map 'calico-config'&gt;  Optional: false
      FELIX_WIREGUARDMTU:                     &lt;set to the key 'veth_mtu' of config map 'calico-config'&gt;  Optional: false
      CALICO_IPV4POOL_CIDR:                   100.96.0.0/11
      CALICO_DISABLE_FILE_LOGGING:            true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:      ACCEPT
      FELIX_IPV6SUPPORT:                      false
      FELIX_LOGSEVERITYSCREEN:                info
      FELIX_HEALTHENABLED:                    true
      FELIX_IPTABLESBACKEND:                  Auto
      FELIX_PROMETHEUSMETRICSENABLED:         false
      FELIX_PROMETHEUSMETRICSPORT:            9091
      FELIX_PROMETHEUSGOMETRICSENABLED:       true
      FELIX_PROMETHEUSPROCESSMETRICSENABLED:  true
      FELIX_WIREGUARDENABLED:                 false
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-mnnrd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  calico-node-token-mnnrd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-mnnrd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     :NoSchedule op=Exists
                 :NoExecute op=Exists
                 CriticalAddonsOnly op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  35m   default-scheduler  Successfully assigned kube-system/calico-node-m255f to ip-10-2-167-145.ec2.internal
  Normal   Pulled     35m   kubelet            Container image &quot;calico/cni:v3.15.3&quot; already present on machine
  Normal   Created    35m   kubelet            Created container upgrade-ipam
  Normal   Started    35m   kubelet            Started container upgrade-ipam
  Normal   Started    35m   kubelet            Started container install-cni
  Normal   Pulled     35m   kubelet            Container image &quot;calico/cni:v3.15.3&quot; already present on machine
  Normal   Created    35m   kubelet            Created container install-cni
  Normal   Pulled     35m   kubelet            Container image &quot;calico/pod2daemon-flexvol:v3.15.3&quot; already present on machine
  Normal   Created    35m   kubelet            Created container flexvol-driver
  Normal   Started    35m   kubelet            Started container flexvol-driver
  Normal   Started    35m   kubelet            Started container calico-node
  Normal   Pulled     35m   kubelet            Container image &quot;calico/node:v3.15.3&quot; already present on machine
  Normal   Created    35m   kubelet            Created container calico-node
  Warning  Unhealthy  35m   kubelet            Readiness probe failed: 2021-01-08 18:18:12.731 [INFO][130] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  35m  kubelet  Readiness probe failed: 2021-01-08 18:18:22.727 [INFO][169] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  35m  kubelet  Readiness probe failed: 2021-01-08 18:18:32.733 [INFO][207] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  35m  kubelet  Readiness probe failed: 2021-01-08 18:18:42.730 [INFO][237] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  35m  kubelet  Readiness probe failed: 2021-01-08 18:18:52.736 [INFO][268] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  34m  kubelet  Readiness probe failed: 2021-01-08 18:19:02.731 [INFO][294] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  34m  kubelet  Readiness probe failed: 2021-01-08 18:19:12.734 [INFO][318] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  34m  kubelet  Readiness probe failed: 2021-01-08 18:19:22.739 [INFO][360] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  34m  kubelet  Readiness probe failed: 2021-01-08 18:19:32.748 [INFO][391] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14
  Warning  Unhealthy  45s (x202 over 34m)  kubelet  (combined from similar events): Readiness probe failed: 2021-01-08 18:53:12.726 [INFO][6053] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 10.2.147.44,10.2.149.115,10.2.150.124,10.2.151.33,10.2.167.162,10.2.172.248,10.2.173.134,10.2.177.100,10.2.181.235,10.2.182.14

</code></pre>
<p>I can ssh into the node and check calico from there</p>
<pre class="lang-sh prettyprint-override"><code>$ sudo ./calicoctl-linux-amd64 node status
Calico process is running.
IPv4 BGP status
+--------------+-------------------+-------+----------+--------------------------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |              INFO              |
+--------------+-------------------+-------+----------+--------------------------------+
| 10.2.147.44  | node-to-node mesh | start | 00:21:18 | Active Socket: Connection      |
|              |                   |       |          | refused                        |
| 10.2.149.115 | node-to-node mesh | start | 00:21:18 | Active Socket: Connection      |
|              |                   |       |          | refused                        |
| 10.2.150.124 | node-to-node mesh | start | 00:21:18 | Active Socket: Connection      |
|              |                   |       |          | refused                        |
| 10.2.151.33  | node-to-node mesh | start | 00:21:18 | Active Socket: Connection      |
|              |                   |       |          | refused                        |
| 10.2.167.162 | node-to-node mesh | start | 00:21:18 | Passive                        |
| 10.2.172.248 | node-to-node mesh | start | 00:21:18 | Passive                        |
| 10.2.173.134 | node-to-node mesh | start | 00:21:18 | Passive                        |
| 10.2.177.100 | node-to-node mesh | start | 00:21:18 | Passive                        |
| 10.2.181.235 | node-to-node mesh | start | 00:21:18 | Passive                        |
| 10.2.182.14  | node-to-node mesh | start | 00:21:18 | Passive                        |
+--------------+-------------------+-------+----------+--------------------------------+
IPv6 BGP status
No IPv6 peers found.
</code></pre>
<p>Here's the calico-node DaemonSet configuration (I assume this was generated by kops and has been untouched)</p>
<pre><code>kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  selfLink: /apis/apps/v1/namespaces/kube-system/daemonsets/calico-node
  uid: 33dfb80a-c840-11e9-af87-02fc30bb40d6
  resourceVersion: '142850829'
  generation: 5
  creationTimestamp: '2019-08-26T20:29:28Z'
  labels:
    k8s-app: calico-node
    role.kubernetes.io/networking: '1'
  annotations:
    deprecated.daemonset.template.generation: '5'
    kubectl.kubernetes.io/last-applied-configuration: '[cut out to save space]'
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: calico-node
        role.kubernetes.io/networking: '1'
    spec:
      volumes:
        - name: lib-modules
          hostPath:
            path: /lib/modules
            type: ''
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
            type: ''
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
            type: ''
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
            type: ''
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
            type: ''
        - name: host-local-net-dir
          hostPath:
            path: /var/lib/cni/networks
            type: ''
        - name: policysync
          hostPath:
            path: /var/run/nodeagent
            type: DirectoryOrCreate
        - name: flexvol-driver-host
          hostPath:
            path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
            type: DirectoryOrCreate
      initContainers:
        - name: upgrade-ipam
          image: 'calico/cni:v3.15.3'
          command:
            - /opt/cni/bin/calico-ipam
            - '-upgrade'
          env:
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
          resources: {}
          volumeMounts:
            - name: host-local-net-dir
              mountPath: /var/lib/cni/networks
            - name: cni-bin-dir
              mountPath: /host/opt/cni/bin
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            procMount: Default
        - name: install-cni
          image: 'calico/cni:v3.15.3'
          command:
            - /install-cni.sh
          env:
            - name: CNI_CONF_NAME
              value: 10-calico.conflist
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: SLEEP
              value: 'false'
          resources: {}
          volumeMounts:
            - name: cni-bin-dir
              mountPath: /host/opt/cni/bin
            - name: cni-net-dir
              mountPath: /host/etc/cni/net.d
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            procMount: Default
        - name: flexvol-driver
          image: 'calico/pod2daemon-flexvol:v3.15.3'
          resources: {}
          volumeMounts:
            - name: flexvol-driver-host
              mountPath: /host/driver
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            procMount: Default
      containers:
        - name: calico-node
          image: 'calico/node:v3.15.3'
          env:
            - name: DATASTORE_TYPE
              value: kubernetes
            - name: WAIT_FOR_DATASTORE
              value: 'true'
            - name: NODENAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            - name: CLUSTER_TYPE
              value: 'kops,bgp'
            - name: IP
              value: autodetect
            - name: CALICO_IPV4POOL_IPIP
              value: Always
            - name: CALICO_IPV4POOL_VXLAN
              value: Never
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: FELIX_VXLANMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: CALICO_IPV4POOL_CIDR
              value: 100.96.0.0/11
            - name: CALICO_DISABLE_FILE_LOGGING
              value: 'true'
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: ACCEPT
            - name: FELIX_IPV6SUPPORT
              value: 'false'
            - name: FELIX_LOGSEVERITYSCREEN
              value: info
            - name: FELIX_HEALTHENABLED
              value: 'true'
            - name: FELIX_IPTABLESBACKEND
              value: Auto
            - name: FELIX_PROMETHEUSMETRICSENABLED
              value: 'false'
            - name: FELIX_PROMETHEUSMETRICSPORT
              value: '9091'
            - name: FELIX_PROMETHEUSGOMETRICSENABLED
              value: 'true'
            - name: FELIX_PROMETHEUSPROCESSMETRICSENABLED
              value: 'true'
            - name: FELIX_WIREGUARDENABLED
              value: 'false'
          resources:
            requests:
              cpu: 100m
          volumeMounts:
            - name: lib-modules
              readOnly: true
              mountPath: /lib/modules
            - name: xtables-lock
              mountPath: /run/xtables.lock
            - name: var-run-calico
              mountPath: /var/run/calico
            - name: var-lib-calico
              mountPath: /var/lib/calico
            - name: policysync
              mountPath: /var/run/nodeagent
          livenessProbe:
            exec:
              command:
                - /bin/calico-node
                - '-felix-live'
                - '-bird-live'
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
                - /bin/calico-node
                - '-felix-ready'
                - '-bird-ready'
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            procMount: Default
      restartPolicy: Always
      terminationGracePeriodSeconds: 0
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: calico-node
      serviceAccount: calico-node
      hostNetwork: true
      securityContext: {}
      schedulerName: default-scheduler
      tolerations:
        - operator: Exists
          effect: NoSchedule
        - key: CriticalAddonsOnly
          operator: Exists
        - operator: Exists
          effect: NoExecute
      priorityClassName: system-node-critical
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  revisionHistoryLimit: 10
status:
  currentNumberScheduled: 1
  numberMisscheduled: 0
  desiredNumberScheduled: 1
  numberReady: 0
  observedGeneration: 5
  updatedNumberScheduled: 1
  numberUnavailable: 1
</code></pre>
<p>There's nothing really useful in the pod logs either; no errors or anything obvious. It mostly looks like this:</p>
<pre><code>2021-01-08 19:08:21.603 [INFO][48] felix/int_dataplane.go 1245: Applying dataplane updates
2021-01-08 19:08:21.603 [INFO][48] felix/ipsets.go 223: Asked to resync with the dataplane on next update. family=&quot;inet&quot;
2021-01-08 19:08:21.603 [INFO][48] felix/ipsets.go 306: Resyncing ipsets with dataplane. family=&quot;inet&quot;
2021-01-08 19:08:21.603 [INFO][48] felix/wireguard.go 578: Wireguard is not enabled
2021-01-08 19:08:21.605 [INFO][48] felix/ipsets.go 356: Finished resync family=&quot;inet&quot; numInconsistenciesFound=0 resyncDuration=1.573324ms
2021-01-08 19:08:21.605 [INFO][48] felix/int_dataplane.go 1259: Finished applying updates to dataplane. msecToApply=2.03915
</code></pre>
<p><strong>Things I tried</strong></p>
<p>Unfortunately, I'm not a networking expert so I didn't get too deep into the specifics of calico.</p>
<p>I have tried rebooting the related pods, rebooting the actual EC2 instance, and deleting the daemonset and re-adding it using the above config.</p>
<p>I can also assure you there are no networking restrictions (firewalls, sec groups, etc) within the internal network that might be blocking connections.</p>
<p>It's also worth pointing out that this cluster was working perfectly before the <code>kops rolling-update</code> attempt.</p>
<p>I'm pretty much at a roadblock here and not sure what else I could try.</p>

hhaslam11
<p>I have solved this by updating all the masters at the same time, without validation</p>
<pre><code>kops rolling-update cluster --cloudonly --instance-group-roles master --master-interval=1s --node-interval=1s
</code></pre>
<p>Everything is working now!</p>
