<p>First of all: I readed other posts like <a href="https://stackoverflow.com/questions/51946393/kubernetes-pod-warning-1-nodes-had-volume-node-affinity-conflict">this</a>.</p>
<p>My staging cluster is allocated on AWS using <strong>spot instances</strong>.</p>
<p>I have arround 50+ pods (runing diferent services / products) and 6 StatefulSets.</p>
<p>I created the StatefulSets this way:</p>
<p>OBS: I do not have PVs and PVCs created manualy, they are being created from the StatfulSet</p>
<pre><code>---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  labels:
    app: redis
spec:
  selector:
    matchLabels:
      app: redis
  serviceName: &quot;redis&quot;
  replicas: 1
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:alpine
        imagePullPolicy: Always
        ports:
        - containerPort: 6379
          name: client
        volumeMounts:
          - name: data
            mountPath: /data
            readOnly: false
  volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          name: redis-gp2
      spec:
        accessModes: [ &quot;ReadWriteOnce&quot; ]
        resources:
          requests:
            storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
  - port: 6379
    name: redis
    targetPort: 6379
  selector:
    app: redis
  type: NodePort    
</code></pre>
<p>I do have node and pod autoscalers configured.</p>
<p>In the past week after deploying some extra micro-services during the &quot;usage peak&quot; the node autoscaler trigged.</p>
<p>During the scale down some pods(StatefulSets) crashed with the error <code>node(s) had volume node affinity conflict</code>.</p>
<p>My first reaction wast to delete and &quot;recreate&quot; the PVs/PVCs with high priority. That &quot;fixed&quot; the pending pods on that time.</p>
<p>Today I forced another scale-up, so I was able to check what was happening.</p>
<p>The problem occurs during the scalle up and take a long time to go back to normal (+/- 30 min) even after the scalling down.</p>
<p>Describe Pod:</p>
<pre><code>Name:                 redis-0
Namespace:            ***-staging
Priority:             1000
Priority Class Name:  prioridade-muito-alta
Node:                 ip-***-***-***-***.sa-east-1.compute.internal/***.***.*.***
Start Time:           Mon, 03 Jan 2022 09:24:13 -0300
Labels:               app=redis
                      controller-revision-hash=redis-6fd5f59c5c
                      statefulset.kubernetes.io/pod-name=redis-0
Annotations:          kubernetes.io/psp: eks.privileged
Status:               Running
IP:                   ***.***.***.***
IPs:
  IP:           ***.***.***.***
Controlled By:  StatefulSet/redis
Containers:
  redis:
    Container ID:   docker://4928f38ed12c206dc5915c863415d3eba98b9592f2ab5c332a900aa2fa2cef64
    Image:          redis:alpine
    Image ID:       docker-pullable://redis@sha256:4bed291aa5efb9f0d77b76ff7d4ab71eee410962965d052552db1fb80576431d
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 03 Jan 2022 09:24:36 -0300
    Ready:          True
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ngc7q (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-redis-0
    ReadOnly:   false
  default-token-***:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  *****
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From                                                  Message
  ----     ------                  ----                 ----                                                  -------
  Warning  FailedScheduling        59m (x4 over 61m)    default-scheduler                                     0/7 nodes are available: 1 Too many pods, 1 node(s) were unschedulable, 5 node(s) had volume node affinity conflict.
  Warning  FailedScheduling        58m                  default-scheduler                                     0/7 nodes are available: 1 Too many pods, 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1641210902}, that the pod didn't tolerate, 1 node(s) were unschedulable, 4 node(s) had volume node affinity conflict.
  Warning  FailedScheduling        58m                  default-scheduler                                     0/7 nodes are available: 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1641210902}, that the pod didn't tolerate, 1 node(s) were unschedulable, 2 Too many pods, 3 node(s) had volume node affinity conflict.
  Warning  FailedScheduling        57m (x2 over 58m)    default-scheduler                                     0/7 nodes are available: 2 Too many pods, 2 node(s) were unschedulable, 3 node(s) had volume node affinity conflict.
  Warning  FailedScheduling        50m (x9 over 57m)    default-scheduler                                     0/6 nodes are available: 1 node(s) were unschedulable, 2 Too many pods, 3 node(s) had volume node affinity conflict.
  Warning  FailedScheduling        48m (x2 over 49m)    default-scheduler                                     0/5 nodes are available: 2 Too many pods, 3 node(s) had volume node affinity conflict.
  Warning  FailedScheduling        35m (x10 over 48m)   default-scheduler                                     0/5 nodes are available: 1 Too many pods, 4 node(s) had volume node affinity conflict.
  Normal   NotTriggerScaleUp       30m (x163 over 58m)  cluster-autoscaler                                    pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 node(s) had volume node affinity conflict
  Warning  FailedScheduling        30m (x3 over 33m)    default-scheduler                                     0/5 nodes are available: 5 node(s) had volume node affinity conflict.
  Normal   SuccessfulAttachVolume  29m                  attachdetach-controller                               AttachVolume.Attach succeeded for volume &quot;pvc-23168a78-2286-40b7-aa71-194ca58e0005&quot;
  Normal   Pulling                 28m                  kubelet, ip-***-***-***-***.sa-east-1.compute.internal  Pulling image &quot;redis:alpine&quot;
  Normal   Pulled                  28m                  kubelet, ip-***-***-***-***.sa-east-1.compute.internal  Successfully pulled image &quot;redis:alpine&quot; in 3.843908086s
  Normal   Created                 28m                  kubelet, ip-***-***-***-***.sa-east-1.compute.internal  Created container redis
  Normal   Started                 28m                  kubelet, ip-***-***-***-***.sa-east-1.compute.internal  Started container redis
</code></pre>
<p>PVC:</p>
<pre><code>Name:          data-redis-0
Namespace:     ***-staging
StorageClass:  gp2
Status:        Bound
Volume:        pvc-23168a78-2286-40b7-aa71-194ca58e0005
Labels:        app=redis
               name=redis-gp2
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
               volume.kubernetes.io/selected-node: ip-***-***-***-***.sa-east-1.compute.internal
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    redis-0
Events:        &lt;none&gt;
</code></pre>
<p>PV:</p>
<pre><code>Name:              pvc-23168a78-2286-40b7-aa71-194ca58e0005
Labels:            failure-domain.beta.kubernetes.io/region=sa-east-1
                   failure-domain.beta.kubernetes.io/zone=sa-east-1b
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      gp2
Status:            Bound
Claim:             ***-staging/data-redis-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:     
  Required Terms:  
    Term 0:        failure-domain.beta.kubernetes.io/zone in [sa-east-1b]
                   failure-domain.beta.kubernetes.io/region in [sa-east-1]
Message:           
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://sa-east-1b/vol-061fd23a65185d42c
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         &lt;none&gt;
</code></pre>
<p>This happend in 4 of my 6 StatefulSets.</p>
<p><strong>Question:</strong></p>
<p>If I create PVs and PVCs manually setting:</p>
<pre><code>volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - sa-east-1
</code></pre>
<p>will the scale up/down not mess up with StatefulSets?</p>
<p>If not what can I do to avoid this problem ?</p>

Felipe Colussi-oliva
<p>First of all, it's better to move <code>allowedTopologies</code> stanza to <code>StorageClass</code>. It's more flexible because you can create multiple zone-specific storage classes.</p>
<p>And yes, this should obviously solve your one problem and create another. You basically want to sacrifice high availability to costs/convenience. It's totally up to you, there is no one-size-fits-all recommendation here but I just want to make sure you know the options.</p>
<p>You may still have volumes not tied to specific zones if you always have enough node capacity in every AZ. This can be <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler" rel="nofollow noreferrer">achieved</a> using cluster-autoscaler. Generally, you create separate node groups per each AZ and autoscaler will do the rest.</p>
<p>Another option is to build distributed storage like Ceph or Portworx that allows to mount volumes from another AZ. That will greatly increase your cross-AZ traffic costs and needs to be maintained properly but I know companies that do that.</p>
