<p>I have an anti-affinity rule that asks kubernetes to schedule pods from the same deployment onto <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#always-co-located-in-the-same-node" rel="nofollow noreferrer">different nodes</a>, we've used it successfully for a long time.</p>

<pre><code>affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: kubernetes.io/hostname
      labelSelector:
        matchExpressions:
        - key: application
          operator: In
          values:
          - {{ $appName }}
        - key: proc  
          operator: In
          values:
          - {{ $procName }}
</code></pre>

<p>I'm trying to update my pod affinity rules to be a strong preference instead of a hard requirement, so that we don't need to expand our cluster if a deployment needs more replicas than there are nodes available.</p>

<pre><code>affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: kubernetes.io/hostname
      weight: 100
      labelSelector:
        matchExpressions:
        - key: application
          operator: In
          values:
          - {{ $appName }}
        - key: proc 
          operator: In
          values:
          - {{ $procName }}
</code></pre>

<p>However, when I try applying the new rules, I get an unexpected error with the topologyKey:</p>

<pre><code>Error: Deployment.apps "core--web" is invalid:
[spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey: Required value: can not be empty,
spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey: Invalid value: "": name part must be non-empty,
spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey: Invalid value: "": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')]
</code></pre>

<p>The scheduler seems to be getting an empty string value for the topology key, even though all my nodes have a label for the specified key which match the regex:</p>

<pre><code>$ kubectl describe nodes | grep kubernetes.io/hostname
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal
kubernetes.io/hostname=ip-10-x-x-x.ec2.internal
</code></pre>

<p>I didn't expect to see a problem like this from a simple change from required to preferred.  What did I screw up to cause the topologyKey error?</p>

<p>There's a slight difference between the syntax of required and preferred, note the reference to <code>podAffinityTerm</code> in the error message path:</p>

<pre><code>spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey
</code></pre>

<p>The correct syntax for preferred scheduling is:</p>

<pre><code>affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        topologyKey: kubernetes.io/hostname
        labelSelector:
          matchExpressions:
          - key: application
            operator: In
            values:
            - {{ $appName }}
          - key: proc
            operator: In
            values:
            - {{ $procName }}
</code></pre>

<p>Note that <code>weight</code> is a top level key, with a sibling of <code>podAffinityTerm</code> which contains the <code>topologyKey</code> and <code>labelSelector</code>.</p>
