<p>I have a <code>jenkins</code> pod, having the label <code>app: jenkins-master</code></p>

<p>This resides on <code>jenkins</code> namespace.</p>

<p>I want an <code>nginx</code> pod of a deployment (on another namespace, <code>default</code>) to be collocated to the above pod.</p>

<p>So I add the following in its spec:</p>

<pre><code>spec:
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          namespaces:
            - all
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - jenkins-master
          topologyKey: "kubernetes.io/os"
</code></pre>

<p>I have a GKE cluster of 8 nodes.</p>

<p>Out of 5-6 times I have created/deleted the deployment, the <code>nginx</code> pod actually <strong>never</strong> landed on the same node as <code>jenkins-master</code>.</p>

<p>I know it is <strong><code>preferred</code></strong> scheduling, but is this behavior normal?</p>

<p>Working on GKE with <code>"v1.15.9-gke.24"</code></p>

<p><strong>edit 1</strong>: I have changed the <code>topologyKey: "kubernetes.io/hostname"</code> as suggested in a couple of answers below but that didn't help much either.</p>

<p><strong>edit 2</strong>: These are the allocated resources for the node that <code>jenkins-master</code> pod is scheduled on</p>

<pre><code>  Resource                   Requests      Limits
  cpu                        1691m (43%)   5013m (127%)
  memory                     4456Mi (33%)  8902Mi (66%)
</code></pre>

<p>Since scheduling is based on requests, I don't understand how the following deployment fails to collocate, the requests I am making are minimal</p>

<pre><code>resources:
  limits:
    memory: "1Gi"
    cpu: "100m"
   requests:
    memory: "100Mi"
    cpu: "50m"
</code></pre>

pkaramol
<p>I think you made a mistake using the <code>topologyKey: "kubernetes.io/os"</code> which is used if you are mixing operating systems in your cluster (for example: mixing Linux and Windows nodes).</p>

<p>You should be using: <code>topologyKey: "kubernetes.io/hostname"</code>, where Kubelet populates this label with the hostname.</p>
