<p>I've got a preemptible node pool which is clearly under-utilized:
<a href="https://i.stack.imgur.com/LtDoh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LtDoh.png" alt="enter image description here" /></a></p>
<p>The node pool hosts a <strong>deployment</strong> with <strong>HPA</strong> with the following setup:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    app: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      initContainers:
      - name: wait-for-database
        image: ### IMAGE ###
        command: ['bash', 'init.sh']
      containers:
      - name: backend
        image: ### IMAGE ###
        command: [&quot;bash&quot;, &quot;entrypoint.sh&quot;]
        imagePullPolicy: Always
        resources:
          requests:
            memory: &quot;200M&quot;
            cpu: &quot;50m&quot;
        ports:
        - name: probe-port
          containerPort: 8080
          hostPort: 8080
        volumeMounts:
          - name: static-shared-data
            mountPath: /static
        readinessProbe:
          httpGet:
            path: /readiness/
            port: probe-port
          failureThreshold: 5
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
      - name: nginx
        image: nginx:alpine
        resources:
          requests:
            memory: &quot;400M&quot;
            cpu: &quot;20m&quot;
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-proxy-config
          mountPath: /etc/nginx/conf.d/default.conf
          subPath: app.conf
        - name: static-shared-data
          mountPath: /static
      volumes:
      - name: nginx-proxy-config
        configMap:
          name: backend-nginx
      - name: static-shared-data
        emptyDir: {}
      nodeSelector:
        cloud.google.com/gke-nodepool: app-dev
      tolerations:
      - effect: NoSchedule
        key: workload
        operator: Equal
        value: dev
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: backend
  namespace: default
spec:
  maxReplicas: 12
  minReplicas: 8
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: backend
  metrics:
  - resource:
      name: cpu
      targetAverageUtilization: 50
    type: Resource
---
</code></pre>
<p>The node pool also has the toleration label.</p>
<p>The HPA utilization shows this:</p>
<pre><code>NAME              REFERENCE                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
backend-develop   Deployment/backend-develop   10%/50%   8         12        8          38d
</code></pre>
<p>But the node pool does not scale down for about a day. No heavy load on this deployment:</p>
<pre><code>NAME                             STATUS   ROLES    AGE     VERSION
gke-dev-app-dev-fee1a901-fvw9    Ready    &lt;none&gt;   22h     v1.14.10-gke.36
gke-dev-app-dev-fee1a901-gls7    Ready    &lt;none&gt;   22h     v1.14.10-gke.36
gke-dev-app-dev-fee1a901-lf3f    Ready    &lt;none&gt;   24h     v1.14.10-gke.36
gke-dev-app-dev-fee1a901-lgw9    Ready    &lt;none&gt;   3d10h   v1.14.10-gke.36
gke-dev-app-dev-fee1a901-qxkz    Ready    &lt;none&gt;   3h35m   v1.14.10-gke.36
gke-dev-app-dev-fee1a901-s10l    Ready    &lt;none&gt;   22h     v1.14.10-gke.36
gke-dev-app-dev-fee1a901-sj4d    Ready    &lt;none&gt;   22h     v1.14.10-gke.36
gke-dev-app-dev-fee1a901-vdnw    Ready    &lt;none&gt;   27h     v1.14.10-gke.36
</code></pre>
<p>There's no <code>affinity</code> settings for this deployment and node pool. Some of the nodes easily pack several same pods, but others just hold one pod for hours, <strong>no scale down</strong> happens.</p>
<p>What could be wrong?</p>

orkenstein
<p>The issue was:</p>
<pre><code>hostPort: 8080
</code></pre>
<p>This lead to <code>FailedScheduling didn't have free ports</code>.
That's why the nodes were kept online.</p>
