<p>We've just had an increase in traffic to our kubernetes cluster and I've noticed that of our 6 application pods, 2 of them are seemingly not used very much. <code>kubectl top pods</code> returns the following</p>
<p><a href="https://i.stack.imgur.com/TyLWx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/TyLWx.png" alt="CPU usage" /></a></p>
<p>You can see of the 6 pods, 4 of them are using more than 50% of the CPU (2 vCPU nodes), but two of them aren't really doing much at all.</p>
<p>Our cluster is setup on AWS, using the ALB ingress controller. The load balancer is configured to use the <code>Least outstanding requests</code> rather than <code>Round robin</code> in an attempt to balance things out a bit more, but we're still seeing this imbalance.</p>
<p>Is there any way of determining why this is happening, or if indeed it actually is a problem? I'm hoping it's normal behaviour rather than an issue but I'd rather investigate it.</p>
<p><strong>App deployment config</strong></p>
<p>This is the configuration of the main application pods. Nothing fancy really</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include &quot;app.fullname&quot; . }}
  labels:
    {{- include &quot;app.labels&quot; . | nindent 4 }}
    app.kubernetes.io/component: web
spec:
  replicas: {{ .Values.app.replicaCount }}
  selector:
    matchLabels:
      app: {{ include &quot;app.fullname&quot; . }}-web

  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath &quot;/config_maps/app-env-vars.yaml&quot;) . | sha256sum }}
      {{- with .Values.podAnnotations }}
        {{- toYaml . | nindent 10 }}
      {{- end }}
      labels:
        {{- include &quot;app.selectorLabels&quot; . | nindent 8 }}
        app.kubernetes.io/component: web
        app: {{ include &quot;app.fullname&quot; . }}-web
    spec:
      serviceAccountName: {{ .Values.serviceAccount.web }}
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - {{ include &quot;app.fullname&quot; . }}-web
              topologyKey: failure-domain.beta.kubernetes.io/zone
      containers:
        - name: {{ .Values.image.name }}
          image: &quot;{{ .Values.image.repository }}:{{ .Values.image.tag }}&quot;
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command:
            - bundle
          args: [&quot;exec&quot;, &quot;puma&quot;, &quot;-p&quot;, &quot;{{ .Values.app.containerPort }}&quot;]
          ports:
            - name: http
              containerPort: {{ .Values.app.containerPort }}
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /healthcheck
              port: {{ .Values.app.containerPort }}
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 5
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          envFrom:
          - configMapRef:
              name: {{ include &quot;app.fullname&quot; . }}-cm-env-vars

          - secretRef:
              name: {{ include &quot;app.fullname&quot; . }}-secret-rails-master-key

          - secretRef:
              name: {{ include &quot;app.fullname&quot; . }}-secret-db-credentials

          - secretRef:
              name: {{ include &quot;app.fullname&quot; . }}-secret-db-url-app

          - secretRef:
              name: {{ include &quot;app.fullname&quot; . }}-secret-redis-credentials
</code></pre>

PaReeOhNos
<p>That's a known issue with Kubernetes.<br />
In short, Kubernetes doesn't load balance long-lived TCP connections.<br />
This <a href="https://learnk8s.io/kubernetes-long-lived-connections" rel="noreferrer">excellent article</a> covers it in details.</p>
<p>The load distribution you service is showing complies exactly with the case.</p>
