<p>We are working on an akka-cluster based application which we run in a Kubernetes cluster. We are now in a situation where we will like the application to scale up in case there is an increase in load on the cluster. We are using <code>HorizontalPodAutoscaler</code> to achieve this. Our manifest files looks like:</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: app
  namespace: some-namespace
  labels:
    componentName: our-component
    app: our-component
    version: some-version
  annotations:
    prometheus.io/scrape: &quot;true&quot;
    prometheus.io/path: &quot;/metrics&quot;
    prometheus.io/port: &quot;9252&quot;
spec:
  serviceName: our-component
  replicas: 2
  selector:
    matchLabels:
      componentName: our-component
      app: our-app
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        componentName: our-component
        app: our-app
    spec:
      containers:
        - name: our-component-container
          image: image-path
          imagePullPolicy: Always
          resources:
            requests:
              cpu: .1
              memory: 500Mi
            limits:
              cpu: 1
              memory: 1Gi
          command:
            - &quot;/microservice/bin/our-component&quot;
          ports:
            - name: remoting
              containerPort: 8080
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /ready
              port: 9085
            initialDelaySeconds: 40
            periodSeconds: 30
            failureThreshold: 3
            timeoutSeconds: 30
          livenessProbe:
            httpGet:
              path: /alive
              port: 9085
            initialDelaySeconds: 130
            periodSeconds: 30
            failureThreshold: 3
            timeoutSeconds: 5
   

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
  namespace: some-namespace
  labels:
    componentName: our-component
    app: our-app
spec:
  minReplicas: 2
  maxReplicas: 8
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: our-component
  targetCPUUtilizationPercentage: 75
</code></pre>
<p>The issue we face is, as soon as we deploy the application it scales to the <code>maxReplicas</code> defined, even if there is no load on the application. Also, the application never seem to scale down.</p>
<p>Can someone who faced a similar issue in their application share their experience of why this happens and if they were able to resolve this?</p>

Karan Khanna
<p>The issue is with the resource request and limits. You have set requested CPU to be &quot;1&quot; whereas limit to be &quot;0.1&quot;. Therefore, what is happening is, as soon as your pod runs, the limit is naturally exceeded, and the autoscaling kicks in and keeps scaling to the max number of replicas.</p>
<p>You need to switch the parameter names so that your <em>request</em> becomes 0.1 and <em>limit</em> becomes 1.0. This way, your pod will start with 0.1 units of CPU shares, and once average usage grows larger than 65% across all pods, you will get more replicas, and if the usage drops, you will have a scale down, just as expected.</p>
<p>As a general rule of thumb, <em>request</em> is less than <em>limit</em>, or at least equal to it. It cannot be higher, because you end up with an infinitely scaling infrastructure.</p>
