<p>We are encountering some Internal Server errors (500) on our web services API, seemingly when new pods are introduced or deleted as part of our autoscaler configuration.</p>
<p>Naturally, this sort of thing is very hard to troubleshoot as you are not always able to control the moments pods start and stop.</p>
<p>Is there a mechanism one could use to instruct the cluster to scale up and down upon?</p>
<p>Our <code>hpa.yaml</code> for interest.</p>
<pre><code>kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
  name: xyz
spec:
  minReplicas: 1
  maxReplicas: 12
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: xyz
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 85
</code></pre>

<p>If you're interested in causing your HPA to forcibly create or destroy Pods for debugging purposes, you can use custom metrics in your Horizontal Pod Autoscaler .yaml definition.</p>
<p>If the problem is that newly created Pods cause errors, you can implement readiness probes in the Pod definition that perform an httpGet check. That way, you can avoid redirecting traffic on faulty Pods until the probe check returns a status = 200.</p>
