<p><br><strong>Question 1.)</strong>
<strong><br>Given the scenario a multi-container pod, where all containers have a defined CPU request:
<br>How would Kubernetes Horizontal Pod Autoscaler calculate CPU Utilization for Multi Container pods?  <br></strong>
Does it average them? (((500m cpu req + 50m cpu req) /2) * X% HPA target cpu utilization <br>
Does it add them? ((500m cpu req + 50m cpu req) * X% HPA target cpu utilization <br>
Does it track them individually? (500m cpu req * X% HPA target cpu utilization = target #1, 50m cpu req * X% HPA target cpu utilization = target #2.) <br>
<br><strong>Question 2.)</strong> <br>
<strong>Given the scenario of a multi-container pod, where 1 container has a defined CPU request and a blank CPU request for the other containers: <br>
How would Kubernetes Horizontal Pod Autoscaler calculate CPU Utilization for Multi Container pods?</strong><br />
Does it work as if you only had a 1 container pod?</p>
<p><strong>Question 3.)</strong> <br>
<strong>Do the answers to questions 1 and 2 change based on the HPA API version?</strong> <br>I noticed stable/nginx-ingress helm chart, chart version 1.10.2, deploys an HPA for me with these specs:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
</code></pre>
<p>(I noticed apiVersion: autoscaling/v2beta2 now exists)</p>
<p><strong>Background Info:</strong>
<br> I recently had an issue with unexpected wild scaling / constantly going back and forth between min and max pods after adding a sidecar(2nd container) to an nginx ingress controller deployment (which is usually a pod with a single container). In my case, it was an oauth2 proxy, although I image istio sidecar container folks might run into this sort of problem all the time as well.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: nginx-ingress-controller #(primary-container)
          resources:
            requests:
              cpu: 500m    #baseline light load usage in my env 
              memory: 2Gi  #according to kubectl top pods
            limits:
              memory: 8Gi  #(oom kill pod if this high, because somethings wrong)
        - name: oauth2-proxy #(newly-added-2nd-sidecar-container)
          resources: 
            requests:
              cpu: 50m
              memory: 50Mi
            limits:
              memory: 4Gi
</code></pre>
<p>I have an HPA (apiVersion: autoscaling/v1) with:</p>
<ul>
<li>min 3 replicas (to preserve HA during rolling updates)</li>
<li>targetCPUUtilizationPercentage = 150%</li>
</ul>
<p><strong>It occurred to me that my misconfiguration leads to unexpected wild scaling was caused by 2 issues:</strong></p>
<ol>
<li>I don't actually understand how HPAs work when the pod has multiple containers</li>
<li>I don't know how to dig deep to get metrics of what's going on.
<br></li>
</ol>
<hr />
<p><strong>To address the first issue: I brainstormed my understanding of how it works in the single container scenario</strong> (and then realized I don't know the multi-container scenario so I decided to ask this question)
<br><br></p>
<p><strong>This is my understanding of how HPA (autoscaling/v1) works when I have 1 container (temporarily ignore the 2nd container in the above deployment spec):</strong>
<br>The HPA would spawn more replicas when the CPU utilization average of all pods shifted from my normal expected load of 500m or less to 750m (150% x 500m request)</p>
<hr />
<p><strong>To address the 2nd issue: I found out how to dig to see concrete numeric value-based metrics vs relative percentage-based metrics to help figure out what's happening behind the scenes:</strong></p>
<pre class="lang-sh prettyprint-override"><code>bash# kubectl describe horizontalpodautoscaler nginx-ingress-controller -n=ingress | grep Metrics: -A 1
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  5% (56m) / 100%
</code></pre>
<p>(Note: kubectl top pods -n=ingress, showed cpu usage of the 5 replicas as 36m, 34m, 88m, 36m, 91m, so that 57m current which ~matches 56m current)</p>
<p>Also now it's a basic proportions Math Problem that allows solving for target static value: <br>
(5% / 56m) = (100% / x m) --&gt; x = 56 * 100 / 5 = 1120m target cpu
<br> (Note: this HPA isn't associated with the deployment mentioned above, that's why the numbers are off.)</p>

<p>Basing on stackoverflow community member answer in other <a href="https://stackoverflow.com/questions/48172151/kubernetes-pod-cpu-usage-calculation-method-for-hpa">case</a></p>

<blockquote>
  <p>"HPA calculates pod cpu utilization as total cpu usage of all containers in pod divided by total request. I don't think that's specified in docs anywhere, but the relevant code is <a href="https://github.com/kubernetes/kubernetes/blob/v1.9.0/pkg/controller/podautoscaler/metrics/utilization.go#L49" rel="nofollow noreferrer">here</a>"</p>
</blockquote>

<p>You have got more informations,with examples in the link above.</p>

<hr>

<p>Basing on documentation </p>

<p><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">Horizontal Pod Autoscaler</a> automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization (or, <strong>with beta support, on some other, application-provided metrics</strong>).</p>

<p>So basically:</p>

<p><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#api-object" rel="nofollow noreferrer">apiVersion</a> autoscaling/v1 HPA base on <strong>cpu</strong>.</p>

<p>apiVersion autoscaling/v2beta2 base on <strong>cpu,memory,custom metrics</strong>.</p>

<p>More informations <a href="https://rancher.com/blog/2018/2018-08-06-k8s-hpa-resource-custom-metrics/" rel="nofollow noreferrer">here</a> </p>
