<p>So I am trying to figure out how can I configure an Horizontal Pod Autoscaler from a custom metric reading from Prometheus that returns CPU usage with percentile 0.95</p>

<p>I have everything set up to use custom metrics with prometheus-adapter, but I don't understand how to create the rule in Prometheus. For example, if I go to Grafana to check some of the Graphs that comes by default I see this metric:</p>

<pre><code>sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name)
</code></pre>

<p>But how can I modify that to be percentile 95? I tried with histogram_quantile function but it says no datapoints found:</p>

<pre><code>histogram_quantile(0.95, sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name))
</code></pre>

<p>But even if that works, will the pod name and namespace be filled by prometheus-adapter or prometheus when using custom metrics?</p>

<p>And every example I find using custom metrics are not related with CPU. So... other question I have is how people is using autoscaling metrics in production? I'm used to scale based on percentiles but I don't understand how is this managed in Kubernetes.</p>

codiaf
<p>If I understand you correctly you don't have to use custom metrics in order to horizontally autoscale your pods. By default, you can automatically scale the number of Kubernetes pods based on the observed CPU utilization. 
Here is the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">official documentation</a> with necessary details.</p>

<blockquote>
  <p>The Horizontal Pod Autoscaler automatically scales the number of pods
  in a replication controller, deployment or replica set based on
  observed CPU utilization (or, with custom metrics support, on some
  other application-provided metrics).</p>
  
  <p>The Horizontal Pod Autoscaler is implemented as a Kubernetes API
  resource and a controller. The resource determines the behavior of the
  controller. The controller periodically adjusts the number of replicas
  in a replication controller or deployment to match the observed
  average CPU utilization to the target specified by user.</p>
</blockquote>

<p>And here you can find the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/" rel="nofollow noreferrer">walkthrough</a> of how to set it up.</p>

<p>Also, <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#autoscale" rel="nofollow noreferrer">here</a> is the <code>kubectl autoscale</code> command documentation. </p>

<p>Example: <code>kubectl autoscale rc foo --max=5 --cpu-percent=80</code></p>

<p><em>Auto scale a replication controller "foo", with the number of pods between 1 and 5, target CPU utilization at 80%</em></p>

<p>I believe that it is the easiest way so no need to complicate it with some custom metrics.</p>

<p>Please let me know if that helped.</p>
