<p>So I am trying to figure out how can I configure an Horizontal Pod Autoscaler from a custom metric reading from Prometheus that returns CPU usage with percentile 0.95</p>

<p>I have everything set up to use custom metrics with prometheus-adapter, but I don't understand how to create the rule in Prometheus. For example, if I go to Grafana to check some of the Graphs that comes by default I see this metric:</p>

<pre><code>sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name)
</code></pre>

<p>But how can I modify that to be percentile 95? I tried with histogram_quantile function but it says no datapoints found:</p>

<pre><code>histogram_quantile(0.95, sum(namespace_pod_name_container_name:container_cpu_usage_seconds_total:sum_rate{namespace="api", pod_name="api-xxxxx9b-bdxx", container_name!="POD", cluster=""}) by (container_name))
</code></pre>

<p>But even if that works, will the pod name and namespace be filled by prometheus-adapter or prometheus when using custom metrics?</p>

<p>And every example I find using custom metrics are not related with CPU. So... other question I have is how people is using autoscaling metrics in production? I'm used to scale based on percentiles but I don't understand how is this managed in Kubernetes.</p>

codiaf
<p>If you want to add HPA based on custom metrics you can use Prometheus adapter. </p>

<p>Prometheus adapter helps you in exposing custom metrics to HPA.</p>

<p>Helm Chart - <a href="https://github.com/helm/charts/tree/master/stable/prometheus-adapter" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/prometheus-adapter</a></p>

<p>Prometheus adapter - <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter" rel="nofollow noreferrer">https://github.com/DirectXMan12/k8s-prometheus-adapter</a></p>

<p>Note - You have to enable 6443 port from public to cluster, because prometheus doesn’t provide override option.</p>

<p><a href="https://github.com/helm/charts/blob/master/stable/prometheus-adapter/templates/custom-metrics-apiserver-deployment.yaml#L34" rel="nofollow noreferrer">https://github.com/helm/charts/blob/master/stable/prometheus-adapter/templates/custom-metrics-apiserver-deployment.yaml#L34</a></p>

<p>Make sure that Prometheus is getting custom metrics data
Install Prometheus adapter on the same kubernetes cluster where you want apply hpa</p>

<pre><code>helm install --name my-release stable/prometheus-adapter -f values.yaml
</code></pre>

<p>Pass the following config file to helm - values.yaml</p>

<pre><code>prometheus-adapter:
  enabled: true
  prometheus:
    url: http://prometheus.namespace.svc.cluster.local
  rules:
    default: true
    custom:
    - seriesQuery: '{__name__="cpu",namespace!="",pod!="",service="svc_name"}'
      seriesFilters: []
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "cpu"
        as: "cpu_95"
      metricsQuery: "histogram_quantile(0.95, sum(irate(&lt;&lt;.Series&gt;&gt;{&lt;&lt;.LabelMatchers&gt;&gt;}[2m])) by (&lt;&lt;.GroupBy&gt;&gt;,le))"
</code></pre>

<p>Above config will expose </p>

<p>cpu metrics as cpu_95 to HPA.</p>

<p>To Verify, if data is exposed properly run following command - </p>

<p>Fetch the data curl raw query command - <code>kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/namespace_name/pods/\*/cpu_95 | jq .</code></p>

<p>HPA config -</p>

<pre><code>apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: test-cpu-manual
  labels:
    app: app_name
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: app_name
  minReplicas: 1
  maxReplicas: 15
  metrics:
  - type: Pods
    pods:
      metricName: cpu_95
      targetAverageValue: 75
</code></pre>
