<p>We want to scale our pods horizontally based on the amount of messages in our Kafka Topic. The standard solution is to publish the metrics to the custom metrics API of Kubernetes. However, due to company guidelines we are not allowed to use the custom metrics API of Kubernetes. We are only allowed to use non-admin functionality. Is there a solution for this with kubernetes-nativ features or do we need to implement a customized solution?</p>

Torben
<p>I'm not exactly sure if this would fit your needs but you could use <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects" rel="nofollow noreferrer">Autoscaling on metrics not related to Kubernetes objects</a>.</p>

<blockquote>
  <p>Applications running on Kubernetes may need to autoscale based on metrics that don’t have an obvious relationship to any object in the Kubernetes cluster, such as metrics describing a hosted service with no direct correlation to Kubernetes namespaces. In Kubernetes 1.10 and later, you can address this use case with  <em>external metrics</em>.</p>
  
  <p>Using external metrics requires knowledge of your monitoring system; the setup is similar to that required when using custom metrics. External metrics allow you to autoscale your cluster based on any metric available in your monitoring system. Just provide a  <code>metric</code>  block with a  <code>name</code>  and  <code>selector</code>, as above, and use the  <code>External</code>  metric type instead of  <code>Object</code>. If multiple time series are matched by the  <code>metricSelector</code>, the sum of their values is used by the HorizontalPodAutoscaler. External metrics support both the  <code>Value</code>  and  <code>AverageValue</code>  target types, which function exactly the same as when you use the  <code>Object</code>  type.</p>
  
  <p>For example if your application processes tasks from a hosted queue service, you could add the following section to your HorizontalPodAutoscaler manifest to specify that you need one worker per 30 outstanding tasks.</p>
  
  <pre class="lang-yaml prettyprint-override"><code>- type: External
 external:
   metric:
     name: queue_messages_ready
     selector: "queue=worker_tasks"
   target:
     type: AverageValue
     averageValue: 30
</code></pre>
  
  <p>When possible, it’s preferable to use the custom metric target types instead of external metrics, since it’s easier for cluster administrators to secure the custom metrics API. The external metrics API potentially allows access to any metric, so cluster administrators should take care when exposing it.</p>
</blockquote>

<p>You may also have a look at <a href="https://github.com/zalando-incubator/kube-metrics-adapter" rel="nofollow noreferrer">zalando-incubator/kube-metrics-adapter</a> and use Prometheus collector <a href="https://github.com/zalando-incubator/kube-metrics-adapter#example-external-metric" rel="nofollow noreferrer">external metrics</a>.</p>

<blockquote>
  <p>This is an example of an HPA configured to get metrics based on a Prometheus query. The query is defined in the annotation  <code>metric-config.external.prometheus-query.prometheus/processed-events-per-second</code>  where  <code>processed-events-per-second</code>  is the query name which will be associated with the result of the query. A matching  <code>query-name</code>  label must be defined in the  <code>matchLabels</code>  of the metric definition. This allows having multiple prometheus queries associated with a single HPA.</p>

<pre><code>apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
 name: myapp-hpa
 annotations:
   # This annotation is optional.
   # If specified, then this prometheus server is used,
   # instead of the prometheus server specified as the CLI argument `--prometheus-server`.
   metric-config.external.prometheus-query.prometheus/prometheus-server: http://prometheus.my-&gt;namespace.svc
   # metric-config.&lt;metricType&gt;.&lt;metricName&gt;.&lt;collectorName&gt;/&lt;configKey&gt;
   # &lt;configKey&gt; == query-name
   metric-config.external.prometheus-query.prometheus/processed-events-per-second: |
     scalar(sum(rate(event-service_events_count{application="event-service",processed="true"}[1m])))
spec:
 scaleTargetRef:
   apiVersion: apps/v1
   kind: Deployment
   name: custom-metrics-consumer
 minReplicas: 1
 maxReplicas: 10
 metrics:
 - type: External
   external:
     metric:
       name: prometheus-query
       selector:
         matchLabels:
           query-name: processed-events-per-second
     target:
       type: AverageValue
       averageValue: "10"
</code></pre>
</blockquote>
