<p>I had been trying to implement Kubernetes HPA using Metrics from Kafka-exporter. Hpa supports Prometheus, so we tried writing the metrics to prometheus instance. From there, we are unclear on the steps to do. Is there an article where it will explain in details ?</p>

<p>I followed <a href="https://medium.com/google-cloud/kubernetes-hpa-autoscaling-with-kafka-metrics-88a671497f07" rel="nofollow noreferrer">https://medium.com/google-cloud/kubernetes-hpa-autoscaling-with-kafka-metrics-88a671497f07</a></p>

<p>for same in GCP and we used stack driver, and the implementation worked like a charm. But, we are struggling in on-premise setup, as stack driver needs to be replaced by Prometheus</p>

<p>When I implemented Kubernetes HPA using Metrics from Kafka-exporter I had a few setbacks which I solved doing the following:  </p>

<ol>
<li>I deployed the kafka-exporter container as a sidecar to the pods I
wanted to scale. I found that the HPA scales the pod it gets the
metrics from.  </li>
<li><p>I used <a href="https://www.weave.works/docs/cloud/latest/tasks/monitor/configuration-k8s/#per-pod-prometheus-annotations" rel="nofollow noreferrer">annotations</a> to make Prometheus scrape the metrics from the pods with exporter.</p></li>
<li><p>Then I verified that the kafka-exporter metrics are getting to Prometheus. If it's not there you can't advance further.</p></li>
<li>I deployed <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter" rel="nofollow noreferrer">prometheus adapter</a> using its <a href="https://github.com/helm/charts/tree/master/stable/prometheus-adapter" rel="nofollow noreferrer">helm chart</a>. The adapter will "translate" Prometheus's metrics into custom Metrics
Api, which will make it visible to HPA.</li>
<li>I made sure that the metrics are visible in k8s by executing <code>kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1</code> from one of
the master nodes.</li>
<li>I created an hpa with the matching metric name.</li>
</ol>

<p>Here is a complete guide explaining <a href="https://medium.com/@ranrubin/horizontal-pod-autoscaling-hpa-triggered-by-kafka-event-f30fe99f3948" rel="nofollow noreferrer">how to implement Kubernetes HPA using Metrics from Kafka-exporter</a></p>

<p>Please comment if you have more questions</p>
