<p>I am going to deploy a Python Flask Server with Docker on Kubernetes using Gunicorn and Gevent/Eventlet as asynchronous workers. The application will:</p>

<ol>
<li>Subscribe to around 20 different topics on Apache Kafka.</li>
<li>Score some machine learning models with that data.</li>
<li>Upload the results to a relational database.</li>
</ol>

<p>Each topic in Kafka will receive 1 message per minute, so the application needs to consume around 20 messages per minute from Kafka. For each message, the handling and execution take around 45 seconds. The question is how I can scale this in a good way? I know that I can add multiple workers in Gunicorn and use multiple replicas of the pod when I deploy to Kubernetes. But is that enough? Will the workload be automatically balanced between the available workers in the different pods? Or what can I do to ensure scalability?</p>

<p>I recommend you set up an <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">HPA</a> Horizontal Pod Autoscaler for your workers.</p>

<p>It will require to set up support for the <a href="https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/" rel="nofollow noreferrer">metrics API</a>. For custom metrics on the later versions of Kubernetes <a href="https://github.com/kubernetes/heapster" rel="nofollow noreferrer">heapster</a> has been deprecated in favor of the <a href="https://github.com/kubernetes-incubator/metrics-server" rel="nofollow noreferrer">metrics server</a></p>

<p>If you are using the public Cloud like AWS, GCP, or Azure I'd also recommend setting up an Autoscaling Group so that you can scale your VMs or server base on metrics like CPU utilization average.</p>

<p>Hope it helps!</p>
