<p>I have been deploying apps to Kubernetes for the last 2 years. And in my org, all our apps(especially stateless) are running in Kubernetes. I still have a fundamental question, just because very recently we found some issues with respect to our few python apps.</p>

<p>Initially when we deployed, our python apps(Written in Flask and Django), we ran it using <code>python app.py</code>. It's known that, because of GIL, python really doesn't have support for system threads, and it will only serve one request at a time, but in case the one request is CPU heavy, it will not be able to process further requests. This is causing sometimes the health API to not work. We have observed that, at this moment, if there is a single request which is not IO and doing some operation, we will hold the CPU and cannot process another request in parallel. And since it's only doing fewer operations, we have observed there is no increase in the CPU utilization also. This has an impact on how <code>HorizontalPodAutoscaler</code> works, its unable to scale the pods.</p>

<p>Because of this, we started using <code>uWSGI</code> in our pods. So basically <code>uWSGI</code> can run multiple pods under the hood and handle multiple requests in parallel, and automatically spin new processes on demand. But here comes another problem, that we have seen, <code>uwsgi</code> is lacking speed in auto-scaling the process tocorrected serve the request and its causing <code>HTTP 503</code> errors, Because of this we are unable to serve our few APIs in 100% availability. </p>

<p>At the same time our all other apps, written in <code>nodejs</code>, <code>java</code> and <code>golang</code>, is giving 100% availability.</p>

<p>I am looking at what is the best way by which I can run a python app in 100%(99.99) availability in Kubernetes, with the following</p>

<blockquote>
  <ol>
  <li>Having health API and liveness API served by the app</li>
  <li>An app running in Kubernetes</li>
  <li>If possible without uwsgi(Single process per pod is the fundamental docker concept)</li>
  <li>If with uwsgi, are there any specific config we can apply for k8s env</li>
  </ol>
</blockquote>

<p>I have exactly the same problem with a python deployment running the Flask application. Most api calls are handled in a matter of seconds, but there are some cpu intensive requests that acquire GIL for 2 minutes.... The pod keep accepting requests, ignores the configured timeouts, ignores a closed connection by the user; then after 1 minute of liveness probes failing, the pod is restarted by kubelet.</p>
<p>So 1 fat request can dramatically drop the availability.</p>
<p>I see two different solutions:</p>
<ol>
<li>have a separate deployment that will host only long running api calls; configure ingress to route requests between these two deployments;</li>
<li>using multiprocessing handle liveness/readyness probes in a main process, every other request must be handled in the child process;</li>
</ol>
<p>There are pros and cons for each solution, maybe I will need a combination of both. Also if I need a steady flow of prometheus metrics, I might need to create a proxy server on the application layer (1 more container on the same pod). Also need to configure ingress to have a single upstream connection to python pods, so that long running request will be queued, whereas short ones will be processed concurrently (yep, python, concurrency, good joke). Not sure tho it will scale well with HPA.</p>
<p>So yeah, running production ready python rest api server on kubernetes is not a piece of cake. Go and java have a much better ecosystem for microservice applications.</p>
<p>PS
here is a good article that shows that there is no need to run your app in kubernetes with WSGI
<a href="https://techblog.appnexus.com/beyond-hello-world-modern-asynchronous-python-in-kubernetes-f2c4ecd4a38d" rel="nofollow noreferrer">https://techblog.appnexus.com/beyond-hello-world-modern-asynchronous-python-in-kubernetes-f2c4ecd4a38d</a></p>
<p>PPS
Im considering to use prometheus exporter for flask. Looks better than running a python client in a separate thread;
<a href="https://github.com/rycus86/prometheus_flask_exporter" rel="nofollow noreferrer">https://github.com/rycus86/prometheus_flask_exporter</a></p>
