title: Deploying model as service using FastAPI and Gunicorn with high throughput
body: <p>I am trying to deploy a machine learning model (based on tensorflow and keras) as a service using FastAPI and Gunicorn, but I am not able to get enough throughput from the API even after increasing number of Gunicorn workers and threads.</p>
  <p>I have tried with the following configs:</p>
  <ol>
  <li><p>1 Worker: gunicorn <strong>model</strong>:app -k uvicorn.workers.UvicornWorker -b hostname:port
  This is giving me a throughput of 15 responses/sec</p>
  </li>
  <li><p>5 workers: gunicorn <strong>model</strong>:app -k uvicorn.workers.UvicornWorker --workers=5 -b hostname:port
  This is giving me a throughput of 30 responses/sec</p>
  </li>
  </ol>
  <p>30 responses/sec is the maximum throughput I am able to get, while I have to scale it to around 300 responses/sec. I have tried increasing the number of threads too, but that did not result in increase in throughput either.</p>
  <p>When I am timing the request-response with single worker: it takes around 80ms for the response to return (Done through Postman)</p>
  <p>I am trying to run this on Linux machine with the following details:</p>
  <ul>
  <li>OS - CentOS</li>
  <li>CPU(s) - 8</li>
  <li>Core(s) per socket - 4</li>
  <li>Thread(s) per core - 2</li>
  <li>Memory - ~65Gig</li>
  </ul>
  <p>The system is almost idle when I am trying to run the service (less than 5% CPU usage).</p>