<p>I have a machine learning model inside a docker image. I pushed the docker image to google container registry and then deploy it inside a Kubernetes pod. There is a fastapi application that runs on Port 8000 and this Fastapi endpoint is public
(call it <code>mymodel:8000</code>).</p>
<p>The structure of fastapi is :</p>
<pre><code>app.get(&quot;/homepage&quot;)
asynd def get_homepage()

app.get(&quot;/model):
aysnc def get_modelpage()

app.post(&quot;/model&quot;):
async def get_results(query: Form(...))
</code></pre>
<p>User can put query and submit them and get results from the machine learning model running inside the docker. I want to limit the number of times a query can be made by all the users combined. So if the query limit is 100, all the users combined can make only 100 queries in total.</p>
<p>I thought of a way to do this:</p>
<p>Store a database that stores the number of times GET and POST method has been called. As soon as the total number of times POST has been called crosses the limit, stop accepting any more queries.</p>
<p>Is there an alternative way of doing this using Kubernetes limits? Such as I can define a <code>limit_api_calls</code> such that the total number of times <code>mymodel:8000</code> is accessed is at max equal to <code>limit_api_calls</code>.</p>
<p>I looked at the documentation and I could only find setting limits for CPUs, Memory and rateLimits.</p>

shubh gupta
<p>There are several approaches that could satisfy your needs.</p>
<ul>
<li>Custom implementation: As you mentioned, keep in a persistence layer the number of API calls received and deny requests after it has been reached.</li>
<li>Use a service mesh: Istio (for instance) will let you limit the number of requests received and act as a circuit breaker.</li>
<li>Use an external Api Manager: Apigee will also let you limit and even charge your users, however if it is only for internal use (not pay per use) I definitely won't recommend it.</li>
</ul>
<p>The tricky part is what you want to happen after the limit has been reached, if it is just a pod you may exit the application to finish and clear it.</p>
<p>Otherwise, if you have a deployment with its replica set and several resources associated with it (like configmaps), you probably want to use some kind of asynchronous alert or polling check to clean up everything related to your deployment. You may want to have a deep look at orchestrators like Airflow (Composer) and use several tools such as Helm for keeping deployments easy.</p>
