<p>I am new to devops world and have a doubt in Kubernetes.
My usecase is as follows.<br />
<strong>My organization is currently hosting all its microservices in Docker containers using Kubernetes container orchestration platform.</strong></p>
<p>How I want my microservice container to be scaled and automated using Kubernetes:</p>
<blockquote>
<p>I have 100s of AI models stored in Amazon S3 bucket. When a request comes from user1 to the container with a particular model name,
corresponding model from the S3 should be fetched and loaded in the container
memory then user1 should start the inference. Similarly if another
request comes from user2 with another model name, <strong>new container should spin off</strong>
and user2 should start the inference. For n model requests, n containers should
spin off and serve each session. Containers should automatically scale
up with different requests.</p>
</blockquote>
<p>Since the above logic is similar to how AWS Lambda function works, I was recommending the same to our devops team. <strong>&quot;They informed that they cannot use AWS Lambda for a single microservice since it is serverless and all other microservices are non-serverless which runs in Kubernetes&quot;</strong></p>
<p><strong>My question is, can the same concept be implemented with Kubernetes?. If so please suggest a method or doc.</strong></p>
<p>Thanks in advance.</p>

<p>I would use <a href="https://github.com/kubeflow/kfserving" rel="nofollow noreferrer">KFServing</a> for serverless inference on k8s. You can install it as a <a href="https://github.com/kubeflow/kfserving#standalone-kfserving-installation" rel="nofollow noreferrer">standalone component</a> or you can run it with the full <a href="https://www.kubeflow.org/" rel="nofollow noreferrer">Kubeflow</a> toolkit. You can inject your models from s3 into the serving containers which are optimized for whichever ML framework or language you might use. It uses Knative &amp; Istio under the hood in order to achieve the &quot;scale-to-zero&quot; functionality.</p>
