<p>Is there a suggested way to serve hundreds of machine learning models in Kubernetes?
Solutions like <a href="https://github.com/kubeflow/kfserving" rel="nofollow noreferrer">Kfserving</a> seem to be more suitable for cases where there is a single trained model, or a few versions of it, and this model serves all requests. For instance a typeahead model that is universal across all users.</p>

<p>But is there a <strong>suggested way to serve hundreds or thousands of such models?</strong> For example, a typeahead model trained specifically on each user's data.</p>

<p>The most naive way to achieve something like that, would be that each typeahead serving container maintains a local cache of models in memory. But then scaling to multiple pods would be a problem because each cache is local to the pod. So each request would need to get routed to the correct pod that has loaded the model.</p>

<p>Also having to maintain such a registry where we know which pod has loaded which model and perform updates on model eviction seems like a lot of work.</p>

Aristos
<p><strong>You can use Catwalk mixed with <a href="https://www.grab.com/sg/" rel="nofollow noreferrer">Grab</a>.</strong></p>

<blockquote>
  <p>Grab has a tremendous amount of data that we can leverage to solve
  complex problems such as fraudulent user activity, and to provide our
  customers personalized experiences on our products. One of the tools
  we are using to make sense of this data is machine learning (ML).</p>
  
  <p>That is how Catwalk is created: an easy-to-use, self-serve, machine
  learning model serving platform for everyone at Grab.</p>
</blockquote>

<p>More infromation about Catwalk you can find here: <a href="https://engineering.grab.com/catwalk-serving-machine-learning-models-at-scale" rel="nofollow noreferrer">Catwalk</a>.</p>

<p><strong>You can serve multiple Machine Learning models using TensorFlow and Google Cloud.</strong> </p>

<blockquote>
  <p>The reason the field of machine learning is experiencing such an epic
  boom is because of its real potential to revolutionize industries and
  change lives for the better. Once machine learning models have been
  trained, the next step is to deploy these models into usage, making
  them accessible to those who need them — be they hospitals,
  self-driving car manufacturers, high-tech farms, banks, airlines, or
  everyday smartphone users. In production, the stakes are high and one
  cannot afford to have a server crash, connection slow down, etc. As
  our customers increase their demand for our machine learning services,
  we want to seamlessly meet that demand, be it at 3AM or 3PM.
  Similarly, if there is a decrease in demand we want to scale down the
  committed resources so as to save cost, because as we all know, cloud
  resources are very expensive.</p>
</blockquote>

<p>More information you cna find here: <a href="https://blog.usejournal.com/machine-learning-in-production-serving-up-multiple-ml-models-at-scale-the-tensorflow-serving-9607eeea30" rel="nofollow noreferrer">machine-learning-serving</a>.</p>

<p><strong>Also you can use <a href="https://docs.seldon.io/projects/seldon-core/en/latest/" rel="nofollow noreferrer">Seldon</a>.</strong></p>

<p>Seldon Core is an open source platform for deploying machine learning models on a Kubernetes cluster.</p>

<p><strong>Features:</strong></p>

<ul>
<li>deploying machine learning models in the cloud or on-premise.</li>
<li>gaining metrics ensuring proper governance and compliance for your
running machine learning models.</li>
<li><p>creating inference graphs made up of multiple components.</p></li>
<li><p>providing a consistent serving layer for models built using<br>
heterogeneous ML toolkits.</p></li>
</ul>

<p>Useful documentation: <a href="https://mapr.com/ebook/kubernetes-for-machine-learning-deep-learning-and-ai/assets/Cloud_EB_Kubernetes_MLDL_SPONSOR.pdf" rel="nofollow noreferrer">Kubernetes-Machine-Learning</a>.</p>
