<p>I have a container (a machine learning application) which is capable of loading pre-trained ml-models stored in a persistent volume. I can ask the application to load a particular model by giving its name via its REST API.</p>
<p>Now I want to scale up this application so that, I can load whatever the model, in any of the replicas (not in all of them) and should be able to parse data from that model.</p>
<p>I know this can be done by having multiple deployments and multiple services pointing to each of those deployments so that each instance will have a separate node port and I can access the REST API of each instance through these different node ports. By doing this, I can keep a record of which model is loaded on which instance in my own.</p>
<p>Is there any recommended way to accomplish this requirement without having multiple deployments but through replicas?  (Like maintaining a single deployment file with replicas and a manual load balancer at the service level)</p>

<p>As you mentioned, the preferred way to achieve this is through multiple deployments and services.</p>
<p>Unless created manually, replicasets are managed by the deployment and you won't be able to have a single deployment with multiple replicasets running  different versions.</p>
<p>It will definitely be easier for you to have one service + deployment per version and a single ingress in front to route the traffic based on some piece of information. It could be a header, sub-domain, path, etc.</p>
<p>To generate all your deployments and services, you could have a look at <a href="https://kustomize.io/" rel="nofollow noreferrer">kustomize</a>.</p>
