<p>I have created 15 servers which contain the same machine learning program. However, each server has different arguments at the runtime. It is determined by the hostname. Each server also has a copy of a 5 1gb pkl files which contain training data.</p>
<p>So for example right now, I have created 15 servers in the cloud with the following names.</p>
<pre><code>ml-node-1
ml-node-2
ml-node-3
..
ml-node-15
</code></pre>
<p>So when my program runs on <code>ml-node-1</code> it looks like this, <code>python3 mlscript.py $HOSTNAME.csv</code> and it will run <code>python3 mlscript.py ml-node-1.csv</code>. Each server will run the script which is meant for its hostname.</p>
<p>My problem is that I have to create 15 copies of the 5gb pkl data in each server before they are run. I find this very inefficient and costly therefore I am looking up Kubernetes as a solution. From the documentation, I can see that containers within a pod can share a persistent volume. This way I might be able to mitigate copying the 5gb pkl data 15 times.</p>
<p>However, I am trying to figure out the naming of the servers/containers. I figure that I would need 1 pod with a shared persistent volume and 15 containers. According to what I can understand from the documentation, <a href="https://kubernetes.io/docs/concepts/containers/container-environment/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/containers/container-environment/</a>, all the containers within the pod will share the same hostname.</p>
<p>How do I differentiate them and give them specific hostnames so that I can still have a different argument running within each container? My docker images are the standard debian image with my machine learning script.</p>

anarchy
<p>Rather than relying on custom hostnames for a scalable multi-host environment like Kubernetes (which is not designed to allow for this), as a more feasible solution (<a href="https://stackoverflow.com/questions/65576309#comment115941183_65576309">suggested</a> by <a href="https://stackoverflow.com/users/137650">MatsLindh</a>) you could write some code in your <code>mlscript.py</code> that generates a unique random key on startup and then watches an <code>input</code> directory for changes in a continuous loop: This directory would contain the available files to be picked up by the different containers, and the script would rename a given file with the generated key when it assigns it to the running server instance in the container, with &quot;unassigned&quot; files (not containing a key with the same format in the name) being considered available for other instances.</p>
<p>This would allow you to scale the workload up or down from Kubernetes as well as have several files processed by the same container instance.</p>
