<p>I want to run an MPI job on my Kubernetes cluster. The context is that I'm actually running a modern, nicely containerised app but part of the workload is a legacy MPI job which isn't going to be re-written anytime soon, and I'd like to fit it into a kubernetes "worldview" as much as possible.</p>

<p>One initial question: has anyone had any success in running MPI jobs on a kube cluster? I've seen <a href="https://github.com/ChristianKniep" rel="noreferrer">Christian Kniep's</a> work in getting MPI jobs to run in docker containers, but he's going down the docker swarm path (with peer discovery using consul running in each container) and I want to stick to kubernetes (which already knows the info of all the peers) and inject this information into the container from the outside. I do have full control over all the parts of the application, e.g. I can choose which MPI implementation to use.</p>

<p>I have a couple of ideas about how to proceed:</p>

<ol>
<li><p>fat containers containing slurm and the application code -> populate
the slurm.conf with appropriate info about the peers at container
startup -> use srun as the container entrypoint to start the jobs</p></li>
<li><p>slimmer containers with only OpenMPI (no slurm) -> populate a
rankfile in the container with info from outside (provided by
kubernetes) -> use mpirun as the container entrypoint</p></li>
<li><p>an even slimmer approach, where I basically "fake" the MPI runtime by
setting a few environment variables (e.g. the OpenMPI ORTE ones) ->
run the mpicc'd binary directly (where it'll find out about its peers
through the env vars)</p></li>
<li><p>some other option</p></li>
<li><p>give up in despair</p></li>
</ol>

<p>I know trying to mix "established" workflows like MPI with the "new hotness" of kubernetes and containers is a bit of an impedance mismatch, but I'm just looking for pointers/gotchas before I go too far down the wrong path. If nothing exists I'm happy to hack on some stuff and push it back upstream.</p>

<p>I tried MPI Jobs on Kubernetes for a few days and solved it by using <code>dnsPolicy:None</code> and <code>dnsConfig</code> (<code>CustomDNS=true</code> feature gate will be needed).</p>

<p>I pushed my manifests (as Helm chart) here.</p>

<p><a href="https://github.com/everpeace/kube-openmpi" rel="nofollow noreferrer">https://github.com/everpeace/kube-openmpi</a></p>

<p>I hope it would help.</p>
