<p>We have a kind of evaluation job which consists of several thousand invocations of a legacy binary with various inputs, each of which running like a minute. The individual runs are perfectly parallelizable (one instance per core).</p>

<p>What is the state of the art to do this in a hybrid cloud scenario?</p>

<p>Kubernetes itself does not seem to provide an interface for prioritizing or managing waiting jobs. Jenkins would be good at these points, but feels like a hack. Of course, we could hack something ourselves, but the problem should be sufficiently generic to already have an out-of-the box solution.</p>

<p>There are a lot of frameworks that helps managing jobs in Kubernetes cluster. The most popular are:</p>

<ul>
<li><a href="https://github.com/argoproj/argo" rel="nofollow noreferrer">Argo</a> for orchestrating parallel jobs on Kubernetes. Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).</li>
<li><a href="https://github.com/apache/airflow" rel="nofollow noreferrer">Airflow</a> - has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Also take a look for <a href="https://airflow.apache.org/kubernetes.html#kubernetes-executor" rel="nofollow noreferrer">kubernetes-executor</a>. </li>
</ul>

<p>I recommend you to look for <a href="https://www.youtube.com/watch?v=oXPgX7G_eow&amp;utm_campaign=Singapore&amp;utm_content=97139107&amp;utm_medium=social&amp;utm_source=twitter&amp;hss_channel=tw-381629790" rel="nofollow noreferrer">this video</a> which describe each of framework and help you decide which is better for you.</p>
