title: Orchestration engines and frameworks?
body: <p>I'm looking for an <strong>orchestration</strong> framework/engine/toolkit with which to replace/upgrade an existing software, mainly because of scalability limitations. By orchestration I mean asynchronous and distributed execution of generic tasks and workflows.</p>
  
  <p>More specifically the requirements are pretty much these:</p>
  
  <ul>
  <li>Wrapping and execution of generic tasks, in Java if language dependent</li>
  <li>API for tasks and workflows on-demand triggering</li>
  <li>Scheduling would be nice as well</li>
  <li>Support for distributed architecture &amp; scalability (mainly for big numbers of small tasks)</li>
  <li>Persistency and resilience</li>
  <li>Advanced workflow configuration capabilities (do <em>this</em>, then <em>these 3 tasks</em> in parallel, then <em>this</em>, having priorities, dependencies...)</li>
  <li>Monitoring and administration UI (or at least API)</li>
  </ul>
  
  <p>The existing system is an old fashion monolithic service (in Java) that has most of that, including the execution logic itself which should remain as untouched as possible.</p>
  
  <p>Does anyone have experience with a similar problem? It seems to me it's supposed to be pretty common, would be strange if I have to implement it myself entirely. I found some questions here (like <a href="https://stackoverflow.com/questions/29117570/orchestrating-microservices?rq=1">this</a> and <a href="https://stackoverflow.com/questions/43069858/orchestrated-vs-choreographed-service-oriented-architecture-in-large-scale?rq=1">this</a>) discussing the <strong>theory</strong> of orchestration and choreography systems, but not real examples of tools <strong>implementing</strong> it. Also I think we're not exactly talking about microservices - the tasks are not prolonged and heavy, they're just many, running in the background executing short jobs of many types. I wouldn't create a service for every job type.</p>
  
  <p>I'm also <strong>not</strong> looking for cloud and container services at this point - to my understanding the deployment is a different issue.</p>
  
  <p>The closest I got is the <a href="https://github.com/Netflix/conductor" rel="noreferrer">Netflix Conductor engine</a>, which answers most of the requirements by running an orchestration server that manages tasks implemented in servlets (or any web services in any language - a plus). However it seems like it's built mainly for arranging heavy tasks in a workflow rather than running a huge number of small tasks, which makes me wonder what would be the overhead of invoking many small tasks in servlets for example.</p>
  
  <p>Does anyone have experience or any input on the Conductor or other tools I could use? Or even my entire approach to the problem?</p>
  
  <p>EDIT: I realize it's kind of a "research advice needed" so let's put it simply in 3 questions:</p>
  
  <ol>
  <li>Am I right to look for an <strong>orchestration</strong> solution for the requirements above?</li>
  <li>Does anyone have experience with the <strong>Netflix Conductor</strong>? Any feedback on it?</li>
  <li>Does it have good competitors?</li>
  </ol>