<p>I am currently working on the development of an API. I want to upload data via a frontend and this data is then processed on a server. I would like to develop the individual steps independently of one another and also maintain them independently. The first step of this dag is quite complex. There are a lot of different testing machines, each providing a slightly different set of raw data. Depending on the machine, a different task has to be started. In this step the file should be loaded and standardized. Afterwards, the data is passed on to several tasks for analysis (preferably also in parallel) and the results are finally stored in a database. The file size varies from a few MB to several GB.</p>
<p>The API will be extended later, performing calculations using the data previously stored in the database. The calculations are intensive in CPU times and memory usage.</p>
<p>Since I have not much knowledge of such complex structures, I have currently checked the following systems.</p>
<p><strong>Argo</strong></p>
<p>It requires its own Kubernetes cluster to deploy the system. Then you could deploy and run the individual tasks in containers in the cluster. The distribution of the resources takes over K8s. However, setting up the cluster is quite complex for me, since the system is to be set up on a cloud hoster at the end, which does not support Kubernetes natively. This would have to be set up manually. Another disadvantage is that the execution is quite slow, because one container has to be pushed and started at a time. Advantage extremely flexible and expandable. Possibly also future-proof, since each container can be maintained individually.</p>
<p><strong>Celery and Prefect</strong></p>
<p>You can easily code the individual processes in Python and define the dependencies. In Celery, you also have a task queue and therefore you can manage many individual tasks effectively. With Prefect, tasks that do not run in Python can be intercepted well via containers. However, there is no resource management and the systems do not appear to be so well extendable.</p>
<p><strong>What other benefits does Argo offer over Prefect/Celery?</strong></p>
<p>Thx</p>

markusdd
<p>Full disclaimer that I work for Prefect but I think Prefect can indeed slot in nicely here where you have an API that hits Prefect's API to create_flow_runs to process that data that came in through the frontend. It also sounds like you need a decentralized solution where the compute happens on different machines. Prefect lets you do this by having multiple agents on those different machines, and then you can invoke flow runs that get picked up by the respective agents.</p>
<p>Yes, you are also right non Python tasks can be abstracted with containers and by using tasks to spin up containers. There is some degree of resource management offered by the agents because you can have them on multiple machines. If you decide to go to Kubernetes also for better resource management, Prefect also supports that and the Kubernetes agent can control the resources per container.</p>
<p>This might be a good <a href="https://discourse.prefect.io/t/can-i-run-a-flow-of-flows-that-triggers-each-child-flow-on-a-different-machine/180" rel="nofollow noreferrer">resource</a> for you</p>
