<p>I am trying to figure out how to set up a <strong>work queue</strong> with <strong>Argo</strong>. The Argo Workflows are computationally expensive. We need to plan for many simultaneous requests. The workflow items are added to the work queue via HTTP requests.</p>
<p>The flow can be demonstrated like this:</p>
<pre class="lang-py prettyprint-override"><code>client  
  =&gt; hasura # user authentication  
    =&gt; redis # work queue
      =&gt; argo events # queue listener
        =&gt; argo workflows 
          =&gt; redis + hasura # inform that workflow has finished
            =&gt; client 
</code></pre>
<p>I have never build a K8s cluster that exceeds its resources. Where do I limit the execution of workflows? Or does Argo Events and Workflows limit these according to the resources in the cluster?</p>
<p>The above example could probably be simplified to the following, but the problem is what happens if the <a href="https://argoproj.github.io/argo/events/#authentication-and-security" rel="noreferrer">processing queue is full</a>?</p>
<pre class="lang-py prettyprint-override"><code>client
  =&gt; argo events # HTTP request listener
    =&gt; argo workflows
</code></pre>

<p>Argo Workflows has no concept of a queue, so it has no way of knowing when the queue is full. If you need queue control, that should happen before submitting workflows.</p>
<p>Once the workflows are submitted, there are a number of ways to limit resource usage.</p>
<ol>
<li><a href="https://github.com/argoproj/argo-workflows/tree/master/examples#kubernetes-resources" rel="nofollow noreferrer">Pod resources</a> - each Workflow step is represented by a Kubernetes Pod. You can set resource requests and limits just like you would with a Pod in a Deployment.</li>
<li><a href="https://github.com/argoproj/argo-workflows/blob/master/examples/parallelism-limit.yaml" rel="nofollow noreferrer">Step parallelism limit</a> - within a Workflow, you can limit the number of steps running concurrently. This can help when a step is particularly resource-intensive.</li>
<li><a href="https://github.com/argoproj/argo-workflows/blob/master/examples/synchronization-wf-level.yaml" rel="nofollow noreferrer">Workflow parallelism limit</a> - you can limit the number of workflows running concurrently by configuring them to us a semaphore.</li>
</ol>
<p>There are a number of other <a href="https://github.com/argoproj/argo-workflows/blob/master/docs/cost-optimisation.md" rel="nofollow noreferrer">performance optimizations</a> like setting Workflow and Pod TTLs and <a href="https://github.com/argoproj/argo-workflows/blob/master/docs/offloading-large-workflows.md" rel="nofollow noreferrer">offloading YAML for large Workflows to a DB instead of keeping them on the cluster</a>.</p>
<p>As far as I know, there is no way to set a Workflow limit so that Argo will reject additional Workflow submissions until more resources are available. This is a problem if you're worried about Kubernetes etcd filling up with too many Workflow definitions.</p>
<p>To keep from blowing up etcd, you'll need another app of some kind sitting in front of Argo to queue Workflows submissions until more resources become available.</p>
