title: Set delay between tasks in group in Celery
body: <p>I have a python app where user can initiate a certain task.</p>
  
  <p>The whole purpose of a task is too execute a given number of POST/GET requests with a particular interval to a given URL.</p>
  
  <p>So user gives N - number of requests, V - number of requests per second.</p>
  
  <p>How is it better to design such a task taking into account that due to a I/O latency the actual r/s speed could bigger or smaller.</p>
  
  <p>First of all I decided to use Celery with Eventlet because otherwise I would need dozen of works which is not acceptable.</p>
  
  <p>My naive approach:</p>
  
  <ul>
  <li>Client starts a task using task.delay()</li>
  <li><p>Inside task I do something like this:</p>
  
  <pre><code>@task
  def task(number_of_requests, time_period):
     for _ in range(number_of_requests):
         start = time.time()
         params_for_concrete_subtask = ...
         # .... do some IO with monkey_patched eventlet requests library
         elapsed = (time.time() - start)
         # If we completed this subtask to fast
         if elapsed &lt; time_period / number_of_requests:
             eventlet.sleep(time_period / number_of_requests)
  </code></pre></li>
  </ul>
  
  <p>A working example is <a href="https://github.com/Glueon/celery_examples/blob/master/tasks.py#L38-L60" rel="nofollow">here</a>.</p>
  
  <p>if we are too fast we try to wait to keep the desired speed. If we are too slow it's ok from client's prospective. We do not violate requests/second requirement. But will this resume correctly if I restart Celery?</p>
  
  <p>I think this should work but I thought there is a better way.
  In Celery I can define a task with a particular rate limit which will almost match my needs guarantee. So I could use Celery <code>group</code> feature and write:</p>
  
  <pre><code>@task(rate_limit=...)
  def task(...):
      #
  
  task_executor = task.s(number_of_requests, time_period)
  group(task_executor(params_for_concrete_task) for params_for_concrete_task in ...).delay()
  </code></pre>
  
  <p>But here I hardcode the the rate_limit which is dynamic and I do not see a way of changing it. I saw an example:</p>
  
  <pre><code>  task.s(....).set(... params ...)
  </code></pre>
  
  <p>But I tried to pass <code>rate_limit</code> to the <code>set</code> method it it did not work.</p>
  
  <p>Another maybe a bettre idea was to use Celery's periodic task scheduler. With the default implementation periods and tasks to be executed periodically is fixed. </p>
  
  <p>I need to be able to dynamically create tasks, which run periodically a given number of times with a specific rate limit. Maybe I need to run my own Scheduler which will take tasks from DB? But I do not see any documentation around this.</p>
  
  <p>Another approach was to try to use a <code>chain</code> function, but I could not figure out is there a delay between tasks parameter.</p>