title: How to parallelize loops with an include in ansible
body: <p>Recently I came across a bottleneck in our ansible playbooks' code. We were deploying our clusters (e.g. a <a href="https://docs.mongodb.com/manual/tutorial/deploy-replica-set/" rel="nofollow noreferrer">mongoDB Replica Set</a>) sequentially - i.e. one VM after another, each waiting for the previous to be up and running. </p>
  
  <p>This slowed down the whole cluster deploy time by a factor of the members on it. </p>
  
  <p>To solve this, I started digging on <a href="https://docs.ansible.com/ansible/2.5/user_guide/playbooks_async.html" rel="nofollow noreferrer">ansible's async actions and pooling</a> and found out a few examples on parallel loops and "<em>fire-and-forget</em>" strategies for scenarios like ours. </p>
  
  <p>The particular thing is, we have defined our own "customize the VM and spawn it" ansible task (<code>create_instance.yml</code>) that gets included and receives the different customization variables from the playbook and abstracts the whole process by running different KVM/shell commands. </p>
  
  <p>Using <a href="http://toroid.org/ansible-parallel-dispatch" rel="nofollow noreferrer">"<em>Parallel task execution in Ansible</em>"</a> as reference, I ended up having something like: </p>
  
  <pre><code>- name: Generate VMs for DB
    hosts: hypervisor_fe
    tags: platform,mongodb
    tasks:
      - include: tasks/create_instance.yml
        vars:
          vm: "{{ item }}"
        with_items: "{{ mongodb.vms }}"
        register: mongo_instances
        async: 7200
        poll: 0
  
      - name: Wait for instance creation to complete
        async_status: jid={{ item.ansible_job_id }}
        register: mongo_jobs
        until: mongo_jobs.finished
        retries: 300
        with_items: "{{ mongo_instances.results }}"
  </code></pre>
  
  <p>However, <strong>this setup does seem to ignore all the new async code and keeps the old, sequential behavior.</strong> I'm guessing this has to do with the no. and granularity of plays inside the imported task. If I instead replace the <code>include</code> for a <strong>single, explicit</strong> long-running task - let's say, e.g.</p>
  
  <pre><code>- name: Test async operation
    shell: ping -c1 {{ item.hostname }} &amp;&amp; sleep 20
  </code></pre>
  
  <p>This does seem to work just fine, running one ping to each <code>item</code> and then moving on to the next action.</p>
  
  <p>Is this assumption right? Does someone has experience with <code>include</code> and async loops in ansible? Do I need to move the async declaration to a single play inside the imported code?</p>