title: Dynamically generating partitions in Airflow
body: <p>I have a set of tasks in a DAG on Airflow for an ETL job that takes data from an open source, transforms it, then makes it available on AWS Athena. The data is split into 5 tables which holds messages about similar events. I partition the data by date and a group id. </p>
  
  <p>My present set up has a task 'add_partition_{0}_{1}'  where {0} is the table and {1} is the group id. This task exists within a nested loop. The outer loop cycles thru the group ids and the inner loop cycles thru the tables.</p>
  
  <p>I currently do partitioning by using Jinja to template an add_partition.sql file. We altered the standard athena operator to allow it to directly read a file. However, the size of the items being looped has been increased so now ~150 partitioning tasks are being created. I would like to be able to have a single add_partitions task which can take the .sql file and generate a list of statements to add each partition in a single command. I would pass the list of group_ids, list of tables and the date to the template and create the statements.</p>
  
  <pre class="lang-sql prettyprint-override"><code>ALTER TABLE {{ params.database }}.{{ params.table_name }} ADD IF NOT EXISTS
    PARTITION (group_id= '{{ params.group }}', date = '{{ ds }}')
  </code></pre>
  
  <p>I would like the statements sent to the operator to be like </p>
  
  <pre class="lang-sql prettyprint-override"><code>ALTER TABLE db.table ADD IF NOT EXISTS
    PARTITION (group_id= '1', date = '2019-09-30')
  ALTER TABLE db.table ADD IF NOT EXISTS
    PARTITION (group_id= '2', date = '2019-09-30')
  ALTER TABLE db.table ADD IF NOT EXISTS
    PARTITION (group_id= '3', date = '2019-09-30')
  </code></pre>
  
  <p>and so on. Is this possible to do using Jinja templating? I have seen that looping is possible using Jinja but I am not sure how to apply it here?</p>