title: Snakemake: rule for using many inputs for one output with multiple sub-groups
body: <p>I have a working pipeline I'm using for downloading, aligning and performing variant calling on public sequencing data. The problem is that it can currently only work on a per-sample basis (<em>i.e</em> sample as each individual sequencing experiment). It doesn't work if I want to perform variant calling on a group of experiments (such as biological and/or technical replicates of a sample). I've tried to solving it, but I couldn't get it work.</p>
  
  <p>Here's a simplification of the alignment rule:</p>
  
  <pre><code>rule alignment:
      input:
          rules.download.output.fastq
      output:
          '{group}/alignment/{sample}.bam'
      shell:
          "bash scripts/02_alignment.sh {wildcards.group} {wildcards.samples}"
  </code></pre>
  
  <p>And the same for the variant calling:</p>
  
  <pre><code>rule variant_calling:
      input:
          rules.alignment.output
      output:
          '{group}/variants/{sample}.vcf.gz'
      shell:
          "bash scripts/03_variant_calling.sh {wildcards.sample} {wildcards.group}"
  </code></pre>
  
  <p>This works just fine, as there is a single <code>.vcf</code> file generated for each aligned <code>.bam</code> file. But what I would like to do is to generate a single <code>.vcf</code> file from an arbitrary number of <code>.bam</code> files. I have a <code>pandas</code> dataframe containing all the <code>sample</code> names and their corresponding <code>group</code>. I would essentially like to change the <code>output</code> of the second rule to be <code>'{group}/variants/{group}.vcf'</code>, but everything I've done has failed in some way.</p>
  
  <p>The idea I had was to provide the rule with all the per-group aligned <code>.bam</code> files as input and then just give the script it runs the directory where they all lie. The problem is that I can't find a way to give an input in this per-group manner: either I give per-sample (as the working pipeline) or I give ALL the <code>.bam</code> files for EACH group variant calling, regardless of what group they actually belong to. I can't use just wildcards, as the <code>{sample}</code> wildcard is not present in the last output. I also tried using functions as input, but that lead to the same problems as above.</p>
  
  <p>The crux of the matter seem to be the layers of grouping: if I wanted to perform variant calling on all aligned <code>.bam</code> files in the dataset as a whole, that would probably work fine, give my above mentioned problems. The problem comes with sub-groups for the dataset as a whole:</p>
  
  <pre><code>  sample1      sample2             sample1      sample2      sample3
       |            |                   |            |            |
       |            |                   |            |            |
       --------------                   ---------------------------
              |                                      |
              |                                      |
            group1                                 group2
  </code></pre>
  
  <p>Any ideas on how to solve this?</p>