text: user3964: I'm using Snakemake and I'm trying to get all my QC reports into multiqc but I get the following error:
  WorkflowError in line 120 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
  'Wildcards' object has no attribute 'sample'
  
  Rule 120 is the start of my multiqc rule which looks like this:
  rule multiqc:
      input:
          expand([os.path.join(analyzed_dir, '{sample}.stat', '{sample}.cnt'),
                  os.path.join(rseqc_dir, '{sample}.bam_stat.txt'),
                  os.path.join(rseqc_dir, '{sample}.clipping_profile.xls'),
                  os.path.join(rseqc_dir, '{sample}.deletion_profile.txt'),
                  os.path.join(rseqc_dir, '{sample}.infer_experiment.txt'),
                  os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                  os.path.join(rseqc_dir, '{sample}.inner_distance.txt'),
                  os.path.join(rseqc_dir, '{sample}.insertion_profile.xls'),
                  os.path.join(rseqc_dir, '{sample}.junction.xls'),
                  os.path.join(rseqc_dir, '{sample}.junctionSaturation_plot.r'),
                  os.path.join(rseqc_dir, '{sample}.mismatch_profile.xls'),
                  os.path.join(rseqc_dir, '{sample}.read_distribution.txt'),
                  os.path.join(rseqc_dir, '{sample}.pos.DupRate.xls'),
                  os.path.join(rseqc_dir, '{sample}.seq.DupRate.xls'),
                  os.path.join(rseqc_dir, '{sample}.GC.xls'),
                  os.path.join(rseqc_dir, '{sample}.NVC.xls'),
                  os.path.join(rseqc_dir, '{sample}.qual.r'),
                  os.path.join(rseqc_dir, '{sample}.RNA_fragment_size.txt'),
                  os.path.join(rseqc_dir, '{sample}.STAR.genome.sorted.summary.txt'),
                  os.path.join(fastqc_dir, '{sample}_R{read}_001_fastqc.html')],
                  sample=samples['samples'], read=reads)
      output:
          os.path.join(qc_dir, 'multiqc_report.html') 
      conda:
          "envs/multiqc.yaml"
      shell:
          '''
          #!/bin/bash
          multiqc . --outdir qc --ignore .snakemake --force
          '''
  
  My "rule all looks like this:"
  rule all:
      input:
          expand([os.path.join(analyzed_dir, '{sample}.genes.results'),
                  os.path.join(qc_dir, 'multiqc_report.html')],
                  sample=samples['samples'])
  
  I don't see anything wrong with it, in fact, it is very similar to my "rule all" I used before and which worked. Can anyone see an error?
  EDIT
  I limited the rule to only one input, I still have the same error:
  rule multiqc:
      input:
          expand(os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                  sample=samples['samples'])
      output:
          os.path.join(qc_dir, 'multiqc_report.html') 
      conda:
          "envs/multiqc.yaml"
      shell:
          '''
          #!/bin/bash
          multiqc . --outdir qc --ignore .snakemake --force
          '''
  
  This is the ouput, you can see it read the samples correctly:
  run_snakemake.sh 
  /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
    from collections import OrderedDict
  Building DAG of jobs...
  /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
    from collections import OrderedDict
  Building DAG of jobs...
  Using shell: /bin/bash
  Provided cluster nodes: 256
  Job counts:
      count   jobs
      1   all
      1   multiqc
      2
  
  [Mon Jan 14 17:46:40 2019]
  rule multiqc:
      input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R02_S2.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R03_S3.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R04_S4.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R05_S5.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R06_S6.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R08_S8.geneBodyCoverage.txt
      output: qc/multiqc_report.html
      jobid: 2
  
  WorkflowError in line 119 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
  'Wildcards' object has no attribute 'sample'
  17:46 nlv24077@kiato /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full > ll qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt
  -rw-rw-r-- 1 nlv24077 2017-0205_illuminaseq 303 Jan  7 16:00 qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt
  
  EDIT 2:
  This is how I start Snakemake:
  > cat run_snakemake.sh
  source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
  snakemake --dag | dot -T svg > dag.svg
  snakemake --cluster-config cluster.json --cluster "qsub -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {wildcards.sample}_{rule}" --jobs 256 --use-conda --rerun-incomplete
  
  EDIT 3:
  Ok, that last remark (by @finswimmer) triggered me, I removed the "{wildcards.sample}_" from the above command and then it works. Or, works, I get a very different error, it is multiqc specific and solved as specified here: https://multiqc.info/docs/#locale-error-messages. When adding the two lines as suggested this error disappears and everything works as expected, even with the long inputs list.
  RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.
  
  Anyway, I'm guessing Snakemake does not understand how to name this task when it uses an entire list of {sample}'s... Is this a bug? What should be used? I guess it is impossible to use {sample} for naming for this rule. It's a shame I have to kill the naming for all rules because using sample_rule is very nice for keeping track.
  
  user240: Snakemake gets wildcards from the parsing the input/output file names.
  For example, if you had
  rule sort:
      input: "{file}.bed"
      output: "{file}.sorted.bed"
      shell: "sort -k1,1 -k2,2n {input} > {output}"
  
  then the wildcards variable would be set as wildcards = {"file": ...}.
  In your multiqc rule, you have the {sample} variable within your expand(), but this variable doesn't exist outside of expand's local context. You don't actually have any wildcards there, nor in your output file (qc_dir and rseqc_dir are variables you define elsewhere, but not "wildcards" as Snakemake interprets them).
  So you're getting the error 'Wildcards' object has no attribute 'sample' because there are no wildcards for that rule.
  Your rule is currently set up to process every sample that you have listed in  samples['sample']. If you want your multiqc rule to work for a single sample, simply remove the sample=samples['sample'] and double brace the {sample} wildcard.
  rule multiqc:
      input:
          expand([os.path.join(analyzed_dir, '{{sample}}.stat', '{{sample}}.cnt'),
                  os.path.join(rseqc_dir, '{{sample}}.bam_stat.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.clipping_profile.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.deletion_profile.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.infer_experiment.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.geneBodyCoverage.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.inner_distance.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.insertion_profile.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.junction.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.junctionSaturation_plot.r'),
                  os.path.join(rseqc_dir, '{{sample}}.mismatch_profile.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.read_distribution.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.pos.DupRate.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.seq.DupRate.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.GC.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.NVC.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.qual.r'),
                  os.path.join(rseqc_dir, '{{sample}}.RNA_fragment_size.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.STAR.genome.sorted.summary.txt'),
                  os.path.join(fastqc_dir, '{{sample}}_R{read}_001_fastqc.html')],
                  read=reads)
      output:
          # this {sample} is in single {} and is recognized as a wildcard
          os.path.join(qc_dir, '{sample}', 'multiqc_report.html') 
      conda:
          "envs/multiqc.yaml"
      shell:
          '''
          #!/bin/bash
          # using {wildcards.sample} here since MultiQC likes directories
          multiqc . --outdir qc/{wildcards.sample} --ignore .snakemake --force
          '''
  
  When wildcards are recognized, you'll see them in the job list.
  Running snakemake 0067_P2018SEQE27R01_S1/multiqc_report.html, for example, should give you something like:
  rule multiqc:
      input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, ...
      output: qc/0067_P2018SEQE27R01_S1/multiqc_report.html
      jobid: 2
      wildcards: sample=0067_P2018SEQE27R01_S1
  
  
original_text: 3964: <p>I'm using Snakemake and I'm trying to get all my QC reports into multiqc but I get the following error:</p>
  
  <pre><code>WorkflowError in line 120 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
  'Wildcards' object has no attribute 'sample'
  </code></pre>
  
  <p>Rule 120 is the start of my multiqc rule which looks like this:</p>
  
  <pre><code>rule multiqc:
      input:
          expand([os.path.join(analyzed_dir, '{sample}.stat', '{sample}.cnt'),
                  os.path.join(rseqc_dir, '{sample}.bam_stat.txt'),
                  os.path.join(rseqc_dir, '{sample}.clipping_profile.xls'),
                  os.path.join(rseqc_dir, '{sample}.deletion_profile.txt'),
                  os.path.join(rseqc_dir, '{sample}.infer_experiment.txt'),
                  os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                  os.path.join(rseqc_dir, '{sample}.inner_distance.txt'),
                  os.path.join(rseqc_dir, '{sample}.insertion_profile.xls'),
                  os.path.join(rseqc_dir, '{sample}.junction.xls'),
                  os.path.join(rseqc_dir, '{sample}.junctionSaturation_plot.r'),
                  os.path.join(rseqc_dir, '{sample}.mismatch_profile.xls'),
                  os.path.join(rseqc_dir, '{sample}.read_distribution.txt'),
                  os.path.join(rseqc_dir, '{sample}.pos.DupRate.xls'),
                  os.path.join(rseqc_dir, '{sample}.seq.DupRate.xls'),
                  os.path.join(rseqc_dir, '{sample}.GC.xls'),
                  os.path.join(rseqc_dir, '{sample}.NVC.xls'),
                  os.path.join(rseqc_dir, '{sample}.qual.r'),
                  os.path.join(rseqc_dir, '{sample}.RNA_fragment_size.txt'),
                  os.path.join(rseqc_dir, '{sample}.STAR.genome.sorted.summary.txt'),
                  os.path.join(fastqc_dir, '{sample}_R{read}_001_fastqc.html')],
                  sample=samples['samples'], read=reads)
      output:
          os.path.join(qc_dir, 'multiqc_report.html') 
      conda:
          "envs/multiqc.yaml"
      shell:
          '''
          #!/bin/bash
          multiqc . --outdir qc --ignore .snakemake --force
          '''
  </code></pre>
  
  <p>My "rule all looks like this:"</p>
  
  <pre><code>rule all:
      input:
          expand([os.path.join(analyzed_dir, '{sample}.genes.results'),
                  os.path.join(qc_dir, 'multiqc_report.html')],
                  sample=samples['samples'])
  </code></pre>
  
  <p>I don't see anything wrong with it, in fact, it is very similar to my "rule all" I used before and which worked. Can anyone see an error?</p>
  
  <p>EDIT</p>
  
  <p>I limited the rule to only one input, I still have the same error:</p>
  
  <pre><code>rule multiqc:
      input:
          expand(os.path.join(rseqc_dir, '{sample}.geneBodyCoverage.txt'),
                  sample=samples['samples'])
      output:
          os.path.join(qc_dir, 'multiqc_report.html') 
      conda:
          "envs/multiqc.yaml"
      shell:
          '''
          #!/bin/bash
          multiqc . --outdir qc --ignore .snakemake --force
          '''
  </code></pre>
  
  <p>This is the ouput, you can see it read the samples correctly:</p>
  
  <pre><code>run_snakemake.sh 
  /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
    from collections import OrderedDict
  Building DAG of jobs...
  /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/site-packages/snakemake/workflow.py:12: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\t'.
    from collections import OrderedDict
  Building DAG of jobs...
  Using shell: /bin/bash
  Provided cluster nodes: 256
  Job counts:
      count   jobs
      1   all
      1   multiqc
      2
  
  [Mon Jan 14 17:46:40 2019]
  rule multiqc:
      input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R02_S2.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R03_S3.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R04_S4.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R05_S5.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R06_S6.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt, qc/rseqc/0067_P2018SEQE27R08_S8.geneBodyCoverage.txt
      output: qc/multiqc_report.html
      jobid: 2
  
  WorkflowError in line 119 of /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full/Snakefile:
  'Wildcards' object has no attribute 'sample'
  17:46 nlv24077@kiato /rst1/2017-0205_illuminaseq/scratch/swo-406/test_snakemake_full &gt; ll qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt
  -rw-rw-r-- 1 nlv24077 2017-0205_illuminaseq 303 Jan  7 16:00 qc/rseqc/0067_P2018SEQE27R07_S7.geneBodyCoverage.txt
  </code></pre>
  
  <p>EDIT 2:</p>
  
  <p>This is how I start Snakemake:</p>
  
  <pre><code>&gt; cat run_snakemake.sh
  source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
  snakemake --dag | dot -T svg &gt; dag.svg
  snakemake --cluster-config cluster.json --cluster "qsub -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {wildcards.sample}_{rule}" --jobs 256 --use-conda --rerun-incomplete
  </code></pre>
  
  <p>EDIT 3:</p>
  
  <p>Ok, that last remark (by @finswimmer) triggered me, I removed the "{wildcards.sample}_" from the above command and then it works. Or, works, I get a very different error, it is multiqc specific and solved as specified here: <a href="https://multiqc.info/docs/#locale-error-messages" rel="nofollow noreferrer">https://multiqc.info/docs/#locale-error-messages</a>. When adding the two lines as suggested this error disappears and everything works as expected, even with the long inputs list.</p>
  
  <pre><code>RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.
  </code></pre>
  
  <p>Anyway, I'm guessing Snakemake does not understand how to name this task when it uses an entire list of {sample}'s... Is this a bug? What should be used? I guess it is impossible to use {sample} for naming for this rule. It's a shame I have to kill the naming for all rules because using sample_rule is very nice for keeping track.</p>
   240: <p>Snakemake gets <code>wildcards</code> from the parsing the input/output file names.
  For example, if you had</p>
  
  <pre><code>rule sort:
      input: "{file}.bed"
      output: "{file}.sorted.bed"
      shell: "sort -k1,1 -k2,2n {input} &gt; {output}"
  </code></pre>
  
  <p>then the <code>wildcards</code> variable would be set as <code>wildcards = {"file": ...}</code>.</p>
  
  <p>In your <code>multiqc</code> rule, you have the <code>{sample}</code> variable within your <code>expand()</code>, but this variable doesn't exist outside of <code>expand</code>'s local context. You don't actually have any wildcards there, nor in your output file (<code>qc_dir</code> and <code>rseqc_dir</code> are variables you define elsewhere, but not "wildcards" as Snakemake interprets them).</p>
  
  <p>So you're getting the error <code>'Wildcards' object has no attribute 'sample'</code> because there are no wildcards for that rule.</p>
  
  <p>Your rule is currently set up to process every sample that you have listed in  <code>samples['sample']</code>. If you want your <code>multiqc</code> rule to work for a single sample, simply remove the <code>sample=samples['sample']</code> and double brace the <code>{sample}</code> wildcard.</p>
  
  <pre><code>rule multiqc:
      input:
          expand([os.path.join(analyzed_dir, '{{sample}}.stat', '{{sample}}.cnt'),
                  os.path.join(rseqc_dir, '{{sample}}.bam_stat.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.clipping_profile.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.deletion_profile.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.infer_experiment.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.geneBodyCoverage.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.inner_distance.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.insertion_profile.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.junction.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.junctionSaturation_plot.r'),
                  os.path.join(rseqc_dir, '{{sample}}.mismatch_profile.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.read_distribution.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.pos.DupRate.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.seq.DupRate.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.GC.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.NVC.xls'),
                  os.path.join(rseqc_dir, '{{sample}}.qual.r'),
                  os.path.join(rseqc_dir, '{{sample}}.RNA_fragment_size.txt'),
                  os.path.join(rseqc_dir, '{{sample}}.STAR.genome.sorted.summary.txt'),
                  os.path.join(fastqc_dir, '{{sample}}_R{read}_001_fastqc.html')],
                  read=reads)
      output:
          # this {sample} is in single {} and is recognized as a wildcard
          os.path.join(qc_dir, '{sample}', 'multiqc_report.html') 
      conda:
          "envs/multiqc.yaml"
      shell:
          '''
          #!/bin/bash
          # using {wildcards.sample} here since MultiQC likes directories
          multiqc . --outdir qc/{wildcards.sample} --ignore .snakemake --force
          '''
  </code></pre>
  
  <p>When wildcards are recognized, you'll see them in the job list.</p>
  
  <p>Running <code>snakemake 0067_P2018SEQE27R01_S1/multiqc_report.html</code>, for example, should give you something like:</p>
  
  <pre><code>rule multiqc:
      input: qc/rseqc/0067_P2018SEQE27R01_S1.geneBodyCoverage.txt, ...
      output: qc/0067_P2018SEQE27R01_S1/multiqc_report.html
      jobid: 2
      wildcards: sample=0067_P2018SEQE27R01_S1
  </code></pre>
  