text: user29: I’m trying to run a Nextflow workflow in a custom Docker container. Without Docker, the workflow succeeds. But running it inside the container leads to an error because a dependent file cannot be found.
  Here’s a minimal script to illustrate the issue (similar to the basic Nextflow script example):
  params.reference = "/data/reference/GRCh38.fa"
  
  reference = file(params.reference)
  fastq_channel = Channel.fromFilePairs('/data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "$reference" 1.fastq.gz 2.fastq.gz >${sample}.sam'
      stat -t "${reference}.bwt"
      """
  }
  
  Running this inside a container will cause the path of the query, /some/fastq/, to be mounted inside the container. However, the path to the database, /data/reference, will not be mounted. Consequently, I get the following execution error:
  
  stat: cannot stat 'GRCh38-chr21.fa.bwt': No such file or directory
  
  Without the container, the command succeeds.
  
  I understand why the error occurs: Nextflow doesn’t track the dependency on $reference inside the command, and thus doesn’t mount its path in the container.
  My question is, how do I fix this? The documentation is silent on this regard, and none of the workflows I can find online address the problem. For example, the GATK4 IARC workflows would presumably have the same issue, and aren’t configured for use with Docker.
  Just adding the reference as a process input is insufficient, since BWA requires various other index files. Am I supposed to add these all manually as inputs?
  
  user662: Process input files need to be declared properly to allow nextflow to create the required docker mounts accordingly. Therefore the reference file should be declared in the process input block, eh: 
  input:
   set sample, file(fastq) from fastq_channel
   file reference 
  
  
  user29: The following workaround seems to work but ideally I don’t want to have to change the script portion of my process so that it works with Docker.
  params.ref = 'data/reference/GRCh38.fa'
  
  ref = file(params.ref)
  ref_index_dir = ref.parent
  fastq_channel = Channel.fromFilePairs('data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
      file ref
      file ref_index_dir
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "reference/$ref" 1.fastq.gz 2.fastq.gz >${sample}.sam'
      stat -t reference/${ref}.bwt && touch ${sample}.sam
      """
  }
  
  Here, reference inside the script section refers to the parent directory name of the reference, and needs to be adjusted accordingly. Having to refer to reference/$ref is annoying but at least it works — both inside a Docker container and outside of it.
  
original_text: 29: <p>I’m trying to run a Nextflow workflow in a custom Docker container. Without Docker, the workflow succeeds. But running it inside the container leads to an error because a dependent file cannot be found.</p>
  
  <p>Here’s a minimal script to illustrate the issue (similar to the <a href="https://www.nextflow.io/docs/latest/basic.html" rel="nofollow noreferrer">basic Nextflow script example</a>):</p>
  
  <pre><code>params.reference = "/data/reference/GRCh38.fa"
  
  reference = file(params.reference)
  fastq_channel = Channel.fromFilePairs('/data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "<span class="math-container">$reference" 1.fastq.gz 2.fastq.gz &gt;$</span>{sample}.sam'
      stat -t "${reference}.bwt"
      """
  }
  </code></pre>
  
  <p>Running this inside a container will cause the path of the query, <code>/some/fastq/</code>, to be mounted inside the container. However, the path to the database, <code>/data/reference</code>, will <em>not</em> be mounted. Consequently, I get the following execution error:</p>
  
  <blockquote>
    <p>stat: cannot stat 'GRCh38-chr21.fa.bwt': No such file or directory</p>
  </blockquote>
  
  <p>Without the container, the command succeeds.</p>
  
  <hr>
  
  <p>I understand <em>why</em> the error occurs: Nextflow doesn’t track the dependency on <code>$reference</code> inside the command, and thus doesn’t mount its path in the container.</p>
  
  <p>My question is, <strong>how do I fix this?</strong> The documentation is silent on this regard, and none of the workflows I can find online address the problem. For example, the <a href="https://github.com/IARCbioinfo" rel="nofollow noreferrer">GATK4 IARC workflows</a> would presumably have the same issue, and aren’t configured for use with Docker.</p>
  
  <p>Just adding the <code>reference</code> as a process input is insufficient, since BWA requires various other index files. Am I supposed to add these all manually as inputs?</p>
   662: <p>Process input files need to be declared properly to allow nextflow to create the required docker mounts accordingly. Therefore the <code>reference</code> file should be declared in the process input block, eh: </p>
  
  <pre><code>input:
   set sample, file(fastq) from fastq_channel
   file reference 
  </code></pre>
   29: <p>The following workaround seems to work but ideally I don’t want to have to change the <code>script</code> portion of my process so that it works with Docker.</p>
  
  <pre><code>params.ref = 'data/reference/GRCh38.fa'
  
  ref = file(params.ref)
  ref_index_dir = ref.parent
  fastq_channel = Channel.fromFilePairs('data/fastq/*_R{1,2}.fastq.gz')
  
  process map_reads {
      input:
      set sample, '?.fastq.gz' from fastq_channel
      file ref
      file ref_index_dir
  
      output:
      file("${sample}.sam") into sam_channel
  
      script:
      """
      echo 'bwa mem -Y "reference/<span class="math-container">$ref" 1.fastq.gz 2.fastq.gz &gt;$</span>{sample}.sam'
      stat -t reference/<span class="math-container">${ref}.bwt &amp;&amp; touch $</span>{sample}.sam
      """
  }
  </code></pre>
  
  <p>Here, <code>reference</code> inside the script section refers to the parent directory name of the reference, and needs to be adjusted accordingly. Having to refer to <code>reference/$ref</code> is annoying but at least it works — both inside a Docker container and outside of it.</p>
  