text: user3839: I need to merge multiple channels each emits multiple file. So far I did something as below:
  process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val("${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr
  """
  PRE_PASHING_QC2.sh "${hrc_sites_file}" "${prefix}" "${chrom}"
  """
  }
  channel.fromPath('/some/path/genetic_map_chr*_combined_b37.txt').into {shape_map;i
  mp_map}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch*.hap.gz').into {shape_ref;
  imp_ref}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.legend.gz').into {s
  hape_legend;imp_legend}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.samples').into {sha
  pe_sample;imp_sample}
  qc1_chr.filter{name, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.into {shape_in;imp_in}
  
  shape_para=shape_in.merge(shape_map).merge(shape_ref).merge(shape_legend).merge(shape_sample)
      
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  each file(penta) from shape_para
  output:
  path("*step10*") into phased
  script:
  def (in, map, ref, legend, sample) = penta
  """
  shapeit2 -T 32 --input-bed "${in}" --input-map "${map}" --input-ref "${ref}" "${legend}" "${sample}" --duohmm -O "${in}.phased" --seed 54321
  """
  }
  
  The "prephasing" process works fine and produce expected output. But the "phasing" process does not produce any output nor any error. Any help?
  I have tried according to @Steve as below:
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -> tuple(group_key.replaceFirst(/^genetic_map_/,""),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -> tuple(group_key.replaceFirst(/^*HRC\.r1-1\.ega\.grch37\./,""),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  """
  shapeit2 --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${sample}" --duohmm --output-max "${chrom}.phased" --seed 54321
  """
  }
  
  But getting no output so far, (attached image)
  I have tried to check individual channel separately using channel.view(), identified the problems, modified the code and checked again. Using the code below I can see individual channel emits 22 lines [chr1-22,[file_list]]. After merging I can see [chr1-22,[joined_file_list].
  channel
      .fromFilePairs( "output3/*.{bed,bim,fam}", size:3 )
      .set { plink_data }
  plink_data.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.map{group_key,file_list -> tuple (group_key.
  replaceFirst(/^*\.step10/,""), file_list)}.set {in_pl}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {
  group_key, file_list -> tuple(group_key.replaceFirst(/^*genetic_map_/,""),file_list.first())}.set {genetic_map}
  //genetic_map.view()
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{ha
  p.gz,legend.gz,samples}',size:3).map {group_key,file_list -> tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,""),file_list)}.set {re
  f_panel}
  //ref_panel.view()
  
  
  
  in_pl.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  process phasing {
  //publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val("${chrom}"),path("${chrom}.{phased.haps,phased.sample,log}") in
  to phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  """
  //shapeit="/cluster/projects/p697/projects/moba_qc_imputation/software/shap
  eit"
  shapeit --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${samp
  le}" --duohmm --output-max "${chrom}.phased" --seed 54321
  """
  }
  
  
  But now I get error:
  No such variable: chrom
  
  But I decalred "chrom" in the input
  The input bed|bim|fam files from channel from FilePairs works but from a output channel of previous process does not. If I run phasing solo and all the inputs from channel from FilePairs it works. But if I run prephasing and phasing together, the prephasing process works but for phasing process not output and no error; Not sure what I am doing wrong.
  hrc_sites_file = file(params.hrc_sites_file)
  
  Channel.fromFilePairs( "${params.input_dir}/*.{bed,bim,fam}", size:3 ).set { plink_data }
  
  Channel.of(1..22).set { chroms }
  params.out='output/'
  
  process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val("${chrom}"),path("chr${chrom}*.{bed,bim,fam}") into qc1_chr
  tuple val("${chrom}"),path("chr${chrom}*log") into qc1_log
  tuple val("${chrom}"),path("*-chr${chrom}.step4-HRC.txt") into qc1_hrc
  path("chr${chrom}.step1b.snplist.txt") into qc1_snps
  path("chr${chrom}.step1a.dupvar") into qc1_dups
  """
  PRE_PASHING_QC2.sh "${hrc_sites_file}" "${prefix}" "${chrom}"
  """
  }
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -> tuple(group_key.replaceFirst(/^*gen
  etic_map_/,""),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list ->
   tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,""),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -> bfiles.collect {it.getBaseName()}.every {it.endsWith("step10")}}.map{group_key,file_list -> tuple (group_key.replaceFirst(/^*\.step10/,""), file_list)}.join(ref_panel).joi
  n(genetic_map).into {shape_in;imp_in}
  imp_in.view()
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from shape_in
  output:
  tuple val("${chrom}"),path("${chrom}.*") into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  """
  shapeit --thread "${task.cpus}" --input-bed "${bed}" "${bim}" "${fam}" --input-map "${input_map}" --input-ref "${haplotype}" "${legend}" "${sample}" --duohmm --output-max "${chrom}.step10" --seed 54321
  """
  }
  
  
  
  user10775: The merge operator is deprecated and will be removed from future Nextflow releases. You'll need instead the join operator to join by chromosome. You can use the fromFilePairs operator to quickly get the group key, formatting it as required:
  channel
      .fromFilePairs("/some/path/HRC.r1-1.EGA.GRCh37.chr*.{hap.gz,legend.gz,samples}", size: 3)
      .map { group_key, file_list ->
          tuple( group_key.replaceFirst(/^HRC\.r1-1\.EGA\.GRCh37\./, ""), file_list )
      }
      .set { ref_panel_files }
  
  
  channel
      .fromFilePairs("/some/path/genetic_map_chr*_combined_b37.txt", size: 1)
      .map { group_key, file_list ->
          tuple( group_key.replaceFirst(/^genetic_map_/, ""), file_list.first() )
      }
      .set { genetic_map_files }
  
  
  qc1_chr
      .map { chrom, bfiles ->
          tuple( chrom, bfiles.findAll { it.baseName.endsWith("step10") } )
      }
      .join( ref_panel_files )
      .join( genetic_map_files )
      .set { phasing_inputs }
  
  
  process phasing {
  
      input:
      tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from phasing_inputs
  
      script:
      def (bed, bim, fam) = input_bed
      def (haplotypes, legend, sample) = input_ref
  
      """
      shapeit2 \\
          --thread "${task.cpus}" \\
          --input-bed "${bed}" "${bim}" "${fam}" \\
          --input-ref "${haplotypes}" "${legend}" "${sample}" \\
          --input-map "${input_map}" \\
          --duohmm \\
          --output-max "${chrom}.phased" \\
          --seed 54321
      """
  }
  
  
original_text: 3839: <p>I need to merge multiple channels each emits multiple file. So far I did something as below:</p>
  <pre><code>process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*.{bed,bim,fam}&quot;) into qc1_chr
  &quot;&quot;&quot;
  PRE_PASHING_QC2.sh &quot;<span class="math-container">${hrc_sites_file}" "$</span>{prefix}&quot; &quot;${chrom}&quot;
  &quot;&quot;&quot;
  }
  channel.fromPath('/some/path/genetic_map_chr*_combined_b37.txt').into {shape_map;i
  mp_map}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch*.hap.gz').into {shape_ref;
  imp_ref}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.legend.gz').into {s
  hape_legend;imp_legend}
  channel.fromPath('/some/path/*hrc.r1-1.ega.grch37.chr*.samples').into {sha
  pe_sample;imp_sample}
  qc1_chr.filter{name, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.into {shape_in;imp_in}
  
  shape_para=shape_in.merge(shape_map).merge(shape_ref).merge(shape_legend).merge(shape_sample)
      
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  each file(penta) from shape_para
  output:
  path(&quot;*step10*&quot;) into phased
  script:
  def (in, map, ref, legend, sample) = penta
  &quot;&quot;&quot;
  shapeit2 -T 32 --input-bed &quot;<span class="math-container">${in}" --input-map "$</span>{map}&quot; --input-ref &quot;<span class="math-container">${ref}" "$</span>{legend}&quot; &quot;<span class="math-container">${sample}" --duohmm -O "$</span>{in}.phased&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p>The &quot;prephasing&quot; process works fine and produce expected output. But the &quot;phasing&quot; process does not produce any output nor any error. Any help?</p>
  <p>I have tried according to <code>@Steve</code> as below:</p>
  <pre><code>channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -&gt; tuple(group_key.replaceFirst(/^genetic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -&gt; tuple(group_key.replaceFirst(/^*HRC\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.{phased.haps,phased.sample,log}&quot;) into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  &quot;&quot;&quot;
  shapeit2 --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{sample}&quot; --duohmm --output-max &quot;${chrom}.phased&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p>But getting no output so far, (attached image<a href="https://i.stack.imgur.com/B1PtL.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/B1PtL.png" alt="phasing process" /></a>)</p>
  <p>I have tried to check individual channel separately using channel.view(), identified the problems, modified the code and checked again. Using the code below I can see individual channel emits 22 lines [chr1-22,[file_list]]. After merging I can see [chr1-22,[joined_file_list].</p>
  <pre><code>channel
      .fromFilePairs( &quot;output3/*.{bed,bim,fam}&quot;, size:3 )
      .set { plink_data }
  plink_data.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.map{group_key,file_list -&gt; tuple (group_key.
  replaceFirst(/^*\.step10/,&quot;&quot;), file_list)}.set {in_pl}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {
  group_key, file_list -&gt; tuple(group_key.replaceFirst(/^*genetic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}
  //genetic_map.view()
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{ha
  p.gz,legend.gz,samples}',size:3).map {group_key,file_list -&gt; tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {re
  f_panel}
  //ref_panel.view()
  
  
  
  in_pl.join(ref_panel).join(genetic_map).into {shape_in;imp_in}
  process phasing {
  //publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(intput_ref), path(input_map) from shape_in
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.{phased.haps,phased.sample,log}&quot;) in
  to phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  &quot;&quot;&quot;
  //shapeit=&quot;/cluster/projects/p697/projects/moba_qc_imputation/software/shap
  eit&quot;
  shapeit --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{samp
  le}&quot; --duohmm --output-max &quot;${chrom}.phased&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p><a href="https://i.stack.imgur.com/CR2sW.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CR2sW.jpg" alt="joined channel example view" /></a>
  But now I get error:</p>
  <pre><code>No such variable: chrom
  </code></pre>
  <p>But I decalred &quot;chrom&quot; in the input</p>
  <p>The input bed|bim|fam files from channel from FilePairs works but from a output channel of previous process does not. If I run <code>phasing</code> solo and all the inputs from channel from FilePairs it works. But if I run <code>prephasing</code> and <code>phasing</code> together, the <code>prephasing</code> process works but for <code>phasing</code> process not output and no error; Not sure what I am doing wrong.</p>
  <pre><code>hrc_sites_file = file(params.hrc_sites_file)
  
  Channel.fromFilePairs( &quot;${params.input_dir}/*.{bed,bim,fam}&quot;, size:3 ).set { plink_data }
  
  Channel.of(1..22).set { chroms }
  params.out='output/'
  
  process prephasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(prefix), path(bfiles) from plink_data
  each chrom from chroms
  path hrc_sites_file
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*.{bed,bim,fam}&quot;) into qc1_chr
  tuple val(&quot;<span class="math-container">${chrom}"),path("chr$</span>{chrom}*log&quot;) into qc1_log
  tuple val(&quot;<span class="math-container">${chrom}"),path("*-chr$</span>{chrom}.step4-HRC.txt&quot;) into qc1_hrc
  path(&quot;chr<span class="math-container">${chrom}.step1b.snplist.txt") into qc1_snps
  path("chr$</span>{chrom}.step1a.dupvar&quot;) into qc1_dups
  &quot;&quot;&quot;
  PRE_PASHING_QC2.sh &quot;<span class="math-container">${hrc_sites_file}" "$</span>{prefix}&quot; &quot;${chrom}&quot;
  &quot;&quot;&quot;
  }
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/1000Genomes/genetic_map_chr*_combined_b37.txt',size:1).map {group_key, file_list -&gt; tuple(group_key.replaceFirst(/^*gen
  etic_map_/,&quot;&quot;),file_list.first())}.set {genetic_map}
  
  channel.fromFilePairs('/cluster/projects/p697/projects/moba_qc_imputation/resources/HRC/haplegendsample/*hrc.r1-1.ega.grch37.chr*.haplotypes.{hap.gz,legend.gz,samples}',size:3).map{group_key,file_list -&gt;
   tuple(group_key.replaceFirst(/^.*.hrc\.r1-1\.ega\.grch37\./,&quot;&quot;),file_list)}.set {ref_panel}
  
  qc1_chr.filter{chrom, bfiles -&gt; bfiles.collect {it.getBaseName()}.every {it.endsWith(&quot;step10&quot;)}}.map{group_key,file_list -&gt; tuple (group_key.replaceFirst(/^*\.step10/,&quot;&quot;), file_list)}.join(ref_panel).joi
  n(genetic_map).into {shape_in;imp_in}
  imp_in.view()
  process phasing {
  publishDir params.out, mode:'copy'
  input:
  tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from shape_in
  output:
  tuple val(&quot;<span class="math-container">${chrom}"),path("$</span>{chrom}.*&quot;) into phased
  script:
  def (bed, bim, fam) = input_bed
  def (haplotype, legend, sample) = input_ref
  &quot;&quot;&quot;
  shapeit --thread &quot;<span class="math-container">${task.cpus}" --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; --input-map &quot;<span class="math-container">${input_map}" --input-ref "$</span>{haplotype}&quot; &quot;<span class="math-container">${legend}" "$</span>{sample}&quot; --duohmm --output-max &quot;${chrom}.step10&quot; --seed 54321
  &quot;&quot;&quot;
  }
  </code></pre>
  <p><a href="https://i.stack.imgur.com/bjxQJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bjxQJ.png" alt="prephasing works but phasing does not" /></a></p>
   10775: <p>The <a href="https://www.nextflow.io/docs/latest/operator.html#merge" rel="nofollow noreferrer">merge operator</a> is deprecated and will be removed from future Nextflow releases. You'll need instead the <a href="https://www.nextflow.io/docs/latest/operator.html#join" rel="nofollow noreferrer">join operator</a> to join by chromosome. You can use the <a href="https://www.nextflow.io/docs/latest/channel.html#fromfilepairs" rel="nofollow noreferrer">fromFilePairs operator</a> to quickly get the group key, formatting it as required:</p>
  <pre><code>channel
      .fromFilePairs(&quot;/some/path/HRC.r1-1.EGA.GRCh37.chr*.{hap.gz,legend.gz,samples}&quot;, size: 3)
      .map { group_key, file_list -&gt;
          tuple( group_key.replaceFirst(/^HRC\.r1-1\.EGA\.GRCh37\./, &quot;&quot;), file_list )
      }
      .set { ref_panel_files }
  
  
  channel
      .fromFilePairs(&quot;/some/path/genetic_map_chr*_combined_b37.txt&quot;, size: 1)
      .map { group_key, file_list -&gt;
          tuple( group_key.replaceFirst(/^genetic_map_/, &quot;&quot;), file_list.first() )
      }
      .set { genetic_map_files }
  
  
  qc1_chr
      .map { chrom, bfiles -&gt;
          tuple( chrom, bfiles.findAll { it.baseName.endsWith(&quot;step10&quot;) } )
      }
      .join( ref_panel_files )
      .join( genetic_map_files )
      .set { phasing_inputs }
  
  
  process phasing {
  
      input:
      tuple val(chrom), path(input_bed), path(input_ref), path(input_map) from phasing_inputs
  
      script:
      def (bed, bim, fam) = input_bed
      def (haplotypes, legend, sample) = input_ref
  
      &quot;&quot;&quot;
      shapeit2 \\
          --thread &quot;<span class="math-container">${task.cpus}" \\
          --input-bed "$</span>{bed}&quot; &quot;<span class="math-container">${bim}" "$</span>{fam}&quot; \\
          --input-ref &quot;<span class="math-container">${haplotypes}" "$</span>{legend}&quot; &quot;<span class="math-container">${sample}" \\
          --input-map "$</span>{input_map}&quot; \\
          --duohmm \\
          --output-max &quot;${chrom}.phased&quot; \\
          --seed 54321
      &quot;&quot;&quot;
  }
  </code></pre>
  