text: user285: I have numerous gzip-compressed paired-end Fastq files with ChIP-seq data that I would like to align with bowtie2. I confirmed that bowtie2 takes .gz files, although I do see this warning when I run for two samples:
  
  Warning: gzbuffer added in zlib v1.2.3.5. Unable to change buffer size
    from default of 8192.
  
  My files are named like so.
  10_S10_L002_R1_001.fastq.gz  12_S12_L002_R1_001.fastq.gz  2_S2_L001_R1_001.fastq.gz  4_S4_L001_R1_001.fastq-003.gz  6_S6_L001_R1_001.fastq-002.gz  8_S8_L002_R1_001.fastq.gz  
  10_S10_L002_R2_001.fastq.gz  12_S12_L002_R2_001.fastq.gz  2_S2_L001_R2_001.fastq.gz  4_S4_L001_R2_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz  8_S8_L002_R2_001.fastq.gz  
  11_S11_L002_R1_001.fastq.gz  1_S1_L001_R1_001.fastq.gz    3_S3_L001_R1_001.fastq.gz  5_S5_L001_R1_001.fastq-002.gz  7_S7_L002_R1_001.fastq.gz      9_S9_L002_R1_001.fastq.gz
  11_S11_L002_R2_001.fastq.gz  1_S1_L001_R2_001.fastq.gz    3_S3_L001_R2_001.fastq.gz  5_S5_L001_R2_001.fastq-003.gz  7_S7_L002_R2_001.fastq.gz      9_S9_L002_R2_001.fastq.gz
  
  The bash commands I tried are below. When I run this test command:
  for i in $(ls *.fastq*.gz | grep -v 6 | rev | cut -c 16- | rev | uniq); do 
      echo $i; 
  done
  
  ...I get this output.
  12_S12_L002_R1_001.fastq.gz  12_S12_L002_R2_001.fastq.gz 
  
  When I run this:
  for i in $(ls *.fastq*.gz | rev | cut -c 16- | rev | uniq); do 
      echo $i 
      bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 -1 ${i}R1_001.fastq.gz -2 ${i}R2_001.fastq.gz | samtools view -bS - >${i%}.bam
  done
  
  I do see one BAM for my paired end files. But as I see the second set of files which are named:
  6_S6_L001_R1_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz
  
  if I run the same script over them, I would get an error as the loop can't read the paired end file.
  So the tedious way is I can make a separate folder for these sets of data with fastq-002.gz and fastq-003.gz and can run the script by tweaking the cut parameter.
  But I would like to know what is the programmatic way of resolving the issue.
  
  user298: A very simple approach, but one which will work with the names you show, is to remove any occurrence of 00N (where N is any number) from right before the .gz. To do so, edit your script to:
  for i in $(ls *.fastq*.gz | sed 's/00[0-9]\.gz/.gz/' | rev | cut -c 16- | rev | uniq); do 
    bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 \
      -1 ${i}R1_001.fastq.gz \
      -2 ${i}R2_001.fastq.gz | 
          samtools view -bS - >${i%}.bam"
  done
  
  
original_text: 285: <p>I have numerous gzip-compressed paired-end Fastq files with ChIP-seq data that I would like to align with bowtie2. I confirmed that bowtie2 takes <code>.gz</code> files, although I do see this warning when I run for two samples:</p>
  
  <blockquote>
    <p>Warning: gzbuffer added in zlib v1.2.3.5. Unable to change buffer size
    from default of 8192.</p>
  </blockquote>
  
  <p>My files are named like so.</p>
  
  <pre><code>10_S10_L002_R1_001.fastq.gz  12_S12_L002_R1_001.fastq.gz  2_S2_L001_R1_001.fastq.gz  4_S4_L001_R1_001.fastq-003.gz  6_S6_L001_R1_001.fastq-002.gz  8_S8_L002_R1_001.fastq.gz  
  10_S10_L002_R2_001.fastq.gz  12_S12_L002_R2_001.fastq.gz  2_S2_L001_R2_001.fastq.gz  4_S4_L001_R2_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz  8_S8_L002_R2_001.fastq.gz  
  11_S11_L002_R1_001.fastq.gz  1_S1_L001_R1_001.fastq.gz    3_S3_L001_R1_001.fastq.gz  5_S5_L001_R1_001.fastq-002.gz  7_S7_L002_R1_001.fastq.gz      9_S9_L002_R1_001.fastq.gz
  11_S11_L002_R2_001.fastq.gz  1_S1_L001_R2_001.fastq.gz    3_S3_L001_R2_001.fastq.gz  5_S5_L001_R2_001.fastq-003.gz  7_S7_L002_R2_001.fastq.gz      9_S9_L002_R2_001.fastq.gz
  </code></pre>
  
  <p>The bash commands I tried are below. When I run this test command:</p>
  
  <pre><code>for i in $(ls *.fastq*.gz | grep -v 6 | rev | cut -c 16- | rev | uniq); do 
      echo $i; 
  done
  </code></pre>
  
  <p>...I get this output.</p>
  
  <pre><code>12_S12_L002_R1_001.fastq.gz  12_S12_L002_R2_001.fastq.gz 
  </code></pre>
  
  <p>When I run this:</p>
  
  <pre><code>for i in $(ls *.fastq*.gz | rev | cut -c 16- | rev | uniq); do 
      echo $i 
      bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 -1 ${i}R1_001.fastq.gz -2 ${i}R2_001.fastq.gz | samtools view -bS - &gt;${i%}.bam
  done
  </code></pre>
  
  <p>I do see one BAM for my paired end files. But as I see the second set of files which are named:</p>
  
  <pre><code>6_S6_L001_R1_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz
  </code></pre>
  
  <p>if I run the same script over them, I would get an error as the loop can't read the paired end file.</p>
  
  <p>So the tedious way is I can make a separate folder for these sets of data with <code>fastq-002.gz</code> and <code>fastq-003.gz</code> and can run the script by tweaking the <code>cut</code> parameter.</p>
  
  <p>But I would like to know what is the programmatic way of resolving the issue.</p>
   298: <p>A very simple approach, but one which will work with the names you show, is to remove any occurrence of <code>00N</code> (where <code>N</code> is any number) from right before the <code>.gz</code>. To do so, edit your script to:</p>
  
  <pre><code>for i in <span class="math-container">$(ls *.fastq*.gz | sed 's/00[0-9]\.gz/.gz/' | rev | cut -c 16- | rev | uniq); do 
    bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 \
      -1 $</span>{i}R1_001.fastq.gz \
      -2 <span class="math-container">${i}R2_001.fastq.gz | 
          samtools view -bS - &gt;$</span>{i%}.bam"
  done
  </code></pre>
  