text: user200: I have an obitools script (de Barba et al. 2016) that I would like to run faster. How would you run it in parallel to cut down on time?
  illuminapairedend -r rawdata_scandinavia_R2.fastq rawdata_scandinavia_R1.fastq | tee rawdata_scandinavia.fastq | obiannotate -S goodAli:'"Alignement" if score>40.00 else "Bad"' | obisplit -t goodAli -p rawdata_scandinavia.
  touch rawdata_scandinavia.Bad.fastq
  touch rawdata_scandinavia.Alignement.fastq
  touch rawdata_scandinavia.fastq
  ngsfilter -t rawdata_scandinavia.ngsfilter -u rawdata_scandinavia.unidentified.fastq rawdata_scandinavia.Alignement.fastq > rawdata_scandinavia.filtered.fastq
  obisplit -p MICROSAT.PCR_ -t experiment rawdata_scandinavia.filtered.fastq
  
  Can someone describe what tools and steps one needs to take to run this job in parallel, given that the functions are not designed to be run in parallel out of the box?
  De Barba, M., Miquel, C., Lobréaux, S., Quenette, P. Y., Swenson, J. E., & Taberlet, P. (2016). High-throughput microsatellite genotyping in ecology: improved accuracy, efficiency, standardization and success with low-quantity and degraded DNA. Molecular Ecology Resources, 1–16. https://doi.org/10.1111/1755-0998.12594
  
  user77: Your only option here is to first split the fastq files into small chunks, process each and then merge them back together. Alternatively, find something else that runs faster.
  
  user200: Here is a bash script and GNU parallel along with some other tools to split-apply-combine the data. It may or may not work out of the box, but it's not far off and it should give the general idea of how one way of how to approach this.
  #!/bin/bash
  
  # ====== USER PROVIDED PARAMETERS ========
  N=20 # number of cores to use
  ngsfilter="rawdata_scandinavia.ngsfilter"
  unidentified="unidentified_samples.fastq"
  
  R1="rawdata_scandinavia_R1.fastq"
  R2="rawdata_scandinavia_R2.fastq"
  # ==== END USER PROVIDED PARAMETERS =====
  
  # fastqutils available @ http://ngsutils.org/modules/fastqutils
  fastqutils split $R1 to_illumina_R1_ $N &
  fastqutils split $R2 to_illumina_R2_ $N &
  wait
  
  R1=$( ls | grep -P "to_illumina_R1_" )
  R2=$( ls | grep -P "to_illumina_R2_" )
  
  # Do alignment of two reads.
  for i in $( seq $N )
  do
      partR1=$( ls | grep -P "^to_illumina_R1_\\.$i\\.fastq$" )
      partR2=$( ls | grep -P "^to_illumina_R2_\\.$i\\.fastq$" )
      illuminapairedend -r $partR1 $partR2 | tee to_ngsfilter_$i.fastq | obiannotate -S goodAli:'"Align" if score>40.00 else "Bad"' | obisplit -t goodAli -p to_ngsfilter_$i. &
  done
  
  # Remove intermediate aligned files.
  echo $R1 | xargs rm -r
  echo $R2 | xargs rm -r
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  input=$( ls | grep -P "to_ngsfilter_[0-9]+\\.Align\\.fastq" )
  
  # For some reasons, files need to be "touched" in order to work. Perhaps
  # it's not closing them...
  touch $unidentified
  
  touchedbyangel=$( ls | grep -P "to_ngsfilter_"  )
  for i in $touchedbyangel
  do
      touch $i
  done
  
  parallel -j$N --result filtered_{} ngsfilter -t $ngsfilter -u $unidentified {} ::: $input
  
  # Remove intermediate results.
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  ls | grep -P "filtered_.*\\.fastq$" | xargs cat > filtered_data.fastq
  
  # remove intermediate results to conserve space
  rm to_ngsfilter*
  
  # Split reads by locus.
  filtered=$( ls | grep -P "^filtered_"  )
  parallel -j$N --result to_split_{} obisplit -p MS.PCR_ -t experiment {} ::: $filtered
  
  ls | grep -P "\\.err$|\\.seq$" | xargs rm -r
  ls | grep -P "^to_split_" | xargs rm -r
  
  # Wasn't able to parallelize this, so it's just pushing tasks into the background.
  obiuniq -m sample MS.PCR_UA_MxRout1_03.fastq > MS.PCR_UA_MxRout1_03.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_06.fastq > MS.PCR_UA_MxRout1_06.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_14.fastq > MS.PCR_UA_MxRout1_14.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_16.fastq > MS.PCR_UA_MxRout1_16.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_17.fastq > MS.PCR_UA_MxRout1_17.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_25.fastq > MS.PCR_UA_MxRout1_25.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_51.fastq > MS.PCR_UA_MxRout1_51.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_57.fastq > MS.PCR_UA_MxRout1_57.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_63.fastq > MS.PCR_UA_MxRout1_63.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_64.fastq > MS.PCR_UA_MxRout1_64.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_65.fastq > MS.PCR_UA_MxRout1_65.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_67.fastq > MS.PCR_UA_MxRout1_67.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_68.fastq > MS.PCR_UA_MxRout1_68.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_ZF.fastq > MS.PCR_UA_MxRout1_ZF.uniq.fasta &
  wait
  
  uniqfiles=$( ls | grep -P ".{2}+\\.uniq\\.fasta$" )
  
  for i in $uniqfiles
  do
      # this section removes the file suffix
      # https://stackoverflow.com/questions/125281/how-do-i-remove-the-file-suffix-and-path-portion-from-a-path-string-in-bash
      tab=${i%.fasta}
      tab=${tab##*/}
      obigrep -p 'count>1' $i | obiannotate -k merged_sample -k count | obiannotate --length | obisort -r -k seq_length | obitab --no-definition --output-seq > $tab.tab
  done
  
  ls | grep -P "\\.uniq\\.fasta" | xargs rm -r
  
  echo "Done. I think."
  
  
original_text: 200: <p>I have an <a href="http://datadryad.org/bitstream/handle/10255/dryad.122664/obitools_script.txt?sequence=1" rel="nofollow noreferrer">obitools script</a> (de Barba et al. 2016) that I would like to run faster. How would you run it in parallel to cut down on time?</p>
  
  <pre><code>illuminapairedend -r rawdata_scandinavia_R2.fastq rawdata_scandinavia_R1.fastq | tee rawdata_scandinavia.fastq | obiannotate -S goodAli:'"Alignement" if score&gt;40.00 else "Bad"' | obisplit -t goodAli -p rawdata_scandinavia.
  touch rawdata_scandinavia.Bad.fastq
  touch rawdata_scandinavia.Alignement.fastq
  touch rawdata_scandinavia.fastq
  ngsfilter -t rawdata_scandinavia.ngsfilter -u rawdata_scandinavia.unidentified.fastq rawdata_scandinavia.Alignement.fastq &gt; rawdata_scandinavia.filtered.fastq
  obisplit -p MICROSAT.PCR_ -t experiment rawdata_scandinavia.filtered.fastq
  </code></pre>
  
  <p>Can someone describe what tools and steps one needs to take to run this job in parallel, given that the functions are not designed to be run in parallel out of the box?</p>
  
  <p>De Barba, M., Miquel, C., Lobréaux, S., Quenette, P. Y., Swenson, J. E., &amp; Taberlet, P. (2016). High-throughput microsatellite genotyping in ecology: improved accuracy, efficiency, standardization and success with low-quantity and degraded DNA. Molecular Ecology Resources, 1–16. <a href="https://doi.org/10.1111/1755-0998.12594" rel="nofollow noreferrer">https://doi.org/10.1111/1755-0998.12594</a></p>
   77: <p>Your only option here is to first split the fastq files into small chunks, process each and then merge them back together. Alternatively, find something else that runs faster.</p>
   200: <p>Here is a bash script and GNU <code>parallel</code> along with some other tools to split-apply-combine the data. It may or may not work out of the box, but it's not far off and it should give the general idea of how one way of how to approach this.</p>
  
  
  
  <pre><code>#!/bin/bash
  
  # ====== USER PROVIDED PARAMETERS ========
  N=20 # number of cores to use
  ngsfilter="rawdata_scandinavia.ngsfilter"
  unidentified="unidentified_samples.fastq"
  
  R1="rawdata_scandinavia_R1.fastq"
  R2="rawdata_scandinavia_R2.fastq"
  # ==== END USER PROVIDED PARAMETERS =====
  
  # fastqutils available @ http://ngsutils.org/modules/fastqutils
  fastqutils split $R1 to_illumina_R1_ $N &amp;
  fastqutils split $R2 to_illumina_R2_ $N &amp;
  wait
  
  R1=$( ls | grep -P "to_illumina_R1_" )
  R2=$( ls | grep -P "to_illumina_R2_" )
  
  # Do alignment of two reads.
  for i in $( seq $N )
  do
      partR1=$( ls | grep -P "^to_illumina_R1_\\.$i\\.fastq$" )
      partR2=$( ls | grep -P "^to_illumina_R2_\\.$i\\.fastq$" )
      illuminapairedend -r $partR1 $partR2 | tee to_ngsfilter_$i.fastq | obiannotate -S goodAli:'"Align" if score&gt;40.00 else "Bad"' | obisplit -t goodAli -p to_ngsfilter_$i. &amp;
  done
  
  # Remove intermediate aligned files.
  echo $R1 | xargs rm -r
  echo $R2 | xargs rm -r
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  input=$( ls | grep -P "to_ngsfilter_[0-9]+\\.Align\\.fastq" )
  
  # For some reasons, files need to be "touched" in order to work. Perhaps
  # it's not closing them...
  touch $unidentified
  
  touchedbyangel=$( ls | grep -P "to_ngsfilter_"  )
  for i in $touchedbyangel
  do
      touch $i
  done
  
  parallel -j$N --result filtered_{} ngsfilter -t $ngsfilter -u $unidentified {} ::: $input
  
  # Remove intermediate results.
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  ls | grep -P "filtered_.*\\.fastq$" | xargs cat &gt; filtered_data.fastq
  
  # remove intermediate results to conserve space
  rm to_ngsfilter*
  
  # Split reads by locus.
  filtered=$( ls | grep -P "^filtered_"  )
  parallel -j$N --result to_split_{} obisplit -p MS.PCR_ -t experiment {} ::: $filtered
  
  ls | grep -P "\\.err$|\\.seq$" | xargs rm -r
  ls | grep -P "^to_split_" | xargs rm -r
  
  # Wasn't able to parallelize this, so it's just pushing tasks into the background.
  obiuniq -m sample MS.PCR_UA_MxRout1_03.fastq &gt; MS.PCR_UA_MxRout1_03.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_06.fastq &gt; MS.PCR_UA_MxRout1_06.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_14.fastq &gt; MS.PCR_UA_MxRout1_14.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_16.fastq &gt; MS.PCR_UA_MxRout1_16.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_17.fastq &gt; MS.PCR_UA_MxRout1_17.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_25.fastq &gt; MS.PCR_UA_MxRout1_25.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_51.fastq &gt; MS.PCR_UA_MxRout1_51.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_57.fastq &gt; MS.PCR_UA_MxRout1_57.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_63.fastq &gt; MS.PCR_UA_MxRout1_63.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_64.fastq &gt; MS.PCR_UA_MxRout1_64.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_65.fastq &gt; MS.PCR_UA_MxRout1_65.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_67.fastq &gt; MS.PCR_UA_MxRout1_67.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_68.fastq &gt; MS.PCR_UA_MxRout1_68.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_ZF.fastq &gt; MS.PCR_UA_MxRout1_ZF.uniq.fasta &amp;
  wait
  
  uniqfiles=$( ls | grep -P ".{2}+\\.uniq\\.fasta$" )
  
  for i in $uniqfiles
  do
      # this section removes the file suffix
      # https://stackoverflow.com/questions/125281/how-do-i-remove-the-file-suffix-and-path-portion-from-a-path-string-in-bash
      tab=${i%.fasta}
      tab=${tab##*/}
      obigrep -p 'count&gt;1' $i | obiannotate -k merged_sample -k count | obiannotate --length | obisort -r -k seq_length | obitab --no-definition --output-seq &gt; $tab.tab
  done
  
  ls | grep -P "\\.uniq\\.fasta" | xargs rm -r
  
  echo "Done. I think."
  </code></pre>
  