text: user3686: Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.
  So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:
  bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* > ERR040140.sam
  
  So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:
  
  So changes are occurring 5th, 7th and 9th column. which are:
  tID:ERR040140
  tLB:2517707
  tSM:ERR040140
  ERR040140_*
  ERR040140.sam
  
  Can anyone help me with loop or something?
  Thank you!
  
  user1310: if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like snakemake. I've also wrote a little tutorial on biostars about this topic, which might be useful for you.
  fin swimmer
  
  user3828: Something like this might do it:
  #!/bin/bash
  
  # make an array to hold your list of samples/base filenames
  samples=( S1 S2 S3 S4 S5 )
  
  # make a loop to go through the sample IDs
  for sample in "${samples[@]}"
  do
    :
    # do something with each sample/file, e.g. make a directory for it
    mkdir "${sample}"_home
  
    # do something else with the sample, e.g. unzip the fastq.gz file
    gunzip "${sample}".fq.gz > "${sample}"_home/"${sample}".fq
  done
  
  One disadvantage with this is that it will process the files one after the other.  If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker.  If you have a SLURM cluster, you can use job array to do something similar.
  If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:
  samples=( `cat samples_list.txt` )
  
  This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.
  
  user1310: Another solution for repeating one command with different parameters is to use parallel.
  Create a tab-delimited file samples.txt, which contains the sample names and the information for your read groups, e.g.:
  Sample1 Lib1
  Sample2 Lib2
  Sample3 Lib3
  
  Then you can use parallel like this:
  parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* > {1}.sam' :::: samples.txt
  
  For each line in samples.txt a new process is started, replacing {1} and {2} with the values in column 1 and 2 of this line.
  fin swimmer
  
original_text: 3686: <p>Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.</p>
  <p>So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:</p>
  <pre><code>bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* &gt; ERR040140.sam
  </code></pre>
  <p>So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:</p>
  <p><a href="https://i.stack.imgur.com/uj6vl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uj6vl.png" alt="command_column_wise" /></a></p>
  <p>So changes are occurring 5th, 7th and 9th column. which are:</p>
  <pre><code>tID:ERR040140
  tLB:2517707
  tSM:ERR040140
  ERR040140_*
  ERR040140.sam
  </code></pre>
  <p>Can anyone help me with loop or something?</p>
  <p>Thank you!</p>
   1310: <p>if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like <a href="https://snakemake.readthedocs.io/en/stable/tutorial/basics.html" rel="noreferrer">snakemake</a>. I've also wrote a little tutorial on <a href="https://www.biostars.org/p/335903/" rel="noreferrer">biostars</a> about this topic, which might be useful for you.</p>
  
  <p>fin swimmer</p>
   3828: <p>Something like this might do it:</p>
  
  <pre><code>#!/bin/bash
  
  # make an array to hold your list of samples/base filenames
  samples=( S1 S2 S3 S4 S5 )
  
  # make a loop to go through the sample IDs
  for sample in "<span class="math-container">${samples[@]}"
  do
    :
    # do something with each sample/file, e.g. make a directory for it
    mkdir "$</span>{sample}"_home
  
    # do something else with the sample, e.g. unzip the fastq.gz file
    gunzip "<span class="math-container">${sample}".fq.gz &gt; "$</span>{sample}"_home/"${sample}".fq
  done
  </code></pre>
  
  <p>One disadvantage with this is that it will process the files one after the other.  If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker.  If you have a SLURM cluster, you can use job array to do something similar.</p>
  
  <p>If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:</p>
  
  <pre><code>samples=( `cat samples_list.txt` )
  </code></pre>
  
  <p>This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.</p>
   1310: <p>Another solution for repeating one command with different parameters is to use <code>parallel</code>.</p>
  
  <p>Create a tab-delimited file <code>samples.txt</code>, which contains the sample names and the information for your read groups, e.g.:</p>
  
  <pre><code>Sample1 Lib1
  Sample2 Lib2
  Sample3 Lib3
  </code></pre>
  
  <p>Then you can use <code>parallel</code> like this:</p>
  
  <pre><code>parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* &gt; {1}.sam' :::: samples.txt
  </code></pre>
  
  <p>For each line in <code>samples.txt</code> a new process is started, replacing <code>{1}</code> and <code>{2}</code> with the values in column 1 and 2 of this line.</p>
  
  <p>fin swimmer</p>
  