text: user779: I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:
  ./kallisto index -i --index --make-unique index.fa 
  
  This successfully creates an index file. Then, I tried this:
  for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
  "$file"; done
  
  While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:
  for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
  $file-aligned  "$file"; done
  
  I get an error that says "Error: incompatible indices. Found version 0, expected version 10. 
  I have now also tried the following:
  for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
  "${file}-aligned" "${file}"; 
  
  but I get an error saying the index file is not found.
  
  I am now running 
  or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 
  
  "$file"; done
  This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly. 
  If I close terminal, start a new session and run the same command, I get
  Error: file not found ""
  
  
  user73: For generating the index, you should be using one of -i or --index, but not both:
  ./kallisto index --make-unique -i index.fa.idx index.fa 
  
  You need to point kallisto to the generated index file (usually has a .idx extension), rather than the fasta file:
  for file in *.fasta;
    do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
  done
  
  
  user1769: Running the jobs in parallel
  Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular tximport route) you can do better using GNU parallel, this will enable you to run as many jobs as you have execution threads on your system simultaneously.  Note that Kalliso only runs multithreaded with -t if you're using bootstrapping.
  
  You'll need to make the index first:
  kallisto index --make-unique -I index.fa.idx index.fa
  
  Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:
  sample_1    sample_1_R1.fastq.gz
  sample_2    sample_2_R1.fastq.gz
  sample_3    sample_3_R1.fastq.gz
  
  Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file. 
  You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:
  ls -1 *.fastq.gz > editme
  
  Next we create a list of jobs to run on our system using a utility script Make_job_list_kallisto.sh which consumes our tab delimited input file list above.  I've used your -l and -s parameters below, note that with paired end data normally these are estimated for you.
  #!/bin/bash -eu
  
  [ $# -ne 2 ] && { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: < input tab delimited list > < output run list file >\n\n" ; exit 1; }
  set -o pipefail
  
  # Get command-line args
  INPUT_LIST=$1
  OUTPUT=$2
  
  # Set counter
  COUNT=1
  END=$(wc -l $INPUT_LIST | awk '{print $1}')
  
  echo " "
  echo " * Input file is: $INPUT_LIST"
  echo " * Number of runs: $END"
  echo " * Output job list for GNU parallel saved to: $OUTPUT"
  echo " "
  
  # Main bit of command-line for job
  CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"
  
  # Main Loop
  [ -e $OUTPUT ] && rm $OUTPUT
  while [ $COUNT -le $END ];
  do
      LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
      # Make file list
      echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
      echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" >> $OUTPUT
      ((COUNT++))
  done
  
  We use the script like so:
   Make_job_list_kallisto.sh input_file_list.txt job_list.txt
  
  Now we have a job list for GNU parallel we can run our jobs in parallel like so:
  parallel --progress --jobs 4 --joblog kallisto_joblog.txt < job_list.txt
  
  This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file kallisto_joblog.txt.  
  
  For paired end data I already have these scripts + similar instructions on GitHub which people might find useful. 
  
original_text: 779: <p>I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:</p>
  
  <pre><code>./kallisto index -i --index --make-unique index.fa 
  </code></pre>
  
  <p>This successfully creates an index file. Then, I tried this:</p>
  
  <pre><code>for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
  "$file"; done
  </code></pre>
  
  <p>While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:</p>
  
  <pre><code>for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
  $file-aligned  "$file"; done
  </code></pre>
  
  <p>I get an error that says "Error: incompatible indices. Found version 0, expected version 10. </p>
  
  <p>I have now also tried the following:</p>
  
  <pre><code>for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
  "${file}-aligned" "${file}"; 
  </code></pre>
  
  <p>but I get an error saying the index file is not found.</p>
  
  <hr>
  
  <p>I am now running </p>
  
  <pre><code>or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 
  </code></pre>
  
  <p>"$file"; done</p>
  
  <p>This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly. </p>
  
  <p>If I close terminal, start a new session and run the same command, I get</p>
  
  <pre><code>Error: file not found ""
  </code></pre>
   73: <p>For generating the index, you should be using one of <code>-i</code> or <code>--index</code>, but not both:</p>
  
  <pre><code>./kallisto index --make-unique -i index.fa.idx index.fa 
  </code></pre>
  
  <p>You need to point kallisto to the generated index file (usually has a <code>.idx</code> extension), rather than the fasta file:</p>
  
  <pre><code>for file in *.fasta;
    do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
  done
  </code></pre>
   1769: <p><strong>Running the jobs in parallel</strong></p>
  
  <p>Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular <a href="http://bioconductor.org/packages/release/bioc/vignettes/tximport/inst/doc/tximport.html" rel="nofollow noreferrer">tximport</a> route) you can do better using <a href="https://www.gnu.org/software/parallel/" rel="nofollow noreferrer">GNU parallel</a>, this will enable you to run as many jobs as you have execution threads on your system simultaneously.  Note that Kalliso only runs multithreaded with <code>-t</code> if you're using bootstrapping.</p>
  
  <ol>
  <li><p>You'll need to make the index first:</p>
  
  <pre><code>kallisto index --make-unique -I index.fa.idx index.fa
  </code></pre></li>
  <li><p>Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:</p>
  
  <pre><code>sample_1    sample_1_R1.fastq.gz
  sample_2    sample_2_R1.fastq.gz
  sample_3    sample_3_R1.fastq.gz
  </code></pre>
  
  <p>Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file. </p>
  
  <p>You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:</p>
  
  <pre><code>ls -1 *.fastq.gz &gt; editme
  </code></pre></li>
  <li><p>Next we create a list of jobs to run on our system using a utility script <code>Make_job_list_kallisto.sh</code> which consumes our tab delimited input file list above.  I've used your <code>-l</code> and <code>-s</code> parameters below, note that with paired end data normally these are estimated for you.</p>
  
  <pre><code>#!/bin/bash -eu
  
  [ $# -ne 2 ] &amp;&amp; { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: &lt; input tab delimited list &gt; &lt; output run list file &gt;\n\n" ; exit 1; }
  set -o pipefail
  
  # Get command-line args
  INPUT_LIST=$1
  OUTPUT=$2
  
  # Set counter
  COUNT=1
  END=$(wc -l $INPUT_LIST | awk '{print $1}')
  
  echo " "
  echo " * Input file is: $INPUT_LIST"
  echo " * Number of runs: $END"
  echo " * Output job list for GNU parallel saved to: $OUTPUT"
  echo " "
  
  # Main bit of command-line for job
  CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"
  
  # Main Loop
  [ -e $OUTPUT ] &amp;&amp; rm $OUTPUT
  while [ $COUNT -le $END ];
  do
      LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
      # Make file list
      echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
      echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" &gt;&gt; $OUTPUT
      ((COUNT++))
  done
  </code></pre>
  
  <p>We use the script like so:</p>
  
  <pre><code> Make_job_list_kallisto.sh input_file_list.txt job_list.txt
  </code></pre></li>
  <li><p>Now we have a job list for GNU parallel we can run our jobs in parallel like so:</p>
  
  <pre><code>parallel --progress --jobs 4 --joblog kallisto_joblog.txt &lt; job_list.txt
  </code></pre>
  
  <p>This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file <code>kallisto_joblog.txt</code>.  </p></li>
  </ol>
  
  <p>For paired end data I already have these scripts + similar instructions on <a href="https://github.com/MattBashton/grolar" rel="nofollow noreferrer">GitHub</a> which people might find useful. </p>
  