title: Awk inside of qsub
body: <p>I have a bash script in which I have a few qsubs. Each of them are waiting for a preivous qsub to be done before starting.</p>
  
  <p>My first qsub consist of sending files in a certain directory to a perl program and having the outfiles printed in a new directory. At the end, I echo the array with all my jobs names. This script works as intented.</p>
  
  <pre><code>mkdir -p /perl_files_dir
  for ID_FILES in `ls Infiles_dir/*.txt`;
  do
  JOB_ID=`echo "perl perl_scirpt.pl $ID_FILES" | qsub -j oe `
  JOB_ID_ARRAY="${JOB_ID_ARRAY}:$JOB_ID" 
  done
  echo $JOB_ID_ARRAY
  </code></pre>
  
  <p>My second qsub is meant to sort all my previous files made with my perl script in a new outfile and to start after all these jobs are done (about 100 jobs) with <code>depend=afterany</code>. Again, this part is working fine.</p>
  
  <pre><code>SORT_JOB=`echo "sort -m -n perl_files_dir/*.txt  &gt;&gt;sorted_file.txt" | qsub -j oe -W depend=afterany$JOB_ID_ARRAY`
  SORT_ARRAY="${SORT_ARRAY}:$SORT_JOB"
  </code></pre>
  
  <p>My issue is that in my sorted file, I have a few columns I wish to remove (2 to 6), so I came up with this last line using <code>awk</code> piped to <code>sed</code> with another <code>depend=afterany</code></p>
  
  <pre><code>SED=`echo "awk '{\$2="";\$3="";\$4="";\$5="";\$6=""; print \$0}' sorted_file.txt \
  | sed 's/     //g' &gt;final_file.txt" | qsub -j oe -W depend=afterany$SORT_ARRAY`
  </code></pre>
  
  <p>This last step creates <code>final_file.txt</code>, but leaves it empty. I added <code>SED=</code> before my echo because it would otherwise give me <code>Command not found</code>.  </p>
  
  <p>I tried without the pipe so it would just print everything. Unfortunately it prints nothing. 
  I assume it is not opening my sorted file and this is why my final file is empty after my sed. If it's the case, then why won't awk read it?  </p>
  
  <p>In my script, I am using variables to define my directories and files (with the correct path). I know my issue is not about find my files or directories since they are perfectly defined at the beginning and used throughout the script. I tried to write the whole path instead of a variable and I get the same results.</p>