text: user12419: I am currently working with fastq files which have hundreds of thousands of lines of text. However, not all of them are sequences I am interested in. My sequences are in one line and have a fixed length of 272 characters.
  I possess a list of smaller sequences, around 2 million text strings in the following formats (which are fragments from the various fastq files I have):
  Format 1
  ATCGATCG
  ATCGATCGATCG
  ACGATCGTGTTACG
  ATTTACGTACGTA
  AATCG
  ACGATACGATACG
  
  Or I also have them in
  Format 2
  ATCGATCGATACGAT|ATACGTGTTACGAT|ATCGATACG|ATACGATGA
  
  (I just wrote random sequences just to give an idea of how the pattern.txt files look like, I know Format 1 and 2 are not the same in this post, however, the files are in the same order - only the syntax is changed.).
  What I've been trying to do is use grep in a Linux environment and use the pattern.txt files to search for the bigger sequences in my fastq files where the pattern is present, and then output it into a .txt file to further work on.
  I've used this command so far (with the Format 2 of patterns):
  grep -E -f patterns.txt target.fastq > output.txt
  
  Now, this works.
  However, I have to limit the number of patterns in my key file and the number of fastq files I can search at once, otherwise I get "grep: memory exhausted" error.
  I also want to print out all the sequences up to and including my pattern, but not past it (I don't care what's after my pattern sequence).
  And for that I have successfully used the following:
  grep -oE "\S+singlepattern" target.fastq > output.txt
  
  However, I can only manage to get this to work if I manually write in the command the pattern sequence I want to look for (thus I can only use a couple). I can't seem to get it to work with the file list.
  So I thought of using a while loop.
  This is what I've written so far in the terminal (it's one continuous line, but I formatted it here for simplicity on the eyes):
  while read line; do
  grep $line target.fastq > output.txt; 
  done < patterns.txt
  
  However, this does absolutely nothing and just spits out a blank output.txt
  I have tried changing the syntax, with quotations on arguments, no quotations, I tried the various -E -f -o -F, egrep, etc but it's all the same. No error message, just blank file.
  I've also tried implementing the \S to the loop as follows:
  while read line; do
  grep -oE "\S+$line" target.fastq > output.txt; 
  done < patterns.txt
  
  Yet, the result is the same.
  Any suggestions on how I can fix this? Or even other methods to get the cut-down sequences (up to and including my pattern) from the fastq files into an output.txt file?
  Thanks in advance!
  
  user12419: Thank you all for your help. I managed to fix it in the end. The error was with how my pattern.txt file was formatted.
  I initially exported the .txt from R Studio by specifying the EOL as \r\n. However, I think this was the issue as then from my laptop to the computer cluster the files seemed to have a windows EOL.
  Thus on the linux server I did
  dos2unix patterns.txt
  
  And since then all of the suggested loops, both while read and the for chunk work beautifully and I get a nice \S+$line exported into the desired output.txt.
  Thanks again all for your advice and help!
  
original_text: 12419: <p>I am currently working with fastq files which have hundreds of thousands of lines of text. However, not all of them are sequences I am interested in. My sequences are in one line and have a fixed length of 272 characters.</p>
  <p>I possess a list of smaller sequences, around 2 million text strings in the following formats (which are fragments from the various fastq files I have):</p>
  <p>Format 1</p>
  <pre><code>ATCGATCG
  ATCGATCGATCG
  ACGATCGTGTTACG
  ATTTACGTACGTA
  AATCG
  ACGATACGATACG
  </code></pre>
  <p>Or I also have them in
  Format 2</p>
  <pre><code>ATCGATCGATACGAT|ATACGTGTTACGAT|ATCGATACG|ATACGATGA
  </code></pre>
  <p>(I just wrote random sequences just to give an idea of how the pattern.txt files look like, I know Format 1 and 2 are not the same in this post, however, the files are in the same order - only the syntax is changed.).</p>
  <p>What I've been trying to do is use grep in a Linux environment and use the pattern.txt files to search for the bigger sequences in my fastq files where the pattern is present, and then output it into a .txt file to further work on.</p>
  <p>I've used this command so far (with the Format 2 of patterns):</p>
  <pre><code>grep -E -f patterns.txt target.fastq &gt; output.txt
  </code></pre>
  <p><strong>Now, this works.</strong></p>
  <p><strong>However,</strong> I have to limit the number of patterns in my key file and the number of fastq files I can search at once, otherwise I get &quot;grep: memory exhausted&quot; error.</p>
  <p>I also want to print out all the sequences up to and including my pattern, but not past it (I don't care what's after my pattern sequence).</p>
  <p>And for that I have <strong>successfully</strong> used the following:</p>
  <pre><code>grep -oE &quot;\S+singlepattern&quot; target.fastq &gt; output.txt
  </code></pre>
  <p>However, I can only manage to get this to work if I manually write in the command the pattern sequence I want to look for (thus I can only use a couple). I can't seem to get it to work with the file list.</p>
  <p>So I thought of using a while loop.
  This is what I've written so far in the terminal (it's one continuous line, but I formatted it here for simplicity on the eyes):</p>
  <pre><code>while read line; do
  grep $line target.fastq &gt; output.txt; 
  done &lt; patterns.txt
  </code></pre>
  <p><strong>However,</strong> this does absolutely nothing and just spits out a blank output.txt</p>
  <p>I have tried changing the syntax, with quotations on arguments, no quotations, I tried the various -E -f -o -F, egrep, etc but it's all the same. No error message, just blank file.</p>
  <p>I've also tried implementing the \S to the loop as follows:</p>
  <pre><code>while read line; do
  grep -oE &quot;\S+$line&quot; target.fastq &gt; output.txt; 
  done &lt; patterns.txt
  </code></pre>
  <p>Yet, the result is the same.</p>
  <p>Any suggestions on how I can fix this? Or even other methods to get the cut-down sequences (up to and including my pattern) from the fastq files into an output.txt file?</p>
  <p>Thanks in advance!</p>
   12419: <p>Thank you all for your help. I managed to fix it in the end. The error was with how my pattern.txt file was formatted.</p>
  <p>I initially exported the .txt from R Studio by specifying the EOL as \r\n. However, I think this was the issue as then from my laptop to the computer cluster the files seemed to have a windows EOL.</p>
  <p>Thus on the linux server I did</p>
  <pre><code>dos2unix patterns.txt
  </code></pre>
  <p>And since then all of the suggested loops, both while read and the for chunk work beautifully and I get a nice <code>\S+$line</code> exported into the desired output.txt.</p>
  <p>Thanks again all for your advice and help!</p>
  