title: How does linux redirect IO work internally
body: <p>When we use the redirect IO operator for a shell script does the operator keep all the data to be written in memory and write it all at once or does write it to file line by line.</p>
  
  <p>Here is what i am working on.</p>
  
  <p>I have about 200 small files ~1000 lines each in a specific format. I want to process (do a regex and change the format a little) each line in all the files and have the new transformed lines in a single combined file.</p>
  
  <p>I have a transformscript.sh that takes a single file and applies the transformation. I run it in the following manner</p>
  
  <pre><code>sh transformscript.sh somefile.txt &gt; newfile.txt
  </code></pre>
  
  <p>This works fine and fast for a single file.</p>
  
  <p>How do i extend to do it for all the files. will it be efficient to change transformscript.sh to take a directory as argument instead of filename and add a for loop to transform all the lines of all the files together. Or should I run the above trnsformscript.sh for each file and create a new file for each one and combine then separately.</p>
  
  <p>Thanks.</p>