title: Is intermediate key-value pair streaming optimized in hadoop
body: <p>The <strong>intermediate key-value pairs</strong> in a mapreduce job are written to <code>mapred.local.dir</code> before being shuffled to the tasktracker node which will run the reduce task. </p>
  
  <p>I know <strong>HFDS is optimized</strong> to write large Blocks of data therefore minimizing the seek-time of a hard disk as compared to a regular filesystem. </p>
  
  <p><strong>Now I was curious if hadoop is optimized for streaming intermediate kev-value pairs to the local filesystem as well?</strong></p>
  
  <p>I am asking this because my application has little input data, but a huge amount of intermediate data and medium size output data. <strong>Is hadoop in my case beneficial or should I consider a different framework?</strong> (Note that my software is very closely related to WordCount, but I emit all substrings instead of all words)</p>
  
  <p>Thanks a lot for any help!</p>
  
  <blockquote>
    <p>EDIT: I reprased the question somewhat since at first glance I give
    the impression that intermediate kv pairs were sent to HDFS, they are sent to the local filesystem of the tasktracker node!</p>
  </blockquote>