title: How to Calculate Temp Data Generated by a Hadoop MapReduce Job
body: <p>Is there a general formula or best practice estimation for the amount of HDFS space that should be left unused for temporary data generated by the MapReduce job? How does this change for different vendors such as Cloudera and MapR? I've have some jobs that took my cluster from 20GB to 60GB and back again for small test jobs when I'm tweaking a Pig script. I am asking this question for cluster planning.</p>
  
  <p>What are the factors that go into this determination? I assume that the number of reducers play a role in the amount of temp data generated, the peculiarities of the job,  (for example, if I have a Pig script with 20 statements, I've notice that the temp data isn't deleted until all 20 statements have been executed; if I want to minimize temp data I should split the 20 statements into 3 script files and run each consecutively), and other factors.</p>