title: How to avoid OutOfMemoryException when running Hadoop?
body: <p>I'm running a Hadoop job over 1,5 TB of data with doing much pattern matching. I have several machines with 16GB RAM each, and I always get <code>OutOfMemoryException</code> on this job with this data (I'm using Hive).</p>
  
  <p>I would like to know how to optimally set option <code>HADOOP_HEAPSIZE</code> in file <code>hadoop-env.sh</code> so, my job would not fail.  Is it even possible, to set this option so my jobs won't fail?</p>
  
  <p>When I set <code>HADOOP_HEAPSIZE</code> to 1,5 GB and removed half of pattern matching from query, job run successfully. So what is this option for, if it doesn't help avoiding job failures?</p>
  
  <p>I ment to do more experimenting with optimal setup, but since those jobs take >10hr to run, I'm asking for your advice.</p>