Take the 2-minute tour ×

I'm wondering where the memory is used in the following job:

  • Hadoop Mapper/Reducer Heap Size: -Xmx2G
  • Streaming API:

    • Mapper: /bin/cat
    • Reducer: wc
  • Input File is a 350MByte file containg a single line full of a's.

This is a simplified version of the real problem we've encountered.

Reading the file from the HDFS and constructing a Text-Object should not amount to more than 700MB Heap - assuming that Text does also use 16-Bit per Character - I'm not sure about that but I could imagine that Text only uses 8-Bit.

So there is these (worst-case) 700MB Line. The Line should fit at least 2x in the Heap but I'm getting always out of memory errors.

Is this a possible Bug in Hadoop (e.g. unaccary copies) or do I just don't understand some required memory intensive steps?

Would be really thankful for any further hints.

share|improve this question
add comment

1 Answer

The memory given to each child JVM running a task can be changed by setting the mapred.child.java.opts property. The default setting is -Xmx200m, which gives each task 200 MB of memory.

When you are saying -


I'm assuming you file has a single line of all a's with a single endline delimiter.

If that is taken up as a value in the map(key, value) function, I think, you might have memory issues, since, you task have can use only 200MB and you have a record in memory which is of 350MB.

share|improve this answer
Thanks. But my problem is that mapred.child.java.opts is -Xmx2048M - so a task can use up to 2GB memory –  mt_ Aug 1 '13 at 14:28
@mt_: How much does you overall system have? And, how much is allocated to the tasktracker and datanode for that machine? –  SSaikia_JtheRocker Aug 1 '13 at 16:11
Each machine has 16GByte memory and is configured for 2 map tasks and 1 reduce task -> no more than 6GB should be used at any time. –  mt_ Aug 1 '13 at 16:32
HADOOP_HEAPSIZE is 5GB - so DataNode and TaskTracker should have each 5GB Heap - this could be the culprit as the value is probably to high –  mt_ Aug 1 '13 at 16:32
This does not seem to have anything do to with the failing job. The task fails on machines with HADOOP_HEAPSIZE set to either 2GB, 4GB, or 5GB. Does not matter much –  mt_ Aug 1 '13 at 16:43
show 1 more comment

Your Answer


