title: StanfordCoreNlp with MapReduce(Error: GC overhead limit exceeded)
body: <p>I have a text file containing a set of document IDs and document contents, delimited by "::". Here's an example</p>
  
  <pre><code>139::This is a sentence in document 139. This is another sentence.
  140::This is a sentence in document 140. This is another sentence.
  </code></pre>
  
  <p>I want to do some named entity recognition with those sentences using StanfordCoreNLP. This has been working fine with a conventional java program. Now I want to do the same using MapReduce. I am trying to load the StanfordCoreNLP classifiers in the setup() method of my mapper, and the map() method does the named entity tagging, as follows:</p>
  
  <pre><code>public class NerMapper extends Mapper&lt;LongWritable, Text, Text, Text&gt;{
  
  StanfordCoreNLP pipeline;
  
  @Override
  protected void setup(Mapper&lt;LongWritable, Text, Text, Text&gt;.Context context)
          throws IOException, InterruptedException {
      // TODO Auto-generated method stub
      super.setup(context);
      Properties props = new Properties();
      props.setProperty("annotators", "tokenize, ssplit, pos, lemma, ner, parse, dcoref, relation");
      pipeline = new StanfordCoreNLP(props);
  }
  
  @Override
  protected void map(LongWritable key, Text value,
          Mapper&lt;LongWritable, Text, Text, Text&gt;.Context context) throws IOException,
          InterruptedException {
      // TODO Auto-generated method stub
      String[] input = value.toString().split("::");
      List&lt;DataTuple&gt; dataTuples = new ArrayList&lt;DataTuple&gt;();
      Annotation annotation = new Annotation(input[1]);
      pipeline.annotate(annotation);
      List&lt;CoreMap&gt; sentences = annotation.get(SentencesAnnotation.class);
      for(CoreMap sentence : sentences){
          //extract named entities
          //write &lt;documentID&gt;::&lt;the named entity itself&gt;::&lt;the named entity tag&gt;
      }
  }
  }
  </code></pre>
  
  <p>On running the job, it fails with the error "GC overhead limit exceeded". I have tried different heap sizes by doing <code>export HADOOP_OPTS="-Xmx892m"</code> before running the job, and I include the StanfordCoreNLP dependencies using the <code>-libjars</code> option of <code>hadoop jar</code> command. The input documents typically consist of just 4-5 normal-sized sentences. I understand that the problem lies in initialization of the classifiers in setup() method, but I have not been able to figure out what exactly is wrong. I would really appreciate any help here! </p>
  
  <p>I'm using hadoop 2.6.0, Stanford CoreNLP 3.4.1 and java 1.7.</p>