title: Nutch + Solr - Indexer causes java.lang.OutOfMemoryError: Java heap space
body: <p>I have configured my 2 servers to run in distributed mode (with Hadoop) and my configuration for crawling process is Nutch 2.2.1 - HBase (as a storage) and Solr. Solr is run by Tomcat. The problem is everytime I try to do the last step - I mean when I want to index data from HBase into Solr. After then this <strong>[1]</strong> error occures. I tried to add CATALINA_OPTS (or JAVA_OPTS) like this: </p>
  
  <blockquote>
    <p>CATALINA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -Xms1g -Xmx6000m
    -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=30 -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled"</p>
  </blockquote>
  
  <p>to Tomcat's catalina.sh script and run server with this script but it didn't help. I also add these <strong>[2]</strong> properties to nutch-site.xml file but it ended up with <code>OutOfMemory</code> again. Can you help me please? </p>
  
  <p><strong>[1]</strong></p>
  
  <pre><code>2014-09-06 22:52:50,683 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: Java heap space 
      at java.util.Arrays.copyOf(Arrays.java:2367) 
      at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) 
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) 
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:587) 
      at java.lang.StringBuffer.append(StringBuffer.java:332) 
      at java.io.StringWriter.write(StringWriter.java:77) 
      at org.apache.solr.common.util.XML.escape(XML.java:204) 
      at org.apache.solr.common.util.XML.escapeCharData(XML.java:77) 
      at org.apache.solr.common.util.XML.writeXML(XML.java:147) 
      at org.apache.solr.client.solrj.util.ClientUtils.writeVal(ClientUtils.java:161) 
      at org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:129) 
      at org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:355) 
      at org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:271) 
      at org.apache.solr.client.solrj.request.RequestWriter.getContentStream(RequestWriter.java:66) 
      at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94) 
      at org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104) 
      at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:247) 
      at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197) 
      at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) 
      at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68) 
      at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54) 
      at org.apache.nutch.indexwriter.solr.SolrIndexWriter.close(SolrIndexWriter.java:96) 
      at org.apache.nutch.indexer.IndexWriters.close(IndexWriters.java:117) 
      at org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:54) 
      at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:650) 
      at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793) 
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779) 
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364) 
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255) 
      at java.security.AccessController.doPrivileged(Native Method) 
      at javax.security.auth.Subject.doAs(Subject.java:415) 
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 
  </code></pre>
  
  <p><strong>[2]</strong></p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;http.content.limit&lt;/name&gt;
    &lt;value&gt;150000000&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
     &lt;name&gt;indexer.max.tokens&lt;/name&gt;
     &lt;value&gt;100000&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;http.timeout&lt;/name&gt;
    &lt;value&gt;50000&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;solr.commit.size&lt;/name&gt;
    &lt;value&gt;100&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>