title: SOLR Indexing Performance: Processing 2.3k Docs/s
body: <p>Im trying to improve performance of my Solr 6.0 Index.</p>
  
  <p>Originally we were indexing 45m rows that was using a select statement joining 7 table and taking 7+ hours to index. This caused us to get a snapshot too old error while the jdbc connection is open for the entire duration of the indexing. Causing our full index to fail.</p>
  
  <p>We were able to archive about 10m rows and build an external table from the original 7 join select. This simplified the query solr was using so a select * from 1 table.</p>
  
  <p>Now are indexing 35m rows using a Select * from ONE_BIG_<em>External</em>-TABLE now and it's taking ~4-5 hrs @ 2.3k docs/s +-250. Since we are using an external table we shouldn't be getting the snap shot too old because of the UNDO stack. </p>
  
  <p><em>We have 77 columns we are indexing.</em></p>
  
  <p>So we found a solution for our initial issue but now I'm looking to increase our indexing speed when doing clean fulls. </p>
  
  <p><a href="https://wiki.apache.org/solr/SolrPerformanceFactors" rel="nofollow noreferrer">Referencing SolrPerformanceFactors</a> I have tried:</p>
  
  <ul>
  <li>Batch Sizes:
  
  <ol>
  <li>2000 - no change</li>
  <li>6000 - no change</li>
  <li>4000 - no change</li>
  </ol></li>
  </ul>
  
  <p>Example:</p>
  
  <pre><code> &lt;dataSource jndiName="xxxxxx batchSize="2000" type="JdbcDataSource"/&gt;
  </code></pre>
  
  <ul>
  <li>Autocommit:
  
  <ol>
  <li>Every 1 hour - no change</li>
  </ol></li>
  <li>MergeFactor:
  
  <ol>
  <li>20 vs 10 default - shed off 20 mins</li>
  </ol></li>
  <li>Indexed Fields:
  
  <ol>
  <li>Cut out 11 indexed fields - nothing</li>
  </ol></li>
  </ul>
  
  <p>EDIT: Adding some information per questions below. I did auto-commits to every hour which didn't help any. Also soft commit every second. I copied a much smaller solr core we have here that had these parameters and they said they have been running well. </p>
  
  <pre><code>    &lt;autoCommit&gt; 
        &lt;maxTime&gt;3600000&lt;/maxTime&gt; 
        &lt;openSearcher&gt;false&lt;/openSearcher&gt; 
      &lt;/autoCommit&gt;
  
  
      &lt;autoSoftCommit&gt; 
        &lt;maxTime&gt;1000&lt;/maxTime&gt; 
      &lt;/autoSoftCommit&gt;
  </code></pre>
  
  <p>Is there any gotchas that I'm missing other than throwing hardware at this? 
  Let me know if you need more info, I'll try answer questions as best as I'm allowed.</p>