title: Lucene 4.x performance issues
body: <p>Over the last few weeks I've been working on upgrading an application from Lucene 3.x to Lucene 4.x in hopes of improving performance. Unfortunately, after going through the full migration process and playing with all sorts of tweaks I found online and in the documentation, Lucene 4 is running significantly slower than Lucene 3 (~50%). I'm pretty much out of ideas at this point, and was wondering if anyone else had any suggestions on how to bring it up to speed. I'm not even looking for a big improvement over 3.x anymore; I'd be happy to just match it and stay on a current release of Lucene.</p>
  
  <h3>&lt;Edit&gt;</h3>
  
  <p>In order to confirm that none of the standard migration changes had a negative effect on performance, I ported my Lucene 4.x version back to Lucene 3.6.2 and kept the newer API rather than using the custom ParallelMultiSearcher and other deprecated methods/classes.</p>
  
  <p>Performance in 3.6.2 is even faster than before:</p>
  
  <ul>
  <li>Old application (Lucene 3.6.0) - ~5700 requests/min</li>
  <li>Updated application with new API and some minor optimizations (Lucene 4.4.0) - ~2900 requests/min</li>
  <li>New version of the application ported back, but retaining optimizations and newer IndexSearcher/etc API (Lucene 3.6.2) - ~6200 requests/min</li>
  </ul>
  
  <p>Since the optimizations and use of the newer Lucene API actually improved performance on 3.6.2, it doesn't make sense for this to be a problem with anything but Lucene. I just don't know what else I can change in my program to fix it.</p>
  
  <h3>&lt;/Edit&gt;</h3>
  
  <h3>Application Information</h3>
  
  <ul>
  <li><p>We have one index that is broken into 20 shards - this provided the best performance in both Lucene 3.x and Lucene 4.x</p></li>
  <li><p>The index currently contains ~150 million documents, all of which are fairly simple and heavily normalized so there are a lot of duplicate tokens. Only one field (an ID) is stored - the others are not retrievable.</p></li>
  <li><p>We have a fixed set of relatively simple queries that are populated with user input and executed - they are comprised of multiple BooleanQueries, TermQueries and TermRangeQueries. Some of them are nested, but only a single level right now.</p></li>
  <li><p>We're not doing anything advanced with results - we just fetch the the scores and the stored ID fields</p></li>
  <li><p>We're using MMapDirectories pointing to index files in a tmpfs. We played with the useUnmap "hack" since we don't open new Directories very often and got a nice boost from that</p></li>
  <li><p>We're using a single IndexSearcher for all queries</p></li>
  <li><p>Our test machines have 94GB of RAM and 64 logical cores</p></li>
  </ul>
  
  <h3>General Processing</h3>
  
  <p>1) Request received by socket listener</p>
  
  <p>2) Up to 4 Query objects are generated and populated with normalized user input (all of the required input for a query must be present or it won't be executed)</p>
  
  <p>3) Queries are executed in parallel using the Fork/Join framework</p>
  
  <ul>
  <li>Subqueries to each shard are executed in parallel using the IndexSearcher w/ExecutorService</li>
  </ul>
  
  <p>4) Aggregation and other simple post-processing</p>
  
  <h3>Other Relevant Info</h3>
  
  <ul>
  <li><p>Indexes were recreated for the 4.x system, but the data is the same. We tried the normal Lucene42 codec as well as an extended one that didn't use compression (per a suggestion on the web)</p></li>
  <li><p>In 3.x we used a modified version of the ParallelMultisearcher, in 4.x we're using the IndexSearcher with an ExecutorService and combining all of our readers in a MultiReader</p></li>
  <li><p>In 3.x we used a ThreadPoolExecutor instead of Fork/Join (Fork/Join performed better in my tests)</p></li>
  </ul>
  
  <h3>4.x Hot Spots</h3>
  
  <p>Method | Self Time (%) | Self Time (ms)| Self Time (CPU in ms)</p>
  
  <p>java.util.concurrent.CountDownLatch.await() | 11.29% | 140887.219  | 0.0  &lt;- this is just from tcp threads waiting for the real work to finish - you can ignore it<br/>
  org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.&lt;init&gt;() | 9.74% | 21594.03  | 121594<br/>
  org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTermsEnum$Frame.&lt;init&gt;() | 9.59% | 119680.956  | 119680<br/>
  org.apache.lucene.codecs.lucene41.ForUtil.readBlock() | 6.91% | 86208.621  | 86208<br/>
  org.apache.lucene.search.DisjunctionScorer.heapAdjust() | 6.68% | 83332.525  | 83332<br/>
  java.util.concurrent.ExecutorCompletionService.take() | 5.29% | 66081.499  | 6153<br/>
  org.apache.lucene.search.DisjunctionSucorer.afterNext() | 4.93% | 61560.872  | 61560<br/>
  org.apache.lucene.search.Tercorer.advance() | 4.53% | 56530.752  | 56530<br/>
  java.nio.DirectByteBuffer.get() | 3.96% | 49470.349  | 49470<br/>
  org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.&lt;init&gt;() | 2.97% | 37051.644  | 37051<br/>
  org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.getFrame() | 2.77% | 34576.54  | 34576<br/>
  org.apache.lucene.codecs.MultiLevelSkipListReader.skipTo() | 2.47% | 30767.711  | 30767<br/>
  org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.newTertate() | 2.23% | 27782.522  | 27782<br/>
  java.net.ServerSocket.accept() | 2.19% | 27380.696  | 0.0<br/>
  org.apache.lucene.search.DisjunctionSucorer.advance() | 1.82% | 22775.325  | 22775<br/>
  org.apache.lucene.search.HitQueue.getSentinelObject() | 1.59% | 19869.871  | 19869<br/>
  org.apache.lucene.store.ByteBufferIndexInput.buildSlice() | 1.43% | 17861.148  | 17861<br/>
  org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.getArc() | 1.35% | 16813.927  | 16813<br/>
  org.apache.lucene.search.DisjunctionSucorer.countMatches() | 1.25% | 15603.283  | 15603<br/>
  org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.refillDocs() | 1.12% | 13929.646  | 13929<br/>
  java.util.concurrent.locks.ReentrantLock.lock() | 1.05% | 13145.631  | 8618<br/>
  org.apache.lucene.util.PriorityQueue.downHeap() | 1.00% | 12513.406  | 12513<br/>
  java.util.TreeMap.get() | 0.89% | 11070.192  | 11070<br/>
  org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.docs() | 0.80% | 10026.117  | 10026<br/>
  org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum$Frame.decodeMetaData() | 0.62% | 7746.05  | 7746<br/>
  org.apache.lucene.codecs.BlockTreeTerReader$FieldReader.iterator() | 0.60% | 7482.395  | 7482<br/>
  org.apache.lucene.codecs.BlockTreeTerReader$FieldReader$SegmentTerEnum.seekExact() | 0.55% | 6863.069  | 6863<br/>
  org.apache.lucene.store.DataInput.clone() | 0.54% | 6721.357  | 6721<br/>
  java.nio.DirectByteBufferR.duplicate() | 0.48% | 5930.226 | 5930<br/>
  org.apache.lucene.util.fst.ByteSequenceOutputs.read() | 0.46% | 5708.354 | 5708<br/>
  org.apache.lucene.util.fst.FST.findTargetArc() | 0.45% | 5601.63 | 5601<br/>
  org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.readTermsBlock() | 0.45% | 5567.914 | 5567<br/>
  org.apache.lucene.store.ByteBufferIndexInput.toString() | 0.39% | 4889.302 | 4889<br/>
  org.apache.lucene.codecs.lucene41.Lucene41SkipReader.&lt;init&gt;() | 0.33% | 4147.285 | 4147<br/>
  org.apache.lucene.search.TermQuery$TermWeight.scorer() | 0.32% | 4045.912 | 4045<br/>
  org.apache.lucene.codecs.MultiLevelSkipListReader.&lt;init&gt;() | 0.31% | 3890.399 | 3890<br/>
  org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock() | 0.31% | 3886.194 | 3886
  <br/><br/><br/>
  If there's any other information you could use that might help, please let me know.</p>