title: How to improve performance on Lucene searches with big indexes?
body: <p>Hope you're all doing well. I have a question on improving the search performance on a Lucene index that is about 6GB in size in my local filesystem. </p>
  
  <p>Using the Lucene FSDirectory, which ends up mapping to an MMapDirectory, I can perform about 100 searches in about 20 seconds, which is a little slow for my use-case. Using the RAMDirectory, it is a lot slower, performing about 100 searches in about 4 minutes.</p>
  
  <p>Some background (use-case):
  I want to match a given address string to the closest matching address string from among 300 million entries. I am only indexing the 300 million entries (not storing them) in Lucene, and then storing a Long value to identify each of the 300 million entries (and to save space by not storing all 300 million addresses).</p>
  
  <p>Implementation:
  I am using the Standard Analyzer to write and search the index. When searching the index, I am only retrieving the top hit for each query, and then from the Document I am retrieving the Long value to identify the address.</p>
  
  <p>Research:
  I have looked into the suggestions presented at <a href="https://wiki.apache.org/lucene-java/ImproveSearchingSpeed" rel="nofollow">https://wiki.apache.org/lucene-java/ImproveSearchingSpeed</a>.</p>
  
  <p>Since I have a Mac, I've tried using the NIOFSDirectory with multiple lightweight-threads (specifically Quasar fibers written in Java, but I've also tried the standard Java thread) using the same IndexSearcher instance (as suggested in the link above), yet the performance does not improve. Note: Using Quasar, a warning that I usually get is that the thread/fiber "is hogging the CPU or blocking a thread". My hope was that using the NIOFSDirectory and positional reads, "which allows multiple threads to read from the same file" (<a href="https://lucene.apache.org/core/5_2_0/core/org/apache/lucene/store/NIOFSDirectory.html" rel="nofollow">https://lucene.apache.org/core/5_2_0/core/org/apache/lucene/store/NIOFSDirectory.html</a>), would help with performance.</p>
  
  <p>Sorry for the long post, if there is any further information that you'd like me to provide, please let me know. Any insights on the current problem, or alternative open-source solutions, would be helpful.</p>
  
  <p>Thank you for your time.</p>