title: Solr NativeFSLock timeouts
body: <p><strong>Setup:</strong>
  I'm running Apache Solr in a jetty container on CentOS 5.7 linux 2.6.32-042stab044.5. The server is a VPS with 4GB dedicated RAM and 2GB of "burst" RAM.</p>
  
  <p>I'm running Solr with the following options (linebreaks for readability):</p>
  
  <pre><code>/opt/jre/bin/java -Dsolr.solr.home=/opt/solr/solr 
                    -Xms2048m -Xmx2048m 
                    -Djetty.logs=/var/log/solr
                    -Djetty.home=/opt/solr
                    -Djava.io.tmpdir=/tmp 
                    -jar /opt/solr/start.jar
                    /opt/solr/etc/jetty-logging.xml 
                    /opt/solr/etc/jetty.xml
  </code></pre>
  
  <p>My Solr config uses ~500 cores, each using 13MB, for a total on disk of 6.8GB.</p>
  
  <p><strong>Problem:</strong>
  About every 2 weeks the solr server starts responding slowly. When I check the stderror log I see lots of entries like this:</p>
  
  <pre><code>Jun 21, 2012 9:36:11 AM org.apache.solr.common.SolrException log
  SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/opt/solr/solr/examplestore/data/index/write.lock
          at org.apache.lucene.store.Lock.obtain(Lock.java:84)
          at org.apache.lucene.index.IndexWriter.&lt;init&gt;(IndexWriter.java:1108)
          at org.apache.solr.update.SolrIndexWriter.&lt;init&gt;(SolrIndexWriter.java:83)
          at org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:101)
          at org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:171)
          at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:219)
          at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61)
          at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
          at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:158)
          at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
          at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
          at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372)
          at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
          at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
          at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
          at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
          at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
          at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
          at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
          at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
          at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
          at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          at org.mortbay.jetty.Server.handle(Server.java:326)
          at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
          at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
          at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
          at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
          at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
          at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
          at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
  </code></pre>
  
  <p>When I check the memory usage, the jetty process is using 3GB, and <code>free -m</code> reports that ~3GB is in use.</p>
  
  <p>My current fix is to restart the jetty process (/etc/init.d/solr restart). The "Lock obtain timed out" exceptions stop and performance is reasonable for another week or two.</p>
  
  <p>The <a href="http://wiki.apache.org/solr/SolrPerformanceFactors#OutOfMemoryErrors" rel="nofollow">SolrPerformance</a> wiki indicates this is a memory issue, which is why I've set the memory usage to 2GB and check the memory use before killing the process. However, I'm not sure that is the problem anymore, as there is memory to spare (3GB used of 4GB dedicated, plus 2GB of "burst" RAM).</p>
  
  <p><strong>Question:</strong>
  What can I do to prevent the "Lock obtain timed out" exceptions and ultimately improve the stability of my Solr setup so I don't have to restart it every 2 weeks?</p>