title: lucene.net IndexWriter and Azure WebJob
body: <p>Ive got an Azure webjob running continuously that triggers based of a queue trigger. The queue contains a list of items that need to be written to my lucene Index. I currently have a lot of items on the queue (over 500k line items) and Im looking for the most effecient way to process it. I keep getting IndexWriter Lock exception when I attempt to 'scale' out the webjob.</p>
  
  <p>Current Setup:</p>
  
  <pre><code>JobHostConfiguration config = new JobHostConfiguration();
              config.Queues.BatchSize = 1;
  
              var host = new JobHost(config);                        
              host.RunAndBlock();
  </code></pre>
  
  <p>Web job function</p>
  
  <pre><code>     public static void AddToSearchIndex([QueueTrigger("indexsearchadd")] List&lt;ListingItem&gt; items, TextWriter log)
                  {
                      var azureDirectory = new AzureDirectory(CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings["StorageConnectionString"].ConnectionString), "megadata");
                      var findexExists = IndexReader.IndexExists(azureDirectory);
                      var count = items.Count;
                      IndexWriter indexWriter = null;
                      int errors = 0;
                      while (indexWriter == null &amp;&amp; errors &lt; 10)
                      {
                          try
                          {
                              indexWriter = new IndexWriter(azureDirectory, new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30), !IndexReader.IndexExists(azureDirectory), new Lucene.Net.Index.IndexWriter.MaxFieldLength(IndexWriter.DEFAULT_MAX_FIELD_LENGTH));
                          }
                          catch (LockObtainFailedException)
                          {
                              log.WriteLine("Lock is taken, Hit 'Y' to clear the lock, or anything else to try again");
                              errors++;
                          }
                      };
                      if (errors &gt;= 10)
                      {
                          azureDirectory.ClearLock("write.lock");
                          indexWriter = new IndexWriter(azureDirectory, new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30), !IndexReader.IndexExists(azureDirectory), new Lucene.Net.Index.IndexWriter.MaxFieldLength(IndexWriter.DEFAULT_MAX_FIELD_LENGTH));
   log.WriteLine("IndexWriter lock obtained, this process has exclusive write access to index");
              indexWriter.SetRAMBufferSizeMB(10.0);
              // Parallel.ForEach(items, (itm) =&gt;
              //{
              foreach (var itm in items)
              {
                  AddtoIndex(itm, indexWriter);
              }
              //});
      }
  </code></pre>
  
  <p>The method that updates the index items basically looks like this:</p>
  
  <pre><code>private static void AddtoIndex(ListingItem item, IndexWriter indexWriter)
          {            
              var doc = new Document();
              doc.Add(new Field("id", item.URL, Field.Store.NO, Field.Index.NOT_ANALYZED, Field.TermVector.NO));
              var title = new Field("Title", item.Title, Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.YES);
   indexWriter.UpdateDocument(new Term("id", item.URL), doc);
  }
  </code></pre>
  
  <p>Things I have tried:</p>
  
  <ol>
  <li>Set the azure config batch size to the maximum 32</li>
  <li>Make the method async and use  Task.WhenAll</li>
  <li>Use parallel for loop</li>
  </ol>
  
  <p>When I try the above, it usually fails with:</p>
  
  <pre><code>Lucene.Net.Store.LockObtainFailedException: Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: AzureLock@write.lock.
   at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout) in d:\Lucene.Net\FullRepo\trunk\src\core\Store\Lock.cs:line 97
   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer
  </code></pre>
  
  <p>Any suggestions on how I can architecturally set up this web job such that it can process more items in the queue instead of doing it one by one? They need to write to the same index?
  Thanks</p>