title: Elasticsearch efficient index management
body: <p>I'm using Elasticsearch in a non-typical way and would like opinions about how efficient this is.</p>
  
  <p>I create and entire set of "docs" with several fields each. I do this in Spark as an RDD of Map[String, Any]. Then I overwrite the index every so often with potentially new data. I suspect the more typical way is to update or add a doc at a time. I do this using the Elasticsearch Spark Scala support with:</p>
  
  <pre><code>val docsRDD: RDD[Map[String, Any]] = getDocs()
  docsRDD.saveToEs(esIndexURI, Map("es.mapping.id" -&gt; "id"))
  </code></pre>
  
  <p>I've done this in two workflows and want to know the trade-offs for them</p>
  
  <ol>
  <li><strong>Delete</strong> index, <strong>create</strong> it, <strong>write</strong> it, then <strong>refresh</strong> to get the index made available. This is because as I understand it the index is not really deleted until the refresh, which will do something like a hot-swap of new index for the old. In other words Elasticsearch works away in the background then swaps in the new fresh index at refresh time. </li>
  <li><strong>Overwrite</strong> the index in place with the Spark code with no delete/create/refresh.</li>
  </ol>
  
  <p>At first I was updating documents in place but that proved impractical and the Spark support seemed to suggest the all-at-once writing method. Both seem to work but I have a small dataset now and wonder how the different approaches (or other suggestions) affect:</p>
  
  <ul>
  <li>Query performance during the operations</li>
  <li>Speed of reindex</li>
  <li>Coherence of the index. Is #2 creating an index with partial updates so queries will be against an incomplete update? Is #1 causing the index to be unavailable for some length of time?</li>
  </ul>