title: Elasticsearch indexing performance getting worse with my own _id
body: <p>I'm building a fairly large index - around 3 Billion documents, 2KB average size, nothing fancy like parent/child relationships. At first the bulk indexing was running fine, but it's now slowing down drastically.</p>
  
  <p>Not sure if hardware information is important for the question, but here it is:</p>
  
  <p>The cluster is sitting currently on a single server with 24 cores, 128GB of RAM and a RAID 10 array with 7200K disks and HW controller with BBU. Unfortunately most of the RAM is occupied (around 80GB) by other daemons.</p>
  
  <p><strong>Here's what's important (at least I think so):</strong></p>
  
  <p>I'm providing my own IDs. I've already read <a href="http://blog.mikemccandless.com/2014/05/choosing-fast-unique-identifier-uuid.html" rel="nofollow">Choosing a fast unique identifier (UUID) for Lucene</a> and it all seems logical to me. My IDs are 64-bit integers and <em>will be</em> sequential eventually, but due to various reasons the initial indexing is done in bulks with completely random IDs.</p>
  
  <p>At first I was indexing around 3000 documents per second (the bottleneck is not ES, but the databases from where the documents are pulled). Currently the server is almost stalling from IO (99% read), because of the constant lookups. I have already indexed around 60% of the documents, which took roughly two weeks.</p>
  
  <p>When this initial indexing finishes, I will provide <em>only sequential</em> IDs with a rate of around 100 docs/s. The main question here is - <strong>Will the performance then be as good as if the whole index was built using sequential IDs?</strong> Becase if the answer is no, then I will abandon the current index and create a new one using the default _id field of ES and my IDs will be in another field. Surely this will require some changes in the application, but my documents are very rarely updated, so it shouldn't be hard.</p>
  
  <p>=== Edit ===</p>
  
  <p>I'm adding a bit more information about my setup:</p>
  
  <p>Number of shards: 6<br>
  ES_HEAP_SIZE: 32g<br>
  Mapping (PHP array, filters, analyzers and tokenizers are excluded for brevity):</p>
  
  <pre><code>    'index' =&gt; 'articles',
      'body' =&gt; [
          'settings' =&gt; [
              'number_of_shards' =&gt; 6,
              'number_of_replicas' =&gt; 0,
              'refresh_interval' =&gt; -1,
              'analysis' =&gt; [
                  'filter' =&gt; [
                  ],
                  'analyzer' =&gt; [
                  ],
                  'tokenizer'=&gt; [
                  ]
              ]
          ],
          'mappings' =&gt; [
              'article' =&gt; [
                  '_source' =&gt; ['enabled' =&gt; false],
                  '_all' =&gt; ['enabled' =&gt; false],
                  '_analyzer' =&gt; ['path' =&gt; 'lang_analyzer'],
                  'properties' =&gt; [
                      'lang_analyzer' =&gt; [
                          'type' =&gt; 'string',
                          'doc_values' =&gt; true,
                          'store' =&gt; false,
                          'index' =&gt; 'no'
                      ],
                      'date' =&gt; [
                          'type' =&gt; 'date',
                          'doc_values' =&gt; true
                      ],
                      'feed_id' =&gt; [
                          'type' =&gt; 'integer'
                      ],
                      'feed_subscribers' =&gt; [
                          'type' =&gt; 'integer'
                      ],
                      'feed_canonical' =&gt; [
                          'type' =&gt; 'boolean'
                      ],
                      'title' =&gt; [
                          'type' =&gt; 'string',
                          'store' =&gt; false,
                      ],
                      'content' =&gt; [
                          'type' =&gt; 'string',
                          'store' =&gt; false,
                      ],
                      'url' =&gt; [
                          'type' =&gt; 'string',
                          'analyzer' =&gt; 'simple',
                          'store' =&gt; false
                      ]
                  ]
              ]
          ]
      ]
  </code></pre>
  
  <p>Config (elasticsearch.yml):</p>
  
  <pre><code>node.master: true
  node.data: true
  plugin.mandatory: analysis-kuromoji,analysis-icu,langdetect-1.4.0.2-1368fbe,analysis-smartcn
  bootstrap.mlockall: true
  action.disable_delete_all_indices: true;
  index.merge.scheduler.max_thread_count: 1
  indices.memory.index_buffer_size: 3gb
  index.translog.flush_threshold_size: 1gb
  index.store.throttle.type: none
  </code></pre>
  
  <p>I have removed the other services from the host and now all 128GB of memory are for ES. It's now not making any read IO while indexing, because the indexes are cached by the OS.  </p>
  
  <p>I am basically indexing documents with autoincrement IDs from MySQL up to a given ID, which I have written down. Those documents are not indexed in sequential order, but very randomly in the whole ID range. There are no duplicate requests (updates) during this indexing phase.</p>
  
  <p><strong>The main question still remains:</strong></p>
  
  <blockquote>
    <p>When I finish bulk indexing all IDs until my threshold and then start indexing new documents only sequentially, will the indexing performance be the same as if the whole index was built with sequential ids?</p>
  </blockquote>