title: Elasticsearch-Py bulk not indexing all documents
body: <p>I am using the elasticsearch-py Python package to interact with Elasticsearch through code. I have a script that is meant to take each document from one index, generate a field + value, then re-index it into a new index.</p>
  
  <p>The issue is that there is 1216 documents in the first index, but only 1000 documents make it to the second one. Typically, it is exactly 1000 documents, occasionally making it higher around 1100, but never making it to the full 1216.</p>
  
  <p>I usually keep the batch_size at 200, but changing it around seems to have some effect on the amount of documents that make it to the second index. Changing it to 400 will typically get a result of 800 documents being transferred. Using parallel_bulk seems to have the same results as using bulk.</p>
  
  <p>I believe the issue is with the generating process I am performing. For each document I am generating its ancestry (they are organized in a tree structure) by recursively getting its parent from the first index. This involves rapid document GET requests interwoven with Bulk API calls to index the documents and Scroll API calls to get the documents from the index in the first place.</p>
  
  <p>Would activity like this cause the documents to not go through? If I remove (comment out) the recursive GET requests, all documents seem to go through every time. I have tried creating multiple Elasticsearch clients, but that wouldn't even help if ES itself is the bottleneck.</p>
  
  <p>Here is the code if you're curious:</p>
  
  <pre><code>def complete_resources():
      for result in helpers.scan(client=es, query=query, index=TEMP_INDEX_NAME):
          resource = result["_source"]
          ancestors = []
          parent = resource.get("parent")
          while parent is not None:
              ancestors.append(parent)
              parent = es.get(
                  index=TEMP_INDEX_NAME,
                  doc_type=TEMPORARY_DOCUMENT_TYPE,
                  id=parent["uid"]
              ).get("_source").get("parent")
          resource["ancestors"] = ancestors
          resource["_id"] = resource["uid"]
          yield resource
  
  </code></pre>
  
  <p>This generator is consumed by helpers.parallel_bulk()</p>
  
  <pre><code>for success, info in helpers.parallel_bulk(
      client=es,
      actions=complete_resources(),
      thread_count=10,
      queue_size=12,
      raise_on_error=False,
      chunk_size=INDEX_BATCH_SIZE,
      index=new_primary_index_name,
      doc_type=PRIMARY_DOCUMENT_TYPE,
  ):
      if success:
          successful += 1
      else:
          failed += 1
          print('A document failed:', info)
  </code></pre>
  
  <p>This gives me the following result:</p>
  
  <pre><code>Time: 7 seconds
  Successful: 1000
  Failed: 0
  </code></pre>