{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18253","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18253/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18253/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18253/events","html_url":"https://github.com/elastic/elasticsearch/issues/18253","id":154118482,"number":18253,"title":"Scan/Scroll performance degrading logarithmically","user":{"login":"jsnod","id":1916150,"avatar_url":"https://avatars.githubusercontent.com/u/1916150?v=3","gravatar_id":"","url":"https://api.github.com/users/jsnod","html_url":"https://github.com/jsnod","followers_url":"https://api.github.com/users/jsnod/followers","following_url":"https://api.github.com/users/jsnod/following{/other_user}","gists_url":"https://api.github.com/users/jsnod/gists{/gist_id}","starred_url":"https://api.github.com/users/jsnod/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsnod/subscriptions","organizations_url":"https://api.github.com/users/jsnod/orgs","repos_url":"https://api.github.com/users/jsnod/repos","events_url":"https://api.github.com/users/jsnod/events{/privacy}","received_events_url":"https://api.github.com/users/jsnod/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Scroll","name":":Scroll","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-05-10T22:10:23Z","updated_at":"2016-05-12T21:56:36Z","closed_at":"2016-05-12T21:56:35Z","body":"**Elasticsearch version**: 1.7.5\r\n**JVM version**: java version \"1.7.0_99\"\r\nOpenJDK Runtime Environment (rhel-2.6.5.0.el6_7-x86_64 u99-b00)\r\nOpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)\r\n**OS version**: CentOS release 6.7 (Final) 2.6.32-573.22.1.el6.x86_64\r\n\r\nHi,\r\nWe are in the process of upgrading from 1.7 to 2.X, but first we must re-index into mappings compatible with 2.X.  We are using the `reindex()` helper method of `elasticsearch-py v1.9.0` to reindex a relatively small 7GB index, and noticed that performance starts out fast then logarithmically degrades over time:\r\n\r\n![99445104-0bca-11e6-85f5-b0e3df98eed8](https://cloud.githubusercontent.com/assets/1916150/15163270/5daefa34-16bc-11e6-87bd-9e00cf94725c.png)\r\n(Note this shows indexing requests, but this is a function of the slowdown in scan/scroll as I'll show below.)\r\n\r\nAs a test, I set up a simple call to `scan()` on the 7GB index and printed some benchmarks.  I aborted the scan after 13 hours, and it hadn't even gotten close to scanning all 27M docs over those 13 hours:\r\n```\r\n  Starting scan...\r\n    Scanned 100,000 docs (batch time: 0:00:07 / total time: 0:00:07 )\r\n    Scanned 200,000 docs (batch time: 0:00:11 / total time: 0:00:18 )\r\n    Scanned 300,000 docs (batch time: 0:00:14 / total time: 0:00:33 )\r\n    Scanned 400,000 docs (batch time: 0:00:18 / total time: 0:00:52 )\r\n    Scanned 500,000 docs (batch time: 0:00:23 / total time: 0:01:15 )\r\n    Scanned 600,000 docs (batch time: 0:00:26 / total time: 0:01:42 )\r\n    Scanned 700,000 docs (batch time: 0:00:30 / total time: 0:02:13 )\r\n    Scanned 800,000 docs (batch time: 0:00:34 / total time: 0:02:47 )\r\n    Scanned 900,000 docs (batch time: 0:00:38 / total time: 0:03:25 )\r\n    Scanned 1,000,000 docs (batch time: 0:00:41 / total time: 0:04:07 )\r\n    Scanned 1,100,000 docs (batch time: 0:00:45 / total time: 0:04:53 )\r\n    Scanned 1,200,000 docs (batch time: 0:00:49 / total time: 0:05:43 )\r\n    Scanned 1,300,000 docs (batch time: 0:00:53 / total time: 0:06:36 )\r\n    Scanned 1,400,000 docs (batch time: 0:00:58 / total time: 0:07:34 )\r\n    Scanned 1,500,000 docs (batch time: 0:01:03 / total time: 0:08:37 )\r\n    Scanned 1,600,000 docs (batch time: 0:01:06 / total time: 0:09:44 )\r\n    Scanned 1,700,000 docs (batch time: 0:01:09 / total time: 0:10:54 )\r\n    Scanned 1,800,000 docs (batch time: 0:01:13 / total time: 0:12:07 )\r\n    Scanned 1,900,000 docs (batch time: 0:01:17 / total time: 0:13:24 )\r\n    Scanned 2,000,000 docs (batch time: 0:01:20 / total time: 0:14:45 )\r\n    Scanned 2,100,000 docs (batch time: 0:01:24 / total time: 0:16:10 )\r\n    Scanned 2,200,000 docs (batch time: 0:01:34 / total time: 0:17:44 )\r\n    Scanned 2,300,000 docs (batch time: 0:01:46 / total time: 0:19:30 )\r\n    Scanned 2,400,000 docs (batch time: 0:01:37 / total time: 0:21:08 )\r\n    Scanned 2,500,000 docs (batch time: 0:01:42 / total time: 0:22:50 )\r\n    Scanned 2,600,000 docs (batch time: 0:01:50 / total time: 0:24:40 )\r\n    Scanned 2,700,000 docs (batch time: 0:01:54 / total time: 0:26:34 )\r\n    Scanned 2,800,000 docs (batch time: 0:01:54 / total time: 0:28:28 )\r\n    Scanned 2,900,000 docs (batch time: 0:01:55 / total time: 0:30:23 )\r\n    Scanned 3,000,000 docs (batch time: 0:02:05 / total time: 0:32:29 )\r\n    Scanned 3,100,000 docs (batch time: 0:02:12 / total time: 0:34:42 )\r\n    Scanned 3,200,000 docs (batch time: 0:02:13 / total time: 0:36:55 )\r\n    Scanned 3,300,000 docs (batch time: 0:02:18 / total time: 0:39:14 )\r\n    Scanned 3,400,000 docs (batch time: 0:02:14 / total time: 0:41:28 )\r\n    Scanned 3,500,000 docs (batch time: 0:02:24 / total time: 0:43:53 )\r\n    Scanned 3,600,000 docs (batch time: 0:02:40 / total time: 0:46:34 )\r\n    Scanned 3,700,000 docs (batch time: 0:02:50 / total time: 0:49:25 )\r\n    Scanned 3,800,000 docs (batch time: 0:02:37 / total time: 0:52:02 )\r\n    Scanned 3,900,000 docs (batch time: 0:02:35 / total time: 0:54:38 )\r\n    Scanned 4,000,000 docs (batch time: 0:02:37 / total time: 0:57:15 )\r\n    Scanned 4,100,000 docs (batch time: 0:02:42 / total time: 0:59:58 )\r\n    Scanned 4,200,000 docs (batch time: 0:02:45 / total time: 1:02:43 )\r\n    Scanned 4,300,000 docs (batch time: 0:02:57 / total time: 1:05:40 )\r\n    Scanned 4,400,000 docs (batch time: 0:03:02 / total time: 1:08:43 )\r\n    Scanned 4,500,000 docs (batch time: 0:03:03 / total time: 1:11:46 )\r\n    Scanned 4,600,000 docs (batch time: 0:03:12 / total time: 1:14:59 )\r\n    Scanned 4,700,000 docs (batch time: 0:03:17 / total time: 1:18:16 )\r\n    Scanned 4,800,000 docs (batch time: 0:03:15 / total time: 1:21:32 )\r\n    Scanned 4,900,000 docs (batch time: 0:03:25 / total time: 1:24:57 )\r\n    Scanned 5,000,000 docs (batch time: 0:03:23 / total time: 1:28:21 )\r\n    Scanned 5,100,000 docs (batch time: 0:03:34 / total time: 1:31:55 )\r\n    Scanned 5,200,000 docs (batch time: 0:03:37 / total time: 1:35:32 )\r\n    Scanned 5,300,000 docs (batch time: 0:03:28 / total time: 1:39:01 )\r\n    Scanned 5,400,000 docs (batch time: 0:03:37 / total time: 1:42:38 )\r\n    Scanned 5,500,000 docs (batch time: 0:03:48 / total time: 1:46:27 )\r\n    Scanned 5,600,000 docs (batch time: 0:03:54 / total time: 1:50:21 )\r\n    Scanned 5,700,000 docs (batch time: 0:03:43 / total time: 1:54:04 )\r\n    Scanned 5,800,000 docs (batch time: 0:03:46 / total time: 1:57:51 )\r\n    Scanned 5,900,000 docs (batch time: 0:04:03 / total time: 2:01:55 )\r\n    Scanned 6,000,000 docs (batch time: 0:04:09 / total time: 2:06:04 )\r\n    Scanned 6,100,000 docs (batch time: 0:04:12 / total time: 2:10:17 )\r\n    Scanned 6,200,000 docs (batch time: 0:04:07 / total time: 2:14:25 )\r\n    Scanned 6,300,000 docs (batch time: 0:04:31 / total time: 2:18:56 )\r\n    Scanned 6,400,000 docs (batch time: 0:04:20 / total time: 2:23:16 )\r\n    Scanned 6,500,000 docs (batch time: 0:04:28 / total time: 2:27:45 )\r\n    Scanned 6,600,000 docs (batch time: 0:04:22 / total time: 2:32:07 )\r\n    Scanned 6,700,000 docs (batch time: 0:04:23 / total time: 2:36:31 )\r\n    Scanned 6,800,000 docs (batch time: 0:04:38 / total time: 2:41:10 )\r\n    Scanned 6,900,000 docs (batch time: 0:04:45 / total time: 2:45:56 )\r\n    Scanned 7,000,000 docs (batch time: 0:04:56 / total time: 2:50:53 )\r\n    Scanned 7,100,000 docs (batch time: 0:04:47 / total time: 2:55:40 )\r\n    Scanned 7,200,000 docs (batch time: 0:04:56 / total time: 3:00:37 )\r\n    Scanned 7,300,000 docs (batch time: 0:04:57 / total time: 3:05:34 )\r\n    Scanned 7,400,000 docs (batch time: 0:05:04 / total time: 3:10:39 )\r\n    Scanned 7,500,000 docs (batch time: 0:04:55 / total time: 3:15:34 )\r\n    Scanned 7,600,000 docs (batch time: 0:05:14 / total time: 3:20:49 )\r\n    Scanned 7,700,000 docs (batch time: 0:05:17 / total time: 3:26:06 )\r\n    Scanned 7,800,000 docs (batch time: 0:05:12 / total time: 3:31:18 )\r\n    Scanned 7,900,000 docs (batch time: 0:05:17 / total time: 3:36:36 )\r\n    Scanned 8,000,000 docs (batch time: 0:05:32 / total time: 3:42:08 )\r\n    Scanned 8,100,000 docs (batch time: 0:05:27 / total time: 3:47:36 )\r\n    Scanned 8,200,000 docs (batch time: 0:05:44 / total time: 3:53:20 )\r\n    Scanned 8,300,000 docs (batch time: 0:05:22 / total time: 3:58:43 )\r\n    Scanned 8,400,000 docs (batch time: 0:05:32 / total time: 4:04:16 )\r\n    Scanned 8,500,000 docs (batch time: 0:05:42 / total time: 4:09:59 )\r\n    Scanned 8,600,000 docs (batch time: 0:05:48 / total time: 4:15:48 )\r\n    Scanned 8,700,000 docs (batch time: 0:05:57 / total time: 4:21:45 )\r\n    Scanned 8,800,000 docs (batch time: 0:06:00 / total time: 4:27:46 )\r\n    Scanned 8,900,000 docs (batch time: 0:05:50 / total time: 4:33:37 )\r\n    Scanned 9,000,000 docs (batch time: 0:06:00 / total time: 4:39:38 )\r\n    Scanned 9,100,000 docs (batch time: 0:05:59 / total time: 4:45:38 )\r\n    Scanned 9,200,000 docs (batch time: 0:06:16 / total time: 4:51:54 )\r\n    Scanned 9,300,000 docs (batch time: 0:06:21 / total time: 4:58:15 )\r\n    Scanned 9,400,000 docs (batch time: 0:06:22 / total time: 5:04:37 )\r\n    Scanned 9,500,000 docs (batch time: 0:06:19 / total time: 5:10:57 )\r\n    Scanned 9,600,000 docs (batch time: 0:06:11 / total time: 5:17:09 )\r\n    Scanned 9,700,000 docs (batch time: 0:06:27 / total time: 5:23:36 )\r\n    Scanned 9,800,000 docs (batch time: 0:06:19 / total time: 5:29:56 )\r\n    Scanned 9,900,000 docs (batch time: 0:06:35 / total time: 5:36:32 )\r\n    Scanned 10,000,000 docs (batch time: 0:06:42 / total time: 5:43:14 )\r\n    Scanned 10,100,000 docs (batch time: 0:06:55 / total time: 5:50:10 )\r\n    Scanned 10,200,000 docs (batch time: 0:06:48 / total time: 5:56:58 )\r\n    Scanned 10,300,000 docs (batch time: 0:06:55 / total time: 6:03:54 )\r\n    Scanned 10,400,000 docs (batch time: 0:06:45 / total time: 6:10:40 )\r\n    Scanned 10,500,000 docs (batch time: 0:06:57 / total time: 6:17:37 )\r\n    Scanned 10,600,000 docs (batch time: 0:07:04 / total time: 6:24:42 )\r\n    Scanned 10,700,000 docs (batch time: 0:07:09 / total time: 6:31:52 )\r\n    Scanned 10,800,000 docs (batch time: 0:07:11 / total time: 6:39:03 )\r\n    Scanned 10,900,000 docs (batch time: 0:07:22 / total time: 6:46:25 )\r\n    Scanned 11,000,000 docs (batch time: 0:07:38 / total time: 6:54:03 )\r\n    Scanned 11,100,000 docs (batch time: 0:07:15 / total time: 7:01:19 )\r\n    Scanned 11,200,000 docs (batch time: 0:07:27 / total time: 7:08:46 )\r\n    Scanned 11,300,000 docs (batch time: 0:07:24 / total time: 7:16:11 )\r\n    Scanned 11,400,000 docs (batch time: 0:07:32 / total time: 7:23:44 )\r\n    Scanned 11,500,000 docs (batch time: 0:07:25 / total time: 7:31:09 )\r\n    Scanned 11,600,000 docs (batch time: 0:07:31 / total time: 7:38:40 )\r\n    Scanned 11,700,000 docs (batch time: 0:07:34 / total time: 7:46:15 )\r\n    Scanned 11,800,000 docs (batch time: 0:07:48 / total time: 7:54:04 )\r\n    Scanned 11,900,000 docs (batch time: 0:07:59 / total time: 8:02:04 )\r\n    Scanned 12,000,000 docs (batch time: 0:07:44 / total time: 8:09:49 )\r\n    Scanned 12,100,000 docs (batch time: 0:07:47 / total time: 8:17:36 )\r\n    Scanned 12,200,000 docs (batch time: 0:08:05 / total time: 8:25:41 )\r\n    Scanned 12,300,000 docs (batch time: 0:08:21 / total time: 8:34:02 )\r\n    Scanned 12,400,000 docs (batch time: 0:08:06 / total time: 8:42:09 )\r\n    Scanned 12,500,000 docs (batch time: 0:08:30 / total time: 8:50:39 )\r\n    Scanned 12,600,000 docs (batch time: 0:08:07 / total time: 8:58:46 )\r\n    Scanned 12,700,000 docs (batch time: 0:08:40 / total time: 9:07:26 )\r\n    Scanned 12,800,000 docs (batch time: 0:09:11 / total time: 9:16:38 )\r\n    Scanned 12,900,000 docs (batch time: 0:08:31 / total time: 9:25:09 )\r\n    Scanned 13,000,000 docs (batch time: 0:08:28 / total time: 9:33:38 )\r\n    Scanned 13,100,000 docs (batch time: 0:08:48 / total time: 9:42:26 )\r\n    Scanned 13,200,000 docs (batch time: 0:09:04 / total time: 9:51:30 )\r\n    Scanned 13,300,000 docs (batch time: 0:08:34 / total time: 10:00:05 )\r\n    Scanned 13,400,000 docs (batch time: 0:09:04 / total time: 10:09:09 )\r\n    Scanned 13,500,000 docs (batch time: 0:08:47 / total time: 10:17:57 )\r\n    Scanned 13,600,000 docs (batch time: 0:08:51 / total time: 10:26:48 )\r\n    Scanned 13,700,000 docs (batch time: 0:08:48 / total time: 10:35:37 )\r\n    Scanned 13,800,000 docs (batch time: 0:09:02 / total time: 10:44:39 )\r\n    Scanned 13,900,000 docs (batch time: 0:09:23 / total time: 10:54:03 )\r\n    Scanned 14,000,000 docs (batch time: 0:09:22 / total time: 11:03:26 )\r\n    Scanned 14,100,000 docs (batch time: 0:09:22 / total time: 11:12:49 )\r\n    Scanned 14,200,000 docs (batch time: 0:09:31 / total time: 11:22:20 )\r\n    Scanned 14,300,000 docs (batch time: 0:09:14 / total time: 11:31:35 )\r\n    Scanned 14,400,000 docs (batch time: 0:09:33 / total time: 11:41:08 )\r\n    Scanned 14,500,000 docs (batch time: 0:09:45 / total time: 11:50:54 )\r\n    Scanned 14,600,000 docs (batch time: 0:09:46 / total time: 12:00:40 )\r\n    Scanned 14,700,000 docs (batch time: 0:09:29 / total time: 12:10:10 )\r\n    Scanned 14,800,000 docs (batch time: 0:09:43 / total time: 12:19:54 )\r\n    Scanned 14,900,000 docs (batch time: 0:09:43 / total time: 12:29:37 )\r\n    Scanned 15,000,000 docs (batch time: 0:09:40 / total time: 12:39:17 )\r\n    Scanned 15,100,000 docs (batch time: 0:10:09 / total time: 12:49:27 )\r\n    Scanned 15,200,000 docs (batch time: 0:10:10 / total time: 12:59:37 )\r\n    Scanned 15,300,000 docs (batch time: 0:10:06 / total time: 13:09:44 )\r\n```\r\n\r\nAs you can see, at the end it was taking 10+ minutes to iterate over a batch of 100k documents.  I don't even want to know how long it would take at the 27 millionth document batch.\r\n\r\nOur cluster is at AWS and is comprised of the following:\r\n* (5) `m4.xlarge` data nodes\r\n* (3) `m3.medium` master nodes\r\n* (1) `m4.large` client node\r\n\r\nThat's 20 CPUs and 80GB RAM amongst the data nodes, which you think would be well enough for a 7GB index.\r\n\r\nWe experimented with various `scan` values (`5s`, `30s`, `5m`, `10m`) and it didn't seem to make a difference.  I enabled slow logging during the scan, but nothing appeared in the logs.  I did grab a `hot_threads` output during the scan:\r\n\r\n```\r\n::: [estest][PiocenKcQCuuF-82oB7EZw][estest.domain.com][inet[/172.31.9.171:9300]]{master=true}\r\n   Hot threads at 2016-05-02T20:23:39.940Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:\r\n\r\n   31.4% (156.9ms out of 500ms) cpu usage by thread 'elasticsearch[estest][search][T#3]'\r\n     2/10 snapshots sharing following 15 elements\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)\r\n       org.elasticsearch.search.scan.ScanContext.execute(ScanContext.java:52)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:120)\r\n       org.elasticsearch.search.SearchService.executeScan(SearchService.java:274)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:520)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:517)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     4/10 snapshots sharing following 19 elements\r\n       org.apache.lucene.search.BitsFilteredDocIdSet.match(BitsFilteredDocIdSet.java:60)\r\n       org.apache.lucene.search.FilteredDocIdSet$2.match(FilteredDocIdSet.java:103)\r\n       org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:60)\r\n       org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)\r\n       org.elasticsearch.search.scan.ScanContext.execute(ScanContext.java:52)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:120)\r\n       org.elasticsearch.search.SearchService.executeScan(SearchService.java:274)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:520)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:517)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 10 elements\r\n       sun.misc.Unsafe.park(Native Method)\r\n       java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)\r\n       java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)\r\n       java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)\r\n       java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)\r\n       org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)\r\n       java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 17 elements\r\n       org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:59)\r\n       org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)\r\n       org.elasticsearch.search.scan.ScanContext.execute(ScanContext.java:52)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:120)\r\n       org.elasticsearch.search.SearchService.executeScan(SearchService.java:274)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:520)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:517)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n\r\n   27.8% (139ms out of 500ms) cpu usage by thread 'elasticsearch[estest][search][T#4]'\r\n     6/10 snapshots sharing following 19 elements\r\n       org.apache.lucene.search.BitsFilteredDocIdSet.match(BitsFilteredDocIdSet.java:60)\r\n       org.apache.lucene.search.FilteredDocIdSet$2.match(FilteredDocIdSet.java:103)\r\n       org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:60)\r\n       org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)\r\n       org.elasticsearch.search.scan.ScanContext.execute(ScanContext.java:52)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:120)\r\n       org.elasticsearch.search.SearchService.executeScan(SearchService.java:274)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:520)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:517)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     4/10 snapshots sharing following 2 elements\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n\r\n   27.7% (138.2ms out of 500ms) cpu usage by thread 'elasticsearch[estest][search][T#1]'\r\n     3/10 snapshots sharing following 19 elements\r\n       org.apache.lucene.search.BitsFilteredDocIdSet.match(BitsFilteredDocIdSet.java:60)\r\n       org.apache.lucene.search.FilteredDocIdSet$2.match(FilteredDocIdSet.java:103)\r\n       org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:60)\r\n       org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)\r\n       org.elasticsearch.search.scan.ScanContext.execute(ScanContext.java:52)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:120)\r\n       org.elasticsearch.search.SearchService.executeScan(SearchService.java:274)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:520)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:517)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 16 elements\r\n       org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.nextDoc(ConstantScoreQuery.java:257)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)\r\n       org.elasticsearch.search.scan.ScanContext.execute(ScanContext.java:52)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:120)\r\n       org.elasticsearch.search.SearchService.executeScan(SearchService.java:274)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:520)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:517)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     2/10 snapshots sharing following 15 elements\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193)\r\n       org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)\r\n       org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621)\r\n       org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)\r\n       org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)\r\n       org.elasticsearch.search.scan.ScanContext.execute(ScanContext.java:52)\r\n       org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:120)\r\n       org.elasticsearch.search.SearchService.executeScan(SearchService.java:274)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:520)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$21.call(SearchServiceTransportAction.java:517)\r\n       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n     3/10 snapshots sharing following 10 elements\r\n       sun.misc.Unsafe.park(Native Method)\r\n       java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)\r\n       java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)\r\n       java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)\r\n       java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)\r\n       org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:162)\r\n       java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n       java.lang.Thread.run(Thread.java:745)\r\n```\r\n**Steps to reproduce**:\r\n 1. Scan over all documents in a large index.  Here's my scan code:\r\n```\r\nfrom elasticsearch import Elasticsearch, exceptions\r\nfrom elasticsearch import helpers as es_helpers\r\n\r\ndocs = es_helpers.scan(\r\n    sourceCluster,\r\n    index=args.sourceIndex,\r\n    scroll=args.scroll,\r\n    fields=('_source', '_parent', '_routing', '_timestamp')\r\n)\r\n\r\ntotalCount = 0\r\nbatchCount = 0\r\nbatchSize = 100000\r\nbatchTime = time()\r\n\r\nfor d in docs:\r\n    batchCount += 1\r\n    totalCount += 1\r\n    if batchCount == batchSize:\r\n        print \"Scanned\", format(totalCount, \",d\"), \"docs (batch time:\", _timeDiff(batchTime), \\\r\n            \"/ total time:\", _timeDiff(start), \")\"\r\n        batchCount = 0\r\n        batchTime = time()\r\n\r\ndef _timeDiff(start):\r\n    m, s = divmod(time() - start, 60)\r\n    h, m = divmod(m, 60)\r\n    return \"%d:%02d:%02d\" % (h, m, s)\r\n```\r\n\r\n**Expected behavior**: Each batch of results from a `scan` call should return in the same amount of time.\r\n\r\n**Actual behavior**: Each batch of results from a `scan` call gets slower and slower the deeper into the results it goes.\r\n\r\nIs this a bug in scan/scroll?  From everything I've read, `scan` is supposed to make diving deep into large result sets fast.  It doesn't appear to be working here.  We are currently stuck on 1.7 until we can get past this, so any help would be appreciated.  We really want to move to 2.X ASAP.  Thanks!\r\n\r\n(Note: I had originally filed a bug in `elasticsearch-py`, but came to the conclusion that the issue is in ES core rather than the Python client.  See original bug for more details and discussion:https://github.com/elastic/elasticsearch-py/issues/397)\r\n"}}