{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-py/issues/369","repository_url":"https://api.github.com/repos/elastic/elasticsearch-py","labels_url":"https://api.github.com/repos/elastic/elasticsearch-py/issues/369/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-py/issues/369/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-py/issues/369/events","html_url":"https://github.com/elastic/elasticsearch-py/issues/369","id":137964952,"number":369,"title":"clear_scroll on reindex","user":{"login":"cp2587","id":5306537,"avatar_url":"https://avatars.githubusercontent.com/u/5306537?v=3","gravatar_id":"","url":"https://api.github.com/users/cp2587","html_url":"https://github.com/cp2587","followers_url":"https://api.github.com/users/cp2587/followers","following_url":"https://api.github.com/users/cp2587/following{/other_user}","gists_url":"https://api.github.com/users/cp2587/gists{/gist_id}","starred_url":"https://api.github.com/users/cp2587/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cp2587/subscriptions","organizations_url":"https://api.github.com/users/cp2587/orgs","repos_url":"https://api.github.com/users/cp2587/repos","events_url":"https://api.github.com/users/cp2587/events{/privacy}","received_events_url":"https://api.github.com/users/cp2587/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-03-02T19:14:51Z","updated_at":"2016-03-03T17:21:19Z","closed_at":"2016-03-03T17:21:19Z","body":"Hi,\r\n\r\nThanks you very much for you work. \r\nI am trying the reindex helper function and i find it quite slow (~150 sec to reindex a very small index of 80k docs). \r\nIs is something normal - my cluster is made of 3 nodes (1 client and 2 data with each one 2GB ram)? How can i achieve better speed (i already set chunk_size to 1000)? \r\nIt seems you don't call clear_scroll at the end of the reindex function (or after each successful scroll chunk retrieve), would it be useful to do it to speed-up the process ?"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-py/issues/comments/191869854","html_url":"https://github.com/elastic/elasticsearch-py/issues/369#issuecomment-191869854","issue_url":"https://api.github.com/repos/elastic/elasticsearch-py/issues/369","id":191869854,"user":{"login":"HonzaKral","id":32132,"avatar_url":"https://avatars.githubusercontent.com/u/32132?v=3","gravatar_id":"","url":"https://api.github.com/users/HonzaKral","html_url":"https://github.com/HonzaKral","followers_url":"https://api.github.com/users/HonzaKral/followers","following_url":"https://api.github.com/users/HonzaKral/following{/other_user}","gists_url":"https://api.github.com/users/HonzaKral/gists{/gist_id}","starred_url":"https://api.github.com/users/HonzaKral/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HonzaKral/subscriptions","organizations_url":"https://api.github.com/users/HonzaKral/orgs","repos_url":"https://api.github.com/users/HonzaKral/repos","events_url":"https://api.github.com/users/HonzaKral/events{/privacy}","received_events_url":"https://api.github.com/users/HonzaKral/received_events","type":"User","site_admin":false},"created_at":"2016-03-03T17:21:19Z","updated_at":"2016-03-03T17:21:19Z","body":"For sending the data in you can use the `parallel_bulk` helper, it uses threads to send data in parallel. Unfortunately there is no way to paralelize the scan easily.\r\n\r\nThere is currently some confusion whether scan is getting cleared automatically so I created an issue to clear it up - https://github.com/elastic/elasticsearch/issues/16929#issuecomment-191807048"}}