{"action":"created","issue":{"url":"https://api.github.com/repos/juttle/juttle-elastic-adapter/issues/63","labels_url":"https://api.github.com/repos/juttle/juttle-elastic-adapter/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/juttle/juttle-elastic-adapter/issues/63/comments","events_url":"https://api.github.com/repos/juttle/juttle-elastic-adapter/issues/63/events","html_url":"https://github.com/juttle/juttle-elastic-adapter/issues/63","id":129685627,"number":63,"title":"Bulk ingest of a large number of data points causes timeouts","user":{"login":"dmehra","id":5817365,"avatar_url":"https://avatars.githubusercontent.com/u/5817365?v=3","gravatar_id":"","url":"https://api.github.com/users/dmehra","html_url":"https://github.com/dmehra","followers_url":"https://api.github.com/users/dmehra/followers","following_url":"https://api.github.com/users/dmehra/following{/other_user}","gists_url":"https://api.github.com/users/dmehra/gists{/gist_id}","starred_url":"https://api.github.com/users/dmehra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmehra/subscriptions","organizations_url":"https://api.github.com/users/dmehra/orgs","repos_url":"https://api.github.com/users/dmehra/repos","events_url":"https://api.github.com/users/dmehra/events{/privacy}","received_events_url":"https://api.github.com/users/dmehra/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/juttle/juttle-elastic-adapter/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-29T07:10:37Z","updated_at":"2016-01-29T22:46:47Z","closed_at":null,"body":"which is to say, we still need better flow control for ingest via the elastic adapter. \r\n\r\nI have an input file of about 200k entries, available [here in s3](https://s3-us-west-2.amazonaws.com/juttle-data/github-tutorial/github_data_joined.json). When I try ingesting it into the default index like this:\r\n\r\n```\r\nread file -file 'github_data_joined.json' | write elastic\r\n```\r\n\r\nI see a lot of timeouts like this:\r\n```\r\nError: insertion failed: Request Timeout after 30000ms [Error: insertion failed: Request Timeout after 30000ms]\r\n```\r\n\r\nMy data is for 6 months of 2015; if I run ingest in 1-month chunks with a few seconds' pause in between, then no timeouts are reported.\r\n\r\nFrom earlier testing of ingest into ES, we were able to sustain 4K event inserts/sec, so my 200k would be over 50 seconds, which exceeds the 30 second request timeout, so the overload situation makes sense. \r\n\r\nWe need some better flow control mechanism to slow down the writes rather than erroring out."},"comment":{"url":"https://api.github.com/repos/juttle/juttle-elastic-adapter/issues/comments/177006082","html_url":"https://github.com/juttle/juttle-elastic-adapter/issues/63#issuecomment-177006082","issue_url":"https://api.github.com/repos/juttle/juttle-elastic-adapter/issues/63","id":177006082,"user":{"login":"dmehra","id":5817365,"avatar_url":"https://avatars.githubusercontent.com/u/5817365?v=3","gravatar_id":"","url":"https://api.github.com/users/dmehra","html_url":"https://github.com/dmehra","followers_url":"https://api.github.com/users/dmehra/followers","following_url":"https://api.github.com/users/dmehra/following{/other_user}","gists_url":"https://api.github.com/users/dmehra/gists{/gist_id}","starred_url":"https://api.github.com/users/dmehra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmehra/subscriptions","organizations_url":"https://api.github.com/users/dmehra/orgs","repos_url":"https://api.github.com/users/dmehra/repos","events_url":"https://api.github.com/users/dmehra/events{/privacy}","received_events_url":"https://api.github.com/users/dmehra/received_events","type":"User","site_admin":false},"created_at":"2016-01-29T22:46:47Z","updated_at":"2016-01-29T22:46:47Z","body":"Additional observation: even trying to ingest my data in 1-month chunks which are sets of about 27k points each is too much; a few thousand points don't get ingested, with the error reported by juttle like this:\r\n\r\nfrom ES 2.1.1\r\n```\r\npoint rejected by Elasticsearch due to: rejected execution of org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1@604c00d3 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6be6ea92[Running, pool size = 1, active threads = 1, queued tasks = 50, completed tasks = 51]] [Error: point rejected by Elasticsearch due to: rejected execution of org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1@604c00d3 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6be6ea92[Running, pool size = 1, active threads = 1, queued tasks = 50, completed tasks = 51]]]\r\n```\r\n\r\nfrom ES 1.5.2\r\n```\r\npoint rejected by Elasticsearch due to: undefined [Error: point rejected by Elasticsearch due to: undefined]\r\n```"}}