{"action":"created","issue":{"url":"https://api.github.com/repos/pelias/openaddresses/issues/328","repository_url":"https://api.github.com/repos/pelias/openaddresses","labels_url":"https://api.github.com/repos/pelias/openaddresses/issues/328/labels{/name}","comments_url":"https://api.github.com/repos/pelias/openaddresses/issues/328/comments","events_url":"https://api.github.com/repos/pelias/openaddresses/issues/328/events","html_url":"https://github.com/pelias/openaddresses/issues/328","id":281041578,"number":328,"title":"Elasticsearch errors when doing the import on a local machine","user":{"login":"sergiyprotsiv","id":1951020,"avatar_url":"https://avatars3.githubusercontent.com/u/1951020?v=4","gravatar_id":"","url":"https://api.github.com/users/sergiyprotsiv","html_url":"https://github.com/sergiyprotsiv","followers_url":"https://api.github.com/users/sergiyprotsiv/followers","following_url":"https://api.github.com/users/sergiyprotsiv/following{/other_user}","gists_url":"https://api.github.com/users/sergiyprotsiv/gists{/gist_id}","starred_url":"https://api.github.com/users/sergiyprotsiv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sergiyprotsiv/subscriptions","organizations_url":"https://api.github.com/users/sergiyprotsiv/orgs","repos_url":"https://api.github.com/users/sergiyprotsiv/repos","events_url":"https://api.github.com/users/sergiyprotsiv/events{/privacy}","received_events_url":"https://api.github.com/users/sergiyprotsiv/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-12-11T14:44:55Z","updated_at":"2018-03-29T19:40:52Z","closed_at":null,"author_association":"NONE","body":"Hi, I'm attempting a worldwide import on a local machine, but have persistent Elasticsearch errors, with ES dropping connections resulting in batch_error and missing data. The typical example is as follows:\r\n\r\n> Elasticsearch WARNING: 2017-12-11T14:05:12Z\r\n  Unable to revive connection: http://localhost:9200/\r\n> Elasticsearch WARNING: 2017-12-11T14:05:12Z\r\n  No living connections\r\n> 2017-12-11T14:05:12.121Z - error: [dbclient] esclient error Error: No Living connections\r\n    at sendReqWithConnection (/mnt/scratch/pelias/openstreetmap/node_modules/elasticsearch/src/lib/transport.js:225:15)\r\n    at next (/mnt/scratch/pelias/openstreetmap/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)\r\n    at nextTickCallbackWith0Args (node.js:419:9)\r\n    at process._tickCallback (node.js:348:13)\r\n> 2017-12-11T14:05:12.121Z - error: [dbclient] invalid resp from es bulk index operation\r\n> 2017-12-11T14:05:12.121Z - info: [dbclient] retrying batch [500]\r\n> 2017-12-11T14:05:13.643Z - info: [dbclient]  paused=true, transient=15, current_length=0, indexed=3754500, batch_ok=7509, batch_retries=0, failed_records=0, venue=1578642, address=2175858, persec=0, batch_error=35\r\n\r\nThe errors happen reliably and within an hour when running multiple importers in parallel. Running sequentially, Geonames and WOF have imported without issues. OA and OSM start throwing errors after a few hours of importing (e.g. after about 600M OSM nodes in the above example, after getting through half of Brazil in OA).\r\n\r\nThe hardware is quite powerful (32 cores, 128 GB RAM, though not SSDs - HDD in RAID0), so I do not believe this is an issue. ES maxes out 2-3 cores during import, but then it suddenly requires more power and starts dropping connections. \r\n\r\nNot an expert in ES (this is my first encounter), but maybe there is some reindexing going on, or maybe even heavy garbage collection. I give ES 30GB memory (via `ES_JAVA_OPTS=\"-Xms30g -Xmx30g\"`) and run it by launching a daemon. ES version is 2.4, the system is Ubuntu 16.04. I tried with both standard settings and with some suggestions found on the web. The ES documentation is missing and seems to be very different across versions with people suggesting undocumented parameters to tweak. The config that worked to import Geonames and WOF is `\r\nthreadpool.bulk.type: fixed\r\nthreadpool.bulk.size: 25\r\nthreadpool.bulk.queue_size: 1000\r\n`\r\n\r\nI imagine that there could be some ES setting to either give it more resources, keep retrying for longer (or ideally just wait more between retries), or maybe changing batch sizes? Alternatively, is there a way to throttle the importers to a given maximum inserts per second?\r\n"},"comment":{"url":"https://api.github.com/repos/pelias/openaddresses/issues/comments/377349370","html_url":"https://github.com/pelias/openaddresses/issues/328#issuecomment-377349370","issue_url":"https://api.github.com/repos/pelias/openaddresses/issues/328","id":377349370,"user":{"login":"orangejulius","id":111716,"avatar_url":"https://avatars3.githubusercontent.com/u/111716?v=4","gravatar_id":"","url":"https://api.github.com/users/orangejulius","html_url":"https://github.com/orangejulius","followers_url":"https://api.github.com/users/orangejulius/followers","following_url":"https://api.github.com/users/orangejulius/following{/other_user}","gists_url":"https://api.github.com/users/orangejulius/gists{/gist_id}","starred_url":"https://api.github.com/users/orangejulius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orangejulius/subscriptions","organizations_url":"https://api.github.com/users/orangejulius/orgs","repos_url":"https://api.github.com/users/orangejulius/repos","events_url":"https://api.github.com/users/orangejulius/events{/privacy}","received_events_url":"https://api.github.com/users/orangejulius/received_events","type":"User","site_admin":false},"created_at":"2018-03-29T19:40:52Z","updated_at":"2018-03-29T19:40:52Z","author_association":"OWNER","body":"Ouch @sergiyprotsiv, that is not the performance we usually see with Pelias. The PiP service _has_ been observed to load that slowly on small AWS EBS which have \"burst\" performance. If thats the case for you let us know and we can help you fix it. In fact if you want to run any aspect of your setup by us for performance considerations, we might be able to help."}}
{"actor":{"display_login":"orangejulius"}}