{"action":"created","issue":{"url":"https://api.github.com/repos/18F/open-data-maker/issues/297","repository_url":"https://api.github.com/repos/18F/open-data-maker","labels_url":"https://api.github.com/repos/18F/open-data-maker/issues/297/labels{/name}","comments_url":"https://api.github.com/repos/18F/open-data-maker/issues/297/comments","events_url":"https://api.github.com/repos/18F/open-data-maker/issues/297/events","html_url":"https://github.com/18F/open-data-maker/pull/297","id":136480229,"number":297,"title":"Elasticsearch bulk index","user":{"login":"ultrasaurus","id":41906,"avatar_url":"https://avatars.githubusercontent.com/u/41906?v=3","gravatar_id":"","url":"https://api.github.com/users/ultrasaurus","html_url":"https://github.com/ultrasaurus","followers_url":"https://api.github.com/users/ultrasaurus/followers","following_url":"https://api.github.com/users/ultrasaurus/following{/other_user}","gists_url":"https://api.github.com/users/ultrasaurus/gists{/gist_id}","starred_url":"https://api.github.com/users/ultrasaurus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ultrasaurus/subscriptions","organizations_url":"https://api.github.com/users/ultrasaurus/orgs","repos_url":"https://api.github.com/users/ultrasaurus/repos","events_url":"https://api.github.com/users/ultrasaurus/events{/privacy}","received_events_url":"https://api.github.com/users/ultrasaurus/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/18F/open-data-maker/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-02-25T19:05:17Z","updated_at":"2016-02-25T19:09:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/18F/open-data-maker/pulls/297","html_url":"https://github.com/18F/open-data-maker/pull/297","diff_url":"https://github.com/18F/open-data-maker/pull/297.diff","patch_url":"https://github.com/18F/open-data-maker/pull/297.patch"},"body":"re-implements #94 \r\nsupersedes #295 and #296 (rebased on dev)\r\n\r\nThis PR cuts indexing time significantly when run on a machine with multiple processors, at the expense of more memory usage. To mitigate the additional memory use that comes with parallel (forked) processes, CSV files are read via IO stream, a row at a time, rather than being slurped entirely into memory and parsed as a String.\r\n\r\nFor reference, the current import consumes about 250MB for the single process.\r\n\r\nExample stats and usage:\r\n\r\n```\r\n# for 1000 rows from 4 files (using college-choice-data)\r\n\r\nNPROCS=1 rake import\r\n# 812 sec (memory 150MB)\r\n\r\nNPROCS=4 CHUNK_SIZE=50 rake import\r\n# 396 sec (memory 180MB x 4 (720MB))\r\n\r\nNPROCS=5 CHUNK_SIZE=100 rake import\r\n# 268 sec (memory 230MB x 5 (1150MB))\r\n```\r\n\r\nAn important caveat with reading CSV as stream: the :force_utf8 feature does not work as currently implemented. My recommendation is to require all CSV files be in UTF-8 format prior to import, possibly offering some easy docs/script to convert encodings."},"comment":{"url":"https://api.github.com/repos/18F/open-data-maker/issues/comments/188934466","html_url":"https://github.com/18F/open-data-maker/pull/297#issuecomment-188934466","issue_url":"https://api.github.com/repos/18F/open-data-maker/issues/297","id":188934466,"user":{"login":"ultrasaurus","id":41906,"avatar_url":"https://avatars.githubusercontent.com/u/41906?v=3","gravatar_id":"","url":"https://api.github.com/users/ultrasaurus","html_url":"https://github.com/ultrasaurus","followers_url":"https://api.github.com/users/ultrasaurus/followers","following_url":"https://api.github.com/users/ultrasaurus/following{/other_user}","gists_url":"https://api.github.com/users/ultrasaurus/gists{/gist_id}","starred_url":"https://api.github.com/users/ultrasaurus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ultrasaurus/subscriptions","organizations_url":"https://api.github.com/users/ultrasaurus/orgs","repos_url":"https://api.github.com/users/ultrasaurus/repos","events_url":"https://api.github.com/users/ultrasaurus/events{/privacy}","received_events_url":"https://api.github.com/users/ultrasaurus/received_events","type":"User","site_admin":false},"created_at":"2016-02-25T19:09:34Z","updated_at":"2016-02-25T19:09:34Z","body":"@pkarman Agree on simple approach of modifying docs on how to run iconv or whatever to convert to UTF8"}}