{"action":"created","issue":{"url":"https://api.github.com/repos/evilmartians/fias/issues/14","repository_url":"https://api.github.com/repos/evilmartians/fias","labels_url":"https://api.github.com/repos/evilmartians/fias/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/evilmartians/fias/issues/14/comments","events_url":"https://api.github.com/repos/evilmartians/fias/issues/14/events","html_url":"https://github.com/evilmartians/fias/issues/14","id":164945429,"number":14,"title":"Memory limit problem","user":{"login":"ex0rcist","id":1577458,"avatar_url":"https://avatars.githubusercontent.com/u/1577458?v=3","gravatar_id":"","url":"https://api.github.com/users/ex0rcist","html_url":"https://github.com/ex0rcist","followers_url":"https://api.github.com/users/ex0rcist/followers","following_url":"https://api.github.com/users/ex0rcist/following{/other_user}","gists_url":"https://api.github.com/users/ex0rcist/gists{/gist_id}","starred_url":"https://api.github.com/users/ex0rcist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ex0rcist/subscriptions","organizations_url":"https://api.github.com/users/ex0rcist/orgs","repos_url":"https://api.github.com/users/ex0rcist/repos","events_url":"https://api.github.com/users/ex0rcist/events{/privacy}","received_events_url":"https://api.github.com/users/ex0rcist/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-11T22:00:50Z","updated_at":"2016-07-12T09:49:10Z","closed_at":null,"body":"Faced an issue of \"NoMemoryError\" when running fias:import\r\n\r\n> NoMemoryError: failed to allocate memory\r\n/home/vagrant/.rvm/gems/ruby-2.2.1/gems/pg_data_encoder-0.1.10/lib/pg_data_encoder/encode_for_copy.rb:21:in `write'\r\n/home/vagrant/.rvm/gems/ruby-2.2.1/gems/pg_data_encoder-0.1.10/lib/pg_data_encoder/encode_for_copy.rb:21:in `block in add'\r\n/home/vagrant/.rvm/gems/ruby-2.2.1/gems/pg_data_encoder-0.1.10/lib/pg_data_encoder/encode_for_copy.rb:17:in `each'\r\n/home/vagrant/.rvm/gems/ruby-2.2.1/gems/pg_data_encoder-0.1.10/lib/pg_data_encoder/encode_for_copy.rb:17:in `each_with_index'\r\n/home/vagrant/.rvm/gems/ruby-2.2.1/gems/pg_data_encoder-0.1.10/lib/pg_data_encoder/encode_for_copy.rb:17:in `add'\r\n/home/vagrant/.rvm/gems/ruby-2.2.1/gems/fias-1.0.2/lib/fias/import/copy.rb:18:in `block in encode'\r\n/home/vagrant/.rvm/gems/ruby-2.2.1/gems/dbf-3.0.5/lib/dbf/table.rb:113:in `block in each'\r\n\r\n\r\nBy default it's writing DBF contents to StringIO, which size can't be more than 2**32 on 32-bit OS, so we are facing \"Memory limit error\" on https://github.com/pbrumm/pg_data_encoder/blob/master/lib/pg_data_encoder/encode_for_copy.rb#L21\r\n\r\nSolved this by adding `use_tempfile: true` to https://github.com/evilmartians/fias/blob/master/lib/fias/import/copy.rb#L11\r\n\r\nAnother solution is to dump every, say, 1 mil of records from dbf to pg (we are facing problem at 1.4mil+)\r\n"},"comment":{"url":"https://api.github.com/repos/evilmartians/fias/issues/comments/231991775","html_url":"https://github.com/evilmartians/fias/issues/14#issuecomment-231991775","issue_url":"https://api.github.com/repos/evilmartians/fias/issues/14","id":231991775,"user":{"login":"gzigzigzeo","id":27830,"avatar_url":"https://avatars.githubusercontent.com/u/27830?v=3","gravatar_id":"","url":"https://api.github.com/users/gzigzigzeo","html_url":"https://github.com/gzigzigzeo","followers_url":"https://api.github.com/users/gzigzigzeo/followers","following_url":"https://api.github.com/users/gzigzigzeo/following{/other_user}","gists_url":"https://api.github.com/users/gzigzigzeo/gists{/gist_id}","starred_url":"https://api.github.com/users/gzigzigzeo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gzigzigzeo/subscriptions","organizations_url":"https://api.github.com/users/gzigzigzeo/orgs","repos_url":"https://api.github.com/users/gzigzigzeo/repos","events_url":"https://api.github.com/users/gzigzigzeo/events{/privacy}","received_events_url":"https://api.github.com/users/gzigzigzeo/received_events","type":"User","site_admin":false},"created_at":"2016-07-12T09:49:10Z","updated_at":"2016-07-12T09:49:10Z","body":":-) Yeah, could you add warning to README. I will try to take a look on batching in this week."}}
{"actor":{"display_login":"gzigzigzeo"}}