{"action":"created","issue":{"url":"https://api.github.com/repos/uniVocity/univocity-parsers/issues/237","repository_url":"https://api.github.com/repos/uniVocity/univocity-parsers","labels_url":"https://api.github.com/repos/uniVocity/univocity-parsers/issues/237/labels{/name}","comments_url":"https://api.github.com/repos/uniVocity/univocity-parsers/issues/237/comments","events_url":"https://api.github.com/repos/uniVocity/univocity-parsers/issues/237/events","html_url":"https://github.com/uniVocity/univocity-parsers/issues/237","id":328426316,"node_id":"MDU6SXNzdWUzMjg0MjYzMTY=","number":237,"title":"heap memory consumption increase significantly after over 10 million rows reading and processing","user":{"login":"jltz922181","id":39824266,"node_id":"MDQ6VXNlcjM5ODI0MjY2","avatar_url":"https://avatars1.githubusercontent.com/u/39824266?v=4","gravatar_id":"","url":"https://api.github.com/users/jltz922181","html_url":"https://github.com/jltz922181","followers_url":"https://api.github.com/users/jltz922181/followers","following_url":"https://api.github.com/users/jltz922181/following{/other_user}","gists_url":"https://api.github.com/users/jltz922181/gists{/gist_id}","starred_url":"https://api.github.com/users/jltz922181/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jltz922181/subscriptions","organizations_url":"https://api.github.com/users/jltz922181/orgs","repos_url":"https://api.github.com/users/jltz922181/repos","events_url":"https://api.github.com/users/jltz922181/events{/privacy}","received_events_url":"https://api.github.com/users/jltz922181/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-01T08:27:33Z","updated_at":"2018-06-01T12:05:36Z","closed_at":null,"author_association":"NONE","body":"Hi, \r\n\r\nI am using the univocity java parser to parse a large  tsv file with size of about 1.2 GB and totally about 20 million rows. I use the Object Row Processor, following the example code in the parser code repository:\r\n\r\nObjectRowProcessor rowProcessor = new ObjectRowProcessor() {\r\n            @Override\r\n            public void rowProcessed(Object[] row, ParsingContext context) {\r\n                //System.out.println(Arrays.toString(row));\r\n                processEachRow(row);\r\n                increaseProcessedRowCounter(); \r\n            }\r\n        };\r\n\r\nI have my own function: processEachRow, to take the row passed from the parser, and then formulate a query to store the row into the backend data store. After that, I increase the processed row counter.\r\n\r\nI found that after processing several million of rows,  the heap memory continues to grow, up to the point that the GC time is so huge that the application gets frozen. \r\n\r\nRight now, I just use the default parser settings and then invoke: \r\n\r\nparser.parse(new FileReader(myDataFileName));\r\n\r\nI wonder what are the necessary parser settings that can have the parser to release the objects that passed to my code, so that the parsed row objects can be GC'ed. \r\n\r\nRegards,\r\n\r\nJun\r\n"},"comment":{"url":"https://api.github.com/repos/uniVocity/univocity-parsers/issues/comments/393860518","html_url":"https://github.com/uniVocity/univocity-parsers/issues/237#issuecomment-393860518","issue_url":"https://api.github.com/repos/uniVocity/univocity-parsers/issues/237","id":393860518,"node_id":"MDEyOklzc3VlQ29tbWVudDM5Mzg2MDUxOA==","user":{"login":"jbax","id":7100478,"node_id":"MDQ6VXNlcjcxMDA0Nzg=","avatar_url":"https://avatars0.githubusercontent.com/u/7100478?v=4","gravatar_id":"","url":"https://api.github.com/users/jbax","html_url":"https://github.com/jbax","followers_url":"https://api.github.com/users/jbax/followers","following_url":"https://api.github.com/users/jbax/following{/other_user}","gists_url":"https://api.github.com/users/jbax/gists{/gist_id}","starred_url":"https://api.github.com/users/jbax/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbax/subscriptions","organizations_url":"https://api.github.com/users/jbax/orgs","repos_url":"https://api.github.com/users/jbax/repos","events_url":"https://api.github.com/users/jbax/events{/privacy}","received_events_url":"https://api.github.com/users/jbax/received_events","type":"User","site_admin":false},"created_at":"2018-06-01T12:05:35Z","updated_at":"2018-06-01T12:05:35Z","author_association":"MEMBER","body":"It should not cause the heap size to increase. Does it happen if you\ncomment out the lines in your row processor?\n\nIf after removing 'processEachRow(row);' and 'increaseProcessedRowCounter();'\nyour heap size stops growing then there is something in your code that's\ncausing the problem.\n\nLet me know if the issue persists and I'll have a look.\n\nThank you for using our parsers!\n\nOn Fri, 1 Jun. 2018, 5:27 am jltz922181, <notifications@github.com> wrote:\n\n> Hi,\n>\n> I am using the univocity java parser to parse a large tsv file with size\n> of about 1.2 GB and totally about 20 million rows. I use the Object Row\n> Processor, following the example code in the parser code repository:\n>\n> ObjectRowProcessor rowProcessor = new ObjectRowProcessor() {\n> @override <https://github.com/override>\n> public void rowProcessed(Object[] row, ParsingContext context) {\n> //System.out.println(Arrays.toString(row));\n> processEachRow(row);\n> increaseProcessedRowCounter();\n> }\n> };\n>\n> I have my own function: processEachRow, to take the row passed from the\n> parser, and then formulate a query to store the row into the backend data\n> store. After that, I increase the processed row counter.\n>\n> I found that after processing several million of rows, the heap memory\n> continues to grow, up to the point that the GC time is so huge that the\n> application gets frozen.\n>\n> Right now, I just use the default parser settings and then invoke:\n>\n> parser.parse(new FileReader(myDataFileName));\n>\n> I wonder what are the necessary parser settings that can have the parser\n> to release the objects that passed to my code, so that the parsed row\n> objects can be GC'ed.\n>\n> Regards,\n>\n> Jun\n>\n> —\n> You are receiving this because you are subscribed to this thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/uniVocity/univocity-parsers/issues/237>, or mute the\n> thread\n> <https://github.com/notifications/unsubscribe-auth/AGxYPgRmvlRpg3zzVjkOn3dtvmz_j58sks5t4Pr5gaJpZM4UWTAX>\n> .\n>\n"}}
{"actor":{"display_login":"jbax"}}