{"action":"closed","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/1242","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/1242/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/1242/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/1242/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/1242","id":139259877,"number":1242,"title":"Increasing heap size while training CNN","user":{"login":"JohnTheDang","id":17123674,"avatar_url":"https://avatars.githubusercontent.com/u/17123674?v=3","gravatar_id":"","url":"https://api.github.com/users/JohnTheDang","html_url":"https://github.com/JohnTheDang","followers_url":"https://api.github.com/users/JohnTheDang/followers","following_url":"https://api.github.com/users/JohnTheDang/following{/other_user}","gists_url":"https://api.github.com/users/JohnTheDang/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnTheDang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnTheDang/subscriptions","organizations_url":"https://api.github.com/users/JohnTheDang/orgs","repos_url":"https://api.github.com/users/JohnTheDang/repos","events_url":"https://api.github.com/users/JohnTheDang/events{/privacy}","received_events_url":"https://api.github.com/users/JohnTheDang/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/labels/bug","name":"bug","color":"d93f0b"}],"state":"closed","locked":false,"assignee":{"login":"raver119","id":12250879,"avatar_url":"https://avatars.githubusercontent.com/u/12250879?v=3","gravatar_id":"","url":"https://api.github.com/users/raver119","html_url":"https://github.com/raver119","followers_url":"https://api.github.com/users/raver119/followers","following_url":"https://api.github.com/users/raver119/following{/other_user}","gists_url":"https://api.github.com/users/raver119/gists{/gist_id}","starred_url":"https://api.github.com/users/raver119/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raver119/subscriptions","organizations_url":"https://api.github.com/users/raver119/orgs","repos_url":"https://api.github.com/users/raver119/repos","events_url":"https://api.github.com/users/raver119/events{/privacy}","received_events_url":"https://api.github.com/users/raver119/received_events","type":"User","site_admin":false},"milestone":null,"comments":23,"created_at":"2016-03-08T11:59:21Z","updated_at":"2016-05-01T12:36:59Z","closed_at":"2016-05-01T12:36:59Z","body":"\r\n\r\nThe memory heap size keep increasing as I train my CNN using spark. My training set is at about 10 mb (around 3000 images), but after training for about 10 epochs, the program always throws some kind of out of memory exception, like Java heap size or GC overhead. I have changed the xmx setting to use 4g, and I have confirmed that the application uses 4g while training by looking at JProfiler connected to IntelliJ. I can unfortunately not include my image data set, as it is medical images that I'm not sure that I can share, but I'll try to include everything else.\r\n\r\nI have added build.sbt, sbt.txt, src, stack trace as txt, in the zip file. Also included an image of the profiling of a run instance, where the uppermost value of the y-axis is 4g. (Green is free space and blue is used space). Adding an image of the GC activity too.\r\n\r\nMemory:\r\n![telemetry](https://cloud.githubusercontent.com/assets/17123674/13601233/d96a0818-e52e-11e5-807e-e31253340a11.png)\r\n\r\nGC: \r\n![telemetry_2](https://cloud.githubusercontent.com/assets/17123674/13601279/34986d2e-e52f-11e5-85fd-ce65640ff49d.png)\r\n\r\n\r\n\r\n[Memory Issue.zip](https://github.com/deeplearning4j/deeplearning4j/files/163095/Memory.Issue.zip)\r\n\r\n\r\nMemory Issue v2\r\n[Memory Issue.zip](https://github.com/deeplearning4j/deeplearning4j/files/163974/Memory.Issue.zip)\r\n\r\n"}}