{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/4815","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/4815/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/4815/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/4815/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/4815","id":306165986,"number":4815,"title":"Weight init creates an array with the size of the whole network","user":{"login":"treo","id":509379,"avatar_url":"https://avatars0.githubusercontent.com/u/509379?v=4","gravatar_id":"","url":"https://api.github.com/users/treo","html_url":"https://github.com/treo","followers_url":"https://api.github.com/users/treo/followers","following_url":"https://api.github.com/users/treo/following{/other_user}","gists_url":"https://api.github.com/users/treo/gists{/gist_id}","starred_url":"https://api.github.com/users/treo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treo/subscriptions","organizations_url":"https://api.github.com/users/treo/orgs","repos_url":"https://api.github.com/users/treo/repos","events_url":"https://api.github.com/users/treo/events{/privacy}","received_events_url":"https://api.github.com/users/treo/received_events","type":"User","site_admin":false},"labels":[{"id":372474058,"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/labels/perf","name":"perf","color":"f9d0c4","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-03-17T15:39:51Z","updated_at":"2018-03-20T03:21:05Z","closed_at":null,"author_association":"MEMBER","body":"#### Issue Description\r\nWhen training on a GPU, it is easily possible to create a network that _should_ fit into memory, but even just initializing it results in an out of memory state.\r\n\r\nI've tracked the problem down to WeightInitUtil, which gets a parameters view, but instead of directly using it, it creates a new one, which is then copied into that view.\r\n\r\nThis results in the problem that model initialization requires twice as much memory, as is needed to just hold that model.\r\n\r\nOne could argue that if you don't have enough memory to hold the model twice, it is just too large anyway, but a 6GB model isn't trainable on an 11GB GPU, and that certainly shouldn't be an issue.\r\n\r\n#### Version Information\r\n\r\n* Deeplearning4j version: 0.9.2-SNAPSHOT\r\n* Windows\r\n* CUDA 8.0\r\n\r\n#### Contributing\r\n\r\nI've modified WeightInitUtil to use the param view directly like that (Xavier weight init):\r\n```java\r\nNd4j.getExecutioner().exec(new GaussianDistribution(paramView), Nd4j.getRandom());\r\nparamView.muli(FastMath.sqrt(2.0 / (fanIn + fanOut)));\r\nreturn paramView.reshape(order, shape);\r\n```\r\n\r\nAnd it seems to work fine. So if there is no reason to keep creating a new array first, I can create a pull request that does the same for all weight init schemes."},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/374460165","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/4815#issuecomment-374460165","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/4815","id":374460165,"user":{"login":"AlexDBlack","id":2360237,"avatar_url":"https://avatars0.githubusercontent.com/u/2360237?v=4","gravatar_id":"","url":"https://api.github.com/users/AlexDBlack","html_url":"https://github.com/AlexDBlack","followers_url":"https://api.github.com/users/AlexDBlack/followers","following_url":"https://api.github.com/users/AlexDBlack/following{/other_user}","gists_url":"https://api.github.com/users/AlexDBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexDBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexDBlack/subscriptions","organizations_url":"https://api.github.com/users/AlexDBlack/orgs","repos_url":"https://api.github.com/users/AlexDBlack/repos","events_url":"https://api.github.com/users/AlexDBlack/events{/privacy}","received_events_url":"https://api.github.com/users/AlexDBlack/received_events","type":"User","site_admin":false},"created_at":"2018-03-20T03:21:05Z","updated_at":"2018-03-20T03:21:05Z","author_association":"OWNER","body":"Yeah, I think the cleanest way long term - would be to implement methods where the result array can be passed in/specified. Specifically:\r\n(a) Nd4j.rand(INDArray), Nd4j.randn(INDArray) etc, and also\r\n(b) Distribution.sample(INDArray)"}}
{"actor":{"display_login":"AlexDBlack"}}