{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2416","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2416/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2416/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2416/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/2416","id":192781665,"number":2416,"title":"v0.7.0 - Cuda: Under certain circumstances ultra slow","user":{"login":"mjakobus","id":17254924,"avatar_url":"https://avatars.githubusercontent.com/u/17254924?v=3","gravatar_id":"","url":"https://api.github.com/users/mjakobus","html_url":"https://github.com/mjakobus","followers_url":"https://api.github.com/users/mjakobus/followers","following_url":"https://api.github.com/users/mjakobus/following{/other_user}","gists_url":"https://api.github.com/users/mjakobus/gists{/gist_id}","starred_url":"https://api.github.com/users/mjakobus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mjakobus/subscriptions","organizations_url":"https://api.github.com/users/mjakobus/orgs","repos_url":"https://api.github.com/users/mjakobus/repos","events_url":"https://api.github.com/users/mjakobus/events{/privacy}","received_events_url":"https://api.github.com/users/mjakobus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-12-01T08:25:32Z","updated_at":"2016-12-01T18:11:56Z","closed_at":null,"body":"Hi @raver119 , \r\n\r\nfor my business case it is unfortunately necessary that I have a lot of java objects in the memory. But these object have no interaction with deeplearning4j or nd4j. Its just a list of objects each one containing another list of objects.\r\nI noticed that the cuda backend is getting really slow if there are a lot of objects loitering around in the memory (java objects, which are not affecting nd4j and which are doing nothing).\r\n\r\n**My measurements:**\r\n\r\n```\r\n        // - GPU Performance (Nvidia GTX 950 2048MB) -\r\n        //Without GarbageData:                  3000 samples/sec\r\n        //With GarbageData (1Mio. Objects):     10 samples/sec\r\n        //With GarbageData (50Mio. Objects):    10 sample/year (?) ;-)\r\n\r\n        // - CPU Performance (Intel Core i5 6500 + MKL & 16Gb RAM) -\r\n        //Without GarbageData:                  16000 samples/sec\r\n        //With GarbageData (1Mio. Objects):     16000 samples/sec\r\n        //With GarbageData (50Mio. Objects):    16000 samples/sec\r\n```\r\n\r\n**System details:**\r\n- Deeplearning4j, nd4j, datavec version: 0.7.0\r\n- Cuda Version 7.5\r\n- Graphics Card: Nvidia Geforce GTX 950 with 2048MB\r\n- Windows 10\r\n\r\n\r\n**Condensed setup of my program (for reproducing the problem)**:\r\nhttps://gist.github.com/mjakobus/cf3f076220e98c3ec727205a4ead1f0f\r\n\r\nAnd the DoublesDataSetShuffleIterator (needed as a workaround for issue #2397)\r\nhttps://gist.github.com/mjakobus/539c85a2866f1165e3f5911dc6a284a1\r\n\r\nYou can turn the creation of the objects on/off with \" boolean createDataGarbage = true;\"\r\n\r\n\r\nThe hotspots in each case are also a bit different:\r\n**Cuda with 1 Mio. Objects**:\r\n![with_garbadge](https://cloud.githubusercontent.com/assets/17254924/20786689/d186010a-b7a7-11e6-9112-bbab7e0b0217.PNG)\r\n\r\n**Cuda without 1 Mio. Objects**:\r\n![without_garbadge](https://cloud.githubusercontent.com/assets/17254924/20786694/d62a0670-b7a7-11e6-99b9-aebe662b9548.PNG)\r\n\r\nBest regards,\r\n Martin"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/264248846","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/2416#issuecomment-264248846","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2416","id":264248846,"user":{"login":"raver119","id":12250879,"avatar_url":"https://avatars.githubusercontent.com/u/12250879?v=3","gravatar_id":"","url":"https://api.github.com/users/raver119","html_url":"https://github.com/raver119","followers_url":"https://api.github.com/users/raver119/followers","following_url":"https://api.github.com/users/raver119/following{/other_user}","gists_url":"https://api.github.com/users/raver119/gists{/gist_id}","starred_url":"https://api.github.com/users/raver119/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raver119/subscriptions","organizations_url":"https://api.github.com/users/raver119/orgs","repos_url":"https://api.github.com/users/raver119/repos","events_url":"https://api.github.com/users/raver119/events{/privacy}","received_events_url":"https://api.github.com/users/raver119/received_events","type":"User","site_admin":false},"created_at":"2016-12-01T18:11:56Z","updated_at":"2016-12-01T18:11:56Z","body":"Native is affected (or at least was affected at the moment i was taking my measurements few days ago) as well, just higher pressure is needed. It's mostly about System.gc() calls being used within backends. We are going to get rid of them, but not for this 0.7.1 hotfix release."}}
{"actor":{"display_login":"raver119"}}