{"action":"opened","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2123","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2123/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2123/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/2123/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/2123","id":178636400,"number":2123,"title":"ParallelWrapper stops after a short time","user":{"login":"Alcalol","id":3757372,"avatar_url":"https://avatars.githubusercontent.com/u/3757372?v=3","gravatar_id":"","url":"https://api.github.com/users/Alcalol","html_url":"https://github.com/Alcalol","followers_url":"https://api.github.com/users/Alcalol/followers","following_url":"https://api.github.com/users/Alcalol/following{/other_user}","gists_url":"https://api.github.com/users/Alcalol/gists{/gist_id}","starred_url":"https://api.github.com/users/Alcalol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Alcalol/subscriptions","organizations_url":"https://api.github.com/users/Alcalol/orgs","repos_url":"https://api.github.com/users/Alcalol/repos","events_url":"https://api.github.com/users/Alcalol/events{/privacy}","received_events_url":"https://api.github.com/users/Alcalol/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-09-22T15:07:24Z","updated_at":"2016-09-22T15:07:24Z","closed_at":null,"body":"When the normal MultiLayerNetwork is used inside a while loop for the iterator, things works as normal.  However when the MultiLayerNetwork is put inside the ParallelWrapper, a couple of strange things happen:\r\n\r\n1. During Epoch 2 batch 1, a short while (maybe 5 minutes in or so) the entirely thing stops working.  The process is still running but CPU usage drops to idle, GPU load drops to 0% and CudaCachingZeroProvider no longer outputs cache info (Both GPU and normal memory appears to remain populated).\r\n\r\n2.  The miniBatchSize was set to 32 and the data has at least 300,000 timesteps, however epoch 1 only has 1 batch run before it moves onto batch 2\r\n\r\nThings I have tried to no avail:\r\n1. Changing batch size up or down\r\n2. Data Size up or down (but still enough rows for more than one batch)\r\n3. Change worker size between 2 and 4\r\n4. Change prefetchbuffer for parallel wrapper up and down\r\n5. Tried various values for averagingFrequency from 1 to 20\r\n6. Tried useLegacyAveraging true or false\r\n7. Tried wrapper with wrapper.fit inside epoch for loop as well as MultiEpochsIterator\r\n8. Tried various combination of CudaEnvironment settings as well as leaving everything as default\r\n\r\n\r\nSide notes:\r\nI have looked inside the iterator to make sure it does contain data and the data appears correct\r\nI am currently running two GTX 780 cards (SLi disabled)\r\nThe computer has 16GB of ram and at no point was all the memory used up (13gb max during tests)\r\nMy java code is inside NetBeans IDE 8.1 with Maven 3.0.5\r\nThe DL4J and ND4J versions are 0.6.0 [ jar] Central"}}
{"actor":{"display_login":"Alcalol"}}