{"action":"created","issue":{"url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/81","repository_url":"https://api.github.com/repos/yahoo/CaffeOnSpark","labels_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/81/labels{/name}","comments_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/81/comments","events_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/81/events","html_url":"https://github.com/yahoo/CaffeOnSpark/issues/81","id":159649148,"number":81,"title":"Getting error while running in yarn mode","user":{"login":"abhaymise","id":8757867,"avatar_url":"https://avatars.githubusercontent.com/u/8757867?v=3","gravatar_id":"","url":"https://api.github.com/users/abhaymise","html_url":"https://github.com/abhaymise","followers_url":"https://api.github.com/users/abhaymise/followers","following_url":"https://api.github.com/users/abhaymise/following{/other_user}","gists_url":"https://api.github.com/users/abhaymise/gists{/gist_id}","starred_url":"https://api.github.com/users/abhaymise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abhaymise/subscriptions","organizations_url":"https://api.github.com/users/abhaymise/orgs","repos_url":"https://api.github.com/users/abhaymise/repos","events_url":"https://api.github.com/users/abhaymise/events{/privacy}","received_events_url":"https://api.github.com/users/abhaymise/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-06-10T14:35:19Z","updated_at":"2016-07-01T04:44:23Z","closed_at":"2016-07-01T04:44:23Z","body":"16/06/10 14:32:20 INFO caffe.CaffeProcessor: my rank is 1\r\n16/06/10 14:32:20 INFO caffe.LMDB: Batch size:64\r\nWARNING: Logging before InitGoogleLogging() is written to STDERR\r\nF0610 14:32:20.571682 10978 CaffeNet.cpp:67] Check failed: d >= 0 (-1 vs. 0) cannot grab GPU device\r\n\r\ncommand used is \r\n\r\nspark-submit --master yarn --deploy-mode cluster     --num-executors ${SPARK_WORKER_INSTANCES}     --files ${CAFFE_ON_SPARK}/data/lenet_memory_solver.prototxt,${CAFFE_ON_SPARK}/data/lenet_memory_train_test.prototxt     --conf spark.driver.extraLibraryPath=\"${LD_LIBRARY_PATH}\"     --conf spark.executorEnv.LD_LIBRARY_PATH=\"${LD_LIBRARY_PATH}\"     --class com.yahoo.ml.caffe.CaffeOnSpark      ${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar         -train         -features accuracy,loss -label label         -conf lenet_memory_solver.prototxt         -devices ${DEVICES}     -connection ethernet         -model hdfs:///mnist.model         -output hdfs:///mnist_features_result\r\n\r\n\r\nPlease guide"},"comment":{"url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/comments/229852739","html_url":"https://github.com/yahoo/CaffeOnSpark/issues/81#issuecomment-229852739","issue_url":"https://api.github.com/repos/yahoo/CaffeOnSpark/issues/81","id":229852739,"user":{"login":"abhaymise","id":8757867,"avatar_url":"https://avatars.githubusercontent.com/u/8757867?v=3","gravatar_id":"","url":"https://api.github.com/users/abhaymise","html_url":"https://github.com/abhaymise","followers_url":"https://api.github.com/users/abhaymise/followers","following_url":"https://api.github.com/users/abhaymise/following{/other_user}","gists_url":"https://api.github.com/users/abhaymise/gists{/gist_id}","starred_url":"https://api.github.com/users/abhaymise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abhaymise/subscriptions","organizations_url":"https://api.github.com/users/abhaymise/orgs","repos_url":"https://api.github.com/users/abhaymise/repos","events_url":"https://api.github.com/users/abhaymise/events{/privacy}","received_events_url":"https://api.github.com/users/abhaymise/received_events","type":"User","site_admin":false},"created_at":"2016-07-01T04:44:23Z","updated_at":"2016-07-01T04:44:23Z","body":"Yeah the issue is fixed now.The number of devices passed as argument was the issue.Devices should be  equal to number of GPUs in each of your machine.In my case i was using a g2.2xlarge machine which only contains one GPU.When i ran it on cluster i thought this number should be increased to three which caused the above error.When made the $DEVICES variable to one it ran.\r\n\r\nThanks mridul for your constant support and creating such a beautiful API."}}
{"actor":{"display_login":"abhaymise"}}