{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/287","repository_url":"https://api.github.com/repos/Microsoft/CNTK","labels_url":"https://api.github.com/repos/Microsoft/CNTK/issues/287/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/CNTK/issues/287/comments","events_url":"https://api.github.com/repos/Microsoft/CNTK/issues/287/events","html_url":"https://github.com/Microsoft/CNTK/issues/287","id":143451315,"number":287,"title":"problem in running multiple gpu on the newest cntk","user":{"login":"summerQing307","id":17537330,"avatar_url":"https://avatars.githubusercontent.com/u/17537330?v=3","gravatar_id":"","url":"https://api.github.com/users/summerQing307","html_url":"https://github.com/summerQing307","followers_url":"https://api.github.com/users/summerQing307/followers","following_url":"https://api.github.com/users/summerQing307/following{/other_user}","gists_url":"https://api.github.com/users/summerQing307/gists{/gist_id}","starred_url":"https://api.github.com/users/summerQing307/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/summerQing307/subscriptions","organizations_url":"https://api.github.com/users/summerQing307/orgs","repos_url":"https://api.github.com/users/summerQing307/repos","events_url":"https://api.github.com/users/summerQing307/events{/privacy}","received_events_url":"https://api.github.com/users/summerQing307/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-03-25T08:12:39Z","updated_at":"2016-03-29T06:52:04Z","closed_at":null,"body":"   Hi,I have updated the newest cntk version today. I was trying to use the cntk to train LM by multiple gpus.\r\n\r\nI have set the config like this:\r\n        ParallelTrain=[\r\n                   parallelizationMethod = \"DataParallelSGD\"\r\n                   parallelizationStartEpoch = 2 \r\n\r\n                    Additional optional parameters are: useZeroThresholdFor1BitQuantization\r\n                    dataParallelSGD = [ \r\n                        gradientBits = 1 \r\n                     ]\r\n            ]\r\n\r\n**But,when i was running,i got a bug like the following:\r\nEXCEPTION occurred: Inside File: Source/Math/Matrix.cpp  Line: 1735  Function: AssignRowSliceValuesOf  -> Feature Not Implemented.**\r\n\r\n\r\n[CALL STACK]\r\n[0x7f1d21fb9edc]    Microsoft::MSR::CNTK::Matrix<float>::  AssignRowSliceValuesOf  (Microsoft::MSR::CNTK::Matrix<float> const&,  unsigned long,  unsigned long) + 0x17c\r\n[0x73629c]          std::pair<unsigned long,unsigned long> Microsoft::MSR::CNTK::DataReaderHelpers::  DecimateMinibatch  <float>(Microsoft::MSR::CNTK::StreamMinibatchInputs const&,  Microsoft::MSR::CNTK::StreamMinibatchInputs&,  std::shared_ptr<Microsoft::MSR::CNTK::MBLayout>,  std::shared_ptr<Microsoft::MSR::CNTK::MBLayout>&,  unsigned long,  unsigned long) + 0x22c\r\n[0x736cf5]          std::pair<unsigned long,unsigned long> Microsoft::MSR::CNTK::DataReaderHelpers::  DecimateMinibatchInPlace  <float>(Microsoft::MSR::CNTK::StreamMinibatchInputs&,  unsigned long,  unsigned long,  std::shared_ptr<Microsoft::MSR::CNTK::MBLayout>) + 0xc5\r\n[0x7373a8]          bool Microsoft::MSR::CNTK::DataReaderHelpers::  GetMinibatchIntoNetwork  <float>(Microsoft::MSR::CNTK::IDataReader&,  std::shared_ptr<Microsoft::MSR::CNTK::ComputationNetwork>,  std::shared_ptr<Microsoft::MSR::CNTK::ComputationNodeBase>,  bool,  bool,  Microsoft::MSR::CNTK::StreamMinibatchInputs&,  unsigned long&,  std::shared_ptr<Microsoft::MSR::CNTK::MPIWrapper> const&) + 0x458\r\n[0x737cb2]          Microsoft::MSR::CNTK::SimpleEvaluator<float>::  Evaluate  (Microsoft::MSR::CNTK::IDataReader*,  std::vector<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t>>,std::allocator<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t>>>> const&,  unsigned long,  unsigned long) + 0x7a2\r\n[0x73ef99]          Microsoft::MSR::CNTK::SGD<float>::  TrainOrAdaptModel  (int,  std::shared_ptr<Microsoft::MSR::CNTK::ComputationNetwork>,  bool,  std::shared_ptr<Microsoft::MSR::CNTK::ComputationNetwork>,  std::shared_ptr<Microsoft::MSR::CNTK::ComputationNodeBase>,  Microsoft::MSR::CNTK::IDataReader*,  Microsoft::MSR::CNTK::IDataReader*) + 0x1759\r\n[0x7409c0]          Microsoft::MSR::CNTK::SGD<float>::  Train  (std::function<std::shared_ptr<Microsoft::MSR::CNTK::ComputationNetwork> (int)>,  int,  Microsoft::MSR::CNTK::IDataReader*,  Microsoft::MSR::CNTK::IDataReader*,  bool) + 0x310  \r\n[0x748828]          void  DoTrain  <Microsoft::MSR::CNTK::ConfigParameters,float>(Microsoft::MSR::CNTK::ConfigParameters const&) + 0x3c8\r\n[0x5c45a7]          void  DoCommands  <float>(Microsoft::MSR::CNTK::ConfigParameters const&,  std::shared_ptr<Microsoft::MSR::CNTK::MPIWrapper> const&) + 0x907\r\n[0x564dfa]          wmainOldCNTKConfig  (int,  wchar_t**)              + 0x90a\r\n[0x565438]          wmain1  (int,  wchar_t**)                          + 0x68\r\n[0x559128]          main                                               + 0xd8\r\n[0x7f1d20343b15]    __libc_start_main                                  + 0xf5\r\n[0x55ce75]\r\nLMSequenceReader: Reading epoch data... 0 sequences read.\r\nFinished Epoch[ 1 of 16]: [Training Set] TrainLossPerSample = 5.8850384; TotalSamplesSeen = 228950; AvgLearningRatePerSample = 0.1; EpochTime=34.074\r\nLMSequenceReader: Reading epoch data... 5276 sequences read.\r\nInside File: Source/Math/Matrix.cpp  Line: 1735  Function: AssignRowSliceValuesOf  -> Feature Not Implemented.\r\n\r\n**I guess the problem lies in the sparse matrix. Could you help me find the real problem and tell me what may i do to solve the problem to achieve training LM on multiple gpus?**"},"comment":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/comments/202743664","html_url":"https://github.com/Microsoft/CNTK/issues/287#issuecomment-202743664","issue_url":"https://api.github.com/repos/Microsoft/CNTK/issues/287","id":202743664,"user":{"login":"summerQing307","id":17537330,"avatar_url":"https://avatars.githubusercontent.com/u/17537330?v=3","gravatar_id":"","url":"https://api.github.com/users/summerQing307","html_url":"https://github.com/summerQing307","followers_url":"https://api.github.com/users/summerQing307/followers","following_url":"https://api.github.com/users/summerQing307/following{/other_user}","gists_url":"https://api.github.com/users/summerQing307/gists{/gist_id}","starred_url":"https://api.github.com/users/summerQing307/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/summerQing307/subscriptions","organizations_url":"https://api.github.com/users/summerQing307/orgs","repos_url":"https://api.github.com/users/summerQing307/repos","events_url":"https://api.github.com/users/summerQing307/events{/privacy}","received_events_url":"https://api.github.com/users/summerQing307/received_events","type":"User","site_admin":false},"created_at":"2016-03-29T06:52:04Z","updated_at":"2016-03-29T06:52:04Z","body":"@saonim I tested by my config and it seemed not the problem. Thank you so much and could you remember when did you see the same error?"}}