{"action":"created","issue":{"url":"https://api.github.com/repos/torchnet/torchnet/issues/56","repository_url":"https://api.github.com/repos/torchnet/torchnet","labels_url":"https://api.github.com/repos/torchnet/torchnet/issues/56/labels{/name}","comments_url":"https://api.github.com/repos/torchnet/torchnet/issues/56/comments","events_url":"https://api.github.com/repos/torchnet/torchnet/issues/56/events","html_url":"https://github.com/torchnet/torchnet/issues/56","id":176766100,"number":56,"title":"Problem when using multiple GPUs","user":{"login":"joeyhng","id":1289801,"avatar_url":"https://avatars.githubusercontent.com/u/1289801?v=3","gravatar_id":"","url":"https://api.github.com/users/joeyhng","html_url":"https://github.com/joeyhng","followers_url":"https://api.github.com/users/joeyhng/followers","following_url":"https://api.github.com/users/joeyhng/following{/other_user}","gists_url":"https://api.github.com/users/joeyhng/gists{/gist_id}","starred_url":"https://api.github.com/users/joeyhng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joeyhng/subscriptions","organizations_url":"https://api.github.com/users/joeyhng/orgs","repos_url":"https://api.github.com/users/joeyhng/repos","events_url":"https://api.github.com/users/joeyhng/events{/privacy}","received_events_url":"https://api.github.com/users/joeyhng/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-09-13T22:23:43Z","updated_at":"2016-12-09T11:46:30Z","closed_at":"2016-10-02T19:34:32Z","body":"Sorry for posting long code here, but it is a pretty weird problem and this is the simplest example for me to possibly reproduce the problem.\n\nI'm trying to use torchnet's ParallelDatasetIterator, but got problem when using multiple GPUs. For the following code, the output should not depend on the torchnet, since it is not using the torchnet code.\n\nThe program works normally when using single GPU:\n\n```\n$ th train_debug.lua -nGPU 1\nNumber of parameters:   11184650    \niteration 1: loss=2.289696  \niteration 2: loss=2.209115  \niteration 3: loss=2.136883  \niteration 4: loss=2.071370  \niteration 5: loss=2.011046\n```\n\nHowever, when using 2 GPUs, the loss goes to nan after the first iteration.\n\n```\n$ th train_debug.lua -nGPU 2\nNumber of parameters:   11184650    \niteration 1: loss=2.952482  \niteration 2: loss=nan   \niteration 3: loss=nan   \niteration 4: loss=nan   \niteration 5: loss=nan\n```\n\nAlthough the code should not really depend on torchnet, when I comment out the torchnet related code, the code works normally again.\n\nCode:\n\n``` lua\nrequire 'nn'\nrequire 'cunn'\nrequire 'cudnn'\nlocal tnt = require 'torchnet'\nlocal optim = require 'optim'\n\ncmd = torch.CmdLine()\ncmd:option('-nGPU', 1, 'GPU ID (only using cuda)')\ncmd:option('-learning_rate', 1e-5, 'lr')\nopt = cmd:parse(arg)\n\nfunction dpt_model(nGPU, model)\n  if nGPU > 1 then\n    local gpus = torch.range(1, nGPU):totable()\n\n    model = nn.DataParallelTable(1, true, false)\n      :add(model, gpus)\n      :threads(function()\n        local cudnn = require 'cudnn'\n      end)\n    model.gradInput = nil\n  end\n  return model:cuda()\nend\n\n\n------------------------------------\n-- start of torchnet related code\n-- Create torchnet ParallelDatasetIterator, but not using it\nlocal index = {}\nfor i=1,10000 do table.insert(index, {path='12345'}) end \nget_data_iterator_func = function ()\n  return tnt.ParallelDatasetIterator{\n    nthread = 8,\n    init    = function() \n      require 'torchnet'\n      torch.setdefaulttensortype('torch.FloatTensor')\n    end,\n    closure = function()\n      return tnt.ListDataset{\n        list = index,\n        load = function (data)\n          return {\n            input = torch.zeros(3,224,224),\n            label = torch.LongTensor{1},\n          }\n        end\n      }:batch(16, 'skip-last')\n    end\n  }\nend\nget_data_iterator = get_data_iterator_func()\ndata_iterator = get_data_iterator()\n-- end of torchnet related code\n------------------------------------\n\n\n-- modify the network a little bit (to trigger the error)\nlocal model = torch.load('models/resnet-18.t7')\nmodel:remove(#model)\nmodel:add(nn.Linear(512, 10):cuda())\n\nmodel = dpt_model(opt.nGPU, model)\ncriterion = nn.CrossEntropyCriterion():cuda()\n\nlocal params, grad_params = model:getParameters()\nprint('Number of parameters: ', params:size(1))\n\n-- same data in each iteration\nlocal x = opt.nGPU > 1 and cutorch.createCudaHostTensor() or torch.CudaTensor()\nlocal y = torch.ones(16):cuda()\nx = x:resize(16,3,224,224):normal(0, 1)\n\nfunction feval(xx)\n  if xx ~= params then params:copy(xx) end\n  grad_params:zero()\n  model:training()\n  local outputs = model:forward(x)\n  local f = criterion:forward(outputs, y)\n  local df_do = criterion:backward(outputs, y)\n  model:backward(x, df_do)\n  return f, grad_params\nend\n\n\nlocal optim_state = {learningRate=opt.learning_rate}\nfor i = 1,5 do\n  local __, loss = optim.adam(feval, params, optim_state)\n  cutorch.synchronize()\n  print(('iteration %d: loss=%f'):format(i, loss[1]))\nend\n```\n\n`models/resnet-18.t7` is downloaded from [pretrianed model of fb.resnet.torch](https://github.com/facebook/fb.resnet.torch/tree/master/pretrained). Very weirdly, the error only triggers when the model file is put in the subdirectory. I'm able to reproduce the error in two different machines. Occasionally it runs without any problem on one of the machines that I tested on, but the problem occurs most of the time.\n\nIt may also be that I'm using DataParallelTable incorrectly. Please let me know if this is the case.\n\nThanks in advance!\n"},"comment":{"url":"https://api.github.com/repos/torchnet/torchnet/issues/comments/265995768","html_url":"https://github.com/torchnet/torchnet/issues/56#issuecomment-265995768","issue_url":"https://api.github.com/repos/torchnet/torchnet/issues/56","id":265995768,"user":{"login":"joeyhng","id":1289801,"avatar_url":"https://avatars.githubusercontent.com/u/1289801?v=3","gravatar_id":"","url":"https://api.github.com/users/joeyhng","html_url":"https://github.com/joeyhng","followers_url":"https://api.github.com/users/joeyhng/followers","following_url":"https://api.github.com/users/joeyhng/following{/other_user}","gists_url":"https://api.github.com/users/joeyhng/gists{/gist_id}","starred_url":"https://api.github.com/users/joeyhng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joeyhng/subscriptions","organizations_url":"https://api.github.com/users/joeyhng/orgs","repos_url":"https://api.github.com/users/joeyhng/repos","events_url":"https://api.github.com/users/joeyhng/events{/privacy}","received_events_url":"https://api.github.com/users/joeyhng/received_events","type":"User","site_admin":false},"created_at":"2016-12-09T11:46:30Z","updated_at":"2016-12-09T11:46:30Z","body":"No, I haven't solved the problem. I'm still interested in solving it as\nwell.\n\nOn Fri, Dec 9, 2016, 4:44 AM Zhenhua Wang <notifications@github.com> wrote:\n\n> Have you solved the problem? I also get nan when training with multiple\n> GPU.\n> Thanks\n>\n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/torchnet/torchnet/issues/56#issuecomment-265850088>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/ABOuSdRa7JZMrYPuSE4AV72SdEZ8GEAoks5rGGwdgaJpZM4J8Ndi>\n> .\n>\n"}}
{"actor":{"display_login":"joeyhng"}}