{"action":"opened","issue":{"url":"https://api.github.com/repos/Element-Research/rnn/issues/113","labels_url":"https://api.github.com/repos/Element-Research/rnn/issues/113/labels{/name}","comments_url":"https://api.github.com/repos/Element-Research/rnn/issues/113/comments","events_url":"https://api.github.com/repos/Element-Research/rnn/issues/113/events","html_url":"https://github.com/Element-Research/rnn/issues/113","id":128059610,"number":113,"title":"Unsuccessful training on multivariate timeseries toy dataset","user":{"login":"zencoding","id":8623645,"avatar_url":"https://avatars.githubusercontent.com/u/8623645?v=3","gravatar_id":"","url":"https://api.github.com/users/zencoding","html_url":"https://github.com/zencoding","followers_url":"https://api.github.com/users/zencoding/followers","following_url":"https://api.github.com/users/zencoding/following{/other_user}","gists_url":"https://api.github.com/users/zencoding/gists{/gist_id}","starred_url":"https://api.github.com/users/zencoding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zencoding/subscriptions","organizations_url":"https://api.github.com/users/zencoding/orgs","repos_url":"https://api.github.com/users/zencoding/repos","events_url":"https://api.github.com/users/zencoding/events{/privacy}","received_events_url":"https://api.github.com/users/zencoding/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-01-22T01:16:42Z","updated_at":"2016-01-22T01:16:42Z","closed_at":null,"body":"This is really not an issue, so please feel free to close it and I will try asking the questions in Google Groups or Stackoverflow\r\n\r\n I have been playing with LSTM for multi variate time series, it is failing to learn for very simple toy example.  My dataset is basically an alternating time series with even sequence as addition and odd sequence as multiplication\r\n\r\nSample input data\r\n\r\n       1        2        3\r\n       2        3        6\r\n       3        4        7\r\n       4        5       20\r\n       5        6       11\r\n       6        7       42\r\n       7        8       15\r\n       8        9       72\r\n       9       10       19\r\n      10       11      110\r\n      11       12       23\r\n\r\nI modified the recurrent-time-series.lua from rnn/examples to change the sequence dataset leaving all other code as is.  The default dataset works fine but my dataset produces very high error rates.  I tried all recommended settings like changing number of numbers, learning rate, etc, I even tried to clone the output of the forward pass as mentioned in some other forum post.  Nothing seems to fix the error rates. Below is the code, it is identical to the standard example except for the sequence dataset, any help to fix will be greatly appreciated\r\n\r\n-- Multi-variate time-series example\r\n\r\nrequire 'rnn'\r\n\r\ncmd = torch.CmdLine()\r\ncmd:text()\r\ncmd:text('Train a multivariate time-series model using RNN')\r\ncmd:option('--rho', 5, 'maximum number of time steps for back-propagate through time (BPTT)')\r\ncmd:option('--multiSize', 3, 'number of random variables as input and output')\r\ncmd:option('--hiddenSize', 100, 'number of hidden units used at output of the recurrent layer')\r\ncmd:option('--dataSize', 1000, 'total number of time-steps in dataset')\r\ncmd:option('--batchSize', 50, 'number of training samples per batch')\r\ncmd:option('--nIterations', 1000, 'max number of training iterations')\r\ncmd:option('--learningRate', 0.0001, 'learning rate')\r\ncmd:text()\r\nlocal opt = cmd:parse(arg or {})\r\n\r\n-- For simplicity, the multi-variate dataset in this example is independently distributed.\r\n-- Toy dataset (task is to predict next vector, given previous vectors) following the normal distribution .\r\n-- Generated by sampling a separate normal distribution for each random variable.\r\n-- note: vX is used as both input X and output Y to save memory\r\nlocal function evalPDF(vMean, vSigma, vX)\r\n   for i=1,vMean:size(1) do\r\n      local b = (vX[i]-vMean[i])/vSigma[i]\r\n      vX[i] = math.exp(-b*b/2)/(vSigma[i]*math.sqrt(2*math.pi))\r\n   end\r\n   return vX\r\nend\r\n\r\nassert(opt.multiSize > 1, \"Multi-variate time-series\")\r\n\r\n-- vBias = torch.randn(opt.multiSize)\r\n-- vMean = torch.Tensor(opt.multiSize):fill(5)\r\n-- vSigma = torch.linspace(1,opt.multiSize,opt.multiSize)\r\n-- sequence = torch.Tensor(opt.dataSize, opt.multiSize)\r\n--\r\n-- j = 0\r\n-- for i=1,opt.dataSize do\r\n--   sequence[{i,{}}]:fill(j)\r\n--   evalPDF(vMean, vSigma, sequence[{i,{}}])\r\n--   sequence[{i,{}}]:add(vBias)\r\n--   j = j + 1\r\n--   if j>10 then j = 0 end\r\n-- end\r\nsequence = torch.Tensor(opt.dataSize,opt.multiSize)\r\nfor j = 1,opt.dataSize do\r\n  if j%2 > 0 then\r\n    sequence[j][1]=j\r\n    sequence[j][2] = j +1\r\n    sequence[j][3] = sequence[j][1] + sequence[j][2]\r\n  end\r\n  if j%2 == 0 then\r\n    sequence[j][1]=j\r\n    sequence[j][2] = j +1\r\n    sequence[j][3] = sequence[j][1] * sequence[j][2]\r\n  end\r\nend\r\nprint('Sequence:'); print(sequence)\r\n\r\n-- batch mode\r\n\r\noffsets = torch.LongTensor(opt.batchSize):random(1,opt.dataSize)\r\n\r\n-- RNN\r\nr = nn.Recurrent(\r\n   opt.hiddenSize, -- size of output\r\n   nn.Linear(opt.multiSize, opt.hiddenSize), -- input layer\r\n   nn.Linear(opt.hiddenSize, opt.hiddenSize), -- recurrent layer\r\n   nn.Sigmoid(), -- transfer function\r\n   opt.rho\r\n)\r\n\r\nrnn = nn.Sequential()\r\n   :add(r)\r\n   :add(nn.Linear(opt.hiddenSize, opt.multiSize))\r\n\r\ncriterion = nn.MSECriterion()\r\n\r\n-- use Sequencer for better data handling\r\nrnn = nn.Sequencer(rnn)\r\n\r\ncriterion = nn.SequencerCriterion(criterion)\r\nprint(\"Model :\")\r\nprint(rnn)\r\n\r\n-- train rnn model\r\nminErr = opt.multiSize -- report min error\r\nminK = 0\r\navgErrs = torch.Tensor(opt.nIterations):fill(0)\r\nfor k = 1, opt.nIterations do\r\n\r\n   -- 1. create a sequence of rho time-steps\r\n\r\n   local inputs, targets = {}, {}\r\n  --  for step = 1, opt.rho do\r\n  --     -- batch of inputs\r\n  --     inputs[step] = inputs[step] or sequence.new()\r\n  --     inputs[step]:index(sequence, 1, offsets)\r\n  --     -- batch of targets\r\n  --     offsets:add(1) -- increase indices by 1\r\n  --     offsets[offsets:gt(opt.dataSize)] = 1\r\n  --     targets[step] = targets[step] or sequence.new()\r\n  --     targets[step]:index(sequence, 1, offsets)\r\n  --  end\r\n  for step = 1, opt.rho do\r\n     -- batch of inputs\r\n     inputs[step] = inputs[step] or sequence.new()\r\n     inputs[step]:index(sequence, 1, offsets)\r\n     -- batch of targets\r\n     offsets:add(1) -- increase indices by 1\r\n     offsets[offsets:gt(opt.dataSize)] = 1\r\n     targets[step] = targets[step] or sequence.new()\r\n     targets[step]:index(sequence, 1, offsets)\r\n  end\r\n\r\n   -- 2. forward sequence through rnn\r\n\r\n   local outputs = rnn:forward(inputs)\r\n   local err = criterion:forward(outputs, targets)\r\n\r\n   -- report errors\r\n\r\n   print('Iter: ' .. k .. '   Err: ' .. err)\r\n   avgErrs[k] = err\r\n   if avgErrs[k] < minErr then\r\n      minErr = avgErrs[k]\r\n      minK = k\r\n   end\r\n\r\n   -- 3. backward sequence through rnn (i.e. backprop through time)\r\n\r\n   rnn:zeroGradParameters()\r\n\r\n   local gradOutputs = criterion:backward(outputs, targets)\r\n   local gradInputs = rnn:backward(inputs, gradOutputs)\r\n\r\n   -- 4. updates parameters\r\n\r\n   rnn:updateParameters(opt.learningRate)\r\nend\r\n\r\nprint('min err: ' .. minErr .. ' on iteration ' .. minK)\r\n\r\n"}}