title: How to deal with GPU memory leakage issues in Torch?
body: <p>My machine's GPU has 2 GBs of memory. When I run the following code for the first time, I get no errors. However, the second time I run the code I get memory error. As a short-time remedy, the only thing I can do is to cast the data into float32 using <code>torch.Tensor.float()</code>. However, the problem still persists and the occupied memory is not released after the process is done, or the process is terminated while running. This is the case for the machine RAM as well. How should one prevent memory leakage in Torch or release the memory?</p>
  
  <pre><code>require 'nn'
  require 'image'
  require 'cunn'
  require 'paths'
  
  
  
  collectgarbage(); collectgarbage()
  if (not paths.filep("cifar10torchsmall.zip")) then
      os.execute('wget -c https://s3.amazonaws.com/torch7/data/cifar10torchsmall.zip')
      os.execute('unzip cifar10torchsmall.zip')
  end
  trainset = torch.load('cifar10-train.t7')
  testset = torch.load('cifar10-test.t7')
  classes = {'airplane', 'automobile', 'bird', 'cat',
             'deer', 'dog', 'frog', 'horse', 'ship', 'truck'}
  
  setmetatable(trainset, 
      {__index = function(t, i) 
                      return {t.data[i], t.label[i]} 
                  end}
  );
  trainset.data = trainset.data:double() -- convert the data from a ByteTensor to a DoubleTensor.
  
  function trainset:size() 
      return self.data:size(1) 
  end
  
  mean = {} -- store the mean, to normalize the test set in the future
  stdv  = {} -- store the standard-deviation for the future
  for i=1,3 do -- over each image channel
      mean[i] = trainset.data[{ {}, {i}, {}, {}  }]:mean() -- mean estimation
      print('Channel ' .. i .. ', Mean: ' .. mean[i])
      trainset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction
  
      stdv[i] = trainset.data[{ {}, {i}, {}, {}  }]:std() -- std estimation
      print('Channel ' .. i .. ', Standard Deviation: ' .. stdv[i])
      trainset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling
  end
  
  
  testset.data = testset.data:double()   -- convert from Byte tensor to Double tensor
  for i=1,3 do -- over each image channel
      testset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction    
      testset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling
  end
  
  trainset.data = trainset.data:cuda()
  testset.data = testset.data:cuda()
  
  net = nn.Sequential()
  net:add(nn.SpatialConvolution(3, 6, 5, 5)) -- 3 input image channels, 6 output channels, 5x5 convolution kernel
  net:add(nn.ReLU())                       -- non-linearity 
  net:add(nn.SpatialMaxPooling(2,2,2,2))     -- A max-pooling operation that looks at 2x2 windows and finds the max.
  net:add(nn.SpatialConvolution(6, 16, 5, 5))
  net:add(nn.ReLU())                       -- non-linearity 
  net:add(nn.SpatialMaxPooling(2,2,2,2))
  net:add(nn.View(16*5*5))                    -- reshapes from a 3D tensor of 16x5x5 into 1D tensor of 16*5*5
  net:add(nn.Linear(16*5*5, 120))             -- fully connected layer (matrix multiplication between input and weights)
  net:add(nn.ReLU())                       -- non-linearity 
  net:add(nn.Linear(120, 84))
  net:add(nn.ReLU())                       -- non-linearity 
  net:add(nn.Linear(84, 10))                   -- 10 is the number of outputs of the network (in this case, 10 digits)
  net:add(nn.LogSoftMax())  
  net = net:cuda()
  
  criterion = nn.ClassNLLCriterion()
  criterion = criterion:cuda()
  
  
  
  pred = net:forward(trainset.data)
  outputEr = criterion:forward(pred, trainset.label:cuda())
  net:zeroGradParameters()
  outputGrad = criterion:backward(pred, trainset.label:cuda())
  collectgarbage()
  inputGrad = net:backward(trainset.data, outputGrad)
  </code></pre>
  
  <p>Side question: Why Torch initializes the network parameters as double although GPUs are pretty slow at computing double-precision operations and there is actually no need of having 64-bit parameter values for almost all neural network applications? How can I initialize a model with float (32-bit) parameter vectors?</p>
  
  <p>I found the answer to the side question. You can easily make torch's default data type as float using the following at the beginning of your code:</p>
  
  <pre><code>torch.setdefaulttensortype('torch.FloatTensor')
  </code></pre>