title: Theano: Initialisation of device gpu failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY
body: <p>I am running the <a href="https://github.com/fchollet/keras/blob/master/examples/kaggle_otto_nn.py">example</a> <code>kaggle_otto_nn.py</code> of <code>Keras</code> with backend of <code>theano</code>. When I set <code>cnmem=1</code>, the following error comes out: </p>
  
  <pre><code>cliu@cliu-ubuntu:keras-examples$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32,lib.cnmem=1 python kaggle_otto_nn.py
  Using Theano backend.
  ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device gpu failed:
  initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1
  
  /usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module.
    "downsample module has been moved to the theano.tensor.signal.pool module.")
  Traceback (most recent call last):
    File "kaggle_otto_nn.py", line 28, in &lt;module&gt;
      from keras.models import Sequential
    File "build/bdist.linux-x86_64/egg/keras/models.py", line 15, in &lt;module&gt;
    File "build/bdist.linux-x86_64/egg/keras/backend/__init__.py", line 46, in &lt;module&gt;
    File "build/bdist.linux-x86_64/egg/keras/backend/theano_backend.py", line 1, in &lt;module&gt;
    File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/__init__.py", line 111, in &lt;module&gt;
      theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
    File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/sandbox/cuda/tests/test_driver.py", line 38, in test_nvidia_driver1
      if not numpy.allclose(f(), a.sum()):
    File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/compile/function_module.py", line 871, in __call__
      storage_map=getattr(self.fn, 'storage_map', None))
    File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/gof/link.py", line 314, in raise_with_op
      reraise(exc_type, exc_value, exc_trace)
    File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/compile/function_module.py", line 859, in __call__
      outputs = self.fn()
  RuntimeError: Cuda error: kernel_reduce_ccontig_node_97496c4d3cf9a06dc4082cc141f918d2_0: out of memory. (grid: 1 x 1; block: 256 x 1 x 1)
  
  Apply node that caused the error: GpuCAReduce{add}{1}(&lt;CudaNdarrayType(float32, vector)&gt;)
  Toposort index: 0
  Inputs types: [CudaNdarrayType(float32, vector)]
  Inputs shapes: [(10000,)]
  Inputs strides: [(1,)]
  Inputs values: ['not shown']
  Outputs clients: [[HostFromGpu(GpuCAReduce{add}{1}.0)]]
  
  HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
  HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
  </code></pre>
  
  <p>It seems like I cannot set the <code>cnmem</code> to a very large value (about > 0.9) since it may cause the GPU's memory overflow. And when I set <code>cnmem=0.9</code>, it's is working correctly.  According to <a href="http://deeplearning.net/software/theano/library/config.html#config.config.lib.cnmem">this</a>, it </p>
  
  <blockquote>
    <p>represents the start size (in MB or % of total GPU memory) of the memory pool.</p>
  </blockquote>
  
  <p>And</p>
  
  <blockquote>
    <p>This could cause memory fragmentation. So if you have a memory error while using cnmem, try to allocate more memory at the start or disable it. If you try this, report your result on :ref<code>theano-dev</code>.</p>
  </blockquote>
  
  <p>But if I got memory error, why should I allocate more memory at the start？ And in my case, allocating more memory at the start seems like causing the error. </p>